v1.4.0_IMPLEMENTATION_ROADMAP

v1.4.0 Implementation Roadmap - Integration into v1.3.0

Date: December 16, 2025
Status: PLANNING
Branch: copilot/review-source-code-gaps

---

🎯 Executive Summary

User Requirement: "Die nächste Implementierungsphase soll vorgezogen werden. v1.4.0 soll noch in v1.3.0 umgesetzt werden"

Response: Integrate v1.4.0 features (Process Mining, Stream Protocol, Video Processor, OLAP Analytics) into v1.3.0 release.

Total Effort: 12-16 days (2-3 weeks)
Features: 4
New Test Cases: 49+
New Benchmarks: 15+

---

📊 Feature Overview

#	Feature	File	Current	Target	Effort	Priority
1	Process Mining	src/analytics/process_mining.cpp	70%	100%	2-3 days	HIGH
2	OLAP Analytics	src/analytics/olap.cpp	50%	100%	3-4 days	HIGH
3	Stream Protocol	src/sharding/stream_protocol.cpp	60%	100%	3-4 days	HIGH
4	Video Processor	src/content/video_processor.cpp	40%	100%	4-5 days	MEDIUM

---

1. Process Mining (70% → 100%)

Current Implementation Status

Already Implemented (✅):

• Event log extraction from collections
• Process discovery (Alpha, Heuristic, Inductive Miner algorithms)
• Directly-Follows Graph (DFG) creation
• Basic conformance checking
• Trace clustering basics
• BPMN/PNML export structure

Lines of Code: ~1,200 lines

To Implement (⏳)

1.1 Bottleneck Detection

Method: detectBottlenecks(const EnhancedProcess& process, double threshold_percentile)

Implementation:

// Analyze performance data to find bottlenecks
// 1. Calculate percentile thresholds for node durations
// 2. Identify nodes exceeding threshold
// 3. Consider waiting times and frequency
// 4. Return ranked list of bottleneck activities

Effort: 4-6 hours

---

1.2 Variant Analysis & Clustering

Method: analyzeVariants(const EventLog& log, int top_n)

Implementation:

// Identify and rank process variants
// 1. Extract unique activity sequences
// 2. Calculate frequency and percentage
// 3. Compute average duration per variant
// 4. Return top-N variants ranked by frequency

Method: clusterVariants(const EventLog& log, int num_clusters)

Implementation:

// Cluster similar variants using vector embeddings
// 1. Convert activity sequences to embeddings
// 2. Use k-means clustering on embeddings
// 3. Group variants by cluster
// 4. Return cluster assignments

Effort: 6-8 hours

---

1.3 Social Network Mining

New Method: extractSocialNetwork(const EventLog& log) -> SocialNetworkGraph

Implementation:

struct SocialNetworkGraph {
    std::map<std::string, std::vector<std::string>> handovers;  // resource -> [resources]
    std::map<std::string, int> workload;                        // resource -> task_count
    std::map<std::string, double> avg_duration;                 // resource -> avg_time_ms
};

// Extract collaboration patterns
// 1. Track resource-to-resource handovers
// 2. Calculate workload distribution
// 3. Identify collaboration patterns
// 4. Build social network graph

Effort: 4-6 hours

---

1.4 Performance Enhancement Functions

Method: enhanceWithPerformance(const DiscoveredProcess& model, const EventLog& log)

Complete Implementation:

// Add performance data to process model
// 1. Calculate node avg_duration from log
// 2. Calculate node avg_waiting time
// 3. Calculate node frequency
// 4. Calculate edge probabilities
// 5. Identify bottlenecks and rework loops

Effort: 4-6 hours

---

Testing Requirements

Google Test Cases (10+):

1. TEST_F(ProcessMiningTest, BottleneckDetection)
2. TEST_F(ProcessMiningTest, VariantAnalysis)
3. TEST_F(ProcessMiningTest, VariantClustering)
4. TEST_F(ProcessMiningTest, SocialNetworkExtraction)
5. TEST_F(ProcessMiningTest, PerformanceEnhancement)
6. TEST_F(ProcessMiningTest, ComplexEventLog)
7. TEST_F(ProcessMiningTest, EdgeCases_EmptyLog)
8. TEST_F(ProcessMiningTest, EdgeCases_SingleCase)
9. TEST_F(ProcessMiningTest, Integration_FullPipeline)
10. TEST_F(ProcessMiningTest, Concurrent_MultiThread)

Google Benchmarks (3-4):

1. BM_ProcessMining_EventLogExtraction
2. BM_ProcessMining_ProcessDiscovery
3. BM_ProcessMining_VariantAnalysis
4. BM_ProcessMining_BottleneckDetection

---

2. OLAP Analytics (50% → 100%)

Current Implementation Status

Already Implemented (✅):

• Simple GROUP BY operations
• Basic aggregations (SUM, COUNT, AVG, MIN, MAX)
• CUBE/ROLLUP/GROUPING SETS structure
• Multi-dimensional grouping framework

Lines of Code: ~800 lines

To Implement (⏳)

2.1 Window Functions

Methods: executeWindowFunction(const WindowFunctionQuery& query)

Window Functions to Implement:

// ROW_NUMBER() - Sequential numbering
// RANK() - Ranking with gaps
// DENSE_RANK() - Ranking without gaps
// NTILE(n) - Divide into N buckets
// LAG(column, offset) - Previous row value
// LEAD(column, offset) - Next row value
// FIRST_VALUE(column) - First in window
// LAST_VALUE(column) - Last in window

Effort: 8-10 hours

---

2.2 Advanced Aggregations

Functions to Implement:

// STDDEV() - Standard deviation
// STDDEV_POP() - Population standard deviation
// VARIANCE() - Variance
// VAR_POP() - Population variance
// PERCENTILE_CONT(p) - Continuous percentile
// PERCENTILE_DISC(p) - Discrete percentile
// MEDIAN() - 50th percentile
// MODE() - Most frequent value

Effort: 6-8 hours

---

2.3 Apache Arrow Integration

New Class: ArrowOLAPEngine

Implementation:

class ArrowOLAPEngine {
public:
    // Convert query results to Arrow format
    arrow::Result<std::shared_ptr<arrow::Table>> executeToArrow(const OLAPQuery& query);
    
    // Columnar aggregation using Arrow compute functions
    arrow::Result<std::shared_ptr<arrow::Array>> computeAggregation(
        const arrow::ChunkedArray& column,
        AggregationType type
    );
    
    // Zero-copy data sharing
    std::shared_ptr<arrow::RecordBatch> getRecordBatch();
};

Effort: 6-8 hours

---

2.4 Columnar Storage Optimization

New Method: optimizeForOLAP(const std::string& collection) -> Status

Implementation:

// Optimize collection for OLAP queries
// 1. Convert to columnar format
// 2. Create column statistics
// 3. Build bitmap indexes
// 4. Configure compression (dictionary encoding, RLE)

Effort: 4-6 hours

---

Testing Requirements

Google Test Cases (12+):

1. TEST_F(OLAPTest, WindowFunction_ROW_NUMBER)
2. TEST_F(OLAPTest, WindowFunction_RANK)
3. TEST_F(OLAPTest, WindowFunction_LAG_LEAD)
4. TEST_F(OLAPTest, Aggregation_STDDEV)
5. TEST_F(OLAPTest, Aggregation_PERCENTILE)
6. TEST_F(OLAPTest, ArrowIntegration_ToTable)
7. TEST_F(OLAPTest, ArrowIntegration_Aggregation)
8. TEST_F(OLAPTest, ColumnarOptimization)
9. TEST_F(OLAPTest, Complex_MultiDimensional)
10. TEST_F(OLAPTest, EdgeCases_NullHandling)
11. TEST_F(OLAPTest, Performance_LargeDataset)
12. TEST_F(OLAPTest, Concurrent_Queries)

Google Benchmarks (4-5):

1. BM_OLAP_SimpleGroupBy
2. BM_OLAP_WindowFunctions
3. BM_OLAP_CubeQuery
4. BM_OLAP_ArrowIntegration
5. BM_OLAP_ColumnarVsRow

---

3. Stream Protocol (60% → 100%)

Current Implementation Status

Already Implemented (✅):

• Frame encoding/decoding
• CRC32 checksum
• Session management
• Basic protocol structure
• Header parsing

Lines of Code: ~600 lines

To Implement (⏳)

3.1 LZ4 Compression

Method: compressFrame(const Frame& frame, CompressionType type)

Implementation:

#ifdef THEMIS_ENABLE_LZ4
std::vector<uint8_t> compressLZ4(const uint8_t* data, size_t size) {
    int max_compressed = LZ4_compressBound(size);
    std::vector<uint8_t> compressed(max_compressed);
    int compressed_size = LZ4_compress_default(
        reinterpret_cast<const char*>(data),
        reinterpret_cast<char*>(compressed.data()),
        size,
        max_compressed
    );
    compressed.resize(compressed_size);
    return compressed;
}
#endif

Effort: 3-4 hours

---

3.2 Zstd Compression

Method: compressZstd(const uint8_t* data, size_t size, int level)

Implementation:

#ifdef THEMIS_ENABLE_ZSTD
std::vector<uint8_t> compressZstd(const uint8_t* data, size_t size, int level = 3) {
    size_t max_compressed = ZSTD_compressBound(size);
    std::vector<uint8_t> compressed(max_compressed);
    size_t compressed_size = ZSTD_compress(
        compressed.data(),
        max_compressed,
        data,
        size,
        level
    );
    compressed.resize(compressed_size);
    return compressed;
}
#endif

Effort: 3-4 hours

---

3.3 AES-256-GCM Encryption

Method: encryptFrame(const Frame& frame, const EncryptionKey& key)

Implementation:

std::vector<uint8_t> encryptAES256GCM(
    const uint8_t* plaintext,
    size_t plaintext_len,
    const uint8_t* key,
    const uint8_t* iv,
    const uint8_t* aad,
    size_t aad_len
) {
    EVP_CIPHER_CTX* ctx = EVP_CIPHER_CTX_new();
    std::vector<uint8_t> ciphertext(plaintext_len + EVP_GCM_TLS_TAG_LEN);
    
    // Initialize encryption
    EVP_EncryptInit_ex(ctx, EVP_aes_256_gcm(), nullptr, key, iv);
    
    // Add AAD
    int len;
    EVP_EncryptUpdate(ctx, nullptr, &len, aad, aad_len);
    
    // Encrypt
    EVP_EncryptUpdate(ctx, ciphertext.data(), &len, plaintext, plaintext_len);
    int ciphertext_len = len;
    
    // Finalize
    EVP_EncryptFinal_ex(ctx, ciphertext.data() + len, &len);
    ciphertext_len += len;
    
    // Get tag
    EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_GCM_GET_TAG, EVP_GCM_TLS_TAG_LEN, 
                        ciphertext.data() + ciphertext_len);
    
    EVP_CIPHER_CTX_free(ctx);
    ciphertext.resize(ciphertext_len + EVP_GCM_TLS_TAG_LEN);
    return ciphertext;
}

Effort: 6-8 hours

---

3.4 Flow Control (Sliding Window)

Method: updateFlowControl(uint32_t ack_seq, uint32_t window_size)

Implementation:

class FlowController {
    uint32_t send_window_size_ = 65536;  // bytes
    uint32_t recv_window_size_ = 65536;
    uint32_t send_seq_ = 0;
    uint32_t ack_seq_ = 0;
    std::map<uint32_t, Frame> unacked_frames_;
    
public:
    bool canSend(size_t frame_size);
    void onFrameSent(const Frame& frame);
    void onAckReceived(uint32_t seq);
    void onWindowUpdate(uint32_t new_size);
    std::vector<Frame> getRetransmitFrames();
};

Effort: 6-8 hours

---

Testing Requirements

Google Test Cases (15+):

1. TEST_F(StreamProtocolTest, LZ4_Compression)
2. TEST_F(StreamProtocolTest, LZ4_Decompression)
3. TEST_F(StreamProtocolTest, Zstd_Compression)
4. TEST_F(StreamProtocolTest, Zstd_Decompression)
5. TEST_F(StreamProtocolTest, AES256GCM_Encryption)
6. TEST_F(StreamProtocolTest, AES256GCM_Decryption)
7. TEST_F(StreamProtocolTest, FlowControl_SlidingWindow)
8. TEST_F(StreamProtocolTest, FlowControl_Acknowledgment)
9. TEST_F(StreamProtocolTest, FlowControl_Retransmission)
10. TEST_F(StreamProtocolTest, Integration_CompressAndEncrypt)
11. TEST_F(StreamProtocolTest, EdgeCases_LargeFrame)
12. TEST_F(StreamProtocolTest, EdgeCases_WindowExhaustion)
13. TEST_F(StreamProtocolTest, Performance_Throughput)
14. TEST_F(StreamProtocolTest, Concurrent_MultiSession)
15. TEST_F(StreamProtocolTest, Security_EncryptionKeyRotation)

Google Benchmarks (3-4):

1. BM_StreamProtocol_LZ4_Compression
2. BM_StreamProtocol_Zstd_Compression
3. BM_StreamProtocol_AES256GCM_Encryption
4. BM_StreamProtocol_EndToEnd_Throughput

---

4. Video Processor (40% → 100%)

Current Implementation Status

Already Implemented (✅):

• Plugin framework structure
• Metadata extraction structure
• MIME type handling
• File format detection

Lines of Code: ~400 lines

To Implement (⏳)

4.1 FFmpeg/LibAV Integration

Method: initialize(const PluginConfig& config)

Implementation:

bool VideoProcessor::initialize(const PluginConfig& config) {
    #ifdef THEMIS_ENABLE_FFMPEG
    // Initialize FFmpeg
    av_register_all();
    avformat_network_init();
    
    // Configure hardware acceleration if available
    if (config.get<bool>("hardware_accel", false)) {
        av_hwdevice_ctx_create(&hw_device_ctx_, AV_HWDEVICE_TYPE_CUDA, nullptr, nullptr, 0);
    }
    #endif
    
    initialized_ = true;
    return true;
}

Effort: 4-6 hours

---

4.2 Keyframe Extraction

Method: extractKeyframes(const std::vector<uint8_t>& data, const ProcessConfig& config)

Implementation:

#ifdef THEMIS_ENABLE_FFMPEG
std::vector<Keyframe> extractKeyframes(const char* filename, int max_keyframes) {
    AVFormatContext* fmt_ctx = avformat_alloc_context();
    avformat_open_input(&fmt_ctx, filename, nullptr, nullptr);
    avformat_find_stream_info(fmt_ctx, nullptr);
    
    int video_stream = av_find_best_stream(fmt_ctx, AVMEDIA_TYPE_VIDEO, -1, -1, nullptr, 0);
    AVCodecContext* codec_ctx = /* ... */;
    
    std::vector<Keyframe> keyframes;
    AVPacket packet;
    AVFrame* frame = av_frame_alloc();
    
    while (av_read_frame(fmt_ctx, &packet) >= 0) {
        if (packet.stream_index == video_stream) {
            if (packet.flags & AV_PKT_FLAG_KEY) {
                // Decode keyframe
                avcodec_send_packet(codec_ctx, &packet);
                avcodec_receive_frame(codec_ctx, frame);
                
                // Extract JPEG thumbnail
                auto thumbnail = convertToJPEG(frame);
                keyframes.push_back({packet.pts, thumbnail});
                
                if (keyframes.size() >= max_keyframes) break;
            }
        }
        av_packet_unref(&packet);
    }
    
    av_frame_free(&frame);
    avformat_close_input(&fmt_ctx);
    return keyframes;
}
#endif

Effort: 8-10 hours

---

4.3 Scene Detection

Method: detectScenes(const std::vector<uint8_t>& data, double threshold)

Implementation:

std::vector<SceneChange> detectScenes(const char* filename, double threshold = 0.3) {
    // Use histogram difference for scene detection
    std::vector<SceneChange> scenes;
    AVFrame* prev_frame = nullptr;
    
    while (/* read frames */) {
        if (prev_frame) {
            double diff = calculateFrameDifference(prev_frame, curr_frame);
            if (diff > threshold) {
                scenes.push_back({timestamp, diff});
            }
        }
        prev_frame = curr_frame;
    }
    
    return scenes;
}

double calculateFrameDifference(AVFrame* f1, AVFrame* f2) {
    // Compare histograms or pixel differences
    int hist1[256] = {0}, hist2[256] = {0};
    
    // Build histograms
    for (int y = 0; y < f1->height; y++) {
        for (int x = 0; x < f1->width; x++) {
            hist1[f1->data[0][y * f1->linesize[0] + x]]++;
            hist2[f2->data[0][y * f2->linesize[0] + x]]++;
        }
    }
    
    // Calculate histogram difference
    double diff = 0.0;
    for (int i = 0; i < 256; i++) {
        diff += std::abs(hist1[i] - hist2[i]);
    }
    return diff / (f1->width * f1->height);
}

Effort: 6-8 hours

---

4.4 Subtitle Extraction

Method: extractSubtitles(const std::vector<uint8_t>& data)

Implementation:

std::vector<Subtitle> extractSubtitles(const char* filename) {
    AVFormatContext* fmt_ctx = /* ... */;
    int subtitle_stream = av_find_best_stream(fmt_ctx, AVMEDIA_TYPE_SUBTITLE, -1, -1, nullptr, 0);
    
    std::vector<Subtitle> subtitles;
    AVPacket packet;
    
    while (av_read_frame(fmt_ctx, &packet) >= 0) {
        if (packet.stream_index == subtitle_stream) {
            AVSubtitle subtitle;
            int got_subtitle;
            avcodec_decode_subtitle2(codec_ctx, &subtitle, &got_subtitle, &packet);
            
            if (got_subtitle) {
                for (unsigned i = 0; i < subtitle.num_rects; i++) {
                    if (subtitle.rects[i]->type == SUBTITLE_TEXT) {
                        subtitles.push_back({
                            subtitle.start_display_time,
                            subtitle.end_display_time,
                            subtitle.rects[i]->text
                        });
                    }
                }
            }
            avsubtitle_free(&subtitle);
        }
        av_packet_unref(&packet);
    }
    
    return subtitles;
}

Effort: 4-6 hours

---

Testing Requirements

Google Test Cases (12+):

1. TEST_F(VideoProcessorTest, FFmpeg_Initialization)
2. TEST_F(VideoProcessorTest, Metadata_Extraction)
3. TEST_F(VideoProcessorTest, Keyframe_Extraction)
4. TEST_F(VideoProcessorTest, Scene_Detection)
5. TEST_F(VideoProcessorTest, Subtitle_Extraction)
6. TEST_F(VideoProcessorTest, Thumbnail_Generation)
7. TEST_F(VideoProcessorTest, MultipleFormats_MP4_WebM_MKV)
8. TEST_F(VideoProcessorTest, HardwareAcceleration)
9. TEST_F(VideoProcessorTest, EdgeCases_CorruptedVideo)
10. TEST_F(VideoProcessorTest, EdgeCases_NoVideoTrack)
11. TEST_F(VideoProcessorTest, Performance_LargeFile)
12. TEST_F(VideoProcessorTest, Concurrent_Processing)

Google Benchmarks (2-3):

1. BM_VideoProcessor_KeyframeExtraction
2. BM_VideoProcessor_SceneDetection
3. BM_VideoProcessor_FullPipeline

---

📅 Implementation Schedule

Week 1: Process Mining + OLAP

Days 1-3: Process Mining (70% → 100%)

• Day 1: Bottleneck detection + variant analysis
• Day 2: Social network mining + performance enhancement
• Day 3: Google Tests + Benchmarks

Days 4-7: OLAP Analytics (50% → 100%)

• Day 4-5: Window functions + advanced aggregations
• Day 6: Apache Arrow integration
• Day 7: Google Tests + Benchmarks

Week 2: Stream Protocol + Video Processor

Days 8-11: Stream Protocol (60% → 100%)

• Day 8: LZ4/Zstd compression
• Day 9: AES-256-GCM encryption
• Day 10: Flow control
• Day 11: Google Tests + Benchmarks

Days 12-16: Video Processor (40% → 100%)

• Day 12-13: FFmpeg integration + keyframe extraction
• Day 14: Scene detection
• Day 15: Subtitle extraction + thumbnail generation
• Day 16: Google Tests + Benchmarks

---

✅ Success Criteria

Code Quality:

• All features 100% implemented

• 90% test coverage maintained

• All Google Test cases passing
• All Google Benchmarks running

Documentation:

• Implementation details documented
• API usage examples provided
• Integration guides complete

Performance:

• Benchmarks show acceptable performance
• No regressions in existing features

---

📊 Expected Final Metrics

Upon v1.3.0 + v1.4.0 Completion:

Metric	Current	Target
Total Features	6	10
Test Cases	45	94+
Benchmark Suites	20	35+
Code Coverage	>90%	>90%
Implementation Lines	1,200	3,200+
Test Lines	1,675	3,500+
Documentation Files	9	11+
Documentation Size	79 KB	110+ KB
Code Quality	92%	94%+

---

Next Step: Begin Process Mining implementation (fastest to complete, high value)