Performance: 680MB / 5.5M lines syslog takes 4 minutes to parse

[daiminghuan]

Environment

• OS: Windows 11
• ctrlb-decompose: release build

Reproduction

time ctrlb-decompose  syslog-2025-07-12_02-53-02

Result

• File size: 680 MB
• Lines: 5,498,990
• Output: 642 patterns
• Time: real 4m1.614s

[daiminghuan]

[adarshsrivastava11]

Hi @daiminghuan, thanks for pointing it out. Could you share the CPU usage during the time the process was running? It will help us better understand what causes the wall clock time to increase.

[daiminghuan]

@adarshsrivastava11 Sure , Sorry, the video is too large to upload, but the CPU usage stayed at around 4% throughout the entire process.

[adarshsrivastava11]

Hi @daiminghuan, thanks for sharing. This gives us a valuable insight that threads are waiting, hence it is not CPU-bound. Would it be possible for you to also add IOPS and memory consumption numbers so we can determine whether IO or memory is the bottleneck?

Meanwhile, we'll also try to reproduce this. Thanks!

[daiminghuan]

@adarshsrivastava11 Here are the IOPS and memory consumption numbers you requested.

Summary

Metric	Value
Wall Clock Time	336.65 seconds
Effective Throughput	2.02 MB/s
Peak Working Set (RSS)	159.92 MB
Peak Private Bytes	161.24 MB
Memory / File Size Ratio	23.6%
Disk Type	HDD

Memory Timeline

Memory grows steadily from 19 MB to 160 MB over 336 seconds — no spikes, no pressure, no swapping:

• 0s: 19 MB
• 60s: 81 MB
• 120s: 100 MB
• 180s: 112 MB
• 240s: 134 MB
• 336s: 160 MB (peak)

Environment

• OS: Windows 11 Pro
• CPU: 13th Gen Intel Core i7-13700 (16 cores)
• RAM: 31.6 GB
• Disk: HDD

Full performance report attached below.

[daiminghuan]

📎 Full Performance Report (perf_results.txt)

================================================================================
ctrlb-decompose Performance Report
================================================================================
Date: 2026-05-25 11:34:50
OS: Microsoft Windows 11 Pro
CPU: 13th Gen Intel(R) Core(TM) i7-13700
RAM: 31.6 GB
Disk: HDD

--- Input ---
File: syslog-2025-07-12_02-53-02
File Size: 679.03 MB (712015872 bytes)

--- Execution ---
Wall Clock Time: 336.65 seconds
Effective Throughput: 2.02 MB/s (file_size / wall_time)

--- Memory Usage ---
Peak Working Set (RSS): 159.92 MB
Peak Private Bytes: 161.24 MB
Samples Collected: 331

--- Memory Timeline (sampled every 1s, showing key points) ---
ElapsedSec | WorkingSet_MB | PrivateBytes_MB
       0.6 |         18.70 |           16.71
      10.8 |         41.11 |           40.04
      30.0 |         66.91 |           66.05
      60.6 |         80.85 |           80.25
      90.1 |         94.15 |           93.72
     120.7 |        100.22 |           99.91
     150.3 |        106.71 |          106.56
     180.9 |        110.79 |          110.79
     210.4 |        114.65 |          114.90
     240.7 |        134.19 |          134.61
     270.0 |        144.29 |          144.73
     300.2 |        152.02 |          152.62
     335.6 |        159.92 |          161.24

--- Analysis ---
- Peak memory / file size ratio: 23.6%
- Throughput: 2.02 MB/s
- At 2.02 MB/s for 679.03 MB file, the process is NOT saturating disk
- Typical SSD sequential read: 500-3000 MB/s; HDD: 80-160 MB/s
- CPU stayed at ~4% suggesting single-threaded, not CPU-bound

================================================================================