Follow-up to #4567.
Explore a reusable, bounded parallel read/decode facility in the journal layer so QMDB and other consumers can benefit. Keep index reconstruction with the consuming data structure.
Preserve ordered results and errors, bounded buffering/backpressure, and cancellation cleanup. QMDB currently projects operations to keys and routes them inside decoder tasks; preserve that ability to avoid buffering full values or moving routing back onto one coordinator. Partition updates, collision handling, and activity-bitmap construction belong in QMDB.
Archive needs separate integration:
- Prunable: rebuilds its indexes from segmented fixed-index replay through
Oversized. Parallel work must preserve durable validation floors, each section's first-invalid-value boundary, and repair ordering.
- Immutable: restores a checkpoint and scans its Freezer table and Ordinal data. Optimizing contiguous-journal replay alone will not accelerate those scans.
Use complete QMDB initialization and archive restart benchmarks to guide the extraction and verify benefits. Include clean startup and prunable recovery with an unmarked suffix, and separate replay/decode costs from index reconstruction and value validation. Preserve the gains from #4567.
Follow-up to #4567.
Explore a reusable, bounded parallel read/decode facility in the journal layer so QMDB and other consumers can benefit. Keep index reconstruction with the consuming data structure.
Preserve ordered results and errors, bounded buffering/backpressure, and cancellation cleanup. QMDB currently projects operations to keys and routes them inside decoder tasks; preserve that ability to avoid buffering full values or moving routing back onto one coordinator. Partition updates, collision handling, and activity-bitmap construction belong in QMDB.
Archive needs separate integration:
Oversized. Parallel work must preserve durable validation floors, each section's first-invalid-value boundary, and repair ordering.Use complete QMDB initialization and archive restart benchmarks to guide the extraction and verify benefits. Include clean startup and prunable recovery with an unmarked suffix, and separate replay/decode costs from index reconstruction and value validation. Preserve the gains from #4567.