Flush throughput ceiling: global delta_commit_lock lets un-flushed WAL grow to ~25 GB → multi-min startup replay

[tonyalaribe]

Summary

Under high ingest across many (project_id, table) shards, flush-to-Delta cannot keep the WAL read-cursor near the tail, so the un-flushed WAL grows without bound (~25 GB / 65k+ entries observed). Commit throughput is capped by a single process-wide delta_commit_lock, not by FLUSH_PARALLELISM. Memory backpressure is keyed to buffer bytes (30 GB cap), not flush lag, so ingest is acked at full rate while the WAL falls arbitrarily behind. On restart, replay must re-read the whole backlog from the persisted cursor to the tail — many minutes → ~10-min restart outage.

Evidence (prod 2026-07-13)

• recover_from_wal: WAL replay: memory pressure at entry 65685 (25313MB buffered) — draining oldest buckets in background (relief #9) — ~25 GB / 65k un-flushed entries, hit the 30 GB budget 9×.
• bootstrap.phase=buffered_write_layer_init elapsed_ms=11733 then multi-minute replay.
• flush_interval=300s → 46,692 entries replayed in 95.6s; after flush_interval=60s the WAL was larger (~65k / 25 GB). Shortening the interval did not help.
• Config: FLUSH_INTERVAL_SECS=60, FLUSH_PARALLELISM=2, BUFFER_MAX_MEMORY_MB=30000, retention 70 min, bucket duration 300s.

Root cause

Every Delta commit across all topics serializes its refresh + commit-log append behind one tokio::sync::Mutex<()> delta_commit_lock (src/database.rs:1089; critical section :3393-3429). Staged parquet encode/upload is moved outside the lock (:3320-3373), so FLUSH_PARALLELISM (buffer_unordered, src/buffered_write_layer.rs:1618-1625) only parallelizes staging — the commit itself is serial, and OCC conflict retries re-serialize under contention (:3433-3441). Once N_topics × serial_commit_cost exceeds the ingest window, flush falls permanently behind. WAL GC can't reclaim it (floor pinned at oldest-unflushed append, buffered_write_layer.rs:1382). The cursor advances only after a successful commit (:207, :1676 → :1900).

The flush interval is not the lever: flush eligibility is gated on the 300s bucket seal (buffered_write_layer.rs:1516; bucket id = event_ts / bucket_duration, mem_buffer.rs:869, duration config.rs:157), not on flush_interval. A shorter interval only polls more often — it can't flush data sooner and doesn't create more/smaller commits (coalescing collapses a cycle per topic, :1557). So 60s is a harmless no-op for throughput.

Backpressure is keyed to memory, not lag: reserve_with_backpressure (:631) blocks only near the 30 GB buffer cap; there is no admission control on unflushed-WAL bytes / cursor lag, so a 25 GB WAL accumulates long before memory backpressure engages. Only coarse safety valve: wal_max_file_count force-flush (:1462).

Proposed fix (highest-leverage first)

1. Make the commit lock per-(project,table) / per-table-shard instead of process-wide (database.rs:1089). Each Delta table has its own log; the global lock needlessly serializes cross-topic appends. This is the actual throughput ceiling — until it's per-table, FLUSH_PARALLELISM does nothing for the commit phase.
2. Then raise FLUSH_PARALLELISM (prod=2 vs default 8) — only meaningful after Issues #1.
3. Add WAL-lag admission control (gate ingest on unflushed-WAL bytes / cursor lag), or lower BUFFER_MAX_MEMORY_MB so memory backpressure engages before the WAL bloats; add a byte/lag force-flush threshold alongside wal_max_file_count.
4. Amortize per-commit fixed cost (larger coalesced batch / rows-per-commit).

Severity / impact

High. Chronic flush-behind → bloated WAL (25 GB observed) → multi-minute replay → ~10-min restart outage. Secondary: if the buffer hits the 30 GB cap while flush is stalled, reserve_with_backpressure rejects writes after the 60s timeout (wal_admit_decouple OFF in prod) — a data-loss seam unless the upstream DLQ retries. Self-reinforcing in a crash-loop: each restart re-replays the whole backlog and re-hits memory pressure.