Bug: Bound durable startup benchmark write batches

[smiggleworth]

Type

Bug

Affected subsystem

Tier 3 durable startup/recovery benchmark fixture construction.

User/operator impact

The canonical benchmark suite cannot produce retained evidence: its 100k startup fixture writes all documents in one storage batch, creating a WAL record whose legitimate decompressed size exceeds Midge's 64 MiB recovery limit. Reopening then reports corruption and aborts the suite.

Current behavior

put_bench_documents builds the full dataset and calls Midge::put_documents once. Canonical run 32521755392 successfully completed the repaired Tier 3 query owner, then startup_reopen/100k failed with LZ4 declared output size 81283353 exceeds 67108864 byte limit.

Expected behavior/outcome

The benchmark fixture must construct representative durable data in bounded batches outside the measured reopen path, then reopen and verify the first/last documents and catalog cardinality without changing Midge's storage format, public API, or atomic batch semantics.

Deterministic evidence

Run 32521755392 failed at benches/support/workloads/lifecycle.rs:49 after the 100k fixture shutdown. Source inspection shows the complete 100k vector is submitted as one write batch. The declared LZ4 output size is about 77.5 MiB, above the storage decoder's 64 MiB limit.

Confidence and support status

High confidence. This is a deterministic fixture/storage-boundary mismatch, not benchmark variance. Product-side batch splitting would change atomicity and is outside this issue; fixture setup can use multiple existing atomic calls because setup is unmeasured.

Acceptance criteria

• Add a focused red/green contract that prevents the durable startup fixture from submitting an unbounded document batch.
• Keep batching outside the measured reopen operation and preserve exact 100k fixture cardinality/boundary checks.
• Prove the focused durable startup benchmark reopens successfully.
• Pass the complete required validation ladder and exact-head backend CI.
• Rerun final-main canonical evidence only after the repair merges.

Priority and dependencies

P0 recovery/evidence blocker discovered in #7 after PR #158 repaired the preceding Tier 3 query blocker.

Next action

Implement bounded fixture-only writes, run the focused startup owner, validate and merge a narrow repair, then redispatch canonical evidence from the new final main SHA.

[smiggleworth]

Repair PR #166 is open at exact head a706e34616a71666f23aee003805efac1a95985e. The source contract is red/green, the exact 100k native-disk startup owner successfully shut down and reopened with 100,000 rows and zero failed operations/leaked workers, and the complete ordered local ladder passed. Exact-head backend CI is pending; #7 and #165 remain open.

[smiggleworth]

Canonical final-main run 32751589448 disproved closure of this repair at commit 4ae7ede94b95fd579ab6fd1a221c766175f33d0d. The repaired Tier 3 query owner completed, but tier3_system_startup again failed while building its durable 100k fixture:

• run: https://github.com/cntryl/cassie/actions/runs/32751589448
• owner-suite duration before failure: 1h19m42s
• failure: prebuilt Tier 3 startup fixture: StorageMissingFamily("midge key not found")
• location: benches/tier3_system_startup.rs:31
• manifest validation/upload: skipped because the unfiltered owner suite failed

A standalone rerun at the same source revision passed, confirming that the 10,000-row cap removed the deterministic oversized WAL record but left an intermittent large-write fixture failure. #7 remains blocked. Next action: tighten the fixture-only write bound with a focused red/green contract, rerun the exact startup owner and full validation ladder, merge the repair, then dispatch canonical evidence from the new final main SHA.