testing: fuzz persisted-format decoders with bounded CI smoke

[smiggleworth]

Summary

Add deterministic fuzz targets for Pants' persisted-format decoders and a bounded scheduled/manual CI smoke lane. The goal is architectural and parser-safety evidence—not an unbounded PR gate or a giant data-volume test.

Evidence

Audited baselines:

• Pants: 28585a10551d9c636f2c945ec39d4f21aab6f1f3 (origin/main)
• Midge: 115bf31a62886e74f4226fe59c97af6e2c3e6c14 (origin/main)

Pants has extensive focused/adversarial xUnit coverage, but no fuzz target/corpus/workflow. Current Midge has maintained fuzz targets and corpora for WAL frame recovery, manifest journal replay, transaction intent replay, and SST block decoding.

Focused tests lock known invariants; fuzzing should explore combinations, truncations, lengths, checksums, and state-machine transitions that were not enumerated manually.

Requirements

• Expose small deterministic internal decoder/replay seams that accept bounded bytes/state and return typed success/corruption outcomes without filesystem, network, wall-clock, or global-state dependencies.
• Add fuzz targets for at least:
  • WAL frame/envelope and transaction-batch recovery;
  • SST block/index/footer/meta decoding;
  • manifest journal replay;
  • transaction intent/spill log replay;
  • lease record decoding if the format is independently parseable.
• Seed each target with committed valid/minimal/corrupt examples derived from existing Midge/Pants compatibility fixtures.
• Preserve semantic assertions beyond "does not crash":
  • valid inputs round-trip or produce the expected logical state;
  • corrupt/truncated/non-canonical inputs fail closed;
  • recovery preserves only valid committed prefixes;
  • no unbounded allocation, recursion, loop, or pathological retained state is permitted for bounded input.
• Add explicit per-input limits for byte length, decoded counts, nested records, declared lengths, and allocation budgets.
• Make every discovered failure reproducible as a normal deterministic test; retain a minimized corpus input with provenance.
• Add a bounded scheduled and manually dispatchable CI lane with fixed per-target time budgets and artifact upload for reproducing failures.
• Keep ordinary PR tests deterministic and fast. A short corpus replay may run on PRs, but open-ended mutation fuzzing must not make PR duration nondeterministic.
• Ensure fuzz jobs never overwrite committed fixtures, mutate a real database, use cloud credentials, or delete uncertain storage.
• Keep the implementation inside the repository's approved test structure. Do not introduce a standalone CLI/tooling project without explicit agreement.

Acceptance criteria

• Every required decoder/replay family has at least one executable fuzz target and a useful seed corpus.
• Known valid fixtures succeed and known corrupt fixtures fail closed before random mutation begins.
• Tests prove declared-length/count bombs are rejected before large allocation or long iteration.
• A seeded synthetic bug/failpoint demonstrates the workflow produces a reproducible artifact and non-zero result.
• Any corpus input can be replayed locally with one documented deterministic command.
• Minimized regressions are promoted to normal xUnit tests before a fuzz finding is closed.
• Scheduled/manual CI has explicit time, input-size, and artifact-retention bounds.
• PR validation remains deterministic; corpus replay does not write the working tree.
• FORMAT v3 and committed compatibility fixtures are not regenerated by fuzz tests.
• Formatting, Release build, focused corpus replay, and full Release tests pass.

Suggested TDD sequence

1. Add direct tests for bounded decoder seams and allocation/count guards.
2. Check in the minimal seed corpora and deterministic replay command.
3. Add one target at a time, beginning with WAL and SST.
4. Prove the artifact/replay loop with an injected test-only failure.
5. Add manifest, intent, and lease targets.
6. Wire bounded scheduled/manual CI and document triage.

Dependencies and non-goals

• Coordinate corpus ownership with compatibility: restore executable bidirectional Midge qualification and refresh the baseline #224 so compatibility fixtures remain authoritative and immutable during verification.
• This issue does not require 260 million keys, a permanent fuzzing service, or live cloud accounts.
• Fuzz success is not a substitute for the deterministic persisted-format contract tests.