T30: 48-hour soak

[thedancingdeveloper]

Run against live fleet traffic for 48 hours.

Acceptance: no ingester restarts; no dropped events; store size growth within expectation; all five panels correct at the end of the window.

[thedancingdeveloper]

Not started — needs a deploy (T29 → D7) and a live fleet.

48 hours against live traffic with no ingester restarts cannot be simulated. What can be said now:

• ingest is idempotent, so a restart mid-soak is recoverable rather than corrupting — a restarted ingester re-reads and inserts only what is new (verified on 5,300 records);
• a missing or not-yet-upgraded source is not an error, so the soak can start before the classifying worker is deployed everywhere;
• SSE connections recycle on a bounded lifetime, so an idle tab open for 48 hours will not pin a connection for the whole soak.

This also carries the unmet half of T23: reproducing the §2.1 baseline of 27,662 from the real 8-day history. That is the check that the ingester's manifest parsing matches reality rather than matching my synthetic fixture — and it is worth doing first, before the 48-hour clock starts, because it is a single ingest run over existing logs and it either matches or exposes a parser bug.

[thedancingdeveloper]

Status update, 2026-08-04. Still blocked. A 48-hour soak is exactly the shape of run the new ceilings and resumable attempts were built for, and exactly the one nothing here can start.

What is now in place that was not. The fit-for-purpose programme landed stages F, D, K, H, L, J and M on fix/validator-rejects-valid-patches (1063 tests, ruff, format and full-project strict mypy all clean). Several of them exist precisely to make a long run survivable:

• Per-item wall-clock and spend ceilings (Stage L). Previously an item that heartbeated forever was indistinguishable from one making progress — the failure mode a long unattended run is most likely to produce. Ceilings default to unlimited, so they must be set explicitly before a run of this kind; doctor warns when they are not.
• Resumable attempts (Stage H). A killed worker resumes at its last durable stage instead of re-paying for the planner and implementer. On the measured fixture that is 3 model calls against 5 for one induced crash.
• A typed outcome taxonomy (Stage K). A reviewer rejection, a crashed worker, a budget stop and a gate that could not run are now four distinguishable states rather than all failed, which is the difference between a readable post-run report and a pile of failures.
• A durable human hold (Stage J) and doctor (Stage D), so a run can be pre-flighted and a question does not silently consume a worker.

What still blocks this. Unchanged: credentials, network access, a real second repository, and the NGMS human decisions that §9 of docs/PROPOSAL-2026-08-fit-for-purpose.md explicitly refuses to answer by assertion. None of it is startable from the repository alone.

Two caveats that matter for this specific run. Stages H and L are the direct-API executor only — a session-mode fleet has no resumable attempts and no per-item budgets. And nothing in the programme is evidence about live behaviour: no provider was contacted at any point, in any stage.

Full context: docs/evidence/2026-08-04-programme-report.md.