T19: 72-hour measurement run

[thedancingdeveloper]

The P1 deliverable is this measurement, not the code.

Run the fleet 72 hours post-change and record, against the §2.1 and §2.5 baselines:

• rate-limit error count, broken down by class (baseline: 27,662 undifferentiated over 8 days)
• delivery rate (baseline: 56%)
• patch-apply rate (baseline: 25–80%, model-dependent)
• review_rejected rate (baseline: 2.5%)

If the apply rate falls to the bottom of the band, that is a finding, not a failure. Record it and re-evaluate the implementer tier before P3 hardcodes the role map. Extra repair rounds can consume the cost saving.

Acceptance: numbers posted in this issue; §2 of the plan updated; explicit go/no-go on the implementer tier.

[thedancingdeveloper]

Not started — this is what P1 is still missing, and it is the deliverable. §9 is explicit: "Half a day of code, then 72 hours of measurement. The measurement is the deliverable, not the code."

The code is in swack-tools/oxidex#417 and is unmerged. Even once merged, the exit criteria stay unmet until a 72-hour run produces:

• rate-limit error count broken down by class, against the §2.1 baseline of 27,662 / 8 days;
• delivery rate and patch-apply rate against the §2.5 baseline of 56% and 25–80%.

The first is now answerable for the first time: model-calls.jsonl carries error_class per attempt, so rpm vs window_cap vs terminal_cap is a group-by rather than a guess. The 27,662 baseline itself is not decomposable — the harness that produced it never read the discriminator — so the comparison is a total against a breakdown, and that limitation is now recorded in docs/AI_HARNESS.md.

Requires the fleet host. Blocked on someone running it.

[thedancingdeveloper]

The instrument now exists; the run still does not.

scripts/rate_limit_report.py is in swack-tools/oxidex#417 (+27 tests). It reads model-calls.jsonl and prints exactly the first exit criterion — 429s by class, per day, optionally per role/model/endpoint — plus the baseline comparison, and it refuses to present a per-class delta because the baseline has no per-class value to compare against.

So T19 is now a single command on the fleet host:

uv run scripts/rate_limit_report.py --since 72h --by role --by day
uv run scripts/rate_limit_report.py --since 72h --json    # for the P2 ingester

Verified this host is not the fleet host: no ~/.oxidex, no config.toml, no fleet-ops worktree, no worker processes. /home/sprooty/oxidex is a dev checkout. So the 72-hour run cannot be started from here, and no amount of tooling changes that.

Remaining, in order:

1. review + merge swack-tools/oxidex#417 (blocked: REVIEW_REQUIRED on oxidex main);
2. deploy the classifying worker to the fleet;
3. run 72 hours;
4. run the report, and fleet_failrate.py for delivery and patch-apply rate against the §2.5 baseline of 56% / 25–80%.

[thedancingdeveloper]

Re-scoped by the generic ruling. This was 'a 72-hour run of a specific fleet, measured against that fleet's 27,662/8-day baseline'. The harness is not tied to that workload, so neither is this task.

What it becomes: run the harness against a real workload, under real load, against a real provider, long enough for the failure model to be exercised. The measurement machinery is ready and generic:

• ModelClient emits one classified record per attempt;
• ingest --events reads that stream with no adapter;
• the dashboard breaks it down by class, worker, endpoint and role;
• --baseline TOTAL:DAYS:LABEL compares against whatever prior measurement that workload has, and still refuses to imply a per-class delta if the baseline predates classification.

What is still unproven, and what the run has to establish:

1. the classifier's kinds match what a real provider actually returns under sustained load — the one live probe already corrected two wrong assumptions, so expect more;
2. parking an endpoint on a cap actually lets a fleet ride out a window unattended;
3. jittered per-worker backoff actually prevents the synchronised-burst behaviour, rather than merely being designed to.

None of that can be simulated. It needs a workload and elapsed time.

[thedancingdeveloper]

Status: this and #44, #51, #56 are one thing gated on the same event — a first real run.

They form a progression, longest last:

#33	72 hours, measuring rate limits by class against the baseline
#44	48 hours with no ingester restarts
#51	7 days unattended
#56	a second, different workload

None can start until an agent has been given a real key and a real task, which has never happened. Every path is tested — 240 tests, driven end to end against real git repositories with scripted models — but scripted is the operative word.

The tooling is ready: rate_limit_report.py produces the breakdown, /api/errors serves it, and /api/control can stop the fleet if it misbehaves.

What is needed is a decision, not code: a first real run costs money and takes elapsed time. Suggest starting deliberately small — one item, one throwaway repository — which proves the chain end to end without committing to 72 hours of spend.

[thedancingdeveloper]

Status update, 2026-08-04. Still blocked, and still the measurement AGENTS.md rule 1 judges this work by.

What is now in place that was not. The fit-for-purpose programme landed stages F, D, K, H, L, J and M on fix/validator-rejects-valid-patches (1063 tests, ruff, format and full-project strict mypy all clean). Several of them exist precisely to make a long run survivable:

• Per-item wall-clock and spend ceilings (Stage L). Previously an item that heartbeated forever was indistinguishable from one making progress — the failure mode a long unattended run is most likely to produce. Ceilings default to unlimited, so they must be set explicitly before a run of this kind; doctor warns when they are not.
• Resumable attempts (Stage H). A killed worker resumes at its last durable stage instead of re-paying for the planner and implementer. On the measured fixture that is 3 model calls against 5 for one induced crash.
• A typed outcome taxonomy (Stage K). A reviewer rejection, a crashed worker, a budget stop and a gate that could not run are now four distinguishable states rather than all failed, which is the difference between a readable post-run report and a pile of failures.
• A durable human hold (Stage J) and doctor (Stage D), so a run can be pre-flighted and a question does not silently consume a worker.

What still blocks this. Unchanged: credentials, network access, a real second repository, and the NGMS human decisions that §9 of docs/PROPOSAL-2026-08-fit-for-purpose.md explicitly refuses to answer by assertion. None of it is startable from the repository alone.

Two caveats that matter for this specific run. Stages H and L are the direct-API executor only — a session-mode fleet has no resumable attempts and no per-item budgets. And nothing in the programme is evidence about live behaviour: no provider was contacted at any point, in any stage.

Full context: docs/evidence/2026-08-04-programme-report.md.