Re-measure two models after the harness fixes, and raise the wall-clock budget by nedonatelli · Pull Request #55 · nedonatelli/sidecar

nedonatelli · 2026-08-05T15:32:30Z

Three commits: the timeout raise, the re-measured baselines, and a sweep driver
that fails fast on a dead backend.

The result is negative, and that is the point

model	before	after
gemma4:e4b	66/70	67/70
ministral-3	61/70	62/70

32% of runs in the previous sweep had a verification call return garbage —
run_tests throwing on the pytest glob in 67 runs, tsc returning 5,301
characters of help text in 121 — and the contamination was concentrated in
exactly these two models, because they are the ones that verify their own work.

Removing all of it moved each by one case.

The defects were real and worth fixing. Their effect on scores was small. That
is only knowable by measuring, and it is worth writing down precisely because
the 32% figure invited a much larger expectation.

ministral's single gained case is no-stub-in-write, a plausible beneficiary of
the route-pin relaxation — it pinned write_file while edit_file on a missing
path is coerced into a create.

The artifact evidence is unambiguous

Ollama degraded mid-sweep when the laptop lid closed. Re-running only that
window:

case	contaminated	clean
`git-diff-not-run-command`	FAIL (1065s)	pass (8s)
`build-python-calculator`	FAIL (1140s)	pass (77s)
`rename-propagates-to-cross-file-caller`	FAIL (633s)	pass (159s)

1065 seconds to 8. Those were never capability failures.

One went the other way: fix-wrong-comparison-operator passed at 610s inside
the window and fails at 158s outside it — the contaminated run got it right by
accident.

Both baselines are stitched, and the files should be read that way

ministral — 53 cases from the interrupted sweep plus the 17 it never
reached. The 53 were clean: no unavailability, nothing over 500s until the
breaker fired.
gemma4 — 62 cases from that sweep plus 6 re-measured.

The merges refuse across a provenance boundary (model, thinking,
maxIterations, scaffold major); both halves of each were thinking-on /
maxIter 50 / scaffold 4.0.0. What provenance cannot see is that the halves
were measured hours apart, so recordedAt on both files describes only the
later half. Recorded in the commit message rather than left for someone to
discover.

Wall clock 300s -> 600s

Two caps were being conflated. DEFAULT_MAX_ITERATIONS was raised 25 -> 50 on
evidence no failing run reached it; this is the other one, never touched. So
"the ceiling is not binding" was true of iterations and false of time.

300s was still binding, measured:

ministral-3 hit it on ~29% of its cases
claude-sonnet-5 hit it on run-tests-after-fix at 300006ms

The frontier-model hit is the argument: a limit a frontier model reaches is not
separating capable models from incapable ones, it is measuring how long a
subprocess takes.

A cap-hit is not automatically a failure — the abort stops the loop and the
workspace is scored as it stands, so a truncated run can still pass. That is why
it went unnoticed while ministral hit it on nearly a third of its cases.

The doc comment above it claimed a 120s default while the code returned 300s;
corrected, and it now names the distinction from the iteration ceiling.

Case validity, independently confirmed

claude-sonnet-5 passes 70 of the 71 cases it completes, the exception being
the timeout above. After the glob fix, the fixture tsconfig, 17 relaxed route
pins, scorer changes and the tree-sitter swap, the cases are sound — so these
numbers measure models, not damage from this week.

Not re-measured

granite4.1:3b, qwen2.5-coder:7b and llama3.2 never executed a case when
Ollama wedged. Their files still hold 2026-08-02 numbers taken under every
defect above, and are NOT comparable to the two updated here.

The new driver probes generation before each model and stops the sweep when the
backend is dead, rather than letting each remaining model spend ~50 minutes
rediscovering it. Also records that caffeinate -dimsu is insufficient with the
lid closed — -s holds only on AC and clamshell sleeps regardless.

npm run check: green.

🤖 Generated with Claude Code

Two different caps were being conflated. `DEFAULT_MAX_ITERATIONS` (the ITERATION ceiling) was raised 25 -> 50 on evidence that no failing run reached it. This is the other one — wall clock per case — never touched, so "the ceiling is not binding" was true of iterations and false of time. 300s was still binding, measured rather than assumed: - ministral-3 hit it on ~29% of its cases in the 2026-08-04 sweep - claude-sonnet-5 hit it on `run-tests-after-fix` at 300006ms The frontier-model hit is the argument. A limit a frontier model reaches is not separating capable models from incapable ones; it is measuring how long a subprocess takes. Also corrects the doc comment, which claimed a 120s default while the code returned 300s, and names the distinction from the iteration ceiling so the next reader does not repeat the conflation. Note for the next full sweep: the outer budget is `min(2 * cases * (timeout + 10s) + 120s, 12h)`. At 600s a 70-case run whose every case maxed out would take 11.7h against that 12h cap — thin where it used to be ample. Incremental flush makes an overrun survivable, so this is a note rather than a second limit changed in the same commit. A cap-hit is not automatically a failure: the abort stops the loop and the workspace is scored as it stands, so a truncated run can still pass. That is why this went unnoticed while ministral hit it on nearly a third of its cases. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

First baselines recorded since the sandbox glob fix, the fixture tsconfig, the 17 relaxed route pins, the scorer tests and the tree-sitter swap. Both moved by exactly +1 case. gemma4:e4b 66/70 -> 67/70 ministral-3 61/70 -> 62/70 That is the headline result and it is a NEGATIVE one. 32% of runs in the old sweep had a verification call return garbage — `run_tests` throwing on the pytest glob in 67 runs, `tsc` returning 5,301 characters of help in 121 — and the contamination was concentrated in exactly these two models, because they are the ones that verify their own work. Removing all of it moved each by a single case. The defects were real and worth fixing; their effect on scores was small. ministral's only gained case is `no-stub-in-write`, a plausible beneficiary of the route-pin relaxation: it pinned `write_file` while `edit_file` on a missing path is coerced into a create. Neither file is a clean single run, and that should be visible to whoever reads them next: - ministral is 53 cases from the interrupted 2026-08-04 sweep plus the 17 it never reached, run separately and merged. The 53 were clean — no unavailability, nothing over 500s until the circuit breaker fired. - gemma4 is 62 cases from that sweep plus 6 re-measured. Ollama degraded mid-run when the laptop lid closed; four cases took 610-1140s against a 300s timeout and two never recorded. Re-running that window is unambiguous about what those were: `git-diff-not-run-command` went FAIL(1065s) -> pass(8s), `build-python-calculator` FAIL(1140s) -> pass(77s), `rename-propagates-to-cross-file-caller` FAIL(633s) -> pass(159s). `fix-wrong-comparison-operator` went the other way — it passed at 610s inside the degraded window and fails at 158s outside it, so the contaminated run got it right by accident. The merges refuse across a provenance boundary (model, thinking, maxIterations, scaffold major). Both halves of each were thinking-on / maxIter 50 / scaffold 4.0.0. What provenance CANNOT see is that the halves were measured hours apart, so `recordedAt` on both files describes only the later half. Case validity confirmed independently: claude-sonnet-5 passes 70 of the 71 it completes, the one exception being a 300s timeout on `run-tests-after-fix` which is what prompted raising that budget. So these numbers measure models, not damage from this week's changes. granite4.1:3b, qwen2.5-coder:7b and llama3.2 are NOT re-measured — their runs never executed a case, and their files still hold 2026-08-02 numbers taken under all the defects above. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The 2026-08-04 sweep lost three of five models to a wedged Ollama. The laptop lid closed, Ollama stopped serving, and granite4.1, qwen2.5-coder and llama3.2 each spent ~50 minutes discovering that one case at a time before their circuit breakers fired. Nothing was recorded for any of them. The per-model circuit breaker works — it aborts rather than grinding — but only after three failed cases, and it does not stop the sweep, so the next model repeats the discovery. This probes generation before each model and stops the whole run when the backend is dead. A reachable /api/tags is not evidence Ollama can serve; only a completed generation is. That distinction cost nine hours once already. Note for future runs: `caffeinate -dimsu` is NOT sufficient with the lid closed. Its `-s` assertion holds only on AC power and clamshell sleeps regardless on this hardware. A run that must survive the night needs the lid open. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions · 2026-08-05T15:34:38Z

✅ All checks passed

Check	Status
Type check	✅ success
Lint	✅ success
Tests	✅ success

Posted by SideCarAI-Bot

granite4.1:3b 53/70 -> 51/70 (-2) qwen2.5-coder:7b 48/70 -> 49/70 (+1) Both ran clean: all 70 cases, zero API-unavailable, one case each at the wall-clock cap. granite finished in 59 minutes, qwen in 55. Measured at the 600s per-case budget, where gemma4 and ministral-3 were measured at 300s. Not a like-for-like comparison across those two groups: a case that would have been cut off at 300s now gets to finish, so part of any gain could be clock rather than capability. Within each model against its own 2026-08-02 predecessor the comparison holds, since those were also 300s — which makes granite's -2 the interesting one, as extra time should not lose cases. Four of five models re-measured now, and the result is consistent: +1, +1, +1, -2. The 32% of runs that had a verification call return garbage was real and worth fixing, but it was not suppressing scores. The old numbers were not meaningfully wrong; they were untrustworthy, which is a different claim. llama3.2 is NOT re-measured. Its run aborted at case 16 when an aborted case threw from isInfraFailure — the record loop has no try around runAgentCase and only handles the `apiUnavailable` RETURN, so the throw killed the model run. The incremental flush then left a 16-case file where a 69-case one had been; restored from a hand-taken backup. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…napshot A baseline file is a single snapshot each run overwrites. That answers "where is this model now" and nothing else. Whether granite's 53 -> 51 is a regression or noise needs the runs before it, and those existed only in git and in copies taken by hand. Twice in one sweep a truncated run replaced a complete baseline: ministral's 61/70 became 50/53 when the backend wedged, llama3.2's 27/69 became 6/16 when an aborted case threw out of the record loop. Both were recoverable only because backups had been made manually. History does not prevent the overwrite — it makes it visible immediately, and keeps the earlier numbers. One JSON line per run in baselines/history.jsonl, appended and never rewritten: model, provenance, cases run against cases available, pass/fail, unavailable count, duration, and the failing case ids so a regression can be diffed without opening the baseline. Written on BOTH paths. The catch records the run with `complete: false` and the abort reason, then rethrows — a run that dies still produced measurements, and why it died is the most useful field in the entry. `complete` is what stops a fragment being read as a collapse: llama3.2's aborted 6/16 next to its 27/69 looks like the model fell apart and is nothing of the sort. Incomplete runs stay in the file, because the record of what happened matters, but `timelineFor` and `deltaAgainstPrevious` exclude them — otherwise an abort between two good runs reports -45 and then +45. appendHistory never throws. History is telemetry; losing it must not fail a run that produced real measurements. That is the opposite trade from the baseline file itself, and deliberate. Mutation-checked: admitting incomplete runs to the timeline fails two tests, letting a write failure propagate fails one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`runAgentCase` throws on infra breakage rather than returning. That is correct for agent.eval.ts, where each case is its own `it` — the throw fails one test and the suite continues. The baseline recorder runs all 70 cases inside a SINGLE `it`, so the same throw ended the model's entire run. llama3.2 died at case 16 of 70 on "This operation was aborted". Because the recorder flushes after every case, the 16 it had reached then overwrote a complete 69-case baseline — recovered only from a copy taken by hand. Both loops now catch it. An infra throw and the `apiUnavailable` return mean the same thing, the backend gave nothing usable, so they are handled identically: skipped, logged, and counted toward the circuit breaker. Anything else rethrows, because a genuine harness bug must still fail the run rather than being absorbed as "the backend had a moment". Recognising the throw needed a marker rather than reusing `isInfraFailure`. Wrapping the original in `new Error(...)` drops `err.name === 'AbortError'`, and the message regex has no `aborted` term — so the classifier returns false on an error it produced itself. There is a test pinning that, which will start failing if the classifier ever learns to recognise its own output, at which point the marker can go. The recorder assertions read the source rather than driving the loop: exercising it needs a live model, and the defect is an unguarded `await` that is invisible in any run where the backend behaves. Mutation-checked: restoring the bare call fails the test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

llama3.2:latest 27/69 -> 27/70 (flat; one more case in the denominator) Completes the set. This run needed the infra-throw fix to finish at all — the previous attempt died at case 16 of 70 when an aborted case threw out of the record loop. The lid closed partway through, stalling four cases at 947-1038s against a 600s cap, which cannot happen while the machine is awake: the timer freezes during sleep and the abort fires on wake. Those four were re-measured and merged: verify-with-diagnostics-after-edit FAIL(1020s) -> pass(176s) write-tests-for-function FAIL(1013s) -> FAIL(135s) rename-function-across-callers FAIL(1038s) -> FAIL( 47s) explain-function-from-source FAIL( 947s) -> FAIL( 84s) Only one of four was an artifact — the inverse of gemma4's window, where three of four flipped. That fits: llama3.2 is the weakest model here, so a failure is more likely to be genuine. `rename-function-across-callers` failed in 47 seconds, and it is the case every model except gemma4 still fails. With all five re-measured the result is a negative one, and it is worth stating plainly because the 32%-of-runs contamination figure invited a much larger expectation: gemma4:e4b 66/70 -> 67/70 +1 ministral-3 61/70 -> 62/70 +1 granite4.1:3b 53/70 -> 51/70 -2 qwen2.5-coder:7b 48/70 -> 49/70 +1 llama3.2 27/69 -> 27/70 0 11 case flips across 280 measurements (~4%). Every flip is unique to one model, and `shell-error-recovery` moved in OPPOSITE directions — granite lost it, qwen gained it. If the harness fixes had a treatment effect the same cases would move across models, particularly ones touching run_tests or tsc. They did not. The instruments were genuinely broken. They were not distorting the scores. Both statements are true and only the second one is surprising. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The timeline feature catching its own first runs. Both llama3.2 runs are here — the full sweep and the four-case window redo — which is exactly the pair a single-snapshot baseline cannot represent: the subset run legitimately shows 4 cases against 70 available, and `complete` plus `casesAvailable` are what stop that reading as a collapse. Starts from now by design. Backfilling the five runs measured before this existed would mean reconstructed timestamps and durations, and a timeline whose early rows are inferred is worse than one that is short and honest. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Found in the feature's own first real data. The llama3.2 window redo recorded `1/4, complete: true` — it ran everything it was asked to and finished, so `complete` does not catch it. A naive trend read 27/69 -> 26/70 -> 1/4 and would call it a collapse. Exactly the fragment-misread the history log exists to prevent, built into the log. Targeted re-runs are routine here (three of them in this sweep alone), so this is the common case rather than an edge one. Entries now carry `filtered`, set when SIDECAR_EVAL_CASE or SIDECAR_EVAL_TAGS narrowed the selection. `timelineFor` and `deltaAgainstPrevious` exclude them. They stay in the file — what happened matters — but they are not points on a trend. The two existing entries were backfilled from `casesAvailable`, which is unambiguous for both. Mutation-checked: admitting filtered runs to the timeline fails two tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Measured, not suspected. Of the eleven cases that flipped between the 2026-08-02 and 2026-08-05 sweeps, FIVE flip on seed alone: shell-error-recovery granite4.1 2/5 seeds fix-wrong-type-annotation granite4.1 3/5 fix-two-independent-bugs gemma4:e4b 4/5 build-python-calculator-cli granite4.1 4/5 thinking-aliased-mutation qwen2.5-coder 4/5 A one-shot baseline records whichever side the coin landed on, and the flip resurfaces later as a phantom regression. `shell-error-recovery` is the clearest case: it moved in OPPOSITE directions for two models between the same two sweeps, which is what a 2-of-5 case does when sampled once. Three more disagree with a same-seed re-run — `rename-function-across-callers` fails 0/5 including seed 42, while the 08-05 sweep recorded it passing at seed 42. So multi-turn runs are not reproducible even seeded: the seed pins each request, but the number and content of requests depends on what came back, so any divergence compounds. Only single-turn cases reproduce. agent.eval.ts has reported flakiness for a while (`[flaky] case: 13/25`). The recorder ignored the same knob, so every baseline in the repo is one sample per case with no reliability attached. SIDECAR_EVAL_TRIALS now works in record mode. Default 1 leaves cost and behaviour untouched; above 1 the recorded `passed` is a MAJORITY of trials (`passes * 2 > results.length`, which at TRIALS=1 is exactly the single result) and the rate is stored beside it, so a marginal case is legible in the file rather than arriving as a surprise. An infra failure on a later trial costs that trial, not the case. Not applied to the committed baselines: re-recording all five at TRIALS=5 is roughly 65 hours. The knob is there for whoever decides that trade, and the seed-sensitivity data for the eleven flipped cases is kept under .sidecar/logs/seed-sensitivity-keep/. What this means for the numbers already recorded: +1/+1/-2/+1/0 across the five models is not measuring the harness fixes. At most three of eleven flips are real; five are coin tosses and three contradict a same-seed re-run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

nedonatelli and others added 3 commits August 5, 2026 11:20

nedonatelli and others added 7 commits August 5, 2026 14:25

nedonatelli merged commit c1f1a41 into main Aug 6, 2026
3 checks passed

nedonatelli deleted the fix/eval-case-timeout branch September 1, 2026 15:10

Navigation Menu

Re-measure two models after the harness fixes, and raise the wall-clock budget - #55

Re-measure two models after the harness fixes, and raise the wall-clock budget#55
nedonatelli merged 10 commits into
mainfrom
fix/eval-case-timeout

nedonatelli commented Aug 5, 2026

Uh oh!

github-actions Bot commented Aug 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

nedonatelli commented Aug 5, 2026

The result is negative, and that is the point

The artifact evidence is unambiguous

Both baselines are stitched, and the files should be read that way

Wall clock 300s -> 600s

Case validity, independently confirmed

Not re-measured

Uh oh!

github-actions Bot commented Aug 5, 2026

✅ All checks passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant