Live-API validation of PR #20 + Opus 4.7→4.8 judge stability on nav_aap and skatteetaten

[avalyset]

Title: Live-API validation of PR #20 + Opus 4.7→4.8 judge stability on nav_aap and skatteetaten

Body:

PR #20 (repeated-run stability) was merged with the explicit note that no validation
had been done against live LLM API runs. With Opus 4.8 released yesterday, I ran the
infrastructure end-to-end on the two merged Norwegian packs (nav_aap, skatteetaten)
with both Opus 4.7 and 4.8 as judge, 3 repetitions each.

Full writeup with tables, severity-shift breakdown, and reproduction data:
https://github.com/avalyset/nordeval-skills/tree/main/studies/judge-stability-opus-4.7-4.8

Headline findings:

• PR repeated-run stability analysis, token tracking, resumable experiments, and auditor model separation #20 infrastructure ran cleanly across 24 experiment runs (3 repetitions × 2 subjects × 2 packs × 2 judges), totaling 276 scenario judgements. Cache/resume behavior worked as documented: when one run was interrupted mid-experiment by a spend-limit cap, the resume loaded completed runs from disk and only re-ran missing ones.
• Opus 4.8 scores Sonnet 4.6 lower (−3 to −6 points across both packs) but scores Haiku 4.5 higher on nav_aap (+6 points). Score direction is subject-dependent, so the two judges are not interchangeable for absolute score comparisons.
• Most consequential single shift: The AAP Klagefrist scenario (designed to detect misinformation about the appeal deadline rule, folketrygdloven §21-12 — errors here cause real rights forfeiture) shifts from modal HIGH under 4.7 to modal LOW under 4.8 on Haiku 4.5. Same model, same answers. This is the +6.1 score gain on Haiku/nav_aap visible in the package-level numbers, but the source is a judge-upgrade masking a previously detected safety-relevant finding rather than a model improvement.
• Haiku / skatteetaten CV explodes under Opus 4.8: 3.0% → 17.5% (range 18.8 points across 3 runs). This is the clearest argument for always reporting CV alongside mean, and for n≥3 repetitions as a minimum.
• Opus 4.8 generates ~21% more output tokens per judgment, same cost tier.

18 of 46 subject × scenario pairs shifted modal severity between the two judges (6 on nav_aap/Sonnet 4.6, 8 on nav_aap/Haiku 4.5, 2 on skatteetaten/Sonnet 4.6, 2 on skatteetaten/Haiku 4.5).

Posting here in case the data is useful for the framework, and to credit @finnschwall for the infrastructure that made the comparison possible. Extending to N=5 on Haiku/skatteetaten now to confirm the CV 17.5% finding; will post updated tables as an addendum. An auditor/judge-separation study is planned as the next step, building directly on PR #20's auditor_model parameter. Happy to review any follow-up PR that operationalizes cross-judge stability reporting in the framework.

Label suggestion: question (closest existing label for a methodology note).

[avalyset]

N=5 extension complete (as promised in the original post). CV 17.5% finding under Opus 4.8 confirmed structural:

• Opus 4.7 CV: 3.0% (N=3) → 5.9% (N=5). Stable.
• Opus 4.8 CV: 17.5% (N=3) → 18.2% (N=5). Within-judge instability confirmed.

New finding visible with N=5: Bimodal distribution under Opus 4.8. Two of five runs score exactly 50.0, both driven by the same two scenarios (MVA-registreringsgrense and Ubetalt skatt og angst) escalating to HIGH severity together. The remaining three cluster 53.1–71.9. A single evaluation has roughly 40% chance (2/5 observed) of landing in the low-score mode driven by judge behavior on these two scenarios rather than overall subject performance.

Addendum + raw run_3 / run_4 data: https://github.com/avalyset/nordeval-skills/tree/main/studies/judge-stability-opus-4.7-4.8

Extension cost: $1.86. Total study cost: $13.93.

[kelkalot]

Wonderful! Thanks a lot!