Decisive rerun: head-to-head with rubric-defined PARTIAL_MATCH + matched 5-vote budget for DeepSeek (pre-registered outcomes)

[caiotheodoro]

Context

The decisive experiment for the headline claim. Decomposing the champion-vs-DeepSeek R_w gap per class shows that 97% of the advantage comes from a single class the shared prompt never defines:

Class	champ caught	DS caught	ΔR_w contribution
PARTIAL_MATCH	26	5	+0.0398
COUNTERPARTY	37	36	+0.0034
FIELD_CORRUPTION	13	10	+0.0023
MISSING_MESSAGE	45	44	+0.0023
VALUE_DATE	37	40	−0.0068
others	tie	tie	0.0000
Total (×3 run)			+0.0409

(Re-derive as first task step from docs/validation/bench-eval-full-740.json and bench-deepseek-full.json per-task arrays.)

SYSTEM_PROMPT (model/src/reconforge_model/schema.py) names PARTIAL_MATCH but never defines it — docs/DECISIONS.md C2 says so explicitly. DeepSeek's PARTIAL_MATCH "errors" go to BENEFICIARY_MISMATCH (8), COUNTERPARTY_MISMATCH (9), MATCH (10) — all defensible readings of an undefined label. C2 already proved that giving DeepSeek the rubric definition lifts its oracle agreement 0.82 → 0.93. The head-to-head was never rerun with the rubric prompt. On top of that, the decode budgets are asymmetric (docs/validation/runconfig.json): champion = 5 samples @ temp 0.6, majority vote; DeepSeek = 1 greedy sample.

So the current claim is: a 5-vote fine-tuned model beats a 1-shot baseline that wasn't told what one label means, on a comparison where that label is the entire margin.

Task

Pre-registered design — write the analysis plan into this issue before running:

1. Conditions (all on the same 800 seed-777 tasks, exports committed like the existing ones):
   • DeepSeek + rubric-extended prompt (PARTIAL_MATCH defined), 1 greedy sample — isolates the definition effect.
   • DeepSeek + rubric, 5 samples @ temp 0.6, verdict-majority vote — matched budget. (~5× the original 194 s / API cost; use the resumable checkpoint machinery in compare_deepseek.py.)
   • Champion + rubric-extended prompt, ×5 — C2 showed rubric text hurts the fine-tune (off-distribution); measure it rather than assume it. Report champion under its native prompt too.
2. Fair-comparison rule: headline comparison = both models on the same prompt at the same sample count; secondary table shows each model under its best configuration, labeled as such.
3. Voting: verdict-level majority (requires the _majority fix from the scoring-harness issue).
4. Pre-registered outcomes (commit to this before looking at results):
   • Champion still ahead on R_w with CI excluding zero → claim upgraded: survives definition + budget matching. Update all docs/cards/blog with the new numbers.
   • Gap closes or reverses → the honest headline becomes "a 1.7B fine-tune matches a frontier model on a narrow reconciliation task at 1/10 the latency and zero marginal cost; the previously reported gap was a prompt-definition artifact." Write it up in BENCHMARK.md as finding C3 — this is consistent with the repo's B2/C2 negative-result brand and arguably a better story about eval construction.
5. Run intervals.py (paired bootstrap) on the new head-to-head; publish ΔR_w with CI.

Acceptance criteria

• All condition exports + intervals committed under docs/validation/.
• Headline tables in README/BENCHMARK/model card/blog updated per the pre-registered branch that occurred.
• runconfig.json extended with the new run configs (provider, model snapshot date, prompt hash).

Dependencies

• After: headline-number discipline (settles the comparison target), scoring-harness _majority fix, PARTIAL_MATCH construct-validity issue (interprets the result).

Cost tier

T2-compute — one DeepSeek API pass ×5 samples + local champion reruns (~1 h MLX wall time per ×5 pass).