Context
Four independent scoring-harness defects, each small, together enough to distort published numbers. Batched here because they share a verification path (unit tests + one re-score of committed exports).
Defects
(a) _majority votes over whole JSON objects. model/src/reconforge_model/benchmark_eval.py::_majority compares full dicts including free-text reason and float confidence. Two samples agreeing on the verdict but differing by a word count as different votes; ties fall back to the first sample. This deflates self_consistency_confidence and therefore contaminates the ECE numbers (0.1175 ×3 / 0.0875 ×5). Fix: vote on (verdict, exception_type) only; recompute ECE on committed exports if sample-level data allows, otherwise flag ECE as pre-fix.
(b) exception_type_accuracy is miscomputed. In every scoring block it disagrees wildly with the confusion-matrix diagonal: champion 0.3176 vs 299/381 ≈ 0.785; DeepSeek 0.2598; a 100%-correct pilot shows 0.5. Locate the bug in forge/src/reconforge_forge/benchmark.py::score_verdicts, fix, regenerate the field in the validation JSONs, or delete the field if redundant with the confusion matrix. It is currently unexplained in every artifact that carries it.
(c) Escalation precision degenerate cases. forge/benchmark.py:110 computes esc_correct / max(n_esc, 1), so zero escalations prints 0.0 instead of N/A — and DeepSeek's published "escalation precision 1.0" is n=2. Emit null + count when n_esc is 0, and always print n next to the value.
(d) "Perfect HIGH recall (COUNTERPARTY_MISMATCH 31/37)" reads as self-contradictory. BENCHMARK.md and the blog mix caught-count (37/37 caught → HIGH recall 1.0) with exact-type accuracy (31/37 typed correctly) in one sentence. State both numbers with their names.
Task
- Write failing unit tests for (a)–(c) first (repo has 162 tests; extend
model/forge suites).
- Fix; regenerate affected fields in
docs/validation/*.json where derivable from committed per-task data.
- Correct the (d) sentences in BENCHMARK.md and the blog.
- Note in BENCHMARK.md which published numbers changed and why.
Acceptance criteria
- Tests cover verdict-only voting, type-accuracy = confusion diagonal, escalation N/A case.
exception_type_accuracy matches the diagonal on every committed export, or the field is removed.- ECE either recomputed post-fix or explicitly labeled pre-fix.
Cost tier
T1-cheap — code + tests + re-scoring; no model runs (ECE recompute may need sample-level rerun — if so, fold into the T2 rerun issue).
Context
Four independent scoring-harness defects, each small, together enough to distort published numbers. Batched here because they share a verification path (unit tests + one re-score of committed exports).
Defects
(a)
_majorityvotes over whole JSON objects.model/src/reconforge_model/benchmark_eval.py::_majoritycompares full dicts including free-textreasonand floatconfidence. Two samples agreeing on the verdict but differing by a word count as different votes; ties fall back to the first sample. This deflatesself_consistency_confidenceand therefore contaminates the ECE numbers (0.1175 ×3 / 0.0875 ×5). Fix: vote on(verdict, exception_type)only; recompute ECE on committed exports if sample-level data allows, otherwise flag ECE as pre-fix.(b)
exception_type_accuracyis miscomputed. In every scoring block it disagrees wildly with the confusion-matrix diagonal: champion 0.3176 vs 299/381 ≈ 0.785; DeepSeek 0.2598; a 100%-correct pilot shows 0.5. Locate the bug inforge/src/reconforge_forge/benchmark.py::score_verdicts, fix, regenerate the field in the validation JSONs, or delete the field if redundant with the confusion matrix. It is currently unexplained in every artifact that carries it.(c) Escalation precision degenerate cases.
forge/benchmark.py:110computesesc_correct / max(n_esc, 1), so zero escalations prints 0.0 instead of N/A — and DeepSeek's published "escalation precision 1.0" is n=2. Emitnull+ count when n_esc is 0, and always print n next to the value.(d) "Perfect HIGH recall (COUNTERPARTY_MISMATCH 31/37)" reads as self-contradictory. BENCHMARK.md and the blog mix caught-count (37/37 caught → HIGH recall 1.0) with exact-type accuracy (31/37 typed correctly) in one sentence. State both numbers with their names.
Task
model/forgesuites).docs/validation/*.jsonwhere derivable from committed per-task data.Acceptance criteria
exception_type_accuracymatches the diagonal on every committed export, or the field is removed.Cost tier
T1-cheap — code + tests + re-scoring; no model runs (ECE recompute may need sample-level rerun — if so, fold into the T2 rerun issue).