Context
Buried inside the B2 negative result is a second, more interesting negative result that was never written up: validation loss carried zero signal about the metric that decided the champion. B2's final val loss (0.090 at step 550) is statistically identical to run 1's (0.091 at step 700), yet B2 is 19 R_w points worse on the benchmark (0.723 vs 0.913). The documented stopping rule — "stop on loss plateau" — is therefore decoupled from the selection criterion. This is exactly the kind of honest methodological finding the repo's B2/C2 sections already trade in, and it strengthens the "benchmark validity is a construction property" thesis: token-level loss on a narrow format converges regardless of whether the model learned the decision boundary that matters.
Evidence
model/data/train-full.log: run 1 val loss 2.764 → 0.091, flat from ~iter 450.model/data/train-b2.log: B2 val loss → 0.090 at step 550.docs/BENCHMARK.md §B2 table: R_w 0.9128 vs 0.7230; MEDIUM recall 0.84 → 0.40; PARTIAL_MATCH correct 26 → 1.docs/DECISIONS.md M10: stopping rule stated as loss plateau, citing train-full.log.
Task
- Extract both loss curves (iter, train loss, val loss) from the two logs into a small committed CSV/JSON under
docs/validation/.
- Write a short section (TRAINING.md, plus a paragraph in the blog): identical val loss, 19-point benchmark gap, conclusion — for this task family, val loss says "the model learned the output format", not "the model ranks classes correctly"; checkpoint selection must use the benchmark metric (which in turn requires the dev/test discipline issue, else selection overfits the eval set).
- Link this to the stopping-rule text in DECISIONS M10 so the rule is restated honestly: loss plateau decides when to stop paying for compute, benchmark decides which checkpoint ships.
Acceptance criteria
- Loss-curve artifact committed; TRAINING.md section written; M10 cross-referenced.
- Blog updated with the finding (it currently implies loss-based stopping produced the good model).
Dependencies
Reads best alongside the held-out-discipline disclosure issue — selecting on benchmark instead of loss is only sound once dev/test are separated.
Cost tier
T1-cheap — log parsing and writing; no runs.
Context
Buried inside the B2 negative result is a second, more interesting negative result that was never written up: validation loss carried zero signal about the metric that decided the champion. B2's final val loss (0.090 at step 550) is statistically identical to run 1's (0.091 at step 700), yet B2 is 19 R_w points worse on the benchmark (0.723 vs 0.913). The documented stopping rule — "stop on loss plateau" — is therefore decoupled from the selection criterion. This is exactly the kind of honest methodological finding the repo's B2/C2 sections already trade in, and it strengthens the "benchmark validity is a construction property" thesis: token-level loss on a narrow format converges regardless of whether the model learned the decision boundary that matters.
Evidence
model/data/train-full.log: run 1 val loss 2.764 → 0.091, flat from ~iter 450.model/data/train-b2.log: B2 val loss → 0.090 at step 550.docs/BENCHMARK.md§B2 table: R_w 0.9128 vs 0.7230; MEDIUM recall 0.84 → 0.40; PARTIAL_MATCH correct 26 → 1.docs/DECISIONS.mdM10: stopping rule stated as loss plateau, citingtrain-full.log.Task
docs/validation/.Acceptance criteria
Dependencies
Reads best alongside the held-out-discipline disclosure issue — selecting on benchmark instead of loss is only sound once dev/test are separated.
Cost tier
T1-cheap — log parsing and writing; no runs.