Add a section-placement eval with a measured baseline

[kevinhermawan]

Summary

Rule-text and gate changes have been landing with no way to tell an improvement from noise. Re-running article-analysis over the same corpus moves 15-25% of articles between sections, so a single before/after comparison proves nothing.

This is the harness that makes the blocked items in IMPROVEMENTS.md (P0-3 materiality ranking, P1-6/7 clustering, P1-5 rule text) actually measurable.

The measured baseline

openai/gpt-4o-mini, three runs, identical code and identical cases:

run 1:  9/20 (45.0%)   wrongly-rejected=8  wrongly-accepted=2
run 2:  8/20 (40.0%)   wrongly-rejected=5  wrongly-accepted=3
run 3: 11/20 (55.0%)   wrongly-rejected=6  wrongly-accepted=1

mean accuracy: 46.7%   range: 40.0%-55.0%

A 15-point spread on the same inputs. That is the whole point. Any single-run "this rule change improved things by 5 points" claim is indistinguishable from doing nothing. The runner prints the range next to the mean and states the bar explicitly.

It reproduces the hand-found defects

The eval independently rediscovers what the 2026-08-21 review found by hand:

• the Nias Selatan land-office courtesy visit accepted into competitiveLandscape
• the four-bank daily FX table accepted at all
• DSSA's coal-exchange and 760Mt permit-expiry articles rejected — for a coal producer
• GOTO's own response to its MSCI removal rejected
• DCII's data-centre land-demand article rejected

That is the evidence the harness measures the right thing.

The cases

20 real articles, labelled from reviews/2026-08-21/: confirmed misplacements, confirmed wrongly-rejected articles, and correctly-placed items as regression guards. expectedSection: null means the article should be rejected.

Nine of twenty carry no body text. That is deliberate, not a fixture defect — most collected sources have no body, so the eval reproduces the condition the classifier actually works under. The runner reports the count so it is not misread.

Usage

OPENROUTER_API_KEY=... pnpm --filter @mediapulse/article-analysis eval:section

EVAL_REPEATS=3 for variance, EVAL_MODEL for any OpenRouter model id. Runs six cases concurrently, so three repeats take about two minutes.

Two honest limits

• The baseline is model-specific. Production reads its model from the Hermes agent config ({{AI_MODEL}}), which I cannot see from here. 46.7% is gpt-4o-mini's number, not necessarily production's. Re-baseline with EVAL_MODEL set to the production model before drawing conclusions about production. The README says this.
• 20 cases is small. Enough to catch regressions on known defects, not enough for a confident absolute accuracy figure. It should grow as more labelled examples accumulate from reviews.

Credentials

The runner reads OPENROUTER_API_KEY from the environment and fails fast when absent. No key is written to any file in this PR.

Closes P2-11 in reviews/2026-08-21/IMPROVEMENTS.md.