Judged questions N01/N09/N13/N14 measure behaviour we deliberately removed

[muthuishere]

The rule that was broken

CLAUDE.md: "Score and speed may only move UP; raising a floor is a reviewed diff."

proof/golden/SCOREBOARD.md recorded newtonsoft 16.5 / 20 at commit 72900ca (2026-07-16). Today the judged tier runs at 13.0 / 20, and passes — because the floor in testdata/questions/ is now 13.0.

The floor was moved down to accommodate a regression. That is the one direction the golden net exists to prevent, and it means the net has been silently green over a real quality loss for ten days.

What regressed

Four questions went 1.0 → 0.0 between 72900ca and now:

Q	question
N01	How do I deserialize a JSON string to an object?
N09	Where does JSON schema validation live?
N13	What is impacted if DeserializeObject changes?
N14	Which tests exercise SerializeObject? (derived tests-for)

N01/N09 are query ranking; N13/N14 are call-edge derived. Two different subsystems, so this may well be two separate regressions that happened to land in the same window.

linux-block is unaffected (16.5 / 20, floor 16.5).

Not caused by this week's work

Verified by re-running the judged tier against stashed changes: the receiver gate (6236c4f), --include-ambiguous (68910de) and the declared-resolutions work (4358f44) are all score-neutral — byte-identical before and after. The regression predates them.

What needs doing

1. Bisect 72900ca → HEAD on the newtonsoft judged tier to find the commit(s).
2. Fix, or consciously accept with a written reason.
3. Raise the floor back to 16.5 — and treat any future lowering as a reviewed diff, per the rule.
4. Work out why lowering a floor was possible without review, since that is the process failure underneath the quality one.

[muthuishere]

Bisected. My conclusion in this issue was wrong, and the truth is more interesting.

I filed this saying a real quality loss had been hidden by lowering a floor. Two bisects say otherwise.

The regressions are real, and there are two of them

Both found by git bisect run on "newtonsoft judged ≥ threshold", with CTX_OPTIMIZE_GRAMMARS pointed at an empty dir so grammar packs could not confound it (verified: HEAD scores 13.0 with and without packs, so packs are not a variable).

commit	date	score	questions lost
72900ca (baseline)	07-16	16.5	—
5c36287 verify / ambiguity-aware resolution	07-17	14.5	N13, N14
d041a2a definitions beat import stubs	07-24	12.5	N01, N09
96c6a87 question stopwords	07-25	13.0	+N16 recovered

So 16.5 WAS achieved — I measured it at 72900ca today. The floor note's claim that 16.5 "was never measured" and "never met" is factually wrong.

But the floor correction was right anyway

Both regressions are side-effects of changes that were correct, and the lost points were being scored for the wrong reasons.

N13 — affected "JsonConvert.DeserializeObject", min_impacts: 5.

The production symbol is labelled Newtonsoft.JsonConvert.DeserializeObject (namespace-qualified). So the question's argument is not an exact label; it falls through to fuzzy, ties among five Tests.JsonConvertTest.DeserializeObject* methods, and today correctly refuses:

ctx-optimize: "JsonConvert.DeserializeObject" has no exact match and several near names score alike — refusing to guess; pick one:
  Tests.JsonConvertTest.DeserializeObject  [method]  …

Before 5c36287, fuzzy silently took matches[0]. The question was passing by measuring the blast radius of a test method, not the production API. A wrong answer scoring 1.0.

And it would still fail if asked correctly — affected with the exact label returns 2 impacts (both contains ancestors), because DeserializeObject is declared many times over in the test suite, so the call edges are AMBIGUOUS and excluded from traversal by design.

N01 — query "deserialize json string object", expect_any: ["DeserializeObject"], k=5. d041a2a downranked module:// stubs (×0.25) and test files (×0.5) after a measured card url_for failure. The expectation is a bare substring match, so a module://DeserializeObject-shaped stub in the top 5 satisfied it. Demote the stub and it drops out.

So what is actually broken

Not the resolver, and not the ranker. The question set. Three of these four questions measure behaviour we deliberately removed:

• N13/N14 pass only if the tool silently guesses among fuzzy ties.
• N01 passes on a substring match that an import stub can satisfy.

A judged suite whose score goes DOWN when hallucination is removed is measuring the wrong thing. That is the defect worth fixing, and it is not a one-line change — hence an ADR rather than a patch.

Corrections to this issue

• ❌ "the floor was lowered to accommodate a regression" — the person who corrected it measured 12.5, saw the floor had never been met on their runs, and concluded it was fabricated. Wrong reasoning, right destination.
• ❌ "the golden net has been silently green over a real quality loss" — it was RED from 2026-07-20 until corrected, exactly as the floor note says.
• ✅ the inconsistency between SCOREBOARD.md (16.5 at 72900ca) and the floor note ("never measured") was real, and worth pulling on.
• ✅ raising the floor back to 16.5 would be wrong — it would demand the hallucinations back.

Retitling and re-scoping accordingly.