Correct §3.7 baselines, held-out count, and steering-strength units

[amhw460]

Corrections to §3.7 and the abstract, found while auditing the section against the result JSONs. Everything here is either a re-measurement or a text fix — no analysis is re-run or re-interpreted, and the matched-null argument is untouched.

The hardcoded baseline

baseline_p in cross_prompt_results.json was never measured. It's a literal in the PROMPTS table of cross_prompt_validation.py:55-60, copied verbatim into the output JSON, which made it look like a result. It disagrees with the first_default_logprob the API actually returned on the same runs.

remeasure_baselines.py (added) measures all five directly. /steer returns HTTP 500 on an empty feature list — which is why the dedicated baseline call in results.json is itself an HTTP 500, and likely why a literal was used in the first place — so it sends one feature at strength 0 and reads the DEFAULT branch, asserting default == steered to confirm the no-op. It held on all five.

prompt	hardcoded	measured
Berlin	0.958	0.9584
Rome	0.974	0.9584
Tokyo	0.990	0.9820
teacher	0.486	0.5631
bird	0.117	0.0759

Only Berlin was right. §3.7.1's table already used the measured numbers; the §3.7.2 caption and §3.7.4 used the hardcoded ones. All three now agree.

Two incidental findings: Rome and Berlin genuinely both sit at 0.9584 (real coincidence, independently reproduced — not a copy-paste). And on the bird prompt " air" is only marginally the argmax, effectively tied with " ____" at 0.0759 — the model is guessing among blank-fillers rather than weakly completing the analogy. §3.7.4 now says so, which strengthens a hedge that was already there.

Other fixes

• Dropped the Bee/ant held-out row. Its recorded baseline_text is "?" — the model never produced "colony"; that was the expected label, not an output. The row couldn't show a circuit destroying an answer that was never given. Four held-out prompts becomes three throughout. The other three are unaffected.
• Steering-strength units in §3.7.2. The titration sweeps per-feature strength −2…−40, which the ×4 multiplier makes −8…−160, but the text compared these against "−20" and then concluded the effect was not a product of "−80". Both scales are now explicit. Also reframed the conclusion: an identical 97% at −8 and at −160 indicates the readout saturates well below −80, which is worth stating plainly rather than reading the flatness as robustness.
• Abstract said the circuit spans all 26 transformer layers. The 180-feature set touches 21 — nothing at L16, L17, L19, L21, or L23. The graphs span all 26, which §3.1 already states correctly; the abstract was conflating the two.
• §3.7.3 source pointer. That table's data is real and matches cell-for-cell, but it lives under graph-analysis/anish/exp7_remaining_analogy_validation/ while the rest of §3.7 comes from experiments/exp8_.../, so it looks unsourced. Added a caption citing it, and noting the : row is genuine degenerate output rather than a placeholder.
• Removed the "218 experiments" framing from the abstract, intro, and §4 synthesis.

Two things deliberately left alone

§2.7 still says "218 individual steering API calls." I count ~280 across the committed JSONs, so the number likely needs revisiting — but §2.7 isn't mine, so I only removed the framing from the sections that are.

§2.3's selection criterion doesn't match the data file, and I didn't touch it. analogical_circuit_180features.json records counting_method: "total_instances", threshold 5 — i.e. ≥5 node instances summed across the graphs, where one graph can contribute several. §2.3's formula defines the threshold as a count of graphs. These select different sets: by the file's own distinct_graphs field, 119 of the 180 appear in all five graphs, 16 in four, 42 in three, and 3 in only two.

Likely cause: the file also carries an out_of_graphs field that is hardcoded to 5 for every feature — it's the denominator, not a count — which reads as "in 5 graphs" at a glance.

This PR updates the abstract and the §3.2 table labels to match the file, but leaves §2.3 as-is, so §2.3 now disagrees with both. Flagging for whoever owns chapter 2 rather than fixing it here. Note also that I couldn't reproduce the 510/277 figures from the five graph JSONs under either rule (I get 516/236/145 by graph count, 536/303/206 by instances), and no script in the repo generates that file — so it's worth confirming those two rows came from the same criterion as the 180.

🤖 Generated with Claude Code