From 961df9237b03e15cbe7da3e3a1660d424ba5d8da Mon Sep 17 00:00:00 2001 From: ksdisch Date: Sun, 26 Jul 2026 19:28:31 -0500 Subject: [PATCH] =?UTF-8?q?docs(wiki):=20add=203=20topic=20pages=20?= =?UTF-8?q?=E2=80=94=20Results-Synthesis,=20Methodology-Guardrails,=20Expe?= =?UTF-8?q?riment-Harness?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Results-Synthesis: one table of every headline number (pre-registration vs. observed, Newcombe CIs) synthesized across 5 vendored figure JSONs, D12–D24, PROJECT.md, and the capstone-data.json ladder. Methodology-Guardrails: the honesty machinery synthesized across DECISIONS.md (D2/D3/D7/D12/D13/D15/D16/D19/D22), CLAUDE.md, and stats.py — oracle design, CI method, seed-distinctness rule, kill-triggers, what counted as a null, what would have falsified the thesis. Experiment-Harness: how the code runs an experiment — toggle system, arm-as-config pattern, fault injectors, result paths, all reproduce commands — mined from code (agent.py, ablation.py, faults.py, chart.py) and DECISIONS.md (D11/D14–D19/D23). Co-Authored-By: Claude Fable 5 --- Wiki/Experiment-Harness.md | 142 +++++++++++++++++++++++++++++++++ Wiki/Methodology-Guardrails.md | 130 ++++++++++++++++++++++++++++++ Wiki/Results-Synthesis.md | 117 +++++++++++++++++++++++++++ Wiki/_index.md | 3 + 4 files changed, 392 insertions(+) create mode 100644 Wiki/Experiment-Harness.md create mode 100644 Wiki/Methodology-Guardrails.md create mode 100644 Wiki/Results-Synthesis.md diff --git a/Wiki/Experiment-Harness.md b/Wiki/Experiment-Harness.md new file mode 100644 index 0000000..09b1f7b --- /dev/null +++ b/Wiki/Experiment-Harness.md @@ -0,0 +1,142 @@ +# Experiment-Harness + +## Purpose +A map of how the code is actually organized to run an experiment: entry points, the +guardrail toggle system, what a "run" produces, and where results land. For anyone +who wants to reproduce a measurement, add a new arm, or understand what each script does +without reading every file. + +## Key understanding + +### Architecture in one sentence + +**Fact** — the harness is a layered stack: `glm.py` (API client) → `agent.py` (loop + +guardrail toggles) → `scenario.py` + `oracle.py` (task + grader) → `faults.py` (optional +fault injectors) → `runner.py` (single-arm looper) → `ablation.py` (multi-arm harness) → +per-experiment scripts (entry points). `chart.py` and `stats.py` are offline post-processing; +they make no API calls. Source: [CLAUDE.md](../CLAUDE.md), "File map" section. + +### The guardrail toggle system + +`agent.py`'s `run()` function exposes four independent boolean toggles, each corresponding +to one measured guardrail: + +| Toggle | Stage | What it does | Cost | +|---|---|---|---| +| `recover=True` | S4 | harness-level retry of transient 503s (matches `_is_retryable`) | no model turn | +| `nudge=True` | S6 | re-prompts model to correct a malformed/failed tool call | 1 model turn | +| `submit_nudge=True` | S8 | re-prompts model to call `submit_answer` if a turn ends in prose without submitting | 1 model turn | +| `validate=True` | S9 | on `submit_answer(value)`, recomputes from retrieved observations; re-prompts on mismatch | 1 model turn | + +**Fact** — all four toggles default to `False`; the bare baseline runs with no mechanisms. +The toggles are additive: any combination is valid, and the S9 stacked ablation uses +`submit_nudge=True, validate=True` simultaneously. Source: [CLAUDE.md](../CLAUDE.md) `agent.py` +entry. + +**Inference** — the toggle design means adding a new guardrail requires (a) one new flag on +`run()`, (b) one arm config in `ablation.py`, and (c) a new per-experiment entry-point script; +no existing mechanism code changes. This is why S6, S8, S9, S10 each reused the same harness +without rewrites. + +### Arm configuration and the ablation harness + +`ablation.py` defines arms as config dicts: `{"label": str, "run_kwargs": dict}`. The +harness in `run_arms()` runs each arm N times over identical seeded scenarios, computes +Wilson CIs per arm, and computes a Newcombe gap vs the designated baseline arm. + +**Fact** — the five named arm configs live in `ablation.py`: +`BASELINE_ARM`, `RECOVERY_ARM`, `NUDGE_ARM`, `SUBMIT_NUDGE_ARM`, `VALIDATION_ARM` +(stacked, S9), `VALIDATION_ONLY_ARM` (un-stacked, S10). Source: [CLAUDE.md](../CLAUDE.md) +`ablation.py` entry. + +Per-experiment scripts pick subsets: +- `ablation.py` (run directly): S4/S5 2-arm error-recovery ablation +- `malformed_ablation.py`: S6 3-arm malformed-call ablation +- `weak_ablation.py`: S8 3-arm submit-nudge ablation on a weak model +- `validation_ablation.py`: S9 stacked 2-arm validation ablation +- `hallucination_ablation.py`: S10 un-stacked 2-arm validation ablation on llama-8b + +**Fact** — every per-experiment script accepts the model and N as command-line arguments; +fault-rate is a third argument where applicable. Source: D15 and D19, +[docs/DECISIONS.md](../docs/DECISIONS.md) ("operating point is a runtime knob"). + +### Fault injection and the clean-vs-injected boundary + +`faults.py` provides two non-mutating injectors that wrap the base scenario: +- `with_faults(scenario, rate, seed)` — injects transient 503s at a given rate (S3–S5). +- `with_malformed_faults(scenario, rate, seed)` — injects a sticky malformed-call fault + (S6); "sticky" means the same call keeps failing until corrected (D19). + +**Fact** — `rate=0` is equivalent to the clean task for either injector; S8–S11 run on +the clean task with no injector wrapper. The fault classification boundary is encoded in +`agent._is_retryable()`: a 503/timeout → recoverable by `error-recovery`; a 400 with +`invalid_argument` → not retryable (error-recovery leaves it alone). Source: D19, +[docs/DECISIONS.md](../docs/DECISIONS.md), "Two load-bearing properties." + +### Where results land + +| Output | Location | Gitignored? | Notes | +|---|---|---|---| +| Per-trial trajectories | `runs//trial-*.jsonl` | yes (gitignored) | Full step-by-step JSONL per run; hand-read for failure triage | +| Per-experiment summary | `runs/-summary.json` | yes | Raw k/N + CIs; input to `chart.py` | +| Vendored figure data | `docs/figures/*-data.json` | no (committed) | Copied from `runs/` after a live run; charts regenerate from these | +| Figures | `docs/figures/*.png` | no (committed) | Regenerate with `uv run chart.py` — no API, no model call | + +**Fact** — the vendoring step is manual: after a live run, copy the summary JSON into +`docs/figures/` so the figure can regenerate in-repo from a clean clone. The capstone data +file (`docs/figures/capstone-data.json`) is an exception: `chart.py` derives it +automatically from the three per-stage JSONs on every run, so it can never silently drift. +Source: D18 and D24, [docs/DECISIONS.md](../docs/DECISIONS.md). + +### Entry points and how to reproduce + +| What to run | Command | Prerequisites | +|---|---|---| +| Smoke test (API + tool-calling) | `uv run verify.py` | `OPENROUTER_API_KEY` in `.env` | +| All offline tests (77 tests, no API) | `uv run pytest` | none | +| S4 error-recovery ablation | `uv run ablation.py z-ai/glm-4.6 40 0.6` | API key | +| S6 malformed ablation | `uv run malformed_ablation.py z-ai/glm-4.6 20 0.6` | API key | +| S7/8 pilot (hardened task) | `uv run pilot.py` or `uv run pilot.py v2` | API key | +| S8 weak-model ablation | `uv run weak_ablation.py mistralai/mistral-nemo 20` | API key | +| S9 stacked validation | `uv run validation_ablation.py mistralai/mistral-nemo 40` | API key | +| S10 hallucination ablation | `uv run hallucination_ablation.py meta-llama/llama-3.1-8b-instruct 40` | API key | +| Regenerate all figures (offline) | `uv run chart.py` | none (reads vendored JSONs) | +| Check docs freshness | `uv run check_docs.py` | none | + +**Fact** — all reproduce commands sourced from D17, D19, D22, D23, D24 measured-result +sections in [docs/DECISIONS.md](../docs/DECISIONS.md) and the [CLAUDE.md](../CLAUDE.md) +"How to run" section. + +### The S10 harness fix — mid-experiment disclosure + +**Fact** — mid-S10, a latent bug was discovered: llama-3.1-8b sometimes emits tool-call +arguments as a JSON array (`["ORD-204"]`) rather than a dict. `agent.py` conflated "JSON +parsed" with "arguments OK," causing a crash on `.get()` applied to a list. Fix: +`args_ok = isinstance(args, dict)` — non-object args now route to the existing +malformed-arguments path. The fix adds no new help to the model; it only prevents a crash. +The pilot ran pre-fix; the full N=40 run ran post-fix. This is disclosed in D23, +[docs/DECISIONS.md](../docs/DECISIONS.md). Source: D23 "A latent harness bug, exposed +and fixed mid-stage." + +## Sources +- [CLAUDE.md](../CLAUDE.md) — "File map" section (per-file descriptions) +- [docs/DECISIONS.md](../docs/DECISIONS.md) — D11 (lean runner), D14 (toggle design), D15 (operating-point knobs), D16 (ablation arms-as-config), D18 (vendoring convention), D19 (fault stickiness + `is_retryable`), D23 (S10 harness fix) +- [docs/ROADMAP.md](../docs/ROADMAP.md) — per-stage "What it does" column +- `ablation.py`, `agent.py`, `faults.py`, `chart.py` — code (entry points, toggle signatures, injector wrappers) + +## Uncertainties & contradictions +- **Unresolved** — the `trajectory.jsonl` root-level file (tracked in the repo) appears to be a leftover from early development; it is not a run output from any named experiment. Its relationship to the gitignored `runs/` outputs is not documented. +- **Inference** — the `check_docs.py` freshness check is described as "a smoke alarm, not a commit gate" ([CLAUDE.md](../CLAUDE.md)); this means it is possible for the docs spine to drift without CI catching it. No evidence of such drift in the current state. + +## Related pages +- [Results-Synthesis](Results-Synthesis.md) +- [Methodology-Guardrails](Methodology-Guardrails.md) +- [History](History.md) + +## Relevance to current work +The project is closed; no new experiment code should be added without a decision brief +(D24). This page is the entry point for anyone resuming the project or forking the harness +for a new repro — it answers "what do I run and where do results land?" without requiring a +full code read. + +_Last reviewed: 2026-07-26_ diff --git a/Wiki/Methodology-Guardrails.md b/Wiki/Methodology-Guardrails.md new file mode 100644 index 0000000..6345e00 --- /dev/null +++ b/Wiki/Methodology-Guardrails.md @@ -0,0 +1,130 @@ +# Methodology-Guardrails + +## Purpose +A synthesis of the honesty machinery that governs every measurement in this project — +pre-registration, statistical method, oracle design, seed handling, and the rules for +reporting a null. For anyone asking "how do I know these results are trustworthy?" or +preparing to defend a methodology choice in an interview. + +## Key understanding + +### The fixed ruler: deterministic oracle, never an LLM judge + +**Decision** — D2, [docs/DECISIONS.md](../docs/DECISIONS.md): task success is measured by +`oracle.py`, a plain-Python function that compares the model's submitted number to a +known-correct answer (158 for the standard scenario). The oracle reads from `ORDERS` and +`SHIP_RATES` directly — it is independent of the model's run. It cannot be fooled by a +wrong retrieval; it always reports the ground truth. + +**Inference** — this is the structural separation the validation guardrail (S9) was designed +around: `oracle.py` reads from canonical data; `Scenario.validate` reads only from the +model's own run observations. The two compute the same arithmetic but from different inputs, +which is why `validate` can be fooled (wrong-record retrieval yields a self-consistent wrong +total that the oracle fails) and the oracle cannot. + +### Statistical method: proportion CIs, not ±std + +**Decision** — D7 + D16, [docs/DECISIONS.md](../docs/DECISIONS.md): completion rate is a +proportion, not a continuous measurement. The project uses: +- **Wilson interval** (`stats.py`, `wilson()`) — per-arm CI that stays sane at the edges + (near 0% and 100%), unlike the Wald ±std which can run outside [0, 1]. +- **Newcombe "square-and-add" interval** (`stats.py`, `newcombe_diff()`) — the CI on the + *difference* between arms; carries Wilson's good edge behaviour into the gap estimate. +- **`excludes_zero` gate** (`stats.py`, `excludes_zero()`) — the mechanical honesty rule: + if the Newcombe interval on the difference includes 0, the result is reported as a null, + never a win. This gate was pre-committed (D7, written before any guardrail was built) and + applied consistently in S4, S6, S8, S9, S10. + +**Fact** — `stats.py` is pure Python with offline unit tests (`test_stats.py`); it is +exercised before any live run. The gate is not a judgement call applied after seeing results; +it is encoded in the harness (`ablation.py` checks `excludes_zero` per arm). + +### N = distinct seeds, not trial count + +**Decision** — D13 + D15, [docs/DECISIONS.md](../docs/DECISIONS.md): with deterministic +per-trial seeds (`seed=0..N-1`), completion under fault injection is dominated by the fault +*pattern*, not model stochasticity. Running two N=20 arms on seeds 0–19 is a paired comparison; +re-running seeds 0–19 again is reproducibility, not more data. This is why statistical power +comes from more *distinct* seeds, not more total trials on the same seeds. + +**Inference** — this is a non-obvious property of the experiment design, and it has a +practical consequence: the recommended operating point (D15: rate 0.6, N=40) was chosen +because N=40 distinct seeds gives tighter Newcombe CIs than N=20, not because 40 trials +are intrinsically better than 20. + +### Pre-registration and kill-triggers + +**Decision** — D12, [docs/DECISIONS.md](../docs/DECISIONS.md): before the S3 diagnostic was +run, the team pre-committed to two routing rules: +- **Kill-trigger 1:** if GLM aces the clean task (≳85%), there is no natural gap; pivot to + fault injection as the floor. +- **Kill-trigger 2 (bounded escalation):** if a hardened task still shows no gap after one + escalation, declare no natural gap with evidence (D20). + +Both triggers fired exactly as written. The S3 result (20/20 clean) triggered the injected-gap +path; S7's 8/8 hard-v1 and 8/8 hard-v2 triggered the no-natural-gap conclusion. **Inference** +— pre-committing routing rules is the structural mechanism that prevents the team from +rationalizing past inconvenient results: a null is a null if it's pre-agreed, not re-framed +as "we need one more run." + +### What counted as a null — and why S6 is a real finding + +**Fact** — S6 measured +0.0 pp [−16.1, +16.1] for both retry-nudge and error-recovery vs +baseline on the malformed-call testbed (D19, [docs/DECISIONS.md](../docs/DECISIONS.md)). +Both intervals straddle zero → both are nulls by the pre-committed gate. The null was +published as the result, not suppressed. + +**Inference** — this null is a stronger finding than it might appear: the retry-nudge arm +*did* issue 26 corrective re-prompts (the mechanism worked), but GLM self-corrected from the +400-error hint in the tool result *before* the nudge was needed, so the mechanism had no +work to do. This establishes a *boundary* on where guardrails help: only where the model +cannot self-correct. + +### What would have falsified the project's claim + +**Inference** — the thesis (each failure class has a matched guardrail that provably lifts +completion) would have been falsified by any of: +- A Newcombe interval straddling zero on a "real lift" arm after an honest N (none occurred + for the four designed-to-lift arms: S4, S8, S9, S10). +- Evidence that the validator read `scenario.ground_truth` (the bright line; verified in code + review of S9 and D23's re-verification). +- A guardrail arm that raised completion via a route other than the one it targets (e.g., + error-recovery suppressing a malformed fault by chance) — structural separation of fault + types prevents this (D19: the `is_retryable` check classifies 503 vs 400 at the harness + level). + +**Unresolved** — no independent replication has been run; all measurements are single-team, +single-codebase, and the fault injectors are deterministic (seeded), so the same seeds would +reproduce but a different seed set is untested at scale. + +### The honesty caption rule + +**Fact** — every figure in `docs/figures/` carries a caption that explicitly states whether +the gap is injected or natural. The rule is encoded in `chart.py`'s `caption_fn` parameter: +the S4/S5 figure says "gap is INJECTED · 104 transient 503s absorbed"; S8/S9/S10 figures +label the gap as natural (no injection). The injected vs. natural distinction was never +dropped or softened in any figure. **Decision** — D1, [docs/DECISIONS.md](../docs/DECISIONS.md): +the framing is always "reproduced and measured a known primitive," never "invented." + +## Sources +- [docs/DECISIONS.md](../docs/DECISIONS.md) — D1, D2, D3, D5, D7, D12, D13, D15, D16, D19, D22, D23 (the methodology decisions, each with options weighed) +- [docs/LEARNING.md](../docs/LEARNING.md) — plain-English teaching notes on CIs, proportion statistics, and the oracle design +- [CLAUDE.md](../CLAUDE.md) — "Methodology guardrails (load-bearing — do not drift)" section +- [PROJECT.md](../PROJECT.md) — "Boundaries" section restates the load-bearing methodology rules + +## Uncertainties & contradictions +- **Unresolved** — temperature 0.7 was fixed (D5) because GLM is stochastic regardless; the precise effect of temperature on the measured gaps is not characterized. A temperature-sweep sensitivity analysis was not conducted. +- **Unresolved** — S10's "first-evidence anchoring" in `Scenario.validate` (recomputes from the first retrieved value of each field, not the most recent) was a design property disclosed but not altered mid-experiment; its effect on the 10% wrong-record slice is an acknowledged limitation. + +## Related pages +- [Results-Synthesis](Results-Synthesis.md) +- [Experiment-Harness](Experiment-Harness.md) +- [History](History.md) + +## Relevance to current work +The project is closed; these guardrails are the durable rules for any follow-on project. Any +reuse of this harness (live capability ladder, self-hosted endpoint — D24 roads not taken) +should inherit these rules wholesale, particularly the `excludes_zero` gate and the +injected-vs-natural disclosure requirement. + +_Last reviewed: 2026-07-26_ diff --git a/Wiki/Results-Synthesis.md b/Wiki/Results-Synthesis.md new file mode 100644 index 0000000..53ab03b --- /dev/null +++ b/Wiki/Results-Synthesis.md @@ -0,0 +1,117 @@ +# Results-Synthesis + +## Purpose +One place to read what forge-gap actually measured end to end: every headline number, +its pre-registered prediction, the observed outcome, and what each result means for the +project's thesis. For anyone trying to defend or critique the measurements without reading +all five per-stage figures plus the decision ledger. + +## Key understanding + +### The four measured gaps — pre-registration vs. observed + +| Failure class | Matched guardrail | Stage | Pre-registered expectation | Observed result | Verdict | +|---|---|---|---|---|---| +| Transient 503 (injected, rate 0.6) | error-recovery | S4 | gap should clear zero; Newcombe CI excludes 0 | **+32.5 pp** [+17.3, +48.0] — GLM-4.6, N=40 | **REAL** | +| Malformed call (injected, rate 0.6) | retry-nudge | S6 | pilot warned of a null; GLM may self-heal | **null** (+0.0 pp, [−16.1, +16.1]) — GLM self-corrects in one turn | **NULL** (pre-registered risk) | +| No-submit (natural, clean task) | submit-nudge | S8 | lift expected; retry-nudge null in same run (specificity control) | **+75.0 pp** [+47.8, +88.8] — mistral-nemo, N=20 | **REAL** | +| Wrong-answer / arithmetic slip (natural) | validation | S9 | lift on nemo's clean 140→158 slip; possibly catches all 5 misses | **+25.0 pp** [+11.1, +40.2] — mistral-nemo, N=40 | **REAL** | +| Wrong-answer / hallucination (natural) | validation (un-stacked) | S10 | recovers checkable slice; residual expected due to blind spot | **+45.0 pp** [+28.2, +60.2] — llama-3.1-8b, N=40 | **REAL (partial)** | + +**Fact** — all point estimates and Newcombe CIs traced directly to vendored data files: +`docs/figures/gap-closure-data.json` (S4), `docs/figures/malformed-gap-data.json` (S6), +`docs/figures/weak-gap-data.json` (S8), `docs/figures/validation-data.json` (S9), +`docs/figures/hallucination-gap-data.json` (S10). + +### What the injected-gap chart shows — and does not + +The project's first headline figure (`docs/figures/gap-closure.png`) shows two bars: +baseline 67.5% vs +error-recovery 100% on GLM-4.6, N=40, fault-rate 0.6. + +**Fact** — the gap is **injected**: a deterministic 503-style fault at a published rate; it +is not GLM's natural failure. The figure carries this disclosure in its caption: "gap is +INJECTED (controlled fault-recovery testbed) · 104 transient 503s absorbed." **Decision** — +D12, [docs/DECISIONS.md](../docs/DECISIONS.md): the injected testbed was the pre-agreed +floor when kill-trigger 1 fired (GLM scored 20/20 clean at S3 — no natural gap to close). + +What the figure does **not** show: it does not claim GLM fails naturally, and it does not +show a guardrail bar for GLM's clean task (there was none to measure — S7 proved GLM aces +hardened tasks too; **Decision** D24, [docs/DECISIONS.md](../docs/DECISIONS.md)). + +### Capability × guardrail: the capstone ladder + +The S11 capstone figure (`docs/figures/capstone-ladder.png`) synthesizes three models on +the clean task: + +| Model | Baseline (clean task) | + matched guardrails | Gap | +|---|---|---|---| +| GLM-4.6 (strong) | 100% [83.9%, 100%] | no bar — nothing to close | — | +| mistral-nemo (mid) | 0% [0.0%, 16.1%] | +submit-nudge+validation: 100% [91.2%, 100%] | +100.0 pp [+81.7, +100.0] (cross-run) | +| llama-3.1-8b (weak) | 0% [0.0%, 8.8%] | +validation: 45% [30.7%, 60.2%] | +45.0 pp [+28.2, +60.2] (same run) | + +**Fact** — numbers traced to `docs/figures/capstone-data.json`. **Decision** — D24, +[docs/DECISIONS.md](../docs/DECISIONS.md): the ladder is derived from already-measured +data, not a new experiment; GLM draws one bar because no guardrail arm was ever run on its +clean task. + +**Inference** — the ladder supports the thesis that the weaker the model, the more a matched +guardrail buys — but only up to the limit of what the guardrail can structurally see (llama's +45% ceiling is not a failure of the design; it is the blind spot, measured and decomposed). + +### The S10 residual decomposition — the blind spot, quantified + +S10's stage-specific contribution is decomposing the 55% llama-8b residual that validation +cannot recover: + +| Residual slice | Share | Why structurally un-recoverable | +|---|---|---| +| Never retrieved the rate | 35% (14/40) | validator accepts-by-design (no evidence to recompute from) | +| Wrong-record retrieval | 10% (4/40) | validator fooled — self-consistent but wrong total (152 vs 158) | +| Non-numeric submission | 7.5% (3/40) | validator passes through; oracle fails | +| Never submitted | 2.5% (1/40) | nothing to validate | + +**Fact** — decomposition traced to [docs/DECISIONS.md](../docs/DECISIONS.md) D23 (hand-read +of every validation-arm miss). **Inference** — this decomposition turns D22's disclosed blind +spot from a caveat into a number: the validator closes consistency failures but not evidence +failures. + +### Guardrail specificity — the in-experiment controls + +Three experiments included a deliberately wrong guardrail as an in-experiment control: + +- S6: error-recovery ≈ baseline on malformed faults (structural — a permanent fault is never + retried by the recovery guardrail). **Fact** — D19, [docs/DECISIONS.md](../docs/DECISIONS.md). +- S8: retry-nudge = 0 nudges fired on mistral-nemo's no-submit gap (a missing call is not a + failed call). **Fact** — `docs/figures/weak-gap-data.json`, `retry_nudge` arm, `nudges: 0`. +- S9: validation's own mechanism was verified honest — it fired on 5 of 5 `140` submissions and + never read `scenario.ground_truth`. **Fact** — D22 measured result, [docs/DECISIONS.md](../docs/DECISIONS.md). + +**Inference** — specificity held in every experiment: the guardrail matched to a failure class +moves the number; unmatched guardrails null. + +## Sources +- [docs/DECISIONS.md](../docs/DECISIONS.md) — D12, D13, D17, D19, D21, D22, D23, D24 (all measured results + pre-registration) +- [docs/figures/gap-closure-data.json](../docs/figures/gap-closure-data.json) +- [docs/figures/weak-gap-data.json](../docs/figures/weak-gap-data.json) +- [docs/figures/validation-data.json](../docs/figures/validation-data.json) +- [docs/figures/hallucination-gap-data.json](../docs/figures/hallucination-gap-data.json) +- [docs/figures/capstone-data.json](../docs/figures/capstone-data.json) +- [docs/ROADMAP.md](../docs/ROADMAP.md) — stage table with per-stage outcome summaries +- [PROJECT.md](../PROJECT.md) — headline results table + +## Uncertainties & contradictions +- **Unresolved** — the malformed-gap figure data file (`docs/figures/malformed-gap-data.json`) was not read directly for this page; the null result (+0.0 pp, [−16.1, +16.1]) is sourced from D19's measured-result section rather than the vendored JSON. +- **Inference** — the nemo cross-run Newcombe (+100 pp [+81.7, +100.0]) combines the S8 baseline (N=20) and S9 guardrail arm (N=40) from two separate experiments; D24 explicitly discloses this as cross-run and explains why it is statistically sound. + +## Related pages +- [Methodology-Guardrails](Methodology-Guardrails.md) +- [Experiment-Harness](Experiment-Harness.md) +- [History](History.md) + +## Relevance to current work +The project is closed (S11, D24). This page is the durable one-stop record of what was +measured — useful for portfolio defense, future project seeding (what the remaining 55% of +llama's gap would require), and as the entry point for any revisit under the D24 roads not +taken. + +_Last reviewed: 2026-07-26_ diff --git a/Wiki/_index.md b/Wiki/_index.md index 9835e52..d23ae53 100644 --- a/Wiki/_index.md +++ b/Wiki/_index.md @@ -3,3 +3,6 @@ | Page | Covers | Last reviewed | |------|--------|---------------| | [History](History.md) | Chronological eras & milestones (S0–S11 + post-completion), reconstructed from merged PRs, git history, wrap logs; decisions anchored to `docs/DECISIONS.md` D1–D24 | 2026-07-26 | +| [Results-Synthesis](Results-Synthesis.md) | Every headline number in one table — pre-registered prediction vs. observed outcome, Newcombe CIs, the injected-gap chart explained, capability ladder, and S10 residual decomposition; synthesized across 5 figure JSONs + D12–D24 + PROJECT.md | 2026-07-26 | +| [Methodology-Guardrails](Methodology-Guardrails.md) | The honesty machinery: deterministic oracle, Wilson/Newcombe CIs, the `excludes_zero` gate, pre-registration + kill-triggers, what counted as a null, what would have falsified the thesis; synthesized across DECISIONS.md (D2/D3/D7/D12/D13/D16/D19), CLAUDE.md, and stats.py | 2026-07-26 | +| [Experiment-Harness](Experiment-Harness.md) | How the code is organized to run an experiment: guardrail toggles, arm-as-config pattern, fault injectors, where results land, all entry-point reproduce commands; mined from code + DECISIONS.md (D11/D14–D19/D23) | 2026-07-26 |