Skip to content

docs(wiki): add topic pages to the project wiki #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ksdisch merged 1 commit into from
Jul 27, 2026
Merged

docs(wiki): add topic pages to the project wiki #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
142 changes: 142 additions & 0 deletions Wiki/Experiment-Harness.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
# Experiment-Harness

## Purpose
A map of how the code is actually organized to run an experiment: entry points, the
guardrail toggle system, what a "run" produces, and where results land. For anyone
who wants to reproduce a measurement, add a new arm, or understand what each script does
without reading every file.

## Key understanding

### Architecture in one sentence

**Fact** — the harness is a layered stack: `glm.py` (API client) → `agent.py` (loop +
guardrail toggles) → `scenario.py` + `oracle.py` (task + grader) → `faults.py` (optional
fault injectors) → `runner.py` (single-arm looper) → `ablation.py` (multi-arm harness) →
per-experiment scripts (entry points). `chart.py` and `stats.py` are offline post-processing;
they make no API calls. Source: [CLAUDE.md](../CLAUDE.md), "File map" section.

### The guardrail toggle system

`agent.py`'s `run()` function exposes four independent boolean toggles, each corresponding
to one measured guardrail:

| Toggle | Stage | What it does | Cost |
|---|---|---|---|
| `recover=True` | S4 | harness-level retry of transient 503s (matches `_is_retryable`) | no model turn |
| `nudge=True` | S6 | re-prompts model to correct a malformed/failed tool call | 1 model turn |
| `submit_nudge=True` | S8 | re-prompts model to call `submit_answer` if a turn ends in prose without submitting | 1 model turn |
| `validate=True` | S9 | on `submit_answer(value)`, recomputes from retrieved observations; re-prompts on mismatch | 1 model turn |

**Fact** — all four toggles default to `False`; the bare baseline runs with no mechanisms.
The toggles are additive: any combination is valid, and the S9 stacked ablation uses
`submit_nudge=True, validate=True` simultaneously. Source: [CLAUDE.md](../CLAUDE.md) `agent.py`
entry.

**Inference** — the toggle design means adding a new guardrail requires (a) one new flag on
`run()`, (b) one arm config in `ablation.py`, and (c) a new per-experiment entry-point script;
no existing mechanism code changes. This is why S6, S8, S9, S10 each reused the same harness
without rewrites.

### Arm configuration and the ablation harness

`ablation.py` defines arms as config dicts: `{"label": str, "run_kwargs": dict}`. The
harness in `run_arms()` runs each arm N times over identical seeded scenarios, computes
Wilson CIs per arm, and computes a Newcombe gap vs the designated baseline arm.

**Fact** — the five named arm configs live in `ablation.py`:
`BASELINE_ARM`, `RECOVERY_ARM`, `NUDGE_ARM`, `SUBMIT_NUDGE_ARM`, `VALIDATION_ARM`
(stacked, S9), `VALIDATION_ONLY_ARM` (un-stacked, S10). Source: [CLAUDE.md](../CLAUDE.md)
`ablation.py` entry.

Per-experiment scripts pick subsets:
- `ablation.py` (run directly): S4/S5 2-arm error-recovery ablation
- `malformed_ablation.py`: S6 3-arm malformed-call ablation
- `weak_ablation.py`: S8 3-arm submit-nudge ablation on a weak model
- `validation_ablation.py`: S9 stacked 2-arm validation ablation
- `hallucination_ablation.py`: S10 un-stacked 2-arm validation ablation on llama-8b

**Fact** — every per-experiment script accepts the model and N as command-line arguments;
fault-rate is a third argument where applicable. Source: D15 and D19,
[docs/DECISIONS.md](../docs/DECISIONS.md) ("operating point is a runtime knob").

### Fault injection and the clean-vs-injected boundary

`faults.py` provides two non-mutating injectors that wrap the base scenario:
- `with_faults(scenario, rate, seed)` — injects transient 503s at a given rate (S3–S5).
- `with_malformed_faults(scenario, rate, seed)` — injects a sticky malformed-call fault
(S6); "sticky" means the same call keeps failing until corrected (D19).

**Fact** — `rate=0` is equivalent to the clean task for either injector; S8–S11 run on
the clean task with no injector wrapper. The fault classification boundary is encoded in
`agent._is_retryable()`: a 503/timeout → recoverable by `error-recovery`; a 400 with
`invalid_argument` → not retryable (error-recovery leaves it alone). Source: D19,
[docs/DECISIONS.md](../docs/DECISIONS.md), "Two load-bearing properties."

### Where results land

| Output | Location | Gitignored? | Notes |
|---|---|---|---|
| Per-trial trajectories | `runs/<experiment>/trial-*.jsonl` | yes (gitignored) | Full step-by-step JSONL per run; hand-read for failure triage |
| Per-experiment summary | `runs/<experiment>-summary.json` | yes | Raw k/N + CIs; input to `chart.py` |
| Vendored figure data | `docs/figures/*-data.json` | no (committed) | Copied from `runs/` after a live run; charts regenerate from these |
| Figures | `docs/figures/*.png` | no (committed) | Regenerate with `uv run chart.py` — no API, no model call |

**Fact** — the vendoring step is manual: after a live run, copy the summary JSON into
`docs/figures/` so the figure can regenerate in-repo from a clean clone. The capstone data
file (`docs/figures/capstone-data.json`) is an exception: `chart.py` derives it
automatically from the three per-stage JSONs on every run, so it can never silently drift.
Source: D18 and D24, [docs/DECISIONS.md](../docs/DECISIONS.md).

### Entry points and how to reproduce

| What to run | Command | Prerequisites |
|---|---|---|
| Smoke test (API + tool-calling) | `uv run verify.py` | `OPENROUTER_API_KEY` in `.env` |
| All offline tests (77 tests, no API) | `uv run pytest` | none |
| S4 error-recovery ablation | `uv run ablation.py z-ai/glm-4.6 40 0.6` | API key |
| S6 malformed ablation | `uv run malformed_ablation.py z-ai/glm-4.6 20 0.6` | API key |
| S7/8 pilot (hardened task) | `uv run pilot.py` or `uv run pilot.py v2` | API key |
| S8 weak-model ablation | `uv run weak_ablation.py mistralai/mistral-nemo 20` | API key |
| S9 stacked validation | `uv run validation_ablation.py mistralai/mistral-nemo 40` | API key |
| S10 hallucination ablation | `uv run hallucination_ablation.py meta-llama/llama-3.1-8b-instruct 40` | API key |
| Regenerate all figures (offline) | `uv run chart.py` | none (reads vendored JSONs) |
| Check docs freshness | `uv run check_docs.py` | none |

**Fact** — all reproduce commands sourced from D17, D19, D22, D23, D24 measured-result
sections in [docs/DECISIONS.md](../docs/DECISIONS.md) and the [CLAUDE.md](../CLAUDE.md)
"How to run" section.

### The S10 harness fix — mid-experiment disclosure

**Fact** — mid-S10, a latent bug was discovered: llama-3.1-8b sometimes emits tool-call
arguments as a JSON array (`["ORD-204"]`) rather than a dict. `agent.py` conflated "JSON
parsed" with "arguments OK," causing a crash on `.get()` applied to a list. Fix:
`args_ok = isinstance(args, dict)` — non-object args now route to the existing
malformed-arguments path. The fix adds no new help to the model; it only prevents a crash.
The pilot ran pre-fix; the full N=40 run ran post-fix. This is disclosed in D23,
[docs/DECISIONS.md](../docs/DECISIONS.md). Source: D23 "A latent harness bug, exposed
and fixed mid-stage."

## Sources
- [CLAUDE.md](../CLAUDE.md) — "File map" section (per-file descriptions)
- [docs/DECISIONS.md](../docs/DECISIONS.md) — D11 (lean runner), D14 (toggle design), D15 (operating-point knobs), D16 (ablation arms-as-config), D18 (vendoring convention), D19 (fault stickiness + `is_retryable`), D23 (S10 harness fix)
- [docs/ROADMAP.md](../docs/ROADMAP.md) — per-stage "What it does" column
- `ablation.py`, `agent.py`, `faults.py`, `chart.py` — code (entry points, toggle signatures, injector wrappers)

## Uncertainties & contradictions
- **Unresolved** — the `trajectory.jsonl` root-level file (tracked in the repo) appears to be a leftover from early development; it is not a run output from any named experiment. Its relationship to the gitignored `runs/` outputs is not documented.
- **Inference** — the `check_docs.py` freshness check is described as "a smoke alarm, not a commit gate" ([CLAUDE.md](../CLAUDE.md)); this means it is possible for the docs spine to drift without CI catching it. No evidence of such drift in the current state.

## Related pages
- [Results-Synthesis](Results-Synthesis.md)
- [Methodology-Guardrails](Methodology-Guardrails.md)
- [History](History.md)

## Relevance to current work
The project is closed; no new experiment code should be added without a decision brief
(D24). This page is the entry point for anyone resuming the project or forking the harness
for a new repro — it answers "what do I run and where do results land?" without requiring a
full code read.

_Last reviewed: 2026-07-26_
130 changes: 130 additions & 0 deletions Wiki/Methodology-Guardrails.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
# Methodology-Guardrails

## Purpose
A synthesis of the honesty machinery that governs every measurement in this project —
pre-registration, statistical method, oracle design, seed handling, and the rules for
reporting a null. For anyone asking "how do I know these results are trustworthy?" or
preparing to defend a methodology choice in an interview.

## Key understanding

### The fixed ruler: deterministic oracle, never an LLM judge

**Decision** — D2, [docs/DECISIONS.md](../docs/DECISIONS.md): task success is measured by
`oracle.py`, a plain-Python function that compares the model's submitted number to a
known-correct answer (158 for the standard scenario). The oracle reads from `ORDERS` and
`SHIP_RATES` directly — it is independent of the model's run. It cannot be fooled by a
wrong retrieval; it always reports the ground truth.

**Inference** — this is the structural separation the validation guardrail (S9) was designed
around: `oracle.py` reads from canonical data; `Scenario.validate` reads only from the
model's own run observations. The two compute the same arithmetic but from different inputs,
which is why `validate` can be fooled (wrong-record retrieval yields a self-consistent wrong
total that the oracle fails) and the oracle cannot.

### Statistical method: proportion CIs, not ±std

**Decision** — D7 + D16, [docs/DECISIONS.md](../docs/DECISIONS.md): completion rate is a
proportion, not a continuous measurement. The project uses:
- **Wilson interval** (`stats.py`, `wilson()`) — per-arm CI that stays sane at the edges
(near 0% and 100%), unlike the Wald ±std which can run outside [0, 1].
- **Newcombe "square-and-add" interval** (`stats.py`, `newcombe_diff()`) — the CI on the
*difference* between arms; carries Wilson's good edge behaviour into the gap estimate.
- **`excludes_zero` gate** (`stats.py`, `excludes_zero()`) — the mechanical honesty rule:
if the Newcombe interval on the difference includes 0, the result is reported as a null,
never a win. This gate was pre-committed (D7, written before any guardrail was built) and
applied consistently in S4, S6, S8, S9, S10.

**Fact** — `stats.py` is pure Python with offline unit tests (`test_stats.py`); it is
exercised before any live run. The gate is not a judgement call applied after seeing results;
it is encoded in the harness (`ablation.py` checks `excludes_zero` per arm).

### N = distinct seeds, not trial count

**Decision** — D13 + D15, [docs/DECISIONS.md](../docs/DECISIONS.md): with deterministic
per-trial seeds (`seed=0..N-1`), completion under fault injection is dominated by the fault
*pattern*, not model stochasticity. Running two N=20 arms on seeds 0–19 is a paired comparison;
re-running seeds 0–19 again is reproducibility, not more data. This is why statistical power
comes from more *distinct* seeds, not more total trials on the same seeds.

**Inference** — this is a non-obvious property of the experiment design, and it has a
practical consequence: the recommended operating point (D15: rate 0.6, N=40) was chosen
because N=40 distinct seeds gives tighter Newcombe CIs than N=20, not because 40 trials
are intrinsically better than 20.

### Pre-registration and kill-triggers

**Decision** — D12, [docs/DECISIONS.md](../docs/DECISIONS.md): before the S3 diagnostic was
run, the team pre-committed to two routing rules:
- **Kill-trigger 1:** if GLM aces the clean task (≳85%), there is no natural gap; pivot to
fault injection as the floor.
- **Kill-trigger 2 (bounded escalation):** if a hardened task still shows no gap after one
escalation, declare no natural gap with evidence (D20).

Both triggers fired exactly as written. The S3 result (20/20 clean) triggered the injected-gap
path; S7's 8/8 hard-v1 and 8/8 hard-v2 triggered the no-natural-gap conclusion. **Inference**
— pre-committing routing rules is the structural mechanism that prevents the team from
rationalizing past inconvenient results: a null is a null if it's pre-agreed, not re-framed
as "we need one more run."

### What counted as a null — and why S6 is a real finding

**Fact** — S6 measured +0.0 pp [−16.1, +16.1] for both retry-nudge and error-recovery vs
baseline on the malformed-call testbed (D19, [docs/DECISIONS.md](../docs/DECISIONS.md)).
Both intervals straddle zero → both are nulls by the pre-committed gate. The null was
published as the result, not suppressed.

**Inference** — this null is a stronger finding than it might appear: the retry-nudge arm
*did* issue 26 corrective re-prompts (the mechanism worked), but GLM self-corrected from the
400-error hint in the tool result *before* the nudge was needed, so the mechanism had no
work to do. This establishes a *boundary* on where guardrails help: only where the model
cannot self-correct.

### What would have falsified the project's claim

**Inference** — the thesis (each failure class has a matched guardrail that provably lifts
completion) would have been falsified by any of:
- A Newcombe interval straddling zero on a "real lift" arm after an honest N (none occurred
for the four designed-to-lift arms: S4, S8, S9, S10).
- Evidence that the validator read `scenario.ground_truth` (the bright line; verified in code
review of S9 and D23's re-verification).
- A guardrail arm that raised completion via a route other than the one it targets (e.g.,
error-recovery suppressing a malformed fault by chance) — structural separation of fault
types prevents this (D19: the `is_retryable` check classifies 503 vs 400 at the harness
level).

**Unresolved** — no independent replication has been run; all measurements are single-team,
single-codebase, and the fault injectors are deterministic (seeded), so the same seeds would
reproduce but a different seed set is untested at scale.

### The honesty caption rule

**Fact** — every figure in `docs/figures/` carries a caption that explicitly states whether
the gap is injected or natural. The rule is encoded in `chart.py`'s `caption_fn` parameter:
the S4/S5 figure says "gap is INJECTED · 104 transient 503s absorbed"; S8/S9/S10 figures
label the gap as natural (no injection). The injected vs. natural distinction was never
dropped or softened in any figure. **Decision** — D1, [docs/DECISIONS.md](../docs/DECISIONS.md):
the framing is always "reproduced and measured a known primitive," never "invented."

## Sources
- [docs/DECISIONS.md](../docs/DECISIONS.md) — D1, D2, D3, D5, D7, D12, D13, D15, D16, D19, D22, D23 (the methodology decisions, each with options weighed)
- [docs/LEARNING.md](../docs/LEARNING.md) — plain-English teaching notes on CIs, proportion statistics, and the oracle design
- [CLAUDE.md](../CLAUDE.md) — "Methodology guardrails (load-bearing — do not drift)" section
- [PROJECT.md](../PROJECT.md) — "Boundaries" section restates the load-bearing methodology rules

## Uncertainties & contradictions
- **Unresolved** — temperature 0.7 was fixed (D5) because GLM is stochastic regardless; the precise effect of temperature on the measured gaps is not characterized. A temperature-sweep sensitivity analysis was not conducted.
- **Unresolved** — S10's "first-evidence anchoring" in `Scenario.validate` (recomputes from the first retrieved value of each field, not the most recent) was a design property disclosed but not altered mid-experiment; its effect on the 10% wrong-record slice is an acknowledged limitation.

## Related pages
- [Results-Synthesis](Results-Synthesis.md)
- [Experiment-Harness](Experiment-Harness.md)
- [History](History.md)

## Relevance to current work
The project is closed; these guardrails are the durable rules for any follow-on project. Any
reuse of this harness (live capability ladder, self-hosted endpoint — D24 roads not taken)
should inherit these rules wholesale, particularly the `excludes_zero` gate and the
injected-vs-natural disclosure requirement.

_Last reviewed: 2026-07-26_
Loading
Loading