Admit Only What Re-Derives

Preregistered, verifier-arbitrated experiments on trust surfaces in persistent agents. Re-derive the claim — don't trust it. Clone it, read the preregistrations, and re-run the checks yourself.

A self-improving agent writes its own skills, remembers across sessions, and reports its own test results — so it is the author, executor, and inspector of its own work. This repository is the experimental record behind one answer to that problem: a host-side gate that admits a skill, a memory, or a "tests pass" verdict only when re-running it reproduces the result. The agent's word is never an input.

Every experiment ships its preregistration (hypothesis, arms, and primary statistic fixed before the run), a deterministic scorer as the sole arbiter, and a frozen report. The dispatch harness is a neutral, direct Anthropic-API call (no agent framework, no project policy) — the conservative test condition. This is the companion artifact to the preprint "Admit Only What Re-Derives: A Preregistered, Verifier-Arbitrated Study of Trust Surfaces in Persistent Agents."

Install

pip install -r requirements.txt

This pulls veriker (the open re-derivation verifier) and relyable (the admission gate built on it) — the two packages that back every experiment's correctness check. The deterministic scorers and the dispatch harness add no other dependencies.

Note (publication timing). veriker is on PyPI. relyable publishes alongside this repository; until it lands on PyPI, install it from its source repo and then uncomment the pin in requirements.txt. The model-free checks below need both imports to resolve but need no API key and spend no tokens.

Reproduce it yourself

Zero-cost — no API key, no model tokens. The scorers are deterministic and the solver dispatch is mocked with canned outputs, so the harness, the gate semantics, and the frozen decision rules are checkable offline:

pytest                      # 312 model-free tests across the experiment suite
pytest exp_z5/test_rd.py exp_z5/test_pipe.py   # just the two newest powered results

These exercise the real deterministic grader (relyable.gate) against mocked solver replies — fail-closed teeth, fixture faithfulness, and the McNemar/decision logic — without ever calling a model.

Full repro — needs an API key. To re-run a live experiment end to end, set ANTHROPIC_API_KEY in your environment and invoke its harness. For example, the self-verification head-to-head (RD, n=200):

export ANTHROPIC_API_KEY=...           # read from the environment only; never logged
python exp_z5/run_rd.py run --base runs_rd_repro --seeds 2000-2199

Every experiment's run_*.py is the same shape: a direct Anthropic Messages-API solver with a neutral system prompt, scored by the experiment's frozen grader. Raw per-seed run cells are not committed (they are regenerable and large); the frozen runs_*/…REPORT.md and RESULTS_*.json carry every preregistered statistic.

---

The experiment ledger

Numbers are quoted from the frozen reports, never from a solver's self-report. PW = powered · PL = pilot.

Surface	Exp	Dir	Tier	Frozen primary
Skills	3	exp3/, runs_exp3/	PW	admit 0.938 vs reject 0.062, p=7.45e-9
Skills	6	exp6/, runs_exp6/	PW	B 0.76 vs C 0.26 (Δ+0.50, p=4.6e-6); B−A +0.06, p=0.68 n.s.
Skills	5	exp5/, runs_exp5/	PL	forged label followed 10/10; evidence opened 0/10
Skills	4	exp4/, runs_exp4/	PL	labeled 1.00 vs unlabeled 0.00, p=0.002
Skills	2	exp2/, runs_exp2/	PL	gate vs self-vettable pile Δ0.00, p=1.0 (honest boundary)
Memory	H3	exp_h/, runs_h3/	PW	V_pure−G +0.950 (1.4e-17) / +0.933 (2.8e-17); G 0.000 flat
Memory	M	exp_m/, runs_exp_m/	PW	M_att_clean−A +0.34, p=1.5e-5; wrapper cost 0.00, p=1.0
Memory	F2	exp_f/, runs_confirm_f2/	PW	M_attested 1.00 vs A 0.00, p=1.86e-9
Memory	D / E / persist-E	exp_d/, exp_e/	PL	persist-E gated 6/6 vs raw 3/6; D 0.90 vs 0.00, p=0.0039
Verdicts	Y	exp_y/, runs_y/	PW	weak A 0.033 → B 0.967 (Δ+0.934, p=7.45e-9); 28/28 genuine re-derivation
Self-verify	RD	exp_z5/ (RD)	PW	gate recovery 0.640 vs self 0.000; McNemar b=0/c=55, p=5.6e-17
Capability	PIPE	exp_z5/ (PIPE)	PW	unaided 0/150; final-only 0/150; checkpointed 0.487 (73/150), p=2.1e-22
Capability	PIPE — K-sweep	exp_z5/ (KSWEEP)	PW	dose-response over K∈{2,3,4}: final-only collapses 0.529→0.092→0.000; checkpointed degrades gracefully 0.950→0.908→0.487 (≈ product of per-stage admits)

The K-sweep is the dose-response companion to PIPE (PREREGISTRATION_PIPE_KSWEEP.md, RESULTS_KSWEEP.json, SYNTHESIS_KSWEEP.md): as the number of simultaneous contested-default traps grows, blind whole-pipeline revision (final-only) collapses toward zero while per-stage checkpointing degrades only as hard stages are added — granularity is the active ingredient, measured across the dose.

Supporting infrastructure: exp_w/ (W, a routing-label gap-narrowing result), exp_z4/ (the second-family generalization — a preregistered null, published in full, plus the shared direct-API dispatch rail the z5 harnesses reuse), arms/ (paired McNemar / Wilson statistics), fixture*/, verifier*/, corpus/, mechanism/. EXPERIMENT_ARC_NOTE.md is the cross-experiment synthesis — start there for the whole story.

Posture

This is a research record, not a product. The gate it studies is relyable, built on the open veriker substrate; both are Apache-2.0 and v0.x experimental, and this repository carries the same honest frame — preregistered claims, bounded exactly to what the frozen statistic supports, with the one published null (exp_z4/) and the superseded pilots disclosed rather than dropped. No funnel, no telemetry, no analytics: clone it and check the work.

The experiments behind this paper are developed by Nexi Technologies, which also publishes the open veriker verifier and the relyable gate they exercise.

Apache-2.0. See LICENSE.