Add paired harness A/B experiment infrastructure

[hututuQQQ]

What

• add a generic two-arm paired harness experiment controller with immutable preregistration
• support pinned official Codex Linux runtime archives alongside the Sigma Code runtime
• normalize Sigma JSONL and Codex ATIF traces into comparable correctness, efficiency, cost, and behavior metrics
• add append-only receipts, balanced arm order, gradual ramp stages, and score-independent stop-loss
• fix task-cluster bootstrap sampling so confidence intervals actually resample tasks with replacement
• compare cross-harness tool sequences through neutral semantic categories instead of harness-specific tool names
• document the workflow and cover it with unit/integration tests

Why

The existing benchmark path could run one harness at a time, but it could not execute a fair same-model paired A/B or compare raw trace behavior across harness formats.

Fairness and safety

• task sampling is a seeded SHA-256 ranking over a complete externally pinned catalog
• no task-name or benchmark-identity branching
• no verifier feedback is exposed to either solving agent
• no consumed attempt is retried
• stop-loss only reacts to blocking infrastructure/control-integrity failures, never scores
• prior stopped experiments remain sealed and are not pooled into the completed result

Verification

• pnpm test — 177 files passed, 1 skipped; 1,614 tests passed, 27 skipped
• pnpm lint
• pnpm test:harbor — 63 passed
• pnpm eval:fairness
• real benchmark-free Harbor AgentFactory construction with the pinned Codex archive
• git diff --check

Completed formal experiment

Frozen design:

• experiment: codex-vs-sigma-gpt56sol-max-tb21-20260814-v12
• preregistration SHA-256: 144a098161b36df196156942798429575e7a244e3acd1c931f3834aa3494c1d8
• consumption identity: f9e079a06f15723dc4559016fe905c262c503519928129977c1ed81263a34e82
• openai-codex/gpt-5.6-sol, reasoning max
• Terminal-Bench 2.1 at revision 5c8eadf1f393183288fa08b8f73ca9a469cc5e00
• 10 neutrally sampled tasks × 5 repetitions × 2 arms; concurrency 4; retries 0
• balanced 25/25 arm order; first-repetition ramp 1 → 4 → 10 tasks
• 100/100 attempts and 50/50 pairs completed; no stop receipt

Outcome

metric	Codex	Sigma
attempt pass rate	80.0% (40/50)	60.0% (30/50)
Wilson 95% interval	67.0%–88.8%	46.2%–72.4%
tasks passing at least once	9/10	8/10
median wall time	415,449 ms	735,000 ms
median uncached tokens	84,645	111,529
median output tokens	9,440	16,971
median reasoning tokens	4,498	8,624
median commands	20	32
median normalized tool calls	20	30.5

Task pass counts across five repetitions:

task	Codex	Sigma
tune-mjcf	5	5
code-from-image	5	5
path-tracing-reverse	5	5
reshard-c4-data	5	4
mteb-leaderboard	5	3
multi-source-data-merger	5	5
filter-js-from-html	1	0
caffe-cifar-10	4	1
sqlite-with-gcov	0	0
fix-ocaml-gc	5	2

Paired statistics

• Codex-only passes: 10; Sigma-only passes: 0; ties: 40
• exact two-sided McNemar p-value: 0.001953125
• the arm-order strata are consistent: Codex-only 6/25 when Sigma ran first and 4/25 when Codex ran first; Sigma-only 0 in both
• all-valid Sigma/Codex wall-time median ratio: 1.535, task-cluster bootstrap 95% interval 1.177–2.514; Sigma was slower in 41/50 pairs
• joint-success wall-time ratio: 1.375 (1.284–1.775); Sigma was slower in 26/30 pairs
• all-valid uncached-token ratio: 1.206 (0.988–1.569); directional but the cluster interval includes 1
• all-valid output-token ratio: 1.194 (1.112–1.562), reasoning-token ratio: 1.247 (1.194–1.659)
• all-valid command ratio: 1.267 (1.000–1.667); joint-success ratio: 1.394 (1.158–2.167)
• raw input-token ratio favors Sigma directionally (0.566, 0.396–1.050), while Sigma emits more output/reasoning tokens; input/cache accounting should not be treated as interchangeable with uncached-token demand

Trace findings

• trace coverage is 50/50 for each arm
• model turns are similar (median 21 vs 21.5; paired Sigma/Codex median ratio 0.9)
• Sigma makes more tool calls (paired median ratio 1.233; worse in 33/50 pairs)
• exact repeated tool-call signatures have median 0 for Codex and 25 for Sigma; Sigma is worse in 50/50 pairs (paired median ratio 8.071)
• after semantic tool-category normalization, paired sequence edit distance has median 0.667 (P25 0.538, P75 0.815), showing materially different execution shapes without conflating exec and shell naming
• cost is intentionally not compared: usable telemetry covers only 10 Codex attempts and 0 Sigma attempts

Interpretation and limits

Within this preregistered sample, Codex is materially better on correctness and wall time. Sigma's weaker result is accompanied by 8 timeouts (Codex: 0), more tool use, substantially more repeated exact calls, and more output/reasoning tokens. Sigma's main favorable direction is lower raw input-token accounting, but its cluster interval crosses parity and this does not translate into lower uncached-token demand or wall time.

The correctness Wilson intervals and McNemar test operate on the 50 attempt pairs; five repetitions from the same task are not fully independent. Efficiency intervals therefore task-cluster-resample all repetitions together. With only 10 sampled tasks, the result is strong evidence for this frozen experiment, not a universal ranking over Terminal-Bench or every workload.