Preregister paired BrowseComp-40 and ten-benchmark diversity suite

[Kitahl]

Status

Prospective / unscored. This PR freezes the next FOIL/profile evaluation design; it does not claim a benchmark result.

Experiment A — paired BrowseComp-40

• 40 fresh BrowseComp items.
• Same items across BASE, FOIL, FOIL_BROWSECOMP_PROFILE, FOIL_GENERAL_PROFILE, FOIL_MM.
• 200 isolated execution units.
• 12 search + 12 source-followup ceiling per unit.
• All BrowseComp rows previously sampled by repository pilots excluded.
• Complete-matrix-only scoring with per-condition Wilson intervals and exact paired discordance/McNemar analysis.

Experiment B — ten-benchmark diversity screen

10 paired items each from BrowseComp, FRAMES, WebWalkerQA, FreshQA, HLE, GPQA-Diamond, ARC-AGI-2, HotpotQA, MuSiQue, and DROP across BASE, FOIL, FOIL_GENERAL_PROFILE, FOIL_MM.

The first 10 BrowseComp items reuse Experiment A executions, yielding 560 unique execution units total.

Frozen before new-suite exposure

• Existing BrowseComp profile freeze: 013a728bfd6f57a8592fc3fc6e098ea52da357d5
• General profile freeze: 124c06b173ba6eff2fe0d23660a1ced8b7b975c2

Integrity gates

• fresh isolated context per (benchmark,item,condition)
• unique isolation session ID
• no sibling-condition outputs visible
• no gold until complete expected prediction matrix exists
• whole paired item invalidated for budget overrun, contamination, wrong prompt, or isolation failure
• explicit failure-code and phase-allocation traces
• prospective suite remains unscored until adapters and executions exist

Current implementation state

• BrowseComp-40 prepare/score adapter implemented.
• Ten-benchmark manifest and structural validator implemented.
• Remaining nine diversity adapters are intentionally marked planned and block execution_ready.

This PR should remain draft until the adapter layer and execution receipts are complete and CI is green on the exact head.