Pre-registered benchmark suite for plan-composition conditions - #54
Conversation
…extraction Pre-registered A/B/C/D benchmark per approved plan (dual-reviewed v2). Judge internals extracted from judge-bounce.sh unchanged (byte-diff golden test) so cross-condition judging can share blinding and evidence verification. Corpus frozen before any generation runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase 2-5 of the pre-registered benchmark: run-benchmark.sh (resumable task x condition matrix with GLM quota ledger and MAX_PATH guard), run-panel.sh (condition C critique panel), judge-matrix.sh + report.sh (three-judge blind pairwise protocol). All test suites stubbed, $0. tests/run-all.sh gains --jobs and --resume with a per-suite checkpoint ledger: interrupted runs continue instead of restarting, and the pass gate is cumulative across runs on the same tree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pr-emitter's orphan-cleanup glob claimed pel-code-sandbox-* dirs it never creates, and code-proposer sandboxes lived in machine-global /tmp — so any concurrent suite (or straggler EXIT trap) could delete a live sandbox mid-canary. Surfaced by run-all --jobs; latent before. Scope proposer sandboxes to the suite's own TMPDIR and drop the foreign glob. Verified: both suites green run concurrently (--jobs 2). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e8bcebbc17
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| **Primary comparison: B-vs-A** — does the cross-vendor Codex bounce beat stock | ||
| Fable? Decision rule, with 8 tasks, using the primary judge's sign test: | ||
|
|
||
| - B "helps" only at **>=7/8 decisive wins** (p<0.05, two-sided). |
There was a problem hiding this comment.
Seven wins in eight independent decisive trials has an exact two-sided binomial p-value of 0.0703125, not p<0.05; only 8/8 reaches a two-sided 5% threshold. As written, the preregistration and report.sh can label a non-significant result as “B helps”; either use an 8/8 threshold or explicitly preregister a one-sided test.
Useful? React with 👍 / 👎.
| if [[ -f "$out" ]] && verdict_is_valid "$out"; then | ||
| log "INFO: $task $seat $x vs $y — verdict already present, skipping." | ||
| SKIPPED=$((SKIPPED + 1)) | ||
| return 0 |
There was a problem hiding this comment.
When a documented --force-cell t3/B reruns generation, every existing pair verdict involving B still passes this shallow validity check and is skipped, even though the stored prompt_sha256 refers to the old sanitized plan. The resulting report combines new generation artifacts with judgments of old artifacts; validate the stored prompt/document hashes against the current task and sanitized inputs before reusing a verdict.
AGENTS.md reference: AGENTS.md:L148-L150
Useful? React with 👍 / 👎.
| for attempt in 1 2; do | ||
| : > "$raw"; : > "$errf" | ||
| ( | ||
| GLM_MODEL="$JUDGE_MODEL_GLM" | ||
| invoke_glm "$prompt_file" "$raw" "$errf" false |
There was a problem hiding this comment.
With the default 8 tasks and 4 surviving conditions, this path dispatches 96 GLM trials without consulting or debiting glm-ledger.json. This contradicts the runbook's promised daily budget/pending-quota behavior, so a normal judging run can exhaust the free-tier allowance mid-batch instead of parking before dispatch; gate and debit each two-trial GLM pair using the shared ledger helpers.
AGENTS.md reference: AGENTS.md:L153-L154
Useful? React with 👍 / 👎.
| if [[ "$RESUME" == true && -f "$LEDGER_DIR/$name.result" ]] \ | ||
| && read -r prev _ < "$LEDGER_DIR/$name.result" && [[ "$prev" == "PASS" ]]; then |
There was a problem hiding this comment.
A ledger entry contains only PASS <elapsed>, so --resume skips the suite even after HEAD, tracked files, or the suite itself changes. Reusing the default .run-ledger after a checkout or edit can therefore report the new tree green without testing it; store and compare a tree/input fingerprint before accepting a prior PASS.
Useful? React with 👍 / 👎.
…benchmarks-plan-694e39
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
1 participant
Summary
Completed pilot
The permitted two-task pilot is complete; the full eight-task batch was not started.
benchmarks/reports/pilot1.mdThe pilot is directional only. Its pre-registered B-vs-A primary result is “No evidence” at N=2. Exploratory condition C led the Fable and Codex rankings and won all six of their pair comparisons; GLM recorded 4.5/6 wins, with position-swap disagreement retained as
position_biasedrather than forced into a winner.Verification
bash benchmarks/run-benchmark.sh --checkbash benchmarks/run-benchmark.sh --dry-runOperational notes