Make batch runs resumable, fault-tolerant, and shardable - #61
Conversation
A fifty-task batch is hours long. One arm that produced no patch aborted the whole run and discarded every remaining cell, and restarting was impossible because prepare refuses a directory that already exists. A cell holding a prediction is now reused, an abandoned clone from an interrupted attempt is re-prepared, and a cell that fails is counted and reported rather than ending the batch. Rerunning the same command retries only what is missing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fifty tasks across five arms is roughly twelve hours serially, which is long enough that an interruption is likely and a mistake is expensive. --shard N/M splits the subset across M processes on disjoint instances, so wall clock falls by about M. Shards write separate prediction files because concurrent appends to one file interleave, and each validates only its own so a sibling mid-write is not reported as corrupt. A finished cell whose line is missing from the current shard's file is re-linked from the cell itself, so a resume under a different shard layout still produces complete prediction files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9eedc40fcb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| input=$(bash "$CODE_DIR/scripts/prepare-swebench-instance.sh" "$instance" "$RUN_ID" "$condition") | ||
| cell="$runs_root/$instance/$condition" | ||
| if (( SHARD_COUNT > 1 )); then | ||
| pred_file="$pred_dir/$condition.s$SHARD_INDEX.jsonl" |
There was a problem hiding this comment.
When a run is resumed with a different shard count, the same index reuses the old filename because M is omitted: for example, 0/2 leaves instances 0, 2, and 4 in A.s0.jsonl, then 0/3 retains those records and adds instance 3. The new s1 and s2 files subsequently duplicate instances 4 and 2, so the shards are no longer disjoint and concatenating them fails duplicate validation. Include the shard count in the filename or rebuild each projection for the current layout.
Useful? React with 👍 / 👎.
| if (( SHARD_COUNT > 1 )); then | ||
| pred_file="$pred_dir/$condition.s$SHARD_INDEX.jsonl" | ||
| else | ||
| pred_file="$pred_dir/$condition.jsonl" |
There was a problem hiding this comment.
A successful sharded condition now exists only as multiple partial files, but the repository's evaluation entry point accepts one prediction file (scripts/evaluate-swebench.sh:43-50) and the site builder retains only the latest evaluator report for each condition (benchmarks/site/build-site-data.py:56-70). Evaluating shards separately therefore makes the last shard supersede the others, while evaluating any one file omits the remaining tasks; no merge path exists in the repository. Produce a consolidated condition JSONL or teach evaluation and aggregation to combine all shard files.
AGENTS.md reference: AGENTS.md:L153-L154
Useful? React with 👍 / 👎.
| if ! jq -e --arg id "$instance" 'select(.instance_id == $id)' "$pred_file" \ | ||
| >/dev/null 2>&1; then | ||
| jq -c . "$cell/prediction.json" >> "$pred_file" |
There was a problem hiding this comment.
If an interruption occurs while the driver is appending its potentially large JSON record, prediction.json may already be complete while the JSONL ends with a partial line. On resume, the lookup fails to parse that file and this branch appends the good record after the corrupt bytes, so final validation still aborts and the supposedly resumable run cannot recover without manual file editing. Detect parse failure separately and reconstruct the projection from completed cell records rather than appending.
Useful? React with 👍 / 👎.
A leaderboard that shows only a score answers half the question. Wall time, reported cost, and cost per resolved task are what both reference benchmarks lead with, and they were missing or wrong here: Codex time and tokens were never read at all, so an arm that spent twenty minutes of Codex effort showed a blank. Both are now taken from the log each phase already writes. An unfinished run was the worse problem. Scoring an arm that has run 21 of 50 tasks against 50 reports it as a 42% failure when it has actually solved everything it reached. An arm is now scored against the tasks it ran, labelled in progress, and the page says plainly that arms which have run different numbers of tasks cannot be compared to each other. A cost figure that covers only an arm's Claude phases is marked, because Codex, GLM and Kimi report no cost and rendering that as $0.00 reads as free rather than as unmeasured. Evaluator reports now carry a run label. Named only by timestamp, reports from two runs of one condition were indistinguishable and a page built across them silently mixed a 50-task subset with a 1-task one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 participant
Fifty tasks across five arms is roughly twelve hours serially. At that length an
interruption is likely, and the runner had no answer for one: a single arm that
produced no patch aborted the batch and discarded every remaining cell, and
restarting was impossible because
prepare-instancerefuses a directory thatalready exists.
Changes
Resume. A cell holding a
prediction.jsonis reused. A cell without one isan abandoned clone from an interrupted attempt and gets re-prepared. Rerunning
the same command retries only what is missing.
Fault tolerance. A cell that fails is counted and reported instead of ending
the run. That is the correct reading: the subset is the denominator either way,
so an arm with no patch scores zero for that task rather than invalidating the
other 49.
Sharding.
--shard N/Msplits the subset across M processes on disjointinstances. Wall clock falls by roughly M; provider concurrency rises by M, which
is the real ceiling on how far it can be pushed.
Shards write separate prediction files, because concurrent appends to one file
interleave, and each validates only its own so a sibling mid-write is not
reported as corrupt. A finished cell whose line is missing from the current
shard's file is re-linked from the cell itself, so a resume under a different
shard layout still yields complete prediction files.
Verification
test-code-bench.shgoes from 38 to 41 assertions: a cell with a prediction isreused rather than rerun, shards partition the subset exactly once each, and a
shard index outside its count is rejected.
The partition test is functional rather than arithmetic — it seeds all five
canary cells, runs both halves of a
0/2and1/2split, and asserts the twoshards between them claim all five instances with neither taking none nor all.
🤖 Generated with Claude Code