docs: SWE-Bench example + fix 5 Copilot review issues - #11
Merged
Conversation
$34, one overnight run, frozen 7B model goes from 32.4% to 76.4% on SWE-Bench Verified — matching Mistral Medium 3.5 and Qwen3.6-27B. Shows 5 generations of reasoned hypothesis: voting → ACI editor → ast-grep dep scan → diagnose-then-retry → TDD mode. Gen 09 is the tell: LLM reads the ledger, diagnoses multi-file dep failures, reaches for ast-grep unprompted. Chinese README updated to match. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Halted JSON: match actual CLI output format and reason string
("max_consecutive_failures reached (4)", not bare key + stats)
- .evolution_state.json: label as "hard-stop state" not "budget counters"
(also tracks iterations and consecutive_failures)
- Ledger file list: add planner_input.json, executor_input.json/output,
evaluator_input.json; clarify halted/ contains full run stats
- --reset CLI comment: "Reset all hard-stop state" not "budget counters"
- Config reference mission: align with SWE-Bench example, drop game-AI mission
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Updates the project documentation to replace the previous “game AI” walkthrough with a SWE-Bench–style agent-harness example, and to align several README details with the repository’s actual hard-stop/ledger behavior and artifacts.
Changes:
- Replaced the README “See it in action” narrative with a SWE-Bench Verified / agent-harness progression example (English + Chinese).
- Expanded ledger artifact listings (planner/executor/evaluator inputs/outputs) and refined hard-stop state/reset descriptions.
- Updated example config snippets to include
history.max_entriesand to align the “mission”/scope with agent-harness work.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 8 comments.
| File | Description |
|---|---|
| README.md | Updates quick start + example narrative to SWE-Bench/agent-harness framing; refines ledger/hard-stop documentation. |
| README.zh.md | Same documentation updates as README.md, localized for Chinese readers. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| commit f8e2a11 "harness: TDD mode (69→76%)" | ||
|
|
||
| [gen 21] STOP — 4 generations with no significant improvement | ||
|
|
| mission: "Improve Qwen3-Coder-7B's SWE-Bench Verified pass rate from 32% toward 80%+ by evolving the agent harness — zero weight changes" | ||
|
|
||
| evidence_sources: | ||
| - type: shell |
| ``` | ||
| ledger/ | ||
| .evolution_state.json ← budget counters; survives restarts | ||
| .evolution_state.json ← hard-stop state: iterations, failures, usd, tokens; survives restarts |
| 0002/ ... | ||
| halted/ | ||
| 20260501T120000Z.json ← written when any hard stop fires | ||
| 20260501T120000Z.json ← full run stats (iterations, usd, tokens) written when any hard stop fires |
|
|
||
| [gen 21] STOP — 连续 4 代无显著改进 | ||
|
|
||
| {"halted": true, "reason": "max_consecutive_failures reached (4)"} |
| evidence_sources: | ||
| - type: shell | ||
| command: "python3 scripts/tournament.py --games 20 --json" | ||
| command: "python3 scripts/run_swebench.py --model qwen3-coder-7b --sample 50 --json" |
| ``` | ||
| ledger/ | ||
| .evolution_state.json ← 预算计数器，进程重启后依然有效 | ||
| .evolution_state.json ← hard-stop 完整状态：迭代数、连续失败数、usd、tokens；进程重启后不丢失 |
| 0002/ ... | ||
| halted/ | ||
| 20260501T120000Z.json ← 任何 hard stop 触发时写入 | ||
| 20260501T120000Z.json ← 任何 hard stop 触发时写入完整运行统计（迭代数、usd、tokens） |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
SWE-Bench example (replaces game AI):
5 Copilot review fixes:
{"halted": true, "reason": "max_consecutive_failures reached (4)"}).evolution_state.json: relabeled as "hard-stop state" (not just "budget counters")planner_input.json,executor_input.json/output,evaluator_input.json--resetcomment: "Reset all hard-stop state" (iterations + failures + budget)Both
README.mdandREADME.zh.mdupdated.🤖 Generated with Claude Code