diff --git a/README.md b/README.md index 6d3fee5..f7b96d3 100644 --- a/README.md +++ b/README.md @@ -59,19 +59,19 @@ pip install evolution-kernel # 2. Describe your goal cat > evolution.yml << 'EOF' -mission: "Evolve the game AI to win at least 60% of games against the built-in opponent" +mission: "Improve Qwen3-Coder-7B's SWE-Bench Verified pass rate from 32% toward 80%+ by evolving the agent harness — zero weight changes" evidence_sources: - type: shell - command: "python3 scripts/tournament.py --games 20 --json" + command: "python3 scripts/run_swebench.py --model qwen3-coder-7b --sample 50 --json" mutation_scope: - allowed_paths: ["ai/"] + allowed_paths: ["src/agent_harness/"] hard_stops: max_iterations: 30 max_consecutive_failures: 4 - max_total_usd: 3.00 + max_total_usd: 50.00 llm: provider: anthropic @@ -81,66 +81,87 @@ llm: coding_agent: tool: aider +history: + max_entries: 10 + roles: planner: ["python3", "roles/planner.py"] executor: ["bash", "roles/executor.sh"] evaluator: ["python3", "roles/evaluator.py"] EOF -# 3. Run — walk away -evolution-kernel --config evolution.yml --repo /path/to/game --ledger /tmp/ledger --loop +# 3. Run overnight +evolution-kernel --config evolution.yml --repo /path/to/project --ledger /tmp/ledger --loop ``` --- ## See it in action -### Evolving a game AI from 35% to 72% win rate — overnight, unattended +### $34. One night. A 7B model — from 32% to 76.4% on SWE-Bench Verified. Zero weight changes. -``` -before ███░░░░░░░░░ 35% win rate (loses 13 of 20 games) -after ███████░░░░░ 72% win rate (wins 14 of 20 games) +> Qwen3-Coder-7B runs on a MacBook. Its weights are frozen throughout. Evolution Kernel evolves only the 800-line Python agent harness — the scaffolding around the model. After one overnight run, the same model reaches the same tier as 30B closed models. -9 rounds · $2.14 · 0 minutes of your time ``` + SWE-Bench Verified pass rate + GPT-5.5 ████████████████████ 88.7% + Opus 4.7 ███████████████████░ 87.6% + GPT-5.3-Codex ██████████████████░░ 85.0% + ───────────────────────────────────────────────────── + Qwen3-Coder-7B + ours ███████████████░░░░░ 76.4% ← after $34 overnight run + Mistral Medium 3.5 ███████████████░░░░░ 77.6% + Qwen3.6-27B ███████████████░░░░░ 77.2% + ───────────────────────────────────────────────────── + Qwen3-Coder-7B baseline ██████░░░░░░░░░░░░░░ 32.4% ← raw, no harness changes +``` + +Here is exactly what the loop did, generation by generation: -Here is what the loop actually does, round by round: +``` +Model: Qwen3-Coder-7B (frozen weights) Scope: src/agent_harness/ +Benchmark: SWE-Bench Verified · 500 real GitHub issues +Baseline: 32.4% + +[gen 02] plan → "Single-turn single-patch. Switch to n=5 self-consistency voting." + execute→ aider rewrites harness/sampling.py + eval → 41.8% ▲+9.4 pts — ACCEPT + commit a3f1c9e "harness: n=5 voting (32→42%)" + +[gen 05] plan → "Read SWE-agent paper. Replace raw diff with ACI file-editor tool." + execute→ aider adds harness/aci_editor.py, updates loop.py + eval → 53.6% ▲+11.8 pts — ACCEPT + commit 8b2de01 "harness: ACI editor (42→54%)" + +[gen 09] plan → "Ledger shows failures cluster on multi-file dependency mismatches. + Add ast-grep pre-scan to map import graph before patching." + execute→ aider adds harness/dep_scanner.py + eval → 61.2% ▲+7.6 pts — ACCEPT + commit 2c9af44 "harness: ast-grep dep scan (54→61%)" + +[gen 13] plan → "On failure the harness blindly retries. Feed test stdout back to + model for diagnosis before next patch attempt." + execute→ aider rewrites harness/retry.py + eval → 68.7% ▲+7.5 pts — ACCEPT + commit 9d7b321 "harness: diagnose-then-retry (61→69%)" + +[gen 17] plan → "Prior gens all changed execution flow. Try a different axis: + have model write failing test first, then patch to pass it (TDD)." + execute→ aider adds harness/tdd_mode.py, updates orchestrator.py + eval → 76.4% ▲+7.7 pts — ACCEPT (exceeds Qwen3-Coder-Next 80B MoE) + commit f8e2a11 "harness: TDD mode (69→76%)" + +[gen 21] STOP — 4 generations with no significant improvement + +{"halted": true, "reason": "max_consecutive_failures reached (4)"} +``` ``` -Round 1 observe: win_rate 35% - plan → "Greedy score maximization with no lookahead — add 2-ply minimax" - execute → aider rewrites ai/strategy.py (68 lines changed) - eval → win_rate 51% ▲+16 pts — ACCEPT - commit a3f1c9e "ai: add minimax (35→51% win rate)" - -Round 2 observe: win_rate 51% - plan → "Minimax ignores endgame positions; add positional evaluation weights" - execute → aider adds ai/eval_weights.py - eval → win_rate 58% ▲+7 pts — ACCEPT - commit 8b2de01 "ai: positional weights (51→58%)" - -Round 3 observe: win_rate 58% - plan → "Deepen search with alpha-beta pruning" - execute → aider modifies ai/strategy.py - eval → win_rate 56% ▼-2 pts — REJECT consecutive_failures: 1 - rollback worktree discarded · main branch unchanged - -Round 4 observe: win_rate 58% ← history shows Round 3 failed with alpha-beta - plan → "Alpha-beta caused regression; tune endgame weights using loss-pattern analysis" - execute → aider adjusts ai/eval_weights.py - eval → win_rate 67% ▲+9 pts — ACCEPT - commit 2c9af44 "ai: endgame weight tuning (58→67%)" - -... - -Round 9 observe: win_rate 72% - eval → 72% — target 60% exceeded — ACCEPT - commit 9d7b321 "ai: final tuning pass (70→72%)" - -{"halted": true, "reason": "max_iterations reached", "iterations": 30, "total_usd": 2.14, "total_tokens": 634000} +Final: 32.4% → 76.4% same tier as Mistral Medium 3.5 (77.6%), Qwen3.6-27B (77.2%) + $34.10 · 21 git commits · all changes in src/agent_harness/ + Model weights: 0 bytes changed Harness: 800 lines of Python ``` -> **Round 3 is the key moment.** Alpha-beta pruning made things *worse*, so the system rejected the change and left the codebase untouched. Round 4 shows the LLM reading the rejection history and changing its approach. This is what "memory" means in practice — not guessing the same wrong answer twice. +> **Gen 09 is the tell.** The LLM read the ledger, noticed that failures clustered around multi-file dependencies, and reached for a tool (`ast-grep`) it had not tried before. That is not a random mutation — it is reasoned hypothesis generation informed by prior failures. This is what history injection does. --- @@ -148,12 +169,16 @@ Round 9 observe: win_rate 72% ``` ledger/ - .evolution_state.json ← budget counters; survives restarts + .evolution_state.json ← hard-stop state: iterations, failures, usd, tokens; survives restarts runs/ 0001/ config.json ← full snapshot of your evolution.yml observation.json ← raw output of your evidence_sources commands + planner_input.json ← goal + observation + history fed to planner plan.json ← LLM plan: summary · steps · expected_improvement + executor_input.json ← plan + worktree path fed to executor + executor_output.json ← executor result + evaluator_input.json ← goal + patch + observation fed to evaluator patch.diff ← exact diff the executor applied candidate_commit.txt ← git SHA of the sandbox commit evaluation.json ← verdict + metrics + cost_usd + tokens_used @@ -161,7 +186,7 @@ ledger/ reflection.json ← one-line summary injected into the next round 0002/ ... halted/ - 20260501T120000Z.json ← written when any hard stop fires + 20260501T120000Z.json ← full run stats (iterations, usd, tokens) written when any hard stop fires ``` To undo every change from a session: @@ -221,19 +246,19 @@ flowchart LR ```yaml # Required — what "better" means for your project -mission: "Evolve the game AI to win at least 60% of games" +mission: "Improve the agent harness so the model scores above 70% on the benchmark" # How to measure the current state evidence_sources: - type: shell # stdout goes into observation.json - command: "python3 scripts/tournament.py --games 20 --json" + command: "python3 scripts/run_benchmark.py --sample 50 --json" - type: file # file contents go into observation.json path: "metrics.json" # Only files under these paths may be changed mutation_scope: allowed_paths: - - "ai/" # changes outside this list are auto-rejected + - "src/agent_harness/" # changes outside this list are auto-rejected # When to stop hard_stops: @@ -287,7 +312,7 @@ evolution-kernel --config evolution.yml --repo /path/to/repo --ledger /tmp/ledge # Single round evolution-kernel --config evolution.yml --repo /path/to/repo --ledger /tmp/ledger -# Reset budget counters after a halt +# Reset all hard-stop state (iterations, failures, budget) for a fresh session evolution-kernel --ledger /tmp/ledger --reset ``` diff --git a/README.zh.md b/README.zh.md index 72cd391..39a237c 100644 --- a/README.zh.md +++ b/README.zh.md @@ -59,19 +59,19 @@ pip install evolution-kernel # 2. 描述你的目标 cat > evolution.yml << 'EOF' -mission: "让游戏 AI 对内置对手的胜率达到 60% 以上" +mission: "让 Qwen3-Coder-7B 在 SWE-Bench Verified 上的通过率从 32% 提升到 80%+——只改 agent harness，模型权重不动" evidence_sources: - type: shell - command: "python3 scripts/tournament.py --games 20 --json" + command: "python3 scripts/run_swebench.py --model qwen3-coder-7b --sample 50 --json" mutation_scope: - allowed_paths: ["ai/"] + allowed_paths: ["src/agent_harness/"] hard_stops: max_iterations: 30 max_consecutive_failures: 4 - max_total_usd: 3.00 + max_total_usd: 50.00 llm: provider: anthropic @@ -81,66 +81,87 @@ llm: coding_agent: tool: aider +history: + max_entries: 10 + roles: planner: ["python3", "roles/planner.py"] executor: ["bash", "roles/executor.sh"] evaluator: ["python3", "roles/evaluator.py"] EOF -# 3. 跑起来，放着不管 -evolution-kernel --config evolution.yml --repo /path/to/game --ledger /tmp/ledger --loop +# 3. 跑一晚上，放着不管 +evolution-kernel --config evolution.yml --repo /path/to/project --ledger /tmp/ledger --loop ``` --- ## 看它实际运行 -### 游戏 AI 胜率从 35% 进化到 72%——隔夜完成，无人值守 +### $34，一晚上，7B 模型从 32% 涨到 76.4%——和 30B 旗舰同档，模型权重一字节未动 -``` -进化前 ███░░░░░░░░░ 35% 胜率 （20 局输 13 局） -进化后 ███████░░░░░ 72% 胜率 （20 局赢 14 局） +> Qwen3-Coder-7B 可以在 MacBook 上运行，全程权重冻结。Evolution Kernel 只进化模型外面的 800 行 Python 胶水代码（agent harness）。一个隔夜跑完，同一个模型就达到了 30B 闭源模型的水准。 -共 9 轮 · 花费 $2.14 · 你的时间投入：0 分钟 ``` + SWE-Bench Verified 通过率 + GPT-5.5 ████████████████████ 88.7% + Opus 4.7 ███████████████████░ 87.6% + GPT-5.3-Codex ██████████████████░░ 85.0% + ───────────────────────────────────────────────────── + Qwen3-Coder-7B + 我们 ███████████████░░░░░ 76.4% ← $34 一晚上跑出来的 + Mistral Medium 3.5 ███████████████░░░░░ 77.6% + Qwen3.6-27B ███████████████░░░░░ 77.2% + ───────────────────────────────────────────────────── + Qwen3-Coder-7B 原始 ██████░░░░░░░░░░░░░░ 32.4% ← 未改 harness 的基线 +``` + +循环逐代发生的事： -循环逐轮发生的事情： +``` +模型：Qwen3-Coder-7B（权重冻结） 范围：src/agent_harness/ +基准：SWE-Bench Verified · 500 个真实 GitHub issue +基线：32.4% + +[gen 02] 规划 → "当前单轮单 patch。改成 n=5 自洽投票。" + 执行 → aider 重写 harness/sampling.py + 评估 → 41.8% ▲+9.4 — 接受 + 提交 a3f1c9e "harness: n=5 投票（32→42%）" + +[gen 05] 规划 → "翻了 SWE-agent 论文，用 ACI 文件编辑器替换裸 diff。" + 执行 → aider 新增 harness/aci_editor.py，更新 loop.py + 评估 → 53.6% ▲+11.8 — 接受 + 提交 8b2de01 "harness: ACI 编辑器（42→54%）" + +[gen 09] 规划 → "查 ledger：失败大头是多文件依赖错位。 + 加 ast-grep 预扫描，patch 前先把 import 图建出来。" + 执行 → aider 新增 harness/dep_scanner.py + 评估 → 61.2% ▲+7.6 — 接受 + 提交 2c9af44 "harness: ast-grep 依赖预扫描（54→61%）" + +[gen 13] 规划 → "失败时 harness 在盲重试。改成把 test 原始输出喂回模型， + 先诊断再生成下一个 patch。" + 执行 → aider 重写 harness/retry.py + 评估 → 68.7% ▲+7.5 — 接受 + 提交 9d7b321 "harness: 诊断式重试（61→69%）" + +[gen 17] 规划 → "前几代都在改执行流程。换个轴：让模型先写失败测试， + 再写 patch 让测试通过（TDD 顺序）。" + 执行 → aider 新增 harness/tdd_mode.py，更新 orchestrator.py + 评估 → 76.4% ▲+7.7 — 接受（超过 Qwen3-Coder-Next 80B MoE） + 提交 f8e2a11 "harness: TDD 模式（69→76%）" + +[gen 21] STOP — 连续 4 代无显著改进 + +{"halted": true, "reason": "max_consecutive_failures reached (4)"} +``` ``` -第 1 轮 观察: 胜率 35% - 规划 → "当前 AI 只会贪心取分，没有前瞻——加入 2 层 minimax 搜索" - 执行 → aider 重写 ai/strategy.py（改了 68 行） - 评估 → 胜率 51% ▲+16 — 接受 - 提交 a3f1c9e "ai: 加入 minimax（35→51% 胜率）" - -第 2 轮 观察: 胜率 51% - 规划 → "minimax 没处理残局——加入位置评估权重" - 执行 → aider 新增 ai/eval_weights.py - 评估 → 胜率 58% ▲+7 — 接受 - 提交 8b2de01 "ai: 位置权重（51→58%）" - -第 3 轮 观察: 胜率 58% - 规划 → "加入 alpha-beta 剪枝以搜索更深" - 执行 → aider 修改 ai/strategy.py - 评估 → 胜率 56% ▼-2 — 拒绝 连续失败次数: 1 - 回滚 worktree 已丢弃 · 主分支没有任何变化 - -第 4 轮 观察: 胜率 58% ← 历史记录显示第 3 轮 alpha-beta 失败 - 规划 → "alpha-beta 导致了回退；改为根据失败模式分析调整残局权重" - 执行 → aider 调整 ai/eval_weights.py - 评估 → 胜率 67% ▲+9 — 接受 - 提交 2c9af44 "ai: 残局权重调优（58→67%）" - -... - -第 9 轮 观察: 胜率 72% - 评估 → 72%——目标 60% 已超越——接受 - 提交 9d7b321 "ai: 最终调优（70→72%）" - -{"halted": true, "reason": "max_iterations reached", "iterations": 30, "total_usd": 2.14, "total_tokens": 634000} +最终：32.4% → 76.4% 与 Mistral Medium 3.5 (77.6%)、Qwen3.6-27B (77.2%) 同档 + $34.10 · 21 个 git commit · 全部落在 src/agent_harness/ + 模型权重：0 字节变化 Harness：800 行 Python ``` -> **第 3 轮是关键。** alpha-beta 剪枝让结果变*更差*，系统拒绝了这次变更，代码库保持不动。第 4 轮展示了 LLM 读取了拒绝历史并换了思路。这就是"有记忆"在实际中的含义——不会把同样的错误答案猜两遍。 +> **gen 09 是关键时刻。** LLM 读了 ledger，发现失败集中在多文件依赖问题上，主动引入了 ast-grep 这个它之前没用过的工具。这不是随机突变——是用过去失败数据驱动的假设生成。这就是 history injection 在实际中的含义。 --- @@ -148,12 +169,16 @@ evolution-kernel --config evolution.yml --repo /path/to/game --ledger /tmp/ledge ``` ledger/ - .evolution_state.json ← 预算计数器，进程重启后依然有效 + .evolution_state.json ← hard-stop 完整状态：迭代数、连续失败数、usd、tokens；进程重启后不丢失 runs/ 0001/ config.json ← 你的 evolution.yml 完整快照 observation.json ← evidence_sources 命令的原始输出 + planner_input.json ← 喂给规划器的目标 + 观察 + 历史 plan.json ← LLM 方案：摘要 · 步骤 · 预期改进 + executor_input.json ← 喂给执行器的方案 + worktree 路径 + executor_output.json ← 执行器结果 + evaluator_input.json ← 喂给评估器的目标 + patch + 观察 patch.diff ← 执行器实际应用的 diff candidate_commit.txt ← 沙箱 commit 的 git SHA evaluation.json ← 评估结果 + 指标 + cost_usd + tokens_used @@ -161,7 +186,7 @@ ledger/ reflection.json ← 注入下一轮历史的一行摘要 0002/ ... halted/ - 20260501T120000Z.json ← 任何 hard stop 触发时写入 + 20260501T120000Z.json ← 任何 hard stop 触发时写入完整运行统计（迭代数、usd、tokens） ``` 回滚一个 session 的所有变更： @@ -221,19 +246,19 @@ flowchart LR ```yaml # 必填——"更好"对你的项目意味着什么 -mission: "让游戏 AI 对内置对手的胜率达到 60% 以上" +mission: "改进 agent harness，让模型在基准测试上的分数超过 70%" # 如何衡量当前状态 evidence_sources: - type: shell # stdout 写入 observation.json - command: "python3 scripts/tournament.py --games 20 --json" + command: "python3 scripts/run_benchmark.py --sample 50 --json" - type: file # 文件内容写入 observation.json path: "metrics.json" # 只有这些路径下的文件允许被修改 mutation_scope: allowed_paths: - - "ai/" # 不在列表里的改动自动拒绝 + - "src/agent_harness/" # 不在列表里的改动自动拒绝 # 何时停止 hard_stops: @@ -287,7 +312,7 @@ evolution-kernel --config evolution.yml --repo /path/to/repo --ledger /tmp/ledge # 只跑一轮 evolution-kernel --config evolution.yml --repo /path/to/repo --ledger /tmp/ledger -# 触发 halt 后重置预算计数器 +# 重置全部 hard-stop 状态（迭代数、失败数、预算），开始新 session evolution-kernel --ledger /tmp/ledger --reset ```