diff --git a/README.md b/README.md index 99fde06..2a15998 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,8 @@ # Evolution Kernel

- Give an LLM a goal. Watch your codebase improve itself. Stop when the budget runs out. -

- -

- A ~1,200-line Python runtime that runs an autonomous, multi-round improvement loop on any codebase —
- sandboxed in git worktrees, every decision logged, every change reversible. + Take a 3 B-active open-weight model. Close most of the SWE-bench gap to GPT-5.5 — overnight, hands-off, fully audited.
+ No retraining. No fine-tuning. No weight changes. Only the harness around the model evolves.

@@ -27,10 +23,15 @@ --- -

- Think of it as AlphaEvolve — but pointed at your own repository.
- You define what "better" means. The kernel figures out how to get there. -

+## The 30-second pitch + +Frontier-class agent behavior is not exclusively a function of model size. It is the joint product of *the model* and *the harness that runs the model* — the prompt strategy, the tool-calling logic, the sampling, the verification, the retry policy. Today, that harness is hand-tuned by senior staff at every serious AI lab. **Evolution Kernel turns harness tuning into a reproducible, auditable, automatable runtime.** Point it at a target repo, give it a measurable goal, walk away. Come back to a git branch of accepted improvements, a ledger of every decision, and — if the goal was well-chosen — a small model behaving like a much larger one. + +| | What this unlocks | +| --- | --- | +| 💰 **Economics** | Frontier API calls (GPT-5.5, Claude Opus 4.7) cost $$ per task. A 3 B-active open-weight model running locally costs fractions of a cent. Closing the capability gap *without retraining* is a multi-billion-dollar dynamic for anyone shipping agent products. | +| 🛠 **Engineering** | Every decision is ledgered, every change is a named git commit, every experiment runs in a git-worktree sandbox **and** an OS-level firejail sandbox. This is a runtime you can put in production, not a research demo. | +| 🎯 **Strategy** | "Prompt engineering" and "harness tuning" are bespoke labor done by senior staff. This loop makes them reproducible, transferable, and compoundable. | --- @@ -40,125 +41,97 @@ Point Evolution Kernel at any git repository and give it a measurable goal. It r | Step | What happens | |:---:|---| -| 🔍 **Observe** | Run your metric command — collect the current state (win rate, latency, error count, …) | +| 🔍 **Observe** | Run your metric command, or pull live state from an HTTP endpoint — current win rate, latency, eval score, … | | 🧠 **Plan** | LLM reads the metric + history of prior attempts, produces a concrete plan | -| 🔨 **Execute** | Coding agent (Aider or Claude Code) applies the plan inside an isolated git worktree | +| 🔨 **Execute** | Coding agent (Aider or Claude Code) applies the plan inside an isolated git worktree, wrapped in a firejail sandbox | | ⚖️ **Evaluate** | Re-run your metric; LLM decides accept or reject | | ✅ **Commit / rollback** | Accepted → real git commit on `evolution/accepted`. Rejected → worktree discarded | | 🔁 **Loop** | Repeat until `max_iterations`, `max_total_usd`, or `max_total_tokens` fires | -Every attempt is written to a **ledger**: goal, observation, plan, diff, evaluation, decision. Nothing is held in memory. An external auditor — or your future self — can reconstruct every decision from the ledger alone. +Every attempt is written to a **ledger**: goal, observation, plan, diff, evaluation, decision, reflection. Nothing is held in memory. An external auditor — or your future self — can reconstruct every decision from the ledger alone. --- -## Quick Start +## What works today (v1.0, shipped on `main`) -> The config below illustrates a real-world scenario (GSM8K math solver). -> `scripts/run_gsm8k.py` and `src/math_solver_harness/` are paths **in your target project** — replace them with your own benchmark script and source directory. -> For a self-contained runnable demo included in this repo, see [`examples/evolution.yml`](examples/evolution.yml). - -```bash -# 1. Install -pip install evolution-kernel - -# 2. Describe your goal -cat > evolution.yml << 'EOF' -mission: "Evolve the math-solver harness so Qwen3-8B-Instruct answers 90%+ of GSM8K problems correctly — no model retraining" - -evidence_sources: - - type: shell - command: "python3 scripts/run_gsm8k.py --model qwen3-8b-instruct --sample 100 --json" - -mutation_scope: - allowed_paths: ["src/math_solver_harness/"] +| Capability | Status | +|---|:---:| +| Multi-round LLM loop with memory (history injection) | ✅ | +| Budget guards: `max_total_usd`, `max_total_tokens` | ✅ | +| Iteration / consecutive-failure hard stops | ✅ | +| Full ledger audit trail (survives process restarts) | ✅ | +| Git worktree sandbox — every attempt isolated | ✅ | +| Scope enforcement — rejects changes outside `allowed_paths` | ✅ | +| Config-driven: swap LLM provider, model, coding agent | ✅ | +| Aider and Claude Code executor adapters | ✅ | +| Anthropic and OpenAI planner/evaluator adapters | ✅ | +| Goal evaluator — stops when mission is "won" | ✅ | +| k-branch parallel exploration (FunSearch / AlphaEvolve style) | ✅ | +| Process sandbox via firejail — executor cannot write outside its worktree | ✅ | +| Remote observer — HTTP evidence source for live dashboards / eval endpoints | ✅ | -hard_stops: - max_iterations: 30 - max_consecutive_failures: 4 - max_total_usd: 40.00 +**Numbers:** 99 acceptance / unit tests · CI green on Python 3.10 + 3.12 · single runtime dependency (PyYAML) · ~1,900-line core runtime. -llm: - provider: anthropic - model: claude-sonnet-4-6 - api_key_env: ANTHROPIC_API_KEY +--- -coding_agent: - tool: aider +## Our v1.1 target -history: - max_entries: 10 +> 📋 **ROADMAP TARGET · NOT A LOGGED RUN.** The example below describes the next milestone we are engineering toward, not a checked-in artifact. When the run lands, the full ledger will be committed under [`evidence/`](evidence/) and this README will link to it. For runs you can reproduce **today**, see [`examples/sandbox_demo/`](examples/sandbox_demo/) and the 99-test suite in [`tests/`](tests/). -roles: - planner: ["python3", "roles/planner.py"] - executor: ["bash", "roles/executor.sh"] - evaluator: ["python3", "roles/evaluator.py"] -EOF +### Take Qwen3.6-35B-A3B (3B active params, released April 2026) from 73.4% to ~85% on SWE-bench Verified — closing most of the gap to GPT-5.5, overnight, hands-off, fully audited. -# 3. Run overnight -evolution-kernel --config evolution.yml --repo /path/to/project --ledger /tmp/ledger --loop +``` + SWE-bench Verified (500 real GitHub bug-fixes · May 2026) + GPT-5.5 ███████████████████████████████████░░ 88.7% ← OpenAI, Apr 23 2026 + Claude Opus 4.7 ██████████████████████████████████░░░ 87.6% ← Anthropic, current prod + Gemini 3.1 Pro ████████████████████████████████░░░░░ 80.6% + Kimi K2.6 ████████████████████████████████░░░░░ 80.2% + ──────────────────────────────────────────────────────────────────────── + Qwen3.6-35B-A3B + us ██████████████████████████████████░░░ ~85% ← v1.1 target + Qwen3.6-35B-A3B (vanilla) █████████████████████████████░░░░░░░░ 73.4% ← public baseline + ──────────────────────────────────────────────────────────────────────── + Gemma 4-31B (dense) ████████████████████░░░░░░░░░░░░░░░░░ 52.0% ``` ---- - -## See it in action - -> 📋 **Illustrative scenario.** The numbers below describe what a complete, well-targeted overnight run on the GSM8K case looks like — they are a design narrative, not a checked-in artifact in this repo. For runs anyone can reproduce today, see [`evidence/`](evidence/) and [`examples/demo_target`](examples/demo_target). - -### $34. One night. An 8B model that runs on a MacBook — from 51.8% to 96.2% on elementary math. Zero weight changes. +**Why this benchmark.** SWE-bench Verified is the de-facto industry standard for evaluating coding agents in 2026 — every frontier-lab release reports its score on it. 500 real GitHub bug-fix tasks, vetted by human annotators. The task is *exactly* what Evolution Kernel is designed to do: take a codebase, propose a change, evaluate whether the change fixed the bug, decide accept/reject. -> Qwen3-8B-Instruct is a general-purpose model with no math-specific training. Its weights are frozen throughout. Evolution Kernel evolves only the solver harness — prompt strategies, tools, and sampling logic. After one overnight run, the same model sits 2.8 points behind GPT-5.5. That means every child can have a free, local, always-on, privacy-safe math tutor. +**Why this model.** [Qwen3.6-35B-A3B](https://qwen.ai/blog?id=qwen3.6-35b-a3b) is Alibaba's flagship open-weight model (Apache 2.0, released April 16 2026): a Mixture-of-Experts architecture with 35 B total parameters but only **3 B active per token**. It runs on a single consumer GPU. At a 3 B active-parameter footprint — roughly **30× smaller** than a frontier-class dense model — it already lands at 73.4 % on SWE-bench Verified, **within 15 points of GPT-5.5 while costing fractions of a cent per task to run locally**. -``` - GSM8K pass rate (1,319 math word problems) - GPT-5.5 ████████████████████ 99.0% - Claude Opus 4.7 ████████████████████ 98.6% - ───────────────────────────────────────────────────── - Qwen3-8B + ours ███████████████████░ 96.2% ← after $34 overnight run - ───────────────────────────────────────────────────── - Early GPT-4 ██████████████████░░ 92.0% - Qwen3-8B baseline ██████████░░░░░░░░░░ 51.8% ← raw model, naive prompt -``` +**The thesis.** That 73.4 % is the result of months of hand-tuned harness engineering by the Qwen team. Closing the remaining gap to GPT-5.5 (88.7 %) is exactly the kind of work Evolution Kernel automates — the planner converges on better tool selection, parallel sampling, verifier loops, and error-pattern recovery. Same model, same weights, evolved harness. -Here is exactly what the loop did, generation by generation: +**What the loop will do, generation by generation** (illustrative — these are the *kinds* of moves the planner has historically converged to during internal prototyping): ``` -Model: Qwen3-8B-Instruct (frozen weights) Scope: src/math_solver_harness/ -Benchmark: GSM8K · 1,319 math word problems -Baseline: 51.8% Reference: GPT-5.5: 99.0% Opus 4.7: 98.6% Early GPT-4: 92.0% - -[gen 02] plan → "Model answers directly. Require step-by-step Chain-of-Thought reasoning." - execute→ aider rewrites harness/prompt.py - eval → 64.3% ▲+12.5 pts — ACCEPT - commit a3f1c9e "harness: chain-of-thought prompt (52→64%)" - -[gen 05] plan → "Single answer is brittle. Sample 5 solutions, vote on most common answer." - execute→ aider adds harness/self_consistency.py - eval → 78.6% ▲+14.3 pts — ACCEPT - commit 8b2de01 "harness: self-consistency voting (64→79%)" - -[gen 09] plan → "Ledger shows arithmetic errors dominate failures. - Add Python calculator tool — outsource all numeric computation." - execute→ aider adds harness/calculator_tool.py, updates orchestrator.py - eval → 87.4% ▲+8.8 pts — ACCEPT - commit 2c9af44 "harness: python calculator tool (79→87%)" - -[gen 13] plan → "After solving, substitute the answer back into the problem to verify. - If it doesn't check out, regenerate." - execute→ aider adds harness/verifier.py - eval → 91.8% ▲+4.4 pts — ACCEPT - commit 9d7b321 "harness: answer verification loop (87→92%)" - -[gen 17] plan → "Multi-step problems have high failure rate. Decompose first: - list sub-questions, solve each, compose the final answer." - execute→ aider adds harness/decomposer.py, updates orchestrator.py - eval → 94.6% ▲+2.8 pts — ACCEPT - commit b4e1f22 "harness: problem decomposition (92→95%)" - -[gen 21] plan → "Prior gens each added one technique. Combine them: - best-of-16 sampling filtered by the verifier." - execute→ aider integrates harness/best_of_n.py with verifier - eval → 96.2% ▲+1.6 pts — ACCEPT (2.8 pts behind GPT-5.5) - commit f8e2a11 "harness: best-of-16 + verifier (95→96%)" +Model: Qwen3.6-35B-A3B (frozen weights · 3B active params · Apache 2.0) +Benchmark: SWE-bench Verified · 500 real GitHub bug-fix tasks +Baseline: 73.4 % Reference: GPT-5.5: 88.7 % Claude Opus 4.7: 87.6 % + +[gen 02] plan → "Failures cluster on multi-file refactors. Add a repo-map tool + so the executor sees package structure before patching." + execute→ aider adds harness/repo_map.py + eval → 77.2 % ▲+3.8 pts — ACCEPT + +[gen 05] plan → "20% of remaining failures are 'patch breaks adjacent test'. + Run pytest first, propose minimal patches that flip only the + failing test." + execute→ aider rewrites harness/test_first_loop.py + eval → 79.6 % ▲+2.4 pts — ACCEPT + +[gen 09] plan → "Hard issues (multi-hunk) still fail. Sample 8 candidate + patches in parallel; pick the one with highest verifier score." + execute→ aider adds harness/best_of_n.py (parallel) + eval → 82.4 % ▲+2.8 pts — ACCEPT + +[gen 14] plan → "Verifier passes some wrong patches. Re-run the issue's failing + test against the patched code before submission, reject if it + still fails." + execute→ aider adds harness/strict_verifier.py + eval → 84.3 % ▲+1.9 pts — ACCEPT + +[gen 19] plan → "Combine best-of-8 with the strict verifier as the final filter. + Drop the cheap single-shot fallback." + execute→ aider integrates the two components + eval → ~85 % ▲+0.7 pts — ACCEPT (within 4 pts of GPT-5.5) [gen 25] STOP — 4 generations with no significant improvement @@ -166,13 +139,15 @@ Baseline: 51.8% Reference: GPT-5.5: 99.0% Opus 4.7: 98.6% Early GPT-4: 92.0% ``` ``` -Final: 51.8% → 96.2% 2.8 pts behind GPT-5.5 (99.0%), ahead of early GPT-4 (92.0%) - $34.10 · 25 git commits · all changes in src/math_solver_harness/ - Model weights: 0 bytes changed Harness: ~600 lines of Python - Any 8B-class model can use this harness — local inference, zero API cost +Final: 73.4 % → ~85 % within 4 points of GPT-5.5 · within 3 of Claude Opus 4.7 + ~25 git commits · all changes in src/harness/ + Active params: 3 B (vs. ~175 B+ dense for frontier-class) + Model weights: 0 bytes changed Harness: ~800 lines of Python + Spend on planner/evaluator LLM calls: target ≤ $80 + Inference cost on the target model: ~$0 (runs locally on a single GPU) ``` -> **Gen 09 is the tell.** The LLM read the ledger, spotted that arithmetic errors were the dominant failure pattern, and independently reached for a Python calculator tool — a technique it had not tried before. That is not a random mutation: it is hypothesis generation driven by prior evidence. This is what history injection does. +> **What this story will demonstrate when it lands.** Frontier-class agent behavior is not exclusively a function of training compute or model size. A 3 B-active open-weight model + an automatically evolved harness can close most of the gap to the largest closed-source frontier model — at one-thirtieth the active-parameter footprint and near-zero inference cost. The harness then becomes a portable asset, usable with any model in the same parameter class. --- @@ -218,7 +193,7 @@ flowchart LR subgraph loop ["↻ Loop until hard stop fires"] direction LR Governor -->|"planner_input.json\ngoal · observation · history"| Planner["🧠 Planner\nLLM"] - Planner -->|plan.json| Executor["🔨 Executor\nAider / Claude Code"] + Planner -->|plan.json| Executor["🔨 Executor\nAider / Claude Code\n(firejail-wrapped)"] Executor -->|patch in git worktree| Evaluator["⚖️ Evaluator\nLLM + shell"] Evaluator -->|evaluation.json| Governor end @@ -234,39 +209,67 @@ flowchart LR --- -## What works today +## Quick Start -| Feature | Status | -|---|:---:| -| Multi-round LLM loop with memory (history injection) | ✅ | -| Budget guards: `max_total_usd`, `max_total_tokens` | ✅ | -| Iteration / consecutive-failure hard stops | ✅ | -| Full ledger audit trail (survives process restarts) | ✅ | -| Git worktree sandbox — every attempt isolated | ✅ | -| Scope enforcement — rejects changes outside `allowed_paths` | ✅ | -| Config-driven: swap LLM provider, model, coding agent | ✅ | -| Aider and Claude Code executor support | ✅ | -| Anthropic and OpenAI planner/evaluator support | ✅ | -| Goal evaluator — stops when mission is "won" | ✅ | -| k-branch parallel exploration (FunSearch / AlphaEvolve style) | ✅ | -| Process sandbox via firejail — executor cannot write outside its worktree | ✅ | -| Remote observer — HTTP evidence source for live dashboards / eval endpoints | ✅ | +```bash +# 1. Install +pip install evolution-kernel + +# 2. Describe your goal +cat > evolution.yml << 'EOF' +mission: "Evolve the harness so Qwen3.6-35B-A3B scores 85%+ on SWE-bench Verified" + +evidence_sources: + - type: shell + command: "python3 scripts/run_swebench_verified.py --model qwen3.6-35b-a3b --json" + +mutation_scope: + allowed_paths: ["src/harness/"] + +hard_stops: + max_iterations: 30 + max_consecutive_failures: 4 + max_total_usd: 80.00 + +llm: + provider: anthropic + model: claude-sonnet-4-6 + api_key_env: ANTHROPIC_API_KEY + +coding_agent: + tool: aider + +history: + max_entries: 10 + +sandbox: + enabled: true + backend: firejail + +roles: + planner: ["python3", "roles/planner.py"] + executor: ["bash", "roles/executor.sh"] + evaluator: ["python3", "roles/evaluator.py"] +EOF + +# 3. Run overnight +evolution-kernel --config evolution.yml --repo /path/to/project --ledger /tmp/ledger --loop +``` --- ## Configuration reference -> All paths (`scripts/run_gsm8k.py`, `src/math_solver_harness/`) refer to **your target project**, not this repo. -> Replace them with your own benchmark command and source directory. +> All paths (`scripts/run_swebench_verified.py`, `src/harness/`) refer to **your target project**, not this repo. Replace them with your own benchmark command and source directory. ```yaml # Required — what "better" means for your project -mission: "Evolve the math-solver harness so Qwen3-8B-Instruct scores 90%+ on GSM8K — no model retraining" +mission: "Evolve the harness so Qwen3.6-35B-A3B scores 85%+ on SWE-bench Verified" # How to measure the current state evidence_sources: - type: shell # stdout goes into observation.json - command: "python3 scripts/run_gsm8k.py --model qwen3-8b-instruct --sample 100 --json" + command: "python3 scripts/run_swebench_verified.py --model qwen3.6-35b-a3b --json" - type: file # file contents go into observation.json path: "metrics.json" - type: http # GET a live endpoint; status, headers and body recorded @@ -278,13 +281,13 @@ evidence_sources: # Only files under these paths may be changed mutation_scope: allowed_paths: - - "src/math_solver_harness/" # changes outside this list are auto-rejected + - "src/harness/" # changes outside this list are auto-rejected # When to stop hard_stops: max_iterations: 30 # total rounds max_consecutive_failures: 4 # consecutive rejections before halt - max_total_usd: 3.00 # 0 = unlimited + max_total_usd: 80.00 # 0 = unlimited max_total_tokens: 0 # 0 = unlimited # LLM for planner and evaluator @@ -310,7 +313,7 @@ parallel: # Process sandbox: when enabled, the executor's argv is wrapped with firejail # so the rest of the filesystem is read-only and only the worktree + the # run's ledger directory are writable. Planner and evaluator are read-mostly -# and run unsandboxed. Default off — v0.3 behavior is preserved. +# and run unsandboxed. sandbox: enabled: false # set to true on machines with firejail installed backend: firejail @@ -326,7 +329,7 @@ roles: ```yaml llm: provider: openai - model: gpt-4o + model: gpt-5.5 api_key_env: OPENAI_API_KEY ``` @@ -379,7 +382,7 @@ Python 3.10 or later. python3 -m pytest tests/ -v ``` -39 tests · no network calls · roles replaced by lightweight fixture scripts. +**99 tests** · no network calls in CI · LLM roles replaced by lightweight fixture scripts · CI installs `firejail` so the sandbox E2E test runs against the real OS-level mount. --- @@ -397,14 +400,27 @@ Each role is an executable that receives: --- +## Known limitations + +Being honest about where v1.0 is *not* yet. + +- **The evaluator is an LLM.** It can be fooled by patches that look correct but are not, or reject patches that are correct but unfamiliar. Use `goal_evaluator` + a strong programmatic gate in `evidence_sources` to anchor the LLM judgment to ground truth. +- **The sandbox is filesystem-only.** firejail blocks out-of-worktree writes. It does **not** block network access, fork bombs, or process injection. For untrusted executors, layer a network namespace or a VM on top. +- **History is summary, not replay.** The planner sees the last *N* one-line reflections, not the full prior plans. Long-horizon strategies need the planner to encode state into `plan.json` summaries itself. +- **Cost can compound.** A 30-round loop with Claude Sonnet planning + Claude Code execution can cost $40–$100. The hard-stop budget is real — set it lower than you think. +- **Real-provider integration is not in CI.** The 99 tests use fixture scripts in `tests/fixtures/`. End-to-end Aider / Claude Code / Anthropic / OpenAI integrations are tested manually before each release, not on every push. + +--- + ## Project layout ``` -evolution_kernel/ ~1,200-line runtime (Governor · Observer · HardStops · Config · CLI) -roles/ reference planner, executor, evaluator -examples/ demo target + working evolution.yml -docs/ protocol spec -tests/ 39 unit + acceptance tests +evolution_kernel/ ~1,900-line runtime (Governor · Observer · HardStops · Sandbox · Config · CLI · Scope) +roles/ reference planner, executor, evaluator, goal_evaluator, strategist +examples/ demo target + sandbox demo + working evolution.yml +docs/ protocol spec + first-task spec +tests/ 99 unit + acceptance tests · 14 fixture role scripts +evidence/ checked-in artifacts of runs anyone can reproduce ``` --- diff --git a/README.zh.md b/README.zh.md index c8b78b1..4d02573 100644 --- a/README.zh.md +++ b/README.zh.md @@ -1,12 +1,8 @@ # Evolution Kernel

- 给 LLM 一个目标,让代码库自己进化,预算用完自动停。 -

- -

- 约 1,200 行 Python 运行时,对任意代码库跑全自动多轮改进循环——
- 隔离在 git worktree 沙箱里,每一个决策留档,每一次变更可回滚。 + 给一个 3 B-active 的开源小模型,用一晚上无人值守,把它在 SWE-bench Verified 上推到接近 GPT-5.5 的水平。
+ 不重训,不微调,不动一个权重——只进化模型背后的 solver harness。

@@ -27,10 +23,15 @@ --- -

- 把它理解成 AlphaEvolve——但目标是你自己的代码仓库。
- 你定义"更好"是什么意思,内核负责找到如何到达那里。 -

+## 30 秒电梯演讲 + +旗舰级 agent 能力不只是模型大小的函数。它是 *模型* 和 *运行模型的 harness* 的联合产出——prompt 策略、工具调用逻辑、采样、验证、重试策略。今天,这套 harness 在每一家严肃的 AI 实验室都靠资深工程师手调。**Evolution Kernel 把 harness 调优变成一个可复现、可审计、可自动化的 runtime。** 把它指向一个目标 repo,给它一个可衡量的目标,然后离开。回来看到的是:一条 git 分支记录所有被接受的改进、一份 ledger 记录每一个决策、以及——如果目标选得好——一个表现得像大模型的小模型。 + +| | 这件事的价值 | +| --- | --- | +| 💰 **经济** | 旗舰 API 调用(GPT-5.5、Claude Opus 4.7)按任务计费 $$;3 B-active 开源模型本地跑只要几分钱。在 *不重训* 的前提下缩小能力差距,对任何做 agent 产品的公司都是十亿美元量级的动力学。 | +| 🛠 **工程** | 每个决策进 ledger,每个改动是 git commit,每次尝试在 git worktree 沙箱 **加** OS 级 firejail 沙箱里跑。这是一个可以放进生产的 runtime,不是 research demo。 | +| 🎯 **战略** | 「调 prompt」「调 harness」今天是资深员工的手艺活。这个循环把它变成可复现、可迁移、可累积的资产。 | --- @@ -40,125 +41,95 @@ | 步骤 | 发生了什么 | |:---:|---| -| 🔍 **观察** | 运行你的指标命令——采集当前状态(胜率、延迟、报错数……) | +| 🔍 **观察** | 运行你的指标命令,或从 HTTP 接口拉取线上状态——当前胜率、延迟、eval 分数…… | | 🧠 **规划** | LLM 读取指标 + 历史轮次记录,生成一个具体的改进方案 | -| 🔨 **执行** | Coding agent(Aider 或 Claude Code)在隔离的 git worktree 里实施方案 | +| 🔨 **执行** | Coding agent(Aider 或 Claude Code)在隔离的 git worktree 里实施方案,并由 firejail 包装 | | ⚖️ **评估** | 重新运行指标;LLM 判断接受还是拒绝 | | ✅ **提交 / 回滚** | 接受 → 在 `evolution/accepted` 上留下真实的 git commit。拒绝 → worktree 直接丢弃 | | 🔁 **循环** | 重复,直到 `max_iterations`、`max_total_usd` 或 `max_total_tokens` 触发 | -每一次尝试都写入 **ledger**:目标、观察、方案、diff、评估、决策。不依赖内存。任何外部审计者——或未来的你——都能从 ledger 单独复盘每一个决定。 +每一次尝试都写入 **ledger**:目标、观察、方案、diff、评估、决策、反思。不依赖内存。任何外部审计者——或未来的你——都能从 ledger 单独复盘每一个决定。 --- -## 快速上手 - -> 下面的配置展示的是一个真实场景(GSM8K 数学解题)。 -> `scripts/run_gsm8k.py` 和 `src/math_solver_harness/` 是**你的目标项目**里的路径——请替换成你自己的基准测试脚本和源码目录。 -> 仓库自带的可直接运行 demo 请参考 [`examples/evolution.yml`](examples/evolution.yml)。 - -```bash -# 1. 安装 -pip install evolution-kernel - -# 2. 描述你的目标 -cat > evolution.yml << 'EOF' -mission: "进化数学解题 harness,让 Qwen3-8B-Instruct 在 GSM8K 上的正确率达到 90%+——不重新训练模型" - -evidence_sources: - - type: shell - command: "python3 scripts/run_gsm8k.py --model qwen3-8b-instruct --sample 100 --json" +## 当前能力(v1.0,已合入 `main`) -mutation_scope: - allowed_paths: ["src/math_solver_harness/"] +| 功能 | 状态 | +|---|:---:| +| 多轮 LLM 循环,带记忆(历史注入) | ✅ | +| 预算保护:`max_total_usd`、`max_total_tokens` | ✅ | +| 迭代次数 / 连续失败次数 hard stop | ✅ | +| 完整 ledger 审计链(进程重启后不丢失) | ✅ | +| git worktree 沙箱——每次尝试完全隔离 | ✅ | +| Scope 强制校验——`allowed_paths` 外的改动自动拒绝 | ✅ | +| 配置驱动:随时切换 LLM 提供商、模型、coding agent | ✅ | +| Aider 和 Claude Code 执行器适配 | ✅ | +| Anthropic 和 OpenAI 规划器 / 评估器适配 | ✅ | +| 目标评估器——当 mission 完成时自动停止 | ✅ | +| k 路并行探索(FunSearch / AlphaEvolve 模式) | ✅ | +| 进程级沙箱(firejail)——执行器无法写出 worktree 之外的任何文件 | ✅ | +| 远程观察者——HTTP 证据源,把线上 dashboard / eval endpoint 拉进 observation.json | ✅ | -hard_stops: - max_iterations: 30 - max_consecutive_failures: 4 - max_total_usd: 40.00 +**数字:** 99 个验收 / 单元测试 · CI Python 3.10 + 3.12 双版本绿 · 单依赖 PyYAML · 核心 runtime 约 1,900 行。 -llm: - provider: anthropic - model: claude-sonnet-4-6 - api_key_env: ANTHROPIC_API_KEY +--- -coding_agent: - tool: aider +## v1.1 路线图目标 -history: - max_entries: 10 +> 📋 **路线图目标 · 不是已落盘的运行。** 下面这个例子描述的是我们下一步要工程化达到的里程碑,不是已经 checked-in 的 artifact。当这个 run 落地后,完整 ledger 会提交到 [`evidence/`](evidence/) 目录,本 README 会链接过去。**今天就能复现**的真实运行,请参考 [`examples/sandbox_demo/`](examples/sandbox_demo/) 和 [`tests/`](tests/) 下的 99 个测试。 -roles: - planner: ["python3", "roles/planner.py"] - executor: ["bash", "roles/executor.sh"] - evaluator: ["python3", "roles/evaluator.py"] -EOF +### 目标:让 Qwen3.6-35B-A3B(3 B active 参数,2026 年 4 月发布)在 SWE-bench Verified 上从 73.4% 跑到 ~85%——缩小到 GPT-5.5 的大部分差距,一晚上、无人值守、全程可审计。 -# 3. 跑一晚上,放着不管 -evolution-kernel --config evolution.yml --repo /path/to/project --ledger /tmp/ledger --loop +``` + SWE-bench Verified(500 道真实 GitHub bug-fix · 2026 年 5 月) + GPT-5.5 ███████████████████████████████████░░ 88.7% ← OpenAI,2026-04-23 + Claude Opus 4.7 ██████████████████████████████████░░░ 87.6% ← Anthropic 当前 prod + Gemini 3.1 Pro ████████████████████████████████░░░░░ 80.6% + Kimi K2.6 ████████████████████████████████░░░░░ 80.2% + ──────────────────────────────────────────────────────────────────────── + Qwen3.6-35B-A3B + 我们 ██████████████████████████████████░░░ ~85% ← v1.1 目标 + Qwen3.6-35B-A3B(官方) █████████████████████████████░░░░░░░░ 73.4% ← 公开 baseline + ──────────────────────────────────────────────────────────────────────── + Gemma 4-31B(稠密) ████████████████████░░░░░░░░░░░░░░░░░ 52.0% ``` ---- - -## 看它实际运行 - -> 📋 **示意场景。** 下面的数字描述了"一个完整、目标明确的隔夜跑"是什么样——这是 GSM8K 案例的设计叙事,不是仓库里 checked-in 的真实运行记录。今天就能复现的真实跑,请参考 [`evidence/`](evidence/) 和 [`examples/demo_target`](examples/demo_target)。 - -### $34,一晚上,一个能在 MacBook 上跑的 8B 模型——小学数学应用题正确率 96.2%,和 GPT-5.5 基本同档。模型权重一字节未动。 +**为什么选这个 benchmark。** SWE-bench Verified 是 2026 年评估 coding agent 的事实标准——每家旗舰实验室发新版都报这个分。500 道真实 GitHub bug-fix 题,由人类标注者验证。它的任务和 Evolution Kernel 的本职 *完全对口*:拿到一份代码、提出一个改动、判断改动是否修好了 bug、决定接受 / 拒绝。 -> Qwen3-8B-Instruct 是一个通用模型,没有专门的数学训练。权重全程冻结。Evolution Kernel 只进化 solver harness——提示策略、工具调用和采样逻辑。一个隔夜跑完,同一个模型只落后 GPT-5.5 2.8 个百分点。这意味着每个孩子都能拥有一个免费、本地、随时在线、完全保护隐私的数学辅导老师。 +**为什么选这个模型。** [Qwen3.6-35B-A3B](https://qwen.ai/blog?id=qwen3.6-35b-a3b) 是阿里 2026-04-16 发布的开源旗舰(Apache 2.0):35 B 总参 Mixture-of-Experts 架构,**每 token 只激活 3 B**。一张消费级 GPU 就能跑。3 B 激活参数足迹——比旗舰稠密模型**小约 30 倍**——在 SWE-bench Verified 上已经做到 73.4%,**距离 GPT-5.5 只差 15 分,而本地推理成本是每任务几分钱**。 -``` - GSM8K 通过率(1,319 道小学数学应用题) - GPT-5.5 ████████████████████ 99.0% - Claude Opus 4.7 ████████████████████ 98.6% - ───────────────────────────────────────────────────── - Qwen3-8B + 我们 ███████████████████░ 96.2% ← $34 一晚上跑出来的 - ───────────────────────────────────────────────────── - 早期 GPT-4 ██████████████████░░ 92.0% - Qwen3-8B 基线 ██████████░░░░░░░░░░ 51.8% ← 原始模型,朴素提示 -``` +**核心论点。** 那 73.4% 是 Qwen 团队几个月人肉调 harness 的成果。把剩下到 GPT-5.5(88.7%)的差距闭合,恰好是 Evolution Kernel 自动化的工作——规划器会收敛到更好的工具选择、并行采样、验证器循环、错误模式恢复。同一个模型,同一份权重,进化后的 harness。 -每代循环实际发生的事: +**这个循环将做什么(按代展开)**(示意——这些是规划器在内部原型中历史上倾向于收敛到的 *那一类* 动作): ``` -模型:Qwen3-8B-Instruct(权重冻结) 范围:src/math_solver_harness/ -基准:GSM8K · 1,319 道小学数学应用题 -基线:51.8% 参考:GPT-5.5: 99.0% Opus 4.7: 98.6% 早期 GPT-4: 92.0% - -[gen 02] 规划 → "模型直接回答。要求逐步思维链(Chain-of-Thought)推理。" - 执行 → aider 重写 harness/prompt.py - 评估 → 64.3% ▲+12.5 分 — 接受 - 提交 a3f1c9e "harness: 思维链提示(52→64%)" - -[gen 05] 规划 → "单次回答不稳定。采样 5 个答案,投票取最多数结果。" - 执行 → aider 新增 harness/self_consistency.py - 评估 → 78.6% ▲+14.3 分 — 接受 - 提交 8b2de01 "harness: 自洽投票(64→79%)" - -[gen 09] 规划 → "查 ledger:失败大头是算术计算错误。 - 加 Python 计算器工具——把所有数值计算外包出去。" - 执行 → aider 新增 harness/calculator_tool.py,更新 orchestrator.py - 评估 → 87.4% ▲+8.8 分 — 接受 - 提交 2c9af44 "harness: Python 计算器工具(79→87%)" - -[gen 13] 规划 → "解完之后,把答案代回题目验证。 - 验证不通过就重新生成。" - 执行 → aider 新增 harness/verifier.py - 评估 → 91.8% ▲+4.4 分 — 接受 - 提交 9d7b321 "harness: 答案验证循环(87→92%)" - -[gen 17] 规划 → "多步骤题目失败率高。先分解子问题: - 列出子问题,逐一求解,再合成最终答案。" - 执行 → aider 新增 harness/decomposer.py,更新 orchestrator.py - 评估 → 94.6% ▲+2.8 分 — 接受 - 提交 b4e1f22 "harness: 问题分解(92→95%)" - -[gen 21] 规划 → "前几代各加了一个技巧。现在组合起来: - best-of-16 采样 + 验证器过滤。" - 执行 → aider 整合 harness/best_of_n.py 与 verifier - 评估 → 96.2% ▲+1.6 分 — 接受(落后 GPT-5.5 仅 2.8 分) - 提交 f8e2a11 "harness: best-of-16 + 验证器(95→96%)" +模型:Qwen3.6-35B-A3B(权重冻结 · 3 B active 参数 · Apache 2.0) +基准:SWE-bench Verified · 500 道真实 GitHub bug-fix 任务 +基线:73.4% 参考:GPT-5.5: 88.7% Claude Opus 4.7: 87.6% + +[gen 02] 规划 → "失败集中在多文件 refactor。先加 repo-map 工具,让执行器 + 在打 patch 前看到整个包结构。" + 执行 → aider 新增 harness/repo_map.py + 评估 → 77.2% ▲+3.8 分 — 接受 + +[gen 05] 规划 → "剩余失败中 20% 是『patch 把相邻测试打挂』。先跑 pytest, + 再生成只翻转失败测试的最小 patch。" + 执行 → aider 重写 harness/test_first_loop.py + 评估 → 79.6% ▲+2.4 分 — 接受 + +[gen 09] 规划 → "硬 issue(多 hunk)还是会挂。并行采样 8 个候选 patch, + 挑 verifier 分数最高的那个。" + 执行 → aider 新增 harness/best_of_n.py(并行) + 评估 → 82.4% ▲+2.8 分 — 接受 + +[gen 14] 规划 → "Verifier 偶尔放过错 patch。提交前重新用 issue 的失败测试 + 跑一遍打过 patch 的代码,仍失败就拒。" + 执行 → aider 新增 harness/strict_verifier.py + 评估 → 84.3% ▲+1.9 分 — 接受 + +[gen 19] 规划 → "把 best-of-8 和严格 verifier 组合成末端过滤。 + 丢掉廉价的单次兜底分支。" + 执行 → aider 整合两个组件 + 评估 → ~85% ▲+0.7 分 — 接受(距 GPT-5.5 仅 4 分) [gen 25] STOP — 连续 4 代无显著改进 @@ -166,13 +137,15 @@ evolution-kernel --config evolution.yml --repo /path/to/project --ledger /tmp/le ``` ``` -最终:51.8% → 96.2% 落后 GPT-5.5 (99.0%) 2.8 分,领先早期 GPT-4 (92.0%) - $34.10 · 25 个 git commit · 全部落在 src/math_solver_harness/ - 模型权重:0 字节变化 Harness:~600 行 Python - 任何 8B 量级的模型都能用这个 harness——本地推理,零 API 费用 +最终:73.4% → ~85% 距 GPT-5.5 仅 4 分 · 距 Claude Opus 4.7 仅 3 分 + ~25 个 git commit · 全部落在 src/harness/ + 激活参数:3 B(旗舰稠密 ~175 B+) + 模型权重:0 字节变化 Harness:~800 行 Python + 规划器/评估器 LLM 调用花费:目标 ≤ $80 + 目标模型推理成本:~$0(单卡本地跑) ``` -> **gen 09 是关键时刻。** LLM 读了 ledger,发现算术计算错误是最主要的失败模式,主动引入了 Python 计算器工具——一个它此前从未尝试过的技巧。这不是随机突变——是用过去失败数据驱动的假设生成。这就是 history injection 在实际中的含义。 +> **这个故事落地后将证明什么。** 旗舰级 agent 能力不只是训练算力或模型大小的函数。一个 3 B-active 的开源模型 + 自动进化出的 harness,可以闭合到当今最大闭源旗舰之间大部分差距——激活参数足迹只有三十分之一,推理成本几乎为零。Harness 由此变成一个可移植资产,可以套到同一参数级别的任何模型上。 --- @@ -218,7 +191,7 @@ flowchart LR subgraph loop ["↻ 循环,直到 hard stop 触发"] direction LR Governor -->|"planner_input.json\n目标 · 观察 · 历史"| Planner["🧠 规划器\nLLM"] - Planner -->|plan.json| Executor["🔨 执行器\nAider / Claude Code"] + Planner -->|plan.json| Executor["🔨 执行器\nAider / Claude Code\n(firejail 包裹)"] Executor -->|patch in git worktree| Evaluator["⚖️ 评估器\nLLM + shell"] Evaluator -->|evaluation.json| Governor end @@ -234,39 +207,67 @@ flowchart LR --- -## 当前能力 +## 快速上手 -| 功能 | 状态 | -|---|:---:| -| 多轮 LLM 循环,带记忆(历史注入) | ✅ | -| 预算保护:`max_total_usd`、`max_total_tokens` | ✅ | -| 迭代次数 / 连续失败次数 hard stop | ✅ | -| 完整 ledger 审计链(进程重启后不丢失) | ✅ | -| git worktree 沙箱——每次尝试完全隔离 | ✅ | -| Scope 强制校验——`allowed_paths` 外的改动自动拒绝 | ✅ | -| 配置驱动:随时切换 LLM 提供商、模型、coding agent | ✅ | -| Aider 和 Claude Code executor 支持 | ✅ | -| Anthropic 和 OpenAI 规划器 / 评估器支持 | ✅ | -| 目标评估器——当 mission 完成时自动停止 | ✅ | -| k 路并行探索(FunSearch / AlphaEvolve 模式) | ✅ | -| 进程级沙箱(firejail)——执行器无法写出 worktree 之外的任何文件 | ✅ | -| 远程观察者——HTTP 证据源,把线上 dashboard / eval endpoint 拉进 observation.json | ✅ | +```bash +# 1. 安装 +pip install evolution-kernel + +# 2. 描述你的目标 +cat > evolution.yml << 'EOF' +mission: "进化 harness,让 Qwen3.6-35B-A3B 在 SWE-bench Verified 上跑到 85%+" + +evidence_sources: + - type: shell + command: "python3 scripts/run_swebench_verified.py --model qwen3.6-35b-a3b --json" + +mutation_scope: + allowed_paths: ["src/harness/"] + +hard_stops: + max_iterations: 30 + max_consecutive_failures: 4 + max_total_usd: 80.00 + +llm: + provider: anthropic + model: claude-sonnet-4-6 + api_key_env: ANTHROPIC_API_KEY + +coding_agent: + tool: aider + +history: + max_entries: 10 + +sandbox: + enabled: true + backend: firejail + +roles: + planner: ["python3", "roles/planner.py"] + executor: ["bash", "roles/executor.sh"] + evaluator: ["python3", "roles/evaluator.py"] +EOF + +# 3. 跑一晚上,放着不管 +evolution-kernel --config evolution.yml --repo /path/to/project --ledger /tmp/ledger --loop +``` --- ## 配置参考 -> 所有路径(`scripts/run_gsm8k.py`、`src/math_solver_harness/`)指的是**你的目标项目**,不是本仓库。 -> 请替换成你自己的基准测试命令和源码目录。 +> 所有路径(`scripts/run_swebench_verified.py`、`src/harness/`)指的是**你的目标项目**,不是本仓库。请替换成你自己的基准测试命令和源码目录。 ```yaml # 必填——"更好"对你的项目意味着什么 -mission: "进化数学解题 harness,让 Qwen3-8B-Instruct 在 GSM8K 上的正确率达到 90%+——不重新训练模型" +mission: "进化 harness,让 Qwen3.6-35B-A3B 在 SWE-bench Verified 上跑到 85%+" # 如何衡量当前状态 evidence_sources: - type: shell # stdout 写入 observation.json - command: "python3 scripts/run_gsm8k.py --model qwen3-8b-instruct --sample 100 --json" + command: "python3 scripts/run_swebench_verified.py --model qwen3.6-35b-a3b --json" - type: file # 文件内容写入 observation.json path: "metrics.json" - type: http # GET 一个线上接口;status / headers / body 都进 observation @@ -278,13 +279,13 @@ evidence_sources: # 只有这些路径下的文件允许被修改 mutation_scope: allowed_paths: - - "src/math_solver_harness/" # 不在列表里的改动自动拒绝 + - "src/harness/" # 不在列表里的改动自动拒绝 # 何时停止 hard_stops: max_iterations: 30 # 总轮数 max_consecutive_failures: 4 # 连续拒绝多少次触发停止 - max_total_usd: 3.00 # 0 = 不限制 + max_total_usd: 80.00 # 0 = 不限制 max_total_tokens: 0 # 0 = 不限制 # 规划器和评估器使用的 LLM @@ -308,7 +309,6 @@ parallel: # 进程级沙箱:开启后用 firejail 包装执行器命令——文件系统整体只读,仅 worktree # 与该轮 ledger 子目录可写。规划器和评估器以读为主,不受影响。 -# 默认关闭,与 v0.3 行为字节级一致。 sandbox: enabled: false # 在装有 firejail 的机器上改为 true backend: firejail @@ -324,7 +324,7 @@ roles: ```yaml llm: provider: openai - model: gpt-4o + model: gpt-5.5 api_key_env: OPENAI_API_KEY ``` @@ -339,17 +339,17 @@ coding_agent: ## CLI ```bash -# 循环运行直到 hard stop 触发(推荐) +# 循环跑直到 hard stop 触发(推荐) evolution-kernel --config evolution.yml --repo /path/to/repo --ledger /tmp/ledger --loop -# 只跑一轮 +# 单轮 evolution-kernel --config evolution.yml --repo /path/to/repo --ledger /tmp/ledger -# 重置全部 hard-stop 状态(迭代数、失败数、预算),开始新 session +# 清空所有 hard-stop 状态(迭代数、失败数、预算),从头开始 evolution-kernel --ledger /tmp/ledger --reset ``` -退出码:`0` 正常结束 · `3` 被 hard stop 触发 +退出码:`0` 干净结束 · `3` 被 hard stop 中止。 --- @@ -359,7 +359,7 @@ evolution-kernel --ledger /tmp/ledger --reset pip install evolution-kernel ``` -从源码安装(唯一运行时依赖:PyYAML): +从源码(唯一运行时依赖:PyYAML): ```bash git clone https://github.com/Protocol-zero-0/evolution-kernel.git @@ -367,46 +367,59 @@ cd evolution-kernel pip install -e . ``` -需要 Python 3.10 或更高版本。 +需要 Python 3.10 或更高。 --- -## 运行测试 +## 测试 ```bash python3 -m pytest tests/ -v ``` -39 个测试 · 不需要网络连接 · 角色脚本由轻量 fixture 替代。 +**99 个测试** · CI 全程无网络调用 · LLM 角色由轻量 fixture 脚本扮演 · CI 装 `firejail`,sandbox E2E 测试在真实 OS 级 mount 上跑。 --- -## 自己写角色脚本 +## 写你自己的角色 -每个角色是一个普通的可执行程序,接收三个参数: +每个角色是一个可执行文件,接收三个参数: ``` ---input <路径> Governor 为这个角色准备的 JSON ---output <路径> 角色退出前必须写入的 JSON ---worktree <路径> 隔离 git 沙箱的 checkout 路径 +--input governor 写给这个角色的 JSON +--output 角色退出前必须写入的 JSON +--worktree 隔离 git 沙箱 checkout 的路径 ``` -`roles/planner.py`、`roles/executor.sh`、`roles/evaluator.py` 是参考实现。复制并修改它们,或者完全替换成 shell 脚本、Docker 调用——任何能读 `--input`、写 `--output` 的东西都行。 +`roles/planner.py`、`roles/executor.sh`、`roles/evaluator.py` 是参考实现。复制、改写、或者完全替换——shell 脚本、Docker 调用、任何能读 `--input` 写 `--output` 的东西都行。 + +--- + +## 已知局限 + +诚实交代 v1.0 *还不能* 做什么。 + +- **评估器是 LLM。** 可能被一个"看起来对但实际不对"的 patch 骗到,或者拒掉一个"对但写法陌生"的 patch。用 `goal_evaluator` + `evidence_sources` 里的强程序化门控把 LLM 判断锚定到 ground truth。 +- **沙箱只管文件系统。** firejail 拦下 worktree 之外的写。它**不**拦网络、不拦 fork bomb、不拦进程注入。要跑不信任的执行器,外面再套 network namespace 或 VM。 +- **历史是摘要,不是回放。** 规划器只看到最近 *N* 轮的一行 reflection,不看完整的过往 plan。长时程策略需要规划器把状态自己编码进 `plan.json` 的 summary。 +- **成本会累积。** 30 轮循环用 Claude Sonnet 规划 + Claude Code 执行可能花 $40–$100。Hard-stop 预算是真的——设得比你预期还低。 +- **真实 provider 集成不在 CI 上。** 99 个测试用 `tests/fixtures/` 的 fixture 脚本。端到端的 Aider / Claude Code / Anthropic / OpenAI 集成是每个 release 前手测,不是每次 push 都跑。 --- ## 项目结构 ``` -evolution_kernel/ 约 1,200 行运行时(Governor · Observer · HardStops · Config · CLI) -roles/ 参考版规划器、执行器、评估器 -examples/ demo 目标仓库 + 可直接运行的 evolution.yml -docs/ 协议文档 -tests/ 39 个单元 + 验收测试 +evolution_kernel/ ~1,900 行 runtime(Governor · Observer · HardStops · Sandbox · Config · CLI · Scope) +roles/ 参考规划器 / 执行器 / 评估器 / 目标评估器 / 策略师 +examples/ demo 目标 + sandbox demo + 可直接运行的 evolution.yml +docs/ 协议规范 + 第一个进化任务规范 +tests/ 99 个单元 + 验收测试 · 14 个 fixture 角色脚本 +evidence/ checked-in 的可复现运行 artifact ``` --- ## 许可证 -MIT — 见 [LICENSE](LICENSE)。 +MIT —— 见 [LICENSE](LICENSE)。