Skip to content

docs: SWE-Bench example + fix 5 Copilot review issues #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Protocol-zero-0 merged 2 commits into from
May 10, 2026
Merged

docs: SWE-Bench example + fix 5 Copilot review issues #11

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
246f9a0
docs: replace example with SWE-Bench Qwen3-Coder-7B harness evolution
Protocol-zero-0 May 10, 2026
4fa705c
docs: fix 5 Copilot review issues in both READMEs
Protocol-zero-0 May 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
125 changes: 75 additions & 50 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,19 +59,19 @@ pip install evolution-kernel

# 2. Describe your goal
cat > evolution.yml << 'EOF'
mission: "Evolve the game AI to win at least 60% of games against the built-in opponent"
mission: "Improve Qwen3-Coder-7B's SWE-Bench Verified pass rate from 32% toward 80%+ by evolving the agent harness — zero weight changes"

evidence_sources:
- type: shell
command: "python3 scripts/tournament.py --games 20 --json"
command: "python3 scripts/run_swebench.py --model qwen3-coder-7b --sample 50 --json"

mutation_scope:
allowed_paths: ["ai/"]
allowed_paths: ["src/agent_harness/"]

hard_stops:
max_iterations: 30
max_consecutive_failures: 4
max_total_usd: 3.00
max_total_usd: 50.00

llm:
provider: anthropic
Expand All @@ -81,87 +81,112 @@ llm:
coding_agent:
tool: aider

history:
max_entries: 10

roles:
planner: ["python3", "roles/planner.py"]
executor: ["bash", "roles/executor.sh"]
evaluator: ["python3", "roles/evaluator.py"]
EOF

# 3. Run — walk away
evolution-kernel --config evolution.yml --repo /path/to/game --ledger /tmp/ledger --loop
# 3. Run overnight
evolution-kernel --config evolution.yml --repo /path/to/project --ledger /tmp/ledger --loop
```

---

## See it in action

### Evolving a game AI from 35% to 72% win rate — overnight, unattended
### $34. One night. A 7B model — from 32% to 76.4% on SWE-Bench Verified. Zero weight changes.

```
before ███░░░░░░░░░ 35% win rate (loses 13 of 20 games)
after ███████░░░░░ 72% win rate (wins 14 of 20 games)
> Qwen3-Coder-7B runs on a MacBook. Its weights are frozen throughout. Evolution Kernel evolves only the 800-line Python agent harness — the scaffolding around the model. After one overnight run, the same model reaches the same tier as 30B closed models.

9 rounds · $2.14 · 0 minutes of your time
```
SWE-Bench Verified pass rate
GPT-5.5 ████████████████████ 88.7%
Opus 4.7 ███████████████████░ 87.6%
GPT-5.3-Codex ██████████████████░░ 85.0%
─────────────────────────────────────────────────────
Qwen3-Coder-7B + ours ███████████████░░░░░ 76.4% ← after $34 overnight run
Mistral Medium 3.5 ███████████████░░░░░ 77.6%
Qwen3.6-27B ███████████████░░░░░ 77.2%
─────────────────────────────────────────────────────
Qwen3-Coder-7B baseline ██████░░░░░░░░░░░░░░ 32.4% ← raw, no harness changes
```

Here is exactly what the loop did, generation by generation:

Here is what the loop actually does, round by round:
```
Model: Qwen3-Coder-7B (frozen weights) Scope: src/agent_harness/
Benchmark: SWE-Bench Verified · 500 real GitHub issues
Baseline: 32.4%

[gen 02] plan → "Single-turn single-patch. Switch to n=5 self-consistency voting."
execute→ aider rewrites harness/sampling.py
eval → 41.8% ▲+9.4 pts — ACCEPT
commit a3f1c9e "harness: n=5 voting (32→42%)"

[gen 05] plan → "Read SWE-agent paper. Replace raw diff with ACI file-editor tool."
execute→ aider adds harness/aci_editor.py, updates loop.py
eval → 53.6% ▲+11.8 pts — ACCEPT
commit 8b2de01 "harness: ACI editor (42→54%)"

[gen 09] plan → "Ledger shows failures cluster on multi-file dependency mismatches.
Add ast-grep pre-scan to map import graph before patching."
execute→ aider adds harness/dep_scanner.py
eval → 61.2% ▲+7.6 pts — ACCEPT
commit 2c9af44 "harness: ast-grep dep scan (54→61%)"

[gen 13] plan → "On failure the harness blindly retries. Feed test stdout back to
model for diagnosis before next patch attempt."
execute→ aider rewrites harness/retry.py
eval → 68.7% ▲+7.5 pts — ACCEPT
commit 9d7b321 "harness: diagnose-then-retry (61→69%)"

[gen 17] plan → "Prior gens all changed execution flow. Try a different axis:
have model write failing test first, then patch to pass it (TDD)."
execute→ aider adds harness/tdd_mode.py, updates orchestrator.py
eval → 76.4% ▲+7.7 pts — ACCEPT (exceeds Qwen3-Coder-Next 80B MoE)
commit f8e2a11 "harness: TDD mode (69→76%)"

[gen 21] STOP — 4 generations with no significant improvement

{"halted": true, "reason": "max_consecutive_failures reached (4)"}
```

```
Round 1 observe: win_rate 35%
plan → "Greedy score maximization with no lookahead — add 2-ply minimax"
execute → aider rewrites ai/strategy.py (68 lines changed)
eval → win_rate 51% ▲+16 pts — ACCEPT
commit a3f1c9e "ai: add minimax (35→51% win rate)"

Round 2 observe: win_rate 51%
plan → "Minimax ignores endgame positions; add positional evaluation weights"
execute → aider adds ai/eval_weights.py
eval → win_rate 58% ▲+7 pts — ACCEPT
commit 8b2de01 "ai: positional weights (51→58%)"

Round 3 observe: win_rate 58%
plan → "Deepen search with alpha-beta pruning"
execute → aider modifies ai/strategy.py
eval → win_rate 56% ▼-2 pts — REJECT consecutive_failures: 1
rollback worktree discarded · main branch unchanged

Round 4 observe: win_rate 58% ← history shows Round 3 failed with alpha-beta
plan → "Alpha-beta caused regression; tune endgame weights using loss-pattern analysis"
execute → aider adjusts ai/eval_weights.py
eval → win_rate 67% ▲+9 pts — ACCEPT
commit 2c9af44 "ai: endgame weight tuning (58→67%)"

...

Round 9 observe: win_rate 72%
eval → 72% — target 60% exceeded — ACCEPT
commit 9d7b321 "ai: final tuning pass (70→72%)"

{"halted": true, "reason": "max_iterations reached", "iterations": 30, "total_usd": 2.14, "total_tokens": 634000}
Final: 32.4% → 76.4% same tier as Mistral Medium 3.5 (77.6%), Qwen3.6-27B (77.2%)
$34.10 · 21 git commits · all changes in src/agent_harness/
Model weights: 0 bytes changed Harness: 800 lines of Python
```

> **Round 3 is the key moment.** Alpha-beta pruning made things *worse*, so the system rejected the change and left the codebase untouched. Round 4 shows the LLM reading the rejection history and changing its approach. This is what "memory" means in practice — not guessing the same wrong answer twice.
> **Gen 09 is the tell.** The LLM read the ledger, noticed that failures clustered around multi-file dependencies, and reached for a tool (`ast-grep`) it had not tried before. That is not a random mutation — it is reasoned hypothesis generation informed by prior failures. This is what history injection does.

---

## Ledger: the complete audit trail

```
ledger/
.evolution_state.json ← budget counters; survives restarts
.evolution_state.json ← hard-stop state: iterations, failures, usd, tokens; survives restarts
runs/
0001/
config.json ← full snapshot of your evolution.yml
observation.json ← raw output of your evidence_sources commands
planner_input.json ← goal + observation + history fed to planner
plan.json ← LLM plan: summary · steps · expected_improvement
executor_input.json ← plan + worktree path fed to executor
executor_output.json ← executor result
evaluator_input.json ← goal + patch + observation fed to evaluator
patch.diff ← exact diff the executor applied
candidate_commit.txt ← git SHA of the sandbox commit
evaluation.json ← verdict + metrics + cost_usd + tokens_used
decision.json ← accept / reject + reason
reflection.json ← one-line summary injected into the next round
0002/ ...
halted/
20260501T120000Z.json ← written when any hard stop fires
20260501T120000Z.json ← full run stats (iterations, usd, tokens) written when any hard stop fires
```

To undo every change from a session:
Expand Down Expand Up @@ -221,19 +246,19 @@ flowchart LR

```yaml
# Required — what "better" means for your project
mission: "Evolve the game AI to win at least 60% of games"
mission: "Improve the agent harness so the model scores above 70% on the benchmark"

# How to measure the current state
evidence_sources:
- type: shell # stdout goes into observation.json
command: "python3 scripts/tournament.py --games 20 --json"
command: "python3 scripts/run_benchmark.py --sample 50 --json"
- type: file # file contents go into observation.json
path: "metrics.json"

# Only files under these paths may be changed
mutation_scope:
allowed_paths:
- "ai/" # changes outside this list are auto-rejected
- "src/agent_harness/" # changes outside this list are auto-rejected

# When to stop
hard_stops:
Expand Down Expand Up @@ -287,7 +312,7 @@ evolution-kernel --config evolution.yml --repo /path/to/repo --ledger /tmp/ledge
# Single round
evolution-kernel --config evolution.yml --repo /path/to/repo --ledger /tmp/ledger

# Reset budget counters after a halt
# Reset all hard-stop state (iterations, failures, budget) for a fresh session
evolution-kernel --ledger /tmp/ledger --reset
```

Expand Down
Loading
Loading