Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,11 @@ jobs:
git config --global user.email "ci@evolution-kernel.local"
git config --global user.name "evolution-kernel-ci"

- name: Install firejail (required for the sandbox E2E test)
run: |
sudo apt-get update
sudo apt-get install -y firejail

- name: Install package (PyYAML is the only runtime dep)
run: python -m pip install -e .

Expand Down
31 changes: 20 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,11 +63,11 @@ pip install evolution-kernel

# 2. Describe your goal
cat > evolution.yml << 'EOF'
mission: "Evolve the math-solver harness so Qwen3-7B-Instruct answers 90%+ of GSM8K problems correctly — no model retraining"
mission: "Evolve the math-solver harness so Qwen3-8B-Instruct answers 90%+ of GSM8K problems correctly — no model retraining"

evidence_sources:
- type: shell
command: "python3 scripts/run_gsm8k.py --model qwen3-7b-instruct --sample 100 --json"
command: "python3 scripts/run_gsm8k.py --model qwen3-8b-instruct --sample 100 --json"

mutation_scope:
allowed_paths: ["src/math_solver_harness/"]
Expand Down Expand Up @@ -102,25 +102,25 @@ evolution-kernel --config evolution.yml --repo /path/to/project --ledger /tmp/le

## See it in action

### $34. One night. A 7B model that runs on a MacBook — from 51.8% to 96.2% on elementary math. Zero weight changes.
### $34. One night. An 8B model that runs on a MacBook — from 51.8% to 96.2% on elementary math. Zero weight changes.

> Qwen3-7B-Instruct is a general-purpose model with no math-specific training. Its weights are frozen throughout. Evolution Kernel evolves only the solver harness — prompt strategies, tools, and sampling logic. After one overnight run, the same model sits 2.8 points behind GPT-5.5. That means every child can have a free, local, always-on, privacy-safe math tutor.
> Qwen3-8B-Instruct is a general-purpose model with no math-specific training. Its weights are frozen throughout. Evolution Kernel evolves only the solver harness — prompt strategies, tools, and sampling logic. After one overnight run, the same model sits 2.8 points behind GPT-5.5. That means every child can have a free, local, always-on, privacy-safe math tutor.

```
GSM8K pass rate (1,319 math word problems)
GPT-5.5 ████████████████████ 99.0%
Claude Opus 4.7 ████████████████████ 98.6%
─────────────────────────────────────────────────────
Qwen3-7B + ours ███████████████████░ 96.2% ← after $34 overnight run
Qwen3-8B + ours ███████████████████░ 96.2% ← after $34 overnight run
─────────────────────────────────────────────────────
Early GPT-4 ██████████████████░░ 92.0%
Qwen3-7B baseline ██████████░░░░░░░░░░ 51.8% ← raw model, naive prompt
Qwen3-8B baseline ██████████░░░░░░░░░░ 51.8% ← raw model, naive prompt
```

Here is exactly what the loop did, generation by generation:

```
Model: Qwen3-7B-Instruct (frozen weights) Scope: src/math_solver_harness/
Model: Qwen3-8B-Instruct (frozen weights) Scope: src/math_solver_harness/
Benchmark: GSM8K · 1,319 math word problems
Baseline: 51.8% Reference: GPT-5.5: 99.0% Opus 4.7: 98.6% Early GPT-4: 92.0%

Expand Down Expand Up @@ -167,7 +167,7 @@ Baseline: 51.8% Reference: GPT-5.5: 99.0% Opus 4.7: 98.6% Early GPT-4: 92.0%
Final: 51.8% → 96.2% 2.8 pts behind GPT-5.5 (99.0%), ahead of early GPT-4 (92.0%)
$34.10 · 25 git commits · all changes in src/math_solver_harness/
Model weights: 0 bytes changed Harness: ~600 lines of Python
Any 7B model can use this harness — local inference, zero API cost
Any 8B-class model can use this harness — local inference, zero API cost
```

> **Gen 09 is the tell.** The LLM read the ledger, spotted that arithmetic errors were the dominant failure pattern, and independently reached for a Python calculator tool — a technique it had not tried before. That is not a random mutation: it is hypothesis generation driven by prior evidence. This is what history injection does.
Expand Down Expand Up @@ -247,7 +247,7 @@ flowchart LR
| Anthropic and OpenAI planner/evaluator support | ✅ |
| Goal evaluator — stops when mission is "won" | ✅ |
| k-branch parallel exploration (FunSearch / AlphaEvolve style) | ✅ |
| Process sandbox (firejail / bwrap) for production safety | 🔧 PR #7 |
| Process sandbox via firejail — executor cannot write outside its worktree | ✅ |

---

Expand All @@ -258,12 +258,12 @@ flowchart LR

```yaml
# Required — what "better" means for your project
mission: "Evolve the math-solver harness so Qwen3-7B-Instruct scores 90%+ on GSM8K — no model retraining"
mission: "Evolve the math-solver harness so Qwen3-8B-Instruct scores 90%+ on GSM8K — no model retraining"

# How to measure the current state
evidence_sources:
- type: shell # stdout goes into observation.json
command: "python3 scripts/run_gsm8k.py --model qwen3-7b-instruct --sample 100 --json"
command: "python3 scripts/run_gsm8k.py --model qwen3-8b-instruct --sample 100 --json"
- type: file # file contents go into observation.json
path: "metrics.json"

Expand Down Expand Up @@ -299,6 +299,15 @@ history:
parallel:
k_branches: 1

# Process sandbox: when enabled, the executor's argv is wrapped with firejail
# so the rest of the filesystem is read-only and only the worktree + the
# run's ledger directory are writable. Planner and evaluator are read-mostly
# and run unsandboxed. Default off — v0.3 behavior is preserved.
sandbox:
enabled: false # set to true on machines with firejail installed
backend: firejail
extra_args: [] # appended verbatim before `--`

roles:
planner: ["python3", "roles/planner.py"]
executor: ["bash", "roles/executor.sh"]
Expand Down
30 changes: 19 additions & 11 deletions README.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,11 +63,11 @@ pip install evolution-kernel

# 2. 描述你的目标
cat > evolution.yml << 'EOF'
mission: "进化数学解题 harness,让 Qwen3-7B-Instruct 在 GSM8K 上的正确率达到 90%+——不重新训练模型"
mission: "进化数学解题 harness,让 Qwen3-8B-Instruct 在 GSM8K 上的正确率达到 90%+——不重新训练模型"

evidence_sources:
- type: shell
command: "python3 scripts/run_gsm8k.py --model qwen3-7b-instruct --sample 100 --json"
command: "python3 scripts/run_gsm8k.py --model qwen3-8b-instruct --sample 100 --json"

mutation_scope:
allowed_paths: ["src/math_solver_harness/"]
Expand Down Expand Up @@ -102,25 +102,25 @@ evolution-kernel --config evolution.yml --repo /path/to/project --ledger /tmp/le

## 看它实际运行

### $34,一晚上,一个能在 MacBook 上跑的 7B 模型——小学数学应用题正确率 96.2%,和 GPT-5.5 基本同档。模型权重一字节未动。
### $34,一晚上,一个能在 MacBook 上跑的 8B 模型——小学数学应用题正确率 96.2%,和 GPT-5.5 基本同档。模型权重一字节未动。

> Qwen3-7B-Instruct 是一个通用模型,没有专门的数学训练。权重全程冻结。Evolution Kernel 只进化 solver harness——提示策略、工具调用和采样逻辑。一个隔夜跑完,同一个模型只落后 GPT-5.5 2.8 个百分点。这意味着每个孩子都能拥有一个免费、本地、随时在线、完全保护隐私的数学辅导老师。
> Qwen3-8B-Instruct 是一个通用模型,没有专门的数学训练。权重全程冻结。Evolution Kernel 只进化 solver harness——提示策略、工具调用和采样逻辑。一个隔夜跑完,同一个模型只落后 GPT-5.5 2.8 个百分点。这意味着每个孩子都能拥有一个免费、本地、随时在线、完全保护隐私的数学辅导老师。

```
GSM8K 通过率(1,319 道小学数学应用题)
GPT-5.5 ████████████████████ 99.0%
Claude Opus 4.7 ████████████████████ 98.6%
─────────────────────────────────────────────────────
Qwen3-7B + 我们 ███████████████████░ 96.2% ← $34 一晚上跑出来的
Qwen3-8B + 我们 ███████████████████░ 96.2% ← $34 一晚上跑出来的
─────────────────────────────────────────────────────
早期 GPT-4 ██████████████████░░ 92.0%
Qwen3-7B 基线 ██████████░░░░░░░░░░ 51.8% ← 原始模型,朴素提示
Qwen3-8B 基线 ██████████░░░░░░░░░░ 51.8% ← 原始模型,朴素提示
```

每代循环实际发生的事:

```
模型:Qwen3-7B-Instruct(权重冻结) 范围:src/math_solver_harness/
模型:Qwen3-8B-Instruct(权重冻结) 范围:src/math_solver_harness/
基准:GSM8K · 1,319 道小学数学应用题
基线:51.8% 参考:GPT-5.5: 99.0% Opus 4.7: 98.6% 早期 GPT-4: 92.0%

Expand Down Expand Up @@ -167,7 +167,7 @@ evolution-kernel --config evolution.yml --repo /path/to/project --ledger /tmp/le
最终:51.8% → 96.2% 落后 GPT-5.5 (99.0%) 2.8 分,领先早期 GPT-4 (92.0%)
$34.10 · 25 个 git commit · 全部落在 src/math_solver_harness/
模型权重:0 字节变化 Harness:~600 行 Python
任何 7B 模型都能用这个 harness——本地推理,零 API 费用
任何 8B 量级的模型都能用这个 harness——本地推理,零 API 费用
```

> **gen 09 是关键时刻。** LLM 读了 ledger,发现算术计算错误是最主要的失败模式,主动引入了 Python 计算器工具——一个它此前从未尝试过的技巧。这不是随机突变——是用过去失败数据驱动的假设生成。这就是 history injection 在实际中的含义。
Expand Down Expand Up @@ -247,7 +247,7 @@ flowchart LR
| Anthropic 和 OpenAI 规划器 / 评估器支持 | ✅ |
| 目标评估器——当 mission 完成时自动停止 | ✅ |
| k 路并行探索(FunSearch / AlphaEvolve 模式) | ✅ |
| 进程级沙箱(firejail / bwrap),面向生产环境 | 🔧 PR #7 |
| 进程级沙箱(firejail)——执行器无法写出 worktree 之外的任何文件 | |

---

Expand All @@ -258,12 +258,12 @@ flowchart LR

```yaml
# 必填——"更好"对你的项目意味着什么
mission: "进化数学解题 harness,让 Qwen3-7B-Instruct 在 GSM8K 上的正确率达到 90%+——不重新训练模型"
mission: "进化数学解题 harness,让 Qwen3-8B-Instruct 在 GSM8K 上的正确率达到 90%+——不重新训练模型"

# 如何衡量当前状态
evidence_sources:
- type: shell # stdout 写入 observation.json
command: "python3 scripts/run_gsm8k.py --model qwen3-7b-instruct --sample 100 --json"
command: "python3 scripts/run_gsm8k.py --model qwen3-8b-instruct --sample 100 --json"
- type: file # 文件内容写入 observation.json
path: "metrics.json"

Expand Down Expand Up @@ -298,6 +298,14 @@ history:
parallel:
k_branches: 1

# 进程级沙箱:开启后用 firejail 包装执行器命令——文件系统整体只读,仅 worktree
# 与该轮 ledger 子目录可写。规划器和评估器以读为主,不受影响。
# 默认关闭,与 v0.3 行为字节级一致。
sandbox:
enabled: false # 在装有 firejail 的机器上改为 true
backend: firejail
extra_args: [] # 追加到 firejail 命令的额外参数(在 `--` 之前)

roles:
planner: ["python3", "roles/planner.py"]
executor: ["bash", "roles/executor.sh"]
Expand Down
1 change: 1 addition & 0 deletions evolution_kernel/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ def _make_governor(args: argparse.Namespace, cfg: EvolutionConfig) -> Governor:
allowed_paths=cfg.mutation_scope.allowed_paths,
config_snapshot=cfg.raw,
history_max_entries=cfg.history.max_entries,
sandbox=cfg.sandbox,
)


Expand Down
36 changes: 36 additions & 0 deletions evolution_kernel/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,11 @@
max_total_usd: 1.00 # 0.0 = unlimited
max_total_tokens: 500000 # 0 = unlimited

sandbox: # process-level isolation for the executor
enabled: false # default off; v0.3 behavior is preserved
backend: firejail # only backend supported in PR7a
extra_args: [] # additional firejail flags, appended before `--`

Validation prefers human-readable errors over raw tracebacks so that bad configs
can be fixed without reading source.
"""
Expand All @@ -44,6 +49,8 @@

import yaml

from .sandbox import SandboxConfig


class ConfigError(ValueError):
"""Raised when the YAML config does not match the expected shape."""
Expand Down Expand Up @@ -124,6 +131,7 @@ class EvolutionConfig:
goal_evaluator: GoalEvaluatorConfig = field(default_factory=GoalEvaluatorConfig)
strategist: StrategistConfig = field(default_factory=StrategistConfig)
parallel: ParallelConfig = field(default_factory=ParallelConfig)
sandbox: SandboxConfig = field(default_factory=SandboxConfig)
raw: Mapping[str, Any] = field(default_factory=dict)


Expand Down Expand Up @@ -159,6 +167,7 @@ def parse_config(raw: Mapping[str, Any]) -> EvolutionConfig:
goal_evaluator = _parse_goal_evaluator(raw.get("goal_evaluator", {}))
strategist = _parse_strategist(raw.get("strategist", {}))
parallel = _parse_parallel(raw.get("parallel", {}))
sandbox = _parse_sandbox(raw.get("sandbox", {}))

return EvolutionConfig(
mission=mission.strip(),
Expand All @@ -172,6 +181,7 @@ def parse_config(raw: Mapping[str, Any]) -> EvolutionConfig:
goal_evaluator=goal_evaluator,
strategist=strategist,
parallel=parallel,
sandbox=sandbox,
raw=dict(raw),
)

Expand Down Expand Up @@ -330,3 +340,29 @@ def _parse_parallel(value: Any) -> ParallelConfig:
if not isinstance(k, int) or isinstance(k, bool) or k < 1:
raise ConfigError("`parallel.k_branches` must be a positive integer")
return ParallelConfig(k_branches=k)


def _parse_sandbox(value: Any) -> SandboxConfig:
if not isinstance(value, Mapping):
raise ConfigError("`sandbox` must be a mapping")
enabled = value.get("enabled", False)
if not isinstance(enabled, bool):
raise ConfigError("`sandbox.enabled` must be a boolean")
backend = value.get("backend", "firejail")
if not isinstance(backend, str) or not backend.strip():
raise ConfigError("`sandbox.backend` must be a non-empty string")
extra_raw = value.get("extra_args", [])
if not isinstance(extra_raw, list):
raise ConfigError("`sandbox.extra_args` must be a list of strings")
extras: list[str] = []
for index, entry in enumerate(extra_raw):
if not isinstance(entry, str) or not entry.strip():
raise ConfigError(
f"`sandbox.extra_args[{index}]` must be a non-empty string"
)
extras.append(entry.strip())
Comment on lines +359 to +363
return SandboxConfig(
enabled=enabled,
backend=backend.strip(),
extra_args=tuple(extras),
)
27 changes: 26 additions & 1 deletion evolution_kernel/governor.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@

from .config import EvidenceSource
from .observer import collect_observation, write_observation
from .sandbox import SandboxConfig, wrap_argv as sandbox_wrap_argv
from .scope import ScopeReport, check_scope


Expand Down Expand Up @@ -53,6 +54,7 @@ def __init__(
allowed_paths: Sequence[str] = (),
config_snapshot: Mapping[str, Any] | None = None,
history_max_entries: int = 10,
sandbox: SandboxConfig | None = None,
) -> None:
self.target_repo = Path(target_repo).resolve()
self.ledger_dir = Path(ledger_dir).resolve()
Expand All @@ -63,6 +65,9 @@ def __init__(
self.allowed_paths = tuple(allowed_paths)
self.config_snapshot = dict(config_snapshot) if config_snapshot else None
self.history_max_entries = history_max_entries
# `sandbox` applies only to the executor invocation (see _run_role).
# Planner and evaluator are read-mostly and unaffected.
self.sandbox = sandbox

def run_once(self, goal: Mapping[str, Any], run_id: str | None = None, strategy: dict | None = None) -> RunResult:
self._ensure_git_repo()
Expand Down Expand Up @@ -118,6 +123,7 @@ def run_once(self, goal: Mapping[str, Any], run_id: str | None = None, strategy:
run_dir / "executor_input.json",
run_dir / "executor_output.json",
worktree,
sandbox=self.sandbox,
)

candidate_commit = self._commit_candidate(worktree, run_id)
Expand Down Expand Up @@ -408,6 +414,7 @@ def _run_single_branch(
run_dir / "executor_input.json",
run_dir / "executor_output.json",
worktree,
sandbox=self.sandbox,
)

candidate_commit = self._commit_candidate(worktree, run_id)
Expand Down Expand Up @@ -569,7 +576,15 @@ def _commit_candidate(self, worktree: Path, run_id: str) -> str | None:
)
return self._git_in(worktree, "rev-parse", "HEAD")

def _run_role(self, role: RoleCommand, input_path: Path, output_path: Path, worktree: Path) -> None:
def _run_role(
self,
role: RoleCommand,
input_path: Path,
output_path: Path,
worktree: Path,
*,
sandbox: SandboxConfig | None = None,
) -> None:
argv = [
*role.argv,
"--input",
Expand All @@ -579,6 +594,16 @@ def _run_role(self, role: RoleCommand, input_path: Path, output_path: Path, work
"--worktree",
str(worktree),
]
if sandbox is not None and sandbox.enabled:
# Allow writes to the run's ledger directory so the role can persist
# its --output JSON, plus any role-declared stdout/stderr capture
# next to it. Everything else stays read-only under the sandbox.
argv = sandbox_wrap_argv(
argv,
worktree=worktree,
writable=[output_path.parent],
config=sandbox,
)
Comment on lines +597 to +606
completed = subprocess.run(argv, cwd=worktree, text=True, capture_output=True, check=False)
if completed.stdout:
(output_path.parent / f"{output_path.stem}.stdout.txt").write_text(completed.stdout, encoding="utf-8")
Expand Down
Loading
Loading