Languages: English | Chinese | Results Site
SWE-bench Essential is a reproducible runner and evaluation workspace for comparing coding agents on a selected 40-task subset of SWE-bench Pro Public. It supports Codex, Claude Code, and PenguinHarness, runs each task in its official Docker environment, exports patches, evaluates predictions incrementally, and records traces, token usage, latency, and cost.
Alongside this cross-agent comparison, the repository examines how manual skills, evolved agent instructions, goal mode, and PenguinHarness versions affect software-engineering performance.
The table summarizes accuracy and resource usage across the current experiments. Accuracy is averaged across repeated runs, time is averaged per task, and token usage and cost are totals for one complete 40-task run.
| Setting | Version and configuration | Accuracy (%) | Avg. time / case (min) | Total tokens (M/run) | Total cost (USD/run) |
|---|---|---|---|---|---|
| Penguin w/o Skill | PenguinHarness v0.1.5 / DeepSeek V4 Flash xhigh / no skill / Goal off | 79.17 | 9.32 | 146.75 | 1.12 |
| Penguin w/ Manual Skill | PenguinHarness v0.1.5 / DeepSeek V4 Flash xhigh / SWE skill / Goal off | 81.67 | 9.48 | 137.99 | 1.06 |
| Penguin w/ Auto Skill | PenguinHarness v0.1.5 / DeepSeek V4 Flash xhigh / evolved AGENTS.md / Goal off |
81.67 | 9.11 | 170.30 | 1.25 |
| Penguin w/ Manual Skill + Goal | PenguinHarness v0.1.5 / DeepSeek V4 Flash xhigh / SWE skill / Goal on | 82.50 | 11.99 | 169.51 | 1.24 |
| Penguin w/o Skill | PenguinHarness v0.0.1 / DeepSeek V4 Flash xhigh / no skill / Goal off | 74.17 | 8.10 | 129.05 | 1.02 |
| Penguin w/ Manual Skill | PenguinHarness v0.0.1 / DeepSeek V4 Flash xhigh / SWE skill / Goal off | 75.00 | 7.45 | 122.20 | 0.98 |
| Claude Code | Claude Code CLI / Claude Opus 4.8 / effort=max | 86.25 | 5.31 | 75.81 | 73.49 |
| Codex | Codex CLI / GPT-5.5 xhigh | 71.25 | 8.55 | 125.60 | 110.04 |
“Manual Skill” is the hand-written SWE skill; “Auto Skill” is the evolved AGENTS.md.
| Path | Purpose |
|---|---|
assets/ |
Essential 40 task list and retained agent/skill materials. |
scripts/agents/ |
Agent launch configuration and output parsing. |
scripts/common/ |
Shared runtime, metrics, logging, and worker-pool utilities. |
scripts/swebench_pro/ |
Resource preparation, prediction, evaluation, and campaign runners. |
result/ |
Generated predictions, traces, patches, metrics, and evaluation outputs. |
AGENTS.md |
Detailed setup, runner options, operating rules, and recovery guidance. |
Docker and uv are required. Put API credentials in .env, then prepare the environment:
uv sync --locked
uv run python scripts/check_env.py
uv run python scripts/build_agent_runtime.py -o .cache/agent-runtime
uv run python scripts/swebench_pro/prepare_resources.pyRun a one-case Penguin smoke test with incremental evaluation:
uv run python scripts/swebench_pro/run_campaign.py \
--agent penguin \
--runs 1 \
--slice 0:1 \
--prediction-workers 1 \
--eval-workers 1Use --agent codex or --agent claude for the other agents. Remove --slice 0:1 and set --runs 3 for the full Essential 40 experiment. See AGENTS.md for runtime components, model credentials, skill injection, case selection, parallel campaigns, resuming runs, and result schemas.
The lightweight checks do not start Docker or call model APIs:
uv run python -m unittest discover -s tests -v
uv run ruff check scripts tests