Basecamp Bench

Basecamp Bench compares coding-agent harnesses on two realistic product tasks: a frontend prototype and a production-shaped API. Implementation agents receive only a concise product directive and the same seed/reference pack. Independent evaluator agents inspect each immutable submission, decide how to run and test it, and return evidence-backed dimension scores under a versioned contract.

The runner preserves raw attempts, computes scores and eligibility itself, and generates a self-contained HTML quality-versus-cost report. FE and BE results remain separate.

How it was built

Basecamp Bench was developed through three successive model-driven passes:

1. Grok 4.5 built the initial working prototype.
2. Claude Fable 5 cleaned up the implementation and improved its structure.
3. GPT-5.6 Sol professionalized the project, hardening the architecture, tests, documentation, safety controls, and release workflow.

Building the project with GPT-5.6 Sol consumed 682 million tokens and $449.33 in model usage. This development cost is separate from the benchmark-run costs reported below.

Results updated September 5, 2026

Basecamp Bench asks each agent to build two parts of a Basecamp 5 clone from a fixed specification: a single-file frontend SPA and a production-shaped backend API. An independent evaluator scores every submission across weighted quality dimensions. Fable 5.1 leads frontend at 8.183, ahead of GPT-6 Astra at 7.607. Astra leads backend at 9.305, ahead of Fable at 8.965. Astra costs about a quarter as much as Fable 5.1 across both tracks ($25.57 versus $92.96). Every model in the table scores higher on backend than frontend.

Model	Frontend	Backend	Total time	Total cost
Fable 5.1	8.183	8.965	3:04:06	$92.96
GPT-6 Astra	7.607	9.305	1:36:31	$25.57
Fable 5	7.578	8.392	2:06:40	$85.87
Grok 4.6	7.104	7.926	57:58	$10.85
Sonnet 5	6.982	7.243	1:27:09	$36.23
Grok 4.5	6.384	7.278	36:48	$9.30
GPT-5.6 Sol	5.765	7.310	59:48	$15.13
Muse Spark 1.3	5.745	6.711	42:24	$4.63
Gemini 3.8 Flash High	5.700	6.483	38:17	$6.07
GPT-5.5	5.670	7.084	44:14	$10.94
Gemini 3.7 Flash High	5.608	5.800	30:59	$7.54

Total time and cost include implementation and evaluation across both tracks.

Original baseline evidence is available in baseline/. Read the full report.

Getting Started

Basecamp Bench runs locally with Python 3.11 or newer and no runtime dependencies. Install and authenticate the agent CLIs you plan to compare: Codex, Claude Code, Grok, Pi, and/or Google Antigravity (agy). External OS isolation is optional for local runs and required for publication; the provided container is one way to supply it.

1. Clone this repository and enter its root.
2. Install the runner and copy the annotated local configuration.

python -m pip install -e .
cp bench.example.toml bench.toml

3. Adjust the selected models or executable paths, inspect the effective configuration, and start a run.

basecamp-bench show-config
basecamp-bench run --harness codex --track fe

Run

Local mode is intended for iteration and defaults to one repetition and one evaluator:

basecamp-bench run --harness codex --track fe

To benchmark GPT-6 Astra, enable the codex-astra block in bench.toml (shown in bench.example.toml) and run basecamp-bench run --harness codex-astra --track fe --track be. It uses gpt-6-astra at high reasoning effort through the Codex CLI; the configured independent evaluator remains unchanged.

Independent implementation attempts run concurrently. As soon as one submission is snapshotted, its evaluator calls run concurrently while other implementations continue. Live progress is written to stderr with attributable build, evaluate, aggregate, report, and final run events; use --quiet to suppress it. The completed run path remains the only stdout output. At most 32 paid agent processes run simultaneously by default; adjust the safety cap with --max-parallel-agents.

Harness and track flags may be repeated or comma-separated. Local jobs use isolated folders beneath runs/<run-id>/workspaces; full_access = true requires --allow-unsafe-host-execution locally or --confirmed-isolated-environment inside an external boundary.

Publication mode requires at least three implementation repetitions, two valid evaluator model IDs, exact or pinned pricing, a distributable reference pack, and confirmed isolation:

basecamp-bench run --mode publication --repetitions 3 \
  --confirmed-isolated-environment

Configure at least two enabled [[evaluators]] entries in bench.toml before running that command. Paid model calls can be substantial; inspect the complete fleet first with basecamp-bench show-config and enforce provider spending limits outside the runner.

Regenerate the report

Reports discover every matching attempt ledger (or legacy leaderboard JSON) beneath the supplied directories. Each ledger stores comparison identity, dimension profile, and raw attempts; statistics are derived at report time, so adding later model runs requires no hand-edited aggregates. Canonical machine artifacts are schema 2.0 JSON attempt ledgers (plus the HTML report). Normal runs do not write leaderboard CSV or Markdown files.

basecamp-bench report runs --output model-performance.html

Optional display-name overrides apply only at render time and never rewrite evidence:

basecamp-bench report runs --output model-performance.html \
  --rename model-id="Friendly Name"

The output is one deterministic, offline HTML decision surface: separate track/contract sections, a plain-language verdict, a cost-versus-quality chart with model labels, a comparison table (score, expected implementation cost per valid result, success rate, tokens, duration, evaluator overhead, total observed cost, and uncertainty when repetitions exist), dimension scores and weights, failures and mixed-eligibility footnotes, and a complete embedded JSON payload with provenance hashes, classifications, source IDs, and raw attempts. Tables remain usable without SVG; imported text is escaped. Methodology prose lives in docs/METHODOLOGY.md. Local reports combine matching benchmark evidence across runner revisions for exploratory comparison; publication reports keep those revisions separate.

Optional CSV and Markdown projections

When a spreadsheet or static Markdown table is needed, project one leaderboard JSON explicitly. Rows are derived only from the loaded attempts through the same aggregator the report uses; CSV/Markdown are never a second source of truth:

basecamp-bench export-tabular \
  runs/<run-id>/leaderboards/leaderboard_<track>_<version>_<sha>.json \
  --output-dir /tmp/tabular

The command writes two deterministic UTF-8 files named from the comparison identity (leaderboard_*.csv and leaderboard_*.md) and refuses to overwrite existing targets. Both schema 2.0 attempt ledgers and committed legacy schema 1.0 leaderboards are accepted as input.

Re-evaluate immutable submissions

Re-evaluation verifies an earlier run and its declared snapshot hashes, then creates a new run with fresh evaluator attempts. The prior run is never changed. Verified prior snapshots and normal implementations share one evaluator and attempt-finalization path; only the submission source differs.

basecamp-bench reevaluate runs/<run-id> --track fe

Current contracts and evaluator configuration apply to the new evaluation; lineage records the prior run manifest and snapshot hashes.

Verify and export

basecamp-bench verify-run runs/<run-id>
basecamp-bench export-run runs/<run-id> basecamp-bench-run.zip

Verification checks the strict manifest shape, identifiers, relative paths, and every declared artifact hash. Export includes only the manifest and declared public artifacts, scans them for likely credentials, rejects symlinks and path escapes, and writes a deterministic archive without overwriting an existing file. Workspaces and private logs are never exported. Local evaluation imports only run provenance builders; publication verification and ZIP export load at the verify/export (and reevaluation) boundary. Public APIs from basecamp_bench.manifest import verify_run, export_run remain stable.

Official baseline

The repository's baseline/ directory contains four verified, shareability-scanned reference runs and one combined self-contained HTML report. It preserves model snapshots, evaluator reports and results, raw attempts, leaderboards, and provenance manifests while excluding private logs, prompts, credentials, and execution workspaces.

for run in baseline/runs/*; do basecamp-bench verify-run "$run"; done
basecamp-bench report baseline/runs --output /tmp/basecamp-bench-report.html \
  --rename 'claude-fable-5=Fable 5' \
  --rename 'claude-sonnet-5=Sonnet 5' \
  --rename 'gpt-5.6-sol=GPT-5.6 Sol' \
  --commentary baseline/commentary.json
cmp baseline/report.html /tmp/basecamp-bench-report.html

New compatible model runs can be compared by regenerating a report from the baseline and additional run directories together. The committed baseline is an exploratory local run; its quality and cost points are auditable, while official publication eligibility and Pareto-frontier claims require the stricter repetition, evaluator, pricing, and isolation rules below.

Method

Each track lives under benchmarks/<track>/:

• prompt.md is passed byte-for-byte to implementation agents. It contains the task only—no benchmark, evaluator, rubric, output filename, or runner instructions.
• eval.md gives evaluator agents the full assessment context and evidence standard without prescribing the submission's language, runtime, filenames, commands, or architecture.
• contract.json is the canonical dimension, anchor, weight, and score policy.

Evaluators receive disposable copies of the original seed and immutable submission plus an exact JSON result schema. The runner rejects missing or unknown dimensions, non-finite/out-of-range scores, identity or hash mismatch, malformed evidence, failed evaluator processes, and any evidence mutation. It then takes the median evaluator score per dimension and computes the weighted overall score.

Sol 5.6 is the default evaluator model. Comparative evaluator trials with Fable 5, Grok 4.5, and Sol 5.6 produced similarly fair judgments; Sol was the most detailed and thorough. Changing the evaluator is therefore a methodology and cost-routing choice that may affect the level of detail more than the general direction of the judgment.

Leaderboard aggregation is scoped by track, contract version, harness, and model. Failed attempts remain visible and contribute to success rate. The frontier's primary cost is median implementation cost per attempt divided by success rate; evaluation cost is reported separately.

See Methodology for claim boundaries and detailed rules.

Safety

Agent CLIs and generated applications are untrusted code. Directory separation, prompt instructions, environment allowlists, hash checks, log caps, and process group cleanup provide integrity and operational controls; some harnesses do not provide an OS security boundary. Use spend-limited credentials and keep personal files, ambient cloud credentials, Docker sockets, SSH agents, browser profiles, and production systems outside the run environment.

The isolated execution guide and reference containers/ recipe provide optional local hardening. Publication mode requires an explicit external-isolation confirmation.

Add models and harnesses

A new model on an existing CLI is configuration only: update the corresponding [harnesses.<id>] model and pricing override if necessary. A new harness needs an adapter implementing command construction, environment allowlisting, working directory, stdin, and output/usage parsing. Adapter tests must cover redaction, malformed output, permissions, timeouts, and vendor output drift.

The optional pi adapter exposes the safe benchmark model ID glm-5.2 and routes it to OpenRouter's z-ai/glm-5.2; set OPENROUTER_API_KEY and use the commented example in bench.example.toml. Pi relies on the per-job workspace boundary in local mode; the optional container recipe adds OS isolation.

The optional opencode adapter exposes safe benchmark model IDs and routes them to OpenCode's native IDs: 0x-alpha maps to opencode/x-preview-f-free and requires the max effort, and muse-spark-1.3 maps to opencode/muse-spark-1.3-contributor-free at minimal through xhigh. The configured effort is passed through as OpenCode's --variant. It reads prompts from stdin, consumes OpenCode's JSON event stream for usage, and installs a per-job config that denies external-directory access in workspace-write mode. Use the commented example in bench.example.toml.

The optional agy adapter supports gemini-3.1-pro at low or high effort and gemini-3.5-flash, gemini-3.7-flash, and gemini-3.8-flash at low, medium, or high effort. It enables Antigravity's terminal sandbox and stages evaluator evidence as disposable workspace copies, preserving the immutable originals. See the commented bench.example.toml entry for setup.

Contract or evaluator-directive changes require a new contract version and changelog entry. Published contract versions are immutable.

Repository map

basecamp_bench/     runner, adapters, validation, aggregation, reporting, CLI
benchmarks/         FE/BE directives, evaluator rubrics, contracts, asset manifest
Repo/               identical seed and reference material supplied to agents
schemas/            public artifact JSON Schemas
docs/               methodology, security, rights, and community documentation
containers/         disposable non-root runner reference
tests/              unit and credential-free fake-harness integration tests

Rights and independence

Project code and original documentation are Apache-2.0. The vendored basecamp-sdk reference material is MIT-licensed at its recorded upstream commit. Personal-account screenshots are documented as Fair use in the hash-matched reference-pack manifest. See asset provenance, third-party notices, and trademarks.

This is an independent evaluation project and is not affiliated with or endorsed by Basecamp.