feat(bench): a Frontier-Bench lane that keeps the adapter untouched

.github/workflows/bench.yml

@@ -58,7 +58,7 @@ jobs:
             exit 0
           fi
           if git diff --name-only "$BASE_SHA"...HEAD \
-            | grep -Eq '^(bench/harbor_adapter/|bench/terminal_bench_analysis/|\.github/workflows/bench\.yml$)'; then
+            | grep -Eq '^(bench/harbor_adapter/|bench/terminal_bench_analysis/|bench/evidence/frontier/|\.github/workflows/bench\.yml$)'; then
             echo "changed=true" >> "$GITHUB_OUTPUT"
           else
             echo "changed=false" >> "$GITHUB_OUTPUT"
@@ -83,3 +83,11 @@ jobs:
         run: |
           uv sync --locked --extra dev
           uv run --no-sync pytest -q
+
+      # The Frontier-Bench run planner. No project to sync and no Harbor to
+      # install: plan.py imports nothing outside the standard library, which is
+      # deliberate — the piece that decides which tasks are allowed to run
+      # should not need the benchmark harness to be testable.
+      - name: frontier planner — pytest
+        if: steps.scope.outputs.changed == 'true'
+        run: uv run --with pytest --no-project pytest -q bench/evidence/frontier/tests

bench/README.md

@@ -5,11 +5,12 @@ with other agents, and standalone. Everything here is **BYOK** (bring your own
 key), makes **no phone-home**, and never hard-codes a secret. Claim runs use a
 secure launcher that consumes the selected credential before Harbor starts.
 
-Four entry points:
+Five entry points:
 
 | Path | What it does | Needs |
 |---|---|---|
 | [`harbor_adapter/`](harbor_adapter/) | A Harbor *installed-agent* adapter — run Stella on Terminal-Bench 2.x / SWE-bench in the same container + verifier as Claude Code, Codex CLI, Terminus, etc. | Docker, `harbor`, a provider key |
+| [`evidence/frontier/`](evidence/frontier/) | **Frontier-Bench** — Harbor's successor to Terminal-Bench, 74 tasks over seven domains. Same adapter, unmodified; a separate Harbor pin, a resource-aware schedule, and GPU tasks excluded by name rather than scored as zeros. | Docker, a provider key (a *submittable* run needs GPUs — see its [SUBMISSION.md](evidence/frontier/SUBMISSION.md)) |
 | [`run_swebench.py`](run_swebench.py) | A standalone SWE-bench *prediction* harness — clone each instance, run Stella, emit the official predictions JSONL. No Harbor. | `git`, a provider key (Docker only for the official scoring step) |
 | [`loop-bench/`](loop-bench/) | A cheap **turn-loop + context-query correctness** harness: runs N tasks on a flash-tier model, budget-capped, and reports loop health (silent-death / zero-work / stuck-loop) and `project_overview`/`graph_query` adoption — the signals the pass-rate number hides. | `cargo`, Docker, `harbor`, a key |
 | [`smoke/smoke_test.py`](smoke/smoke_test.py) | An **offline, zero-cost** self-test of the adapter wiring for CI. | just the built `stella` binary |

bench/evidence/frontier/README.md

@@ -0,0 +1,107 @@
+# Frontier-Bench (dev-baseline lane)
+
+Runs Stella on [Frontier-Bench](https://github.com/harbor-framework/frontier-bench)
+— Harbor's successor to Terminal-Bench, 74 tasks across seven domains, where the
+best published agents clear roughly a third.
+
+**The adapter is unchanged.** `stella_harbor:StellaAgent` runs this benchmark
+exactly as it runs Terminal-Bench 2.1, with no edit to anything under
+`bench/harbor_adapter/`. That was a design constraint, not a happy accident: the
+adapter's Python tree is digest-frozen for the audited claim path, so a new
+benchmark has to earn its place without perturbing it. Everything specific to
+Frontier-Bench lives in this directory.
+
+## Run it
+
+```bash
+export TB_REPO=/path/to/stella TB_ROOT=/abs/scratch OPENROUTER_API_KEY=...
+bench/evidence/run/build_sut.sh              # shared with the Terminal-Bench lane
+bench/evidence/frontier/setup_venv.sh        # this lane's Harbor pin
+bench/evidence/frontier/fetch_dataset.sh     # pinned dataset + resource plan
+bench/evidence/frontier/warm_images.sh       # pull the base images the builds need
+bench/evidence/frontier/sentinel.sh          # two cheap trials, end to end
+bench/evidence/frontier/primary.sh small fb-small-01
+```
+
+## What differs from Terminal-Bench, and why each difference exists
+
+**A second Harbor pin (0.20.0), not the audited 0.6.1.** Every one of the 74
+tasks declares `environment_mode = "separate"` under `[verifier]`. Harbor 0.6.1
+has no such field, and pydantic's default `extra="ignore"` drops it *silently* —
+0.6.1 runs the whole set with the verifier sharing the agent's container instead
+of the separate one the task asked for, produces rewards, and warns about
+nothing. Plausible numbers answering the wrong question are worse than an error.
+This lane gets its own venv; `bench/harbor_adapter/.venv` stays on 0.6.1 so
+Terminal-Bench reruns keep measuring what they measured.
+
+**Base images are warmed, not task images pulled.** No Frontier-Bench task has a
+`docker_image`; all 74 ship an `environment/Dockerfile` (12 also a compose file)
+and build in place. `warm_images.sh` therefore pulls the `FROM` bases that
+`plan.py` collects. The reasoning is the Terminal-Bench prepull's, unchanged:
+with `--max-retries 0` a registry hiccup is a permanent reward-0 row.
+
+**Resource tiers, not an A/B split.** Terminal-Bench's one-line rule — over 4 GB
+or one CPU goes serial — puts 58 of these 74 in the serial phase. Frontier-Bench
+declares 1–16 CPUs, 2–32 GB memory, and 4 GB–1 TB storage. `plan.py` tiers tasks
+by footprint and gives each tier the concurrency the host can actually support.
+
+**GPU tasks are excluded by default and named.** Four tasks (`exam-pdf-eval`,
+`fp8-rmsnorm-gemm`, `jax-speedrun-gpu`, `math-eval-grader`) declare `gpus = 1`.
+On a GPU-less host they do not error — the container starts, the work is
+impossible, the verifier returns 0.0, and the row is arithmetically identical to
+Stella genuinely failing. Silently that is ~5.4 points of pass rate. They are
+excluded unless `FB_ALLOW_GPU=1` *and* Docker exposes an nvidia runtime, and the
+plan prints every exclusion with its reason.
+
+**A higher budget default ($2.50/trial).** Declared agent timeouts run 30 minutes
+to 8 hours against expert time estimates measured in days. Terminal-Bench's
+$0.60 cap would truncate most of this set and record the truncation as failure.
+`STELLA_BUDGET=` (explicitly empty) means no cap.
+
+## The sentinel's two gates
+
+Stage 1 runs the synthetic fixture and demands reward 1.0 — it is
+oracle-solvable, so anything less is a broken harness. Stage 2 runs the cheapest
+real task and demands only that the trial *ran*: binary verified in-container, a
+status reported, a reward produced, no infrastructure exception. Requiring 1.0
+there would gate on model quality instead of plumbing; a real task that ran and
+scored 0.0 passes stage 2, correctly.
+
+## Known local limits
+
+`plan.py` reports what the host can take. On a 10 GB / 6-CPU Docker VM (a
+typical Mac) it admits **48 of 74** tasks — 4 GPU-excluded and 22 over the memory
+budget. That is a real constraint, not a bug, and it is why a submittable run
+needs a bigger machine: see [SUBMISSION.md](SUBMISSION.md).
+
+The memory exclusion is deliberately conservative. `memory_mb` is a cap Docker
+accepts even when the VM is smaller, so an 8 GB task on a 10 GB daemon usually
+starts — and then swaps and gets OOM-killed partway in, arriving as a reward-0
+row that looks exactly like a genuine failure. Lower `FB_MEMORY_HEADROOM_MB`
+(default 2048) to attempt them anyway; every exclusion is named with its reason,
+so nothing disappears quietly either way.
+
+## Two Harbor majors means the CLI is a moving contract
+
+`--agent-import-path` is 0.6.1 spelling; 0.20.0 folds it into `--agent`, which
+takes either a built-in name or an import path. This lane uses `--agent`, the
+Terminal-Bench lane keeps the old flag, and both are correct for their pin. The
+preflight's `fb_assert_cli_flags` asserts every flag these scripts pass is still
+advertised, so the next rename costs one message rather than a run — it would
+otherwise surface as `No such option` at the first trial, after the venv build,
+dataset download, image warm and preflight had all passed.
+
+The trial result schema, by contrast, is unchanged: `task_name`,
+`agent_result`, `verifier_result.rewards`, and `exception_info` are identical in
+both versions, so the sentinel's gates read the same fields either way.
+
+## Harbor version and the test suite
+
+`bench/harbor_adapter/tests/` passes 132/132 under 0.6.1 and 131/132 under
+0.20.0. The single difference is
+`test_hashes_exact_uploaded_binary_and_records_source_commit`, which asserts
+`agent._harbor_version_value == "0.6.1"` — the Terminal-Bench claim's audited
+constant, correctly failing when a different Harbor is installed. CI runs the
+suite in the 0.6.1 venv and stays green. Nothing in the adapter needed changing:
+every import resolves, the `name`/`install`/`run` interface is unchanged, and
+`setup_venv.sh` re-proves that on every venv build.

bench/evidence/frontier/SUBMISSION.md

@@ -0,0 +1,96 @@
+# Getting Stella onto the Frontier-Bench leaderboard
+
+What the leaderboard requires, what this lane already satisfies, and the two
+things that genuinely block a submission today. Researched 2026-07-31 against
+the frontier-bench repo, its pending leaderboard PR, and the live pipelines on
+the sibling benchmarks; re-check before acting, because the intake is still
+moving.
+
+## The intake is not open yet
+
+Frontier-Bench has no merged self-serve submission pipeline as of 2026-07-31.
+The tooling exists as [PR #1405](https://github.com/harbor-framework/frontier-bench/pull/1405)
+("Add leaderboard package and PR-submit CI for Harbor Hub") and has only been
+exercised end-to-end on a fork. Two "Leaderboard Submission" PRs against
+upstream (#1408 Claude Fable 5, #1419 Kimi K3) were both closed by their own
+author as opened against the wrong repo during that fork test. The identical
+pipeline is already live for `harbor-index` and `terminal-bench-2-1`, so the
+mechanics below are stable in shape even though the door is shut.
+
+**Consequence for us:** the run can be produced now; the PR cannot be filed
+until #1405 merges. Producing the run first is still the right order — it is the
+long pole by a wide margin.
+
+## What a submission is
+
+Two parts. A public job upload, then a one-file PR.
+
+```bash
+harbor run -d frontier-bench/frontier-bench --agent <agent> -m <provider/model> --upload --public
+cd leaderboard && uv run lb submit https://hub.harborframework.com/jobs/<uuid> [...]
+```
+
+`lb submit` writes one JSON per unique (agent, agent version, model, reasoning
+effort) into `leaderboard/submissions/` and opens one PR per file — CI requires
+exactly one added file. The PR carries only job links and metadata; CI re-derives
+every trial from the uploaded job, so trajectories must be public. Verification
+is audit-based, not re-execution: static analysis, then promotion into
+leaderboard-owned copies, then an LLM judge that reviews *every* trajectory for
+reward hacking, then merge.
+
+## Requirements, and where this lane stands
+
+| Requirement | Status |
+|---|---|
+| Dataset pinned to the leaderboard's `DATASET_REF` | **Done.** `env.sh` pins `sha256:97fd2ba3…`, the leaderboard's own ref |
+| Harbor ≥ 0.20.0 | **Done.** This lane pins 0.20.0 |
+| Default execution settings; no timeout or resource overrides | **Done.** Nothing here overrides either; tiering only chooses what runs *concurrently* |
+| All 74 tasks, no subsetting | **Blocked by hardware** — see below |
+| ≥ 5 trials per task | **Supported, off by default.** `FB_ATTEMPTS=5`; default 1 is the dev baseline |
+| `--upload --public` | **Not wired.** Deliberate: uploading publishes trajectories, which is a decision to take explicitly, not a flag to inherit |
+| Agent + model disclosure metadata | Supplied at `lb submit` time, not by this lane |
+
+Note the metric: **accuracy = trials with reward > 0, over all trials**. Not
+reward == 1.0, and errored trials count as reward 0 rather than being dropped.
+Infrastructure flakiness therefore lands directly in the score, which is exactly
+why `warm_images.sh` and the preflight exist.
+
+## The two real blockers
+
+**1. GPUs.** Four tasks need one, and the benchmark authors ran them on a single
+H100 each. Subsetting is not allowed for a submission, so those four cannot be
+excluded the way the dev baseline excludes them — they must actually run.
+
+**2. Host size.** On a 10 GB / 6-CPU Docker VM, `plan.py` admits 48 of 74 tasks;
+22 more are simply over the memory budget. The full set declares up to 32 GB
+memory, 16 CPUs, and ~1.9 TB of storage in aggregate.
+
+Both point the same way: **a submission run does not happen on a Mac.** The
+benchmark's own guidance is Modal (`uv tool install 'harbor[modal]'`, `--env
+modal`), with Daytona as an alternative that recently added GPU support. Harbor's
+repo CI defaults to `env: modal`.
+
+## Cost, honestly
+
+74 tasks × 5 trials = 370 trials, against declared agent timeouts averaging
+around two hours. At this lane's $2.50/trial default that is roughly $925 in
+model spend before Modal compute, and the budget cap would truncate the longer
+tasks — for a leaderboard run you would want `STELLA_BUDGET=` (uncapped), which
+removes the ceiling on that estimate. This is a four-figure decision, not a
+weekend experiment, and it should be preregistered like the Terminal-Bench claim
+was.
+
+## Suggested order
+
+1. Land this lane; run the sentinel locally to prove the plumbing on the new
+   Harbor pin. Cheap.
+2. Run the dev baseline on the 48 locally-runnable tasks at one attempt. Cheap
+   enough to be worth it, and it tells you where Stella actually lands before
+   you spend four figures.
+3. Only if that number justifies it: preregister, move to Modal with GPU access,
+   run all 74 at five attempts uncapped with `--upload --public`.
+4. File the PR once #1405 merges.
+
+Steps 1 and 2 are what this directory is for. Step 3 needs a Modal environment
+this lane does not yet configure — `primary.sh` hardcodes `--env docker`, and
+adding a Modal path is the next piece of work, not something already done.