diff --git a/.github/workflows/bench.yml b/.github/workflows/bench.yml index d82ebbff0..34c21e62c 100644 --- a/.github/workflows/bench.yml +++ b/.github/workflows/bench.yml @@ -58,7 +58,7 @@ jobs: exit 0 fi if git diff --name-only "$BASE_SHA"...HEAD \ - | grep -Eq '^(bench/harbor_adapter/|bench/terminal_bench_analysis/|\.github/workflows/bench\.yml$)'; then + | grep -Eq '^(bench/harbor_adapter/|bench/terminal_bench_analysis/|bench/evidence/frontier/|\.github/workflows/bench\.yml$)'; then echo "changed=true" >> "$GITHUB_OUTPUT" else echo "changed=false" >> "$GITHUB_OUTPUT" @@ -83,3 +83,11 @@ jobs: run: | uv sync --locked --extra dev uv run --no-sync pytest -q + + # The Frontier-Bench run planner. No project to sync and no Harbor to + # install: plan.py imports nothing outside the standard library, which is + # deliberate — the piece that decides which tasks are allowed to run + # should not need the benchmark harness to be testable. + - name: frontier planner — pytest + if: steps.scope.outputs.changed == 'true' + run: uv run --with pytest --no-project pytest -q bench/evidence/frontier/tests diff --git a/bench/README.md b/bench/README.md index 9af409ea5..2e6578eba 100644 --- a/bench/README.md +++ b/bench/README.md @@ -5,11 +5,12 @@ with other agents, and standalone. Everything here is **BYOK** (bring your own key), makes **no phone-home**, and never hard-codes a secret. Claim runs use a secure launcher that consumes the selected credential before Harbor starts. -Four entry points: +Five entry points: | Path | What it does | Needs | |---|---|---| | [`harbor_adapter/`](harbor_adapter/) | A Harbor *installed-agent* adapter — run Stella on Terminal-Bench 2.x / SWE-bench in the same container + verifier as Claude Code, Codex CLI, Terminus, etc. | Docker, `harbor`, a provider key | +| [`evidence/frontier/`](evidence/frontier/) | **Frontier-Bench** — Harbor's successor to Terminal-Bench, 74 tasks over seven domains. Same adapter, unmodified; a separate Harbor pin, a resource-aware schedule, and GPU tasks excluded by name rather than scored as zeros. | Docker, a provider key (a *submittable* run needs GPUs — see its [SUBMISSION.md](evidence/frontier/SUBMISSION.md)) | | [`run_swebench.py`](run_swebench.py) | A standalone SWE-bench *prediction* harness — clone each instance, run Stella, emit the official predictions JSONL. No Harbor. | `git`, a provider key (Docker only for the official scoring step) | | [`loop-bench/`](loop-bench/) | A cheap **turn-loop + context-query correctness** harness: runs N tasks on a flash-tier model, budget-capped, and reports loop health (silent-death / zero-work / stuck-loop) and `project_overview`/`graph_query` adoption — the signals the pass-rate number hides. | `cargo`, Docker, `harbor`, a key | | [`smoke/smoke_test.py`](smoke/smoke_test.py) | An **offline, zero-cost** self-test of the adapter wiring for CI. | just the built `stella` binary | diff --git a/bench/evidence/frontier/README.md b/bench/evidence/frontier/README.md new file mode 100644 index 000000000..a33dcb6ba --- /dev/null +++ b/bench/evidence/frontier/README.md @@ -0,0 +1,107 @@ +# Frontier-Bench (dev-baseline lane) + +Runs Stella on [Frontier-Bench](https://github.com/harbor-framework/frontier-bench) +— Harbor's successor to Terminal-Bench, 74 tasks across seven domains, where the +best published agents clear roughly a third. + +**The adapter is unchanged.** `stella_harbor:StellaAgent` runs this benchmark +exactly as it runs Terminal-Bench 2.1, with no edit to anything under +`bench/harbor_adapter/`. That was a design constraint, not a happy accident: the +adapter's Python tree is digest-frozen for the audited claim path, so a new +benchmark has to earn its place without perturbing it. Everything specific to +Frontier-Bench lives in this directory. + +## Run it + +```bash +export TB_REPO=/path/to/stella TB_ROOT=/abs/scratch OPENROUTER_API_KEY=... +bench/evidence/run/build_sut.sh # shared with the Terminal-Bench lane +bench/evidence/frontier/setup_venv.sh # this lane's Harbor pin +bench/evidence/frontier/fetch_dataset.sh # pinned dataset + resource plan +bench/evidence/frontier/warm_images.sh # pull the base images the builds need +bench/evidence/frontier/sentinel.sh # two cheap trials, end to end +bench/evidence/frontier/primary.sh small fb-small-01 +``` + +## What differs from Terminal-Bench, and why each difference exists + +**A second Harbor pin (0.20.0), not the audited 0.6.1.** Every one of the 74 +tasks declares `environment_mode = "separate"` under `[verifier]`. Harbor 0.6.1 +has no such field, and pydantic's default `extra="ignore"` drops it *silently* — +0.6.1 runs the whole set with the verifier sharing the agent's container instead +of the separate one the task asked for, produces rewards, and warns about +nothing. Plausible numbers answering the wrong question are worse than an error. +This lane gets its own venv; `bench/harbor_adapter/.venv` stays on 0.6.1 so +Terminal-Bench reruns keep measuring what they measured. + +**Base images are warmed, not task images pulled.** No Frontier-Bench task has a +`docker_image`; all 74 ship an `environment/Dockerfile` (12 also a compose file) +and build in place. `warm_images.sh` therefore pulls the `FROM` bases that +`plan.py` collects. The reasoning is the Terminal-Bench prepull's, unchanged: +with `--max-retries 0` a registry hiccup is a permanent reward-0 row. + +**Resource tiers, not an A/B split.** Terminal-Bench's one-line rule — over 4 GB +or one CPU goes serial — puts 58 of these 74 in the serial phase. Frontier-Bench +declares 1–16 CPUs, 2–32 GB memory, and 4 GB–1 TB storage. `plan.py` tiers tasks +by footprint and gives each tier the concurrency the host can actually support. + +**GPU tasks are excluded by default and named.** Four tasks (`exam-pdf-eval`, +`fp8-rmsnorm-gemm`, `jax-speedrun-gpu`, `math-eval-grader`) declare `gpus = 1`. +On a GPU-less host they do not error — the container starts, the work is +impossible, the verifier returns 0.0, and the row is arithmetically identical to +Stella genuinely failing. Silently that is ~5.4 points of pass rate. They are +excluded unless `FB_ALLOW_GPU=1` *and* Docker exposes an nvidia runtime, and the +plan prints every exclusion with its reason. + +**A higher budget default ($2.50/trial).** Declared agent timeouts run 30 minutes +to 8 hours against expert time estimates measured in days. Terminal-Bench's +$0.60 cap would truncate most of this set and record the truncation as failure. +`STELLA_BUDGET=` (explicitly empty) means no cap. + +## The sentinel's two gates + +Stage 1 runs the synthetic fixture and demands reward 1.0 — it is +oracle-solvable, so anything less is a broken harness. Stage 2 runs the cheapest +real task and demands only that the trial *ran*: binary verified in-container, a +status reported, a reward produced, no infrastructure exception. Requiring 1.0 +there would gate on model quality instead of plumbing; a real task that ran and +scored 0.0 passes stage 2, correctly. + +## Known local limits + +`plan.py` reports what the host can take. On a 10 GB / 6-CPU Docker VM (a +typical Mac) it admits **48 of 74** tasks — 4 GPU-excluded and 22 over the memory +budget. That is a real constraint, not a bug, and it is why a submittable run +needs a bigger machine: see [SUBMISSION.md](SUBMISSION.md). + +The memory exclusion is deliberately conservative. `memory_mb` is a cap Docker +accepts even when the VM is smaller, so an 8 GB task on a 10 GB daemon usually +starts — and then swaps and gets OOM-killed partway in, arriving as a reward-0 +row that looks exactly like a genuine failure. Lower `FB_MEMORY_HEADROOM_MB` +(default 2048) to attempt them anyway; every exclusion is named with its reason, +so nothing disappears quietly either way. + +## Two Harbor majors means the CLI is a moving contract + +`--agent-import-path` is 0.6.1 spelling; 0.20.0 folds it into `--agent`, which +takes either a built-in name or an import path. This lane uses `--agent`, the +Terminal-Bench lane keeps the old flag, and both are correct for their pin. The +preflight's `fb_assert_cli_flags` asserts every flag these scripts pass is still +advertised, so the next rename costs one message rather than a run — it would +otherwise surface as `No such option` at the first trial, after the venv build, +dataset download, image warm and preflight had all passed. + +The trial result schema, by contrast, is unchanged: `task_name`, +`agent_result`, `verifier_result.rewards`, and `exception_info` are identical in +both versions, so the sentinel's gates read the same fields either way. + +## Harbor version and the test suite + +`bench/harbor_adapter/tests/` passes 132/132 under 0.6.1 and 131/132 under +0.20.0. The single difference is +`test_hashes_exact_uploaded_binary_and_records_source_commit`, which asserts +`agent._harbor_version_value == "0.6.1"` — the Terminal-Bench claim's audited +constant, correctly failing when a different Harbor is installed. CI runs the +suite in the 0.6.1 venv and stays green. Nothing in the adapter needed changing: +every import resolves, the `name`/`install`/`run` interface is unchanged, and +`setup_venv.sh` re-proves that on every venv build. diff --git a/bench/evidence/frontier/SUBMISSION.md b/bench/evidence/frontier/SUBMISSION.md new file mode 100644 index 000000000..0a894e9d3 --- /dev/null +++ b/bench/evidence/frontier/SUBMISSION.md @@ -0,0 +1,96 @@ +# Getting Stella onto the Frontier-Bench leaderboard + +What the leaderboard requires, what this lane already satisfies, and the two +things that genuinely block a submission today. Researched 2026-07-31 against +the frontier-bench repo, its pending leaderboard PR, and the live pipelines on +the sibling benchmarks; re-check before acting, because the intake is still +moving. + +## The intake is not open yet + +Frontier-Bench has no merged self-serve submission pipeline as of 2026-07-31. +The tooling exists as [PR #1405](https://github.com/harbor-framework/frontier-bench/pull/1405) +("Add leaderboard package and PR-submit CI for Harbor Hub") and has only been +exercised end-to-end on a fork. Two "Leaderboard Submission" PRs against +upstream (#1408 Claude Fable 5, #1419 Kimi K3) were both closed by their own +author as opened against the wrong repo during that fork test. The identical +pipeline is already live for `harbor-index` and `terminal-bench-2-1`, so the +mechanics below are stable in shape even though the door is shut. + +**Consequence for us:** the run can be produced now; the PR cannot be filed +until #1405 merges. Producing the run first is still the right order — it is the +long pole by a wide margin. + +## What a submission is + +Two parts. A public job upload, then a one-file PR. + +```bash +harbor run -d frontier-bench/frontier-bench --agent -m --upload --public +cd leaderboard && uv run lb submit https://hub.harborframework.com/jobs/ [...] +``` + +`lb submit` writes one JSON per unique (agent, agent version, model, reasoning +effort) into `leaderboard/submissions/` and opens one PR per file — CI requires +exactly one added file. The PR carries only job links and metadata; CI re-derives +every trial from the uploaded job, so trajectories must be public. Verification +is audit-based, not re-execution: static analysis, then promotion into +leaderboard-owned copies, then an LLM judge that reviews *every* trajectory for +reward hacking, then merge. + +## Requirements, and where this lane stands + +| Requirement | Status | +|---|---| +| Dataset pinned to the leaderboard's `DATASET_REF` | **Done.** `env.sh` pins `sha256:97fd2ba3…`, the leaderboard's own ref | +| Harbor ≥ 0.20.0 | **Done.** This lane pins 0.20.0 | +| Default execution settings; no timeout or resource overrides | **Done.** Nothing here overrides either; tiering only chooses what runs *concurrently* | +| All 74 tasks, no subsetting | **Blocked by hardware** — see below | +| ≥ 5 trials per task | **Supported, off by default.** `FB_ATTEMPTS=5`; default 1 is the dev baseline | +| `--upload --public` | **Not wired.** Deliberate: uploading publishes trajectories, which is a decision to take explicitly, not a flag to inherit | +| Agent + model disclosure metadata | Supplied at `lb submit` time, not by this lane | + +Note the metric: **accuracy = trials with reward > 0, over all trials**. Not +reward == 1.0, and errored trials count as reward 0 rather than being dropped. +Infrastructure flakiness therefore lands directly in the score, which is exactly +why `warm_images.sh` and the preflight exist. + +## The two real blockers + +**1. GPUs.** Four tasks need one, and the benchmark authors ran them on a single +H100 each. Subsetting is not allowed for a submission, so those four cannot be +excluded the way the dev baseline excludes them — they must actually run. + +**2. Host size.** On a 10 GB / 6-CPU Docker VM, `plan.py` admits 48 of 74 tasks; +22 more are simply over the memory budget. The full set declares up to 32 GB +memory, 16 CPUs, and ~1.9 TB of storage in aggregate. + +Both point the same way: **a submission run does not happen on a Mac.** The +benchmark's own guidance is Modal (`uv tool install 'harbor[modal]'`, `--env +modal`), with Daytona as an alternative that recently added GPU support. Harbor's +repo CI defaults to `env: modal`. + +## Cost, honestly + +74 tasks × 5 trials = 370 trials, against declared agent timeouts averaging +around two hours. At this lane's $2.50/trial default that is roughly $925 in +model spend before Modal compute, and the budget cap would truncate the longer +tasks — for a leaderboard run you would want `STELLA_BUDGET=` (uncapped), which +removes the ceiling on that estimate. This is a four-figure decision, not a +weekend experiment, and it should be preregistered like the Terminal-Bench claim +was. + +## Suggested order + +1. Land this lane; run the sentinel locally to prove the plumbing on the new + Harbor pin. Cheap. +2. Run the dev baseline on the 48 locally-runnable tasks at one attempt. Cheap + enough to be worth it, and it tells you where Stella actually lands before + you spend four figures. +3. Only if that number justifies it: preregister, move to Modal with GPU access, + run all 74 at five attempts uncapped with `--upload --public`. +4. File the PR once #1405 merges. + +Steps 1 and 2 are what this directory is for. Step 3 needs a Modal environment +this lane does not yet configure — `primary.sh` hardcodes `--env docker`, and +adding a Modal path is the next piece of work, not something already done. diff --git a/bench/evidence/frontier/env.sh b/bench/evidence/frontier/env.sh new file mode 100755 index 000000000..3c4751726 --- /dev/null +++ b/bench/evidence/frontier/env.sh @@ -0,0 +1,172 @@ +#!/bin/bash +# Shared environment + preflight for a Frontier-Bench development-baseline run. +# Sourced by every other script here. Never echoes a credential. +# +# Frontier-Bench is Harbor's successor to Terminal-Bench: 74 tasks across seven +# domains, where Terminal-Bench 2.1's ceiling had already been cleared. The +# agent side is identical — the same `stella_harbor:StellaAgent` installed-agent +# runs both, unmodified — so everything specific to this benchmark lives in this +# directory and nothing under `bench/harbor_adapter/` changes. That is +# deliberate: the adapter's Python tree is digest-frozen for the audited claim +# path, and a benchmark that only differs in *which dataset and how it is +# scheduled* must not perturb it. +# +# The shared half — the glibc floor, the target triple, the SUT path, the +# portability assert, the forced amd64 platform — is sourced from the +# Terminal-Bench lane rather than copied, so the two cannot drift. The halves +# that genuinely differ are re-declared below with the reason. +set -uo pipefail + +FB_HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# The frozen dataset, pinned by content digest. An unversioned name is not a +# freeze, and `@latest` least of all: Frontier-Bench is explicitly a *moving* +# benchmark — the announcement's whole point is that tasks are added and revised +# continuously — so the name that resolved to 74 tasks today resolves to a +# different set next month. Terminal-Bench could get away with a name because it +# had stopped moving. This one cannot. +# +# This exact digest is the leaderboard's own `DATASET_REF`. Resolving the +# dataset by name today yields +# sha256:63f363a191f0a0429fd1c5b318080616bab839473ce27e39f44868d327b03a89 +# instead, which exports a *byte-identical* 74-task tree — verified by diffing +# both exports, no file differs. They are two version records over the same +# content. The leaderboard's static analysis compares the ref *string* a job +# recorded, not the tasks it contains, so a run pinned to the equally-valid +# other digest is rejected on submission while being, in substance, the same +# run. Pin the string CI is looking for. +# +# Re-resolve for a newer snapshot with: +# harbor download frontier-bench/frontier-bench --export -o # prints it +export FB_DATASET="${FB_DATASET:-frontier-bench/frontier-bench@sha256:97fd2ba3aabdda16823a1a8ea695a3875e50e800caa60b450686deedc7171763}" +export FB_TASK_PREFIX="frontier-bench" +export FB_TASK_COUNT="${FB_TASK_COUNT:-74}" + +# Trials per task. One is right for a development baseline, where the question +# is "does the harness work and roughly where do we land". It is *not* a +# leaderboard submission: that requires every task covered at five or more +# trials each, and CI counts them. Raising this multiplies cost by the same +# factor, so it is a deliberate act, not a default. +export FB_ATTEMPTS="${FB_ATTEMPTS:-1}" +export FB_SUBMISSION_MIN_ATTEMPTS=5 + +# Frontier-Bench tasks run far longer than Terminal-Bench ones: declared agent +# timeouts run from 30 minutes to 8 hours, with a 2-hour mode, against expert +# time estimates measured in whole days. Terminal-Bench's $0.60 per-trial cap +# would truncate most of this set mid-task and report the truncation as a task +# failure, which is a measurement error, not a thrifty default. `-` and not `:-` +# for the same reason as the Terminal-Bench lane: an explicitly empty value +# stays empty and means *no cap*, which is the posture a leaderboard-facing run +# has to be able to take. +export STELLA_BUDGET="${STELLA_BUDGET-2.50}" + +# What the sourced Terminal-Bench env would otherwise default. Set before the +# source so its `${TB_DATASET:-...}` sees a value and leaves it alone; nothing +# here reads TB_DATASET afterwards, but leaving it pointing at Terminal-Bench +# while this lane runs Frontier-Bench is the kind of stale-variable trap that +# ends up in a log line and then in a claim. +export TB_DATASET="$FB_DATASET" + +# shellcheck source=bench/evidence/run/env.sh +source "$FB_HERE/../run/env.sh" + +# --- Everything below overrides the Terminal-Bench lane on purpose. --- + +# Harbor 0.6.1 is the Terminal-Bench claim's audited constant. It cannot be this +# lane's constant. Every one of the 74 Frontier-Bench tasks declares +# `environment_mode = "separate"` under `[verifier]`, and 0.6.1's VerifierConfig +# has no such field — pydantic's default `extra="ignore"` drops it silently, so +# 0.6.1 runs the whole set with the verifier sharing the agent's container +# instead of the separate one the task asked for. It does not warn, and the run +# still produces rewards. Those rewards answer a different question than the +# leaderboard's, which is the worst possible failure mode: plausible numbers. +# +# So this lane pins its own Harbor, in its own virtualenv, and asserts it. The +# Terminal-Bench venv at bench/harbor_adapter/.venv is left untouched. +export FB_HARBOR_VERSION="${FB_HARBOR_VERSION:-0.20.0}" +export VENV="$FB_HERE/.venv" +export PATH="$VENV/bin:$PATH" +export PYTHONPATH="$TB_REPO/bench/harbor_adapter" + +# Distinct from the Terminal-Bench lane's, so the two can share one TB_ROOT +# without one run's dataset export, resource plan, or job tree overwriting the +# other's. +export DATASET_DIR="$TB_ROOT/dataset-frontier" +export JOBS="$TB_ROOT/jobs-frontier" +export FB_PLAN="$TB_ROOT/frontier-plan.json" +mkdir -p "$JOBS" + +# Four tasks (exam-pdf-eval, fp8-rmsnorm-gemm, jax-speedrun-gpu, +# math-eval-grader) declare `gpus = 1`. On a host without a GPU they do not +# error in any way that is distinguishable from Stella failing: the container +# comes up, the work is impossible, the verifier returns 0.0, and the row is +# arithmetically identical to a genuine miss. Terminal-Bench had no such tasks +# and so no such hazard. +# +# The default is therefore to exclude them and say so, loudly, in the score's +# denominator — not to include them and quietly absorb four guaranteed zeros. +# Set FB_ALLOW_GPU=1 only on a host that actually has one; `plan.py` checks. +export FB_ALLOW_GPU="${FB_ALLOW_GPU:-0}" + +# Memory the plan leaves to the Docker daemon itself rather than handing to a +# task. A task declaring more than what remains is excluded as unrunnable. +# +# This is deliberately conservative: `memory_mb` is a cap Docker will happily +# accept on a smaller VM, so an 8 GB task on a 10 GB daemon usually *starts* — +# and then competes with the daemon, swaps, and gets OOM-killed partway in, +# which arrives as a reward-0 row indistinguishable from a genuine failure. +# Excluding it and saying so is the honest version of the same outcome. Lower +# this to attempt those tasks anyway; the exclusion list names every one. +export FB_MEMORY_HEADROOM_MB="${FB_MEMORY_HEADROOM_MB:-2048}" + +# Assert `harbor run` still advertises every flag these scripts pass it. +# +# Running two Harbor major versions in one repo means the CLI is a moving +# contract, and it has already moved once: 0.6.1's `--agent-import-path` was +# folded into `--agent` by 0.20.0, which accepts either a built-in name or an +# import path. The Terminal-Bench lane still passes the old spelling and is +# still correct to, because it is pinned to 0.6.1. This lane would have died on +# `No such option` at the first trial of the first run. +# +# Checking the help text costs one subprocess and turns that into one message. +fb_assert_cli_flags() { + local help missing="" + # Scrub ANSI first. Harbor renders help through Rich, which styles each + # option individually and leaves escape sequences *inside* the flag names — + # a literal search for `--env` over the raw 42 KB finds nothing, so an + # unscrubbed version of this check reports every flag missing and fails + # closed on a perfectly good CLI. + help="$(harbor run --help 2>&1 | LC_ALL=C sed $'s/\033\[[0-9;?]*[a-zA-Z]//g')" || + { echo "FATAL: \`harbor run --help\` failed"; return 1; } + for flag in --env --dataset --path --agent --model --job-name --jobs-dir \ + --include-task-name --n-attempts --n-concurrent --max-retries; do + case "$help" in *"$flag"*) ;; *) missing="$missing $flag" ;; esac + done + test -z "$missing" || { + echo "FATAL: harbor $(harbor --version) does not advertise:$missing" + echo " The CLI contract moved. Update this lane's scripts before running." + return 1; } +} + +fb_preflight() { + test -n "${OPENROUTER_API_KEY:-}" || { echo "FATAL: OPENROUTER_API_KEY unset"; return 1; } + test -x "$STELLA_BINARY" || { echo "FATAL: no SUT binary at $STELLA_BINARY (run ../run/build_sut.sh)"; return 1; } + # Reused, not reimplemented: one definition of the glibc floor for both lanes. + assert_portable_binary || return 1 + test "${#STELLA_SOURCE_COMMIT}" = 40 || { echo "FATAL: STELLA_SOURCE_COMMIT is not a full SHA"; return 1; } + test "$(command -v harbor)" = "$VENV/bin/harbor" || { + echo "FATAL: wrong harbor on PATH — expected $VENV/bin/harbor" + echo " Create this lane's venv with: bench/evidence/frontier/setup_venv.sh" + return 1; } + test "$(harbor --version)" = "$FB_HARBOR_VERSION" || { + echo "FATAL: harbor $(harbor --version) != $FB_HARBOR_VERSION" + echo " 0.6.1 silently ignores this dataset's verifier environment_mode." + return 1; } + fb_assert_cli_flags || return 1 + docker info >/dev/null 2>&1 || { echo "FATAL: docker unreachable"; return 1; } +} + +# The Terminal-Bench lane's `preflight` is still defined and still asserts +# harbor 0.6.1, which is false here. Point the familiar name at this lane's +# check so a script that calls `preflight` out of habit fails safe. +preflight() { fb_preflight "$@"; } diff --git a/bench/evidence/frontier/fetch_dataset.sh b/bench/evidence/frontier/fetch_dataset.sh new file mode 100755 index 000000000..1bed48757 --- /dev/null +++ b/bench/evidence/frontier/fetch_dataset.sh @@ -0,0 +1,21 @@ +#!/bin/bash +# Fetch the pinned dataset and build the resource-aware run plan. +set -euo pipefail +source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/env.sh" + +test "$(command -v harbor)" = "$VENV/bin/harbor" || { + echo "FATAL: run setup_venv.sh first"; exit 1; } + +harbor download "$FB_DATASET" --export -o "$DATASET_DIR" + +# Shell glob, not fd/find: these scripts run on whatever bench host is handy, +# and the only tool they may assume is the shell itself. +got=0 +for f in "$DATASET_DIR"/*/*/task.toml; do [ -f "$f" ] && got=$((got+1)); done +test "$got" = "$FB_TASK_COUNT" || { + echo "FATAL: dataset exported $got tasks, expected $FB_TASK_COUNT." + echo " The digest pin and FB_TASK_COUNT disagree — one of them is stale." + exit 1; } + +python3 "$(dirname "${BASH_SOURCE[0]}")/plan.py" "$DATASET_DIR" \ + -o "$FB_PLAN" --images-out "$TB_ROOT/frontier-images.txt" diff --git a/bench/evidence/frontier/plan.py b/bench/evidence/frontier/plan.py new file mode 100755 index 000000000..9dc2b7769 --- /dev/null +++ b/bench/evidence/frontier/plan.py @@ -0,0 +1,284 @@ +#!/usr/bin/env python3 +"""Turn an exported Frontier-Bench dataset into a runnable schedule. + +Terminal-Bench's dev-baseline runner classified tasks with one line — anything +over 4 GB or one CPU went into a serial phase — because Terminal-Bench 2.1 was +homogeneous enough for that to work. Frontier-Bench is not. Its 74 tasks declare +1 to 16 CPUs, 2 to 32 GB of memory, 4 GB to 1 TB of storage, and four of them +want a GPU. Applying the Terminal-Bench rule puts 58 of 74 in the serial phase, +which is not a schedule so much as a surrender. + +Three things this file exists to prevent, each of which produces a *plausible +wrong number* rather than an error: + +1. A GPU task on a GPU-less host. The container starts, the work is impossible, + the verifier returns 0.0. That row is arithmetically identical to Stella + genuinely failing the task. Four of them silently cost ~5.4 points of pass + rate. They are excluded by default and named in the plan, so the denominator + is a stated choice instead of an accident. + +2. A task that cannot fit on the host at all. Same failure shape: not an error, + a zero. Excluded, named, and counted separately from tasks that ran. + +3. Oversubscription. Two 16 GB tasks scheduled together on a 16 GB Docker VM do + not fail cleanly — they fail as an unrelated-looking scatter of OOM kills, + build timeouts, and verifier errors spread across whichever tasks happened to + be running. Tasks are tiered by footprint and each tier gets the concurrency + its own headroom allows. + +It also collects the base images to warm. Terminal-Bench tasks named a prebuilt +``docker_image``; every Frontier-Bench task instead ships an +``environment/Dockerfile`` (12 of them a compose file too) and builds. The thing +to pull ahead of a measured run is therefore each Dockerfile's ``FROM`` bases, +not a task image — same reasoning as the Terminal-Bench prepull, which is that +with ``--max-retries 0`` a registry hiccup becomes a permanent reward-0 row, and +image availability is a precondition of the measurement rather than a term in +it. + +Usage:: + + python3 plan.py -o [--allow-gpu] [--headroom-mb N] +""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import shutil +import subprocess +import sys +from pathlib import Path + +if sys.version_info < (3, 11): # tomllib is what makes this parse rather than guess + sys.exit("FATAL: plan.py needs Python 3.11+ (source env.sh so the lane venv is on PATH)") + +import tomllib + +# Tier boundaries are memory ceilings in MB. A task lands in the first tier it +# fits. The names are load-bearing only as labels; the concurrency each tier +# gets is computed from the host, not from the name. +TIERS: tuple[tuple[str, int], ...] = ( + ("small", 2048), + ("medium", 4096), + ("large", 8192), + ("xlarge", 1 << 30), +) + +# Concurrency is capped regardless of how much headroom arithmetic allows. +# Docker's own build and network machinery, not task memory, is the next +# bottleneck past this point, and a stuck build is far more expensive to debug +# than a slightly longer run. +MAX_CONCURRENCY = 4 + +_FROM_RE = re.compile(r"^\s*FROM\s+(?:--\S+\s+)*(\S+)(?:\s+AS\s+(\S+))?", re.IGNORECASE) +_COMPOSE_IMAGE_RE = re.compile(r"^\s*image:\s*[\"']?([^\s\"']+)", re.MULTILINE) + + +# `.Path` and not the runtime entry itself: Go's `index` returns the zero value +# for a missing map key, and a zero-value struct is truthy in a template, so +# `{{if index .Runtimes "nvidia"}}` reports a GPU on every host alive. The +# registered binary's path is empty exactly when the runtime is absent. +_HOST_FORMAT = '{{.MemTotal}}|{{.NCPU}}|{{(index .Runtimes "nvidia").Path}}' + + +def _host_capacity() -> dict: + """What the Docker daemon says it can actually give a container.""" + cap = {"memory_mb": None, "cpus": None, "gpu": False, "source": "unavailable"} + try: + out = subprocess.run( + ["docker", "info", "--format", _HOST_FORMAT], + capture_output=True, text=True, timeout=60, check=True, + ).stdout.strip() + mem, cpus, nvidia_path = out.split("|", 2) + cap["memory_mb"] = int(mem) // (1024 * 1024) + cap["cpus"] = int(cpus) + # A GPU on the host that Docker cannot pass through is, for our + # purposes, no GPU at all. + cap["gpu"] = bool(nvidia_path.strip()) + cap["source"] = "docker info" + except Exception as exc: # noqa: BLE001 - any failure means "cannot verify" + cap["error"] = f"{type(exc).__name__}: {exc}" + return cap + + +def _read_task(task_dir: Path) -> dict: + cfg = tomllib.loads((task_dir / "task.toml").read_text()) + env = cfg.get("environment", {}) or {} + name = (cfg.get("task", {}) or {}).get("name") or task_dir.name + meta = cfg.get("metadata", {}) or {} + compose = [ + p for p in (task_dir / "environment").glob("docker-compose.y*ml") + ] if (task_dir / "environment").is_dir() else [] + return { + "name": name, + "slug": task_dir.name, + "cpus": int(env.get("cpus", 1)), + "memory_mb": int(env.get("memory_mb", 2048)), + "storage_mb": int(env.get("storage_mb", 0)), + "gpus": int(env.get("gpus", 0)), + "agent_timeout_sec": float((cfg.get("agent", {}) or {}).get("timeout_sec", 0) or 0), + "category": meta.get("category", "?"), + "compose": bool(compose), + "base_images": _base_images(task_dir, compose), + } + + +def _base_images(task_dir: Path, compose: list[Path]) -> list[str]: + """Every externally-pulled image this task's build depends on. + + Multi-stage builds name intermediate stages in later ``FROM`` lines + (``FROM builder``); those are local and must not be pulled. A stage name has + no registry path, tag, or digest, so anything without ``/``, ``:`` or ``@`` + is dropped — along with ``scratch``, which is not an image at all. + """ + images: set[str] = set() + stages: set[str] = set() + for dockerfile in sorted(task_dir.rglob("Dockerfile*")): + text = dockerfile.read_text(errors="replace") + for line in text.splitlines(): + m = _FROM_RE.match(line) + if not m: + continue + ref, alias = m.group(1), m.group(2) + if alias: + stages.add(alias.lower()) + images.add(ref) + for composefile in compose: + images.update(_COMPOSE_IMAGE_RE.findall(composefile.read_text(errors="replace"))) + + resolved = set() + for ref in images: + low = ref.lower() + if low in stages or low == "scratch" or ref.startswith("$"): + continue + if not any(c in ref for c in ("/", ":", "@")): + continue # a bare word with no registry coordinates is a build stage + resolved.add(ref) + return sorted(resolved) + + +def _tier(mem_mb: int) -> str: + for name, ceiling in TIERS: + if mem_mb <= ceiling: + return name + return TIERS[-1][0] + + +def build_plan(dataset_dir: Path, *, allow_gpu: bool, memory_headroom_mb: int) -> dict: + roots = sorted(p.parent for p in dataset_dir.glob("*/*/task.toml")) + if not roots: + roots = sorted(p.parent for p in dataset_dir.glob("*/task.toml")) + if not roots: + sys.exit(f"FATAL: no task.toml under {dataset_dir}") + + tasks = [_read_task(p) for p in roots] + host = _host_capacity() + host_mem = host.get("memory_mb") + host_cpus = host.get("cpus") + + usable_mem = (host_mem - memory_headroom_mb) if host_mem else None + excluded: list[dict] = [] + runnable: list[dict] = [] + + for t in tasks: + if t["gpus"] > 0 and not allow_gpu: + excluded.append({**t, "reason": "requires a GPU; FB_ALLOW_GPU is not set"}) + continue + if t["gpus"] > 0 and not host["gpu"]: + excluded.append({**t, "reason": "requires a GPU; Docker exposes no nvidia runtime"}) + continue + if usable_mem is not None and t["memory_mb"] > usable_mem: + excluded.append({ + **t, + "reason": f"needs {t['memory_mb']} MB; host offers {usable_mem} MB after headroom", + }) + continue + if host_cpus is not None and t["cpus"] > host_cpus: + excluded.append({**t, "reason": f"needs {t['cpus']} CPUs; host has {host_cpus}"}) + continue + runnable.append(t) + + tiers: dict[str, dict] = {} + for t in runnable: + tiers.setdefault(_tier(t["memory_mb"]), {"tasks": []})["tasks"].append(t["slug"]) + + for name, tier in tiers.items(): + members = [t for t in runnable if t["slug"] in set(tier["tasks"])] + peak_mem = max(t["memory_mb"] for t in members) + peak_cpus = max(t["cpus"] for t in members) + by_mem = (usable_mem // peak_mem) if usable_mem else 1 + by_cpu = (host_cpus // peak_cpus) if host_cpus else 1 + tier["concurrency"] = max(1, min(MAX_CONCURRENCY, by_mem, by_cpu)) + tier["peak_memory_mb"] = peak_mem + tier["peak_cpus"] = peak_cpus + tier["tasks"].sort() + + images = sorted({img for t in runnable for img in t["base_images"]}) + + # Storage is advisory: the declared figure is a ceiling the task may never + # approach, and on macOS the Docker VM's free space is not visible from the + # host anyway. Reporting it beats pretending to enforce it. + free_mb = shutil.disk_usage(dataset_dir).free // (1024 * 1024) + + return { + "dataset_dir": str(dataset_dir), + "host": host, + "memory_headroom_mb": memory_headroom_mb, + "allow_gpu": allow_gpu, + "counts": { + "total": len(tasks), + "runnable": len(runnable), + "excluded": len(excluded), + }, + "tiers": {k: tiers[k] for k, _ in TIERS if k in tiers}, + "excluded": sorted(excluded, key=lambda t: t["slug"]), + "base_images": images, + "declared_storage_mb": sum(t["storage_mb"] for t in runnable), + "host_free_mb_at_dataset": free_mb, + "runnable_tasks": sorted(t["slug"] for t in runnable), + } + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("dataset_dir", type=Path) + ap.add_argument("-o", "--output", type=Path, required=True) + ap.add_argument("--allow-gpu", action="store_true", + default=os.environ.get("FB_ALLOW_GPU") == "1") + # Memory, not storage. The two were one knob in the first draft of this + # file and a 20 GB *storage* headroom subtracted from a 10 GB Docker VM's + # *memory* excluded all 74 tasks with a negative budget — the plan was + # empty and said so politely. + ap.add_argument("--memory-headroom-mb", type=int, + default=int(os.environ.get("FB_MEMORY_HEADROOM_MB", "2048"))) + ap.add_argument("--images-out", type=Path, default=None) + args = ap.parse_args() + + plan = build_plan(args.dataset_dir, allow_gpu=args.allow_gpu, + memory_headroom_mb=args.memory_headroom_mb) + args.output.write_text(json.dumps(plan, indent=1) + "\n") + if args.images_out: + args.images_out.write_text("\n".join(plan["base_images"]) + "\n") + + host = plan["host"] + print(f"host: {host.get('memory_mb')} MB / {host.get('cpus')} CPUs " + f"/ gpu={host.get('gpu')} ({host.get('source')})") + for name, tier in plan["tiers"].items(): + print(f" tier {name:<7} tasks={len(tier['tasks']):<3} " + f"concurrency={tier['concurrency']} peak={tier['peak_memory_mb']}MB/" + f"{tier['peak_cpus']}cpu") + c = plan["counts"] + print(f"runnable={c['runnable']}/{c['total']} excluded={c['excluded']} " + f"images={len(plan['base_images'])}") + for t in plan["excluded"]: + print(f" EXCLUDED {t['slug']}: {t['reason']}") + if plan["excluded"]: + print("NOTE: excluded tasks are absent from the run, not scored as failures.") + print(" Any published number must state the denominator it used.") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/bench/evidence/frontier/primary.sh b/bench/evidence/frontier/primary.sh new file mode 100755 index 000000000..40042d4e4 --- /dev/null +++ b/bench/evidence/frontier/primary.sh @@ -0,0 +1,72 @@ +#!/bin/bash +# The measured run for one resource tier. Publish the preregistration before +# calling this. +set -uo pipefail +source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/env.sh" +fb_preflight || exit 1 + +TIER="${1:?usage: primary.sh }" +JOB="${2:?usage: primary.sh }" +test ! -e "$JOBS/$JOB" || { echo "FATAL: $JOBS/$JOB exists — a run never resumes"; exit 1; } +test -f "$FB_PLAN" || { echo "FATAL: no plan at $FB_PLAN (run fetch_dataset.sh)"; exit 1; } + +read -r N CONC </dev/null || { echo "FATAL: no tasks for tier $TIER"; exit 1; } + +CONC="${FB_CONCURRENCY:-$CONC}" + +# macOS ships bash 3.2: no mapfile, no arrays from process substitution. +INCLUDES="" +while IFS= read -r t; do + [ -n "$t" ] || continue + INCLUDES="$INCLUDES --include-task-name $FB_TASK_PREFIX/$t" +done < "$TB_ROOT/frontier-$TIER.tasks" + +echo "tier=$TIER job=$JOB tasks=$N concurrency=$CONC attempts=$FB_ATTEMPTS" +echo "sut=$STELLA_SOURCE_COMMIT model=$TB_MODEL budget/trial=${STELLA_BUDGET:-uncapped}" +echo "dataset=$FB_DATASET" +echo "harbor=$(harbor --version) started=$(date -u +%Y-%m-%dT%H:%M:%SZ)" + +# Say plainly what this run is and is not. A tiered, GPU-excluding, single +# attempt run is a development baseline; the leaderboard wants all 74 tasks at +# five-plus attempts with no subsetting. Both are legitimate — reporting one as +# the other is not, and the distinction is easiest to lose at the moment a +# number finally appears. +if [ "$TIER" != "all" ] || [ "$FB_ATTEMPTS" -lt "$FB_SUBMISSION_MIN_ATTEMPTS" ]; then + echo "posture=development-baseline (not submittable: see SUBMISSION.md)" +else + echo "posture=submission-shaped" +fi + +cd "$TB_REPO" || exit 1 +# INCLUDES is deliberately unquoted: a pre-built flag list, one word per token, +# every task name a fixed [a-z0-9-] slug from the digest-pinned dataset. +# shellcheck disable=SC2086 +harbor run \ + --env docker \ + --dataset "$FB_DATASET" \ + $INCLUDES \ + --agent stella_harbor:StellaAgent \ + --model "$TB_MODEL" \ + --job-name "$JOB" \ + --jobs-dir "$JOBS" \ + --n-attempts "$FB_ATTEMPTS" --n-concurrent "$CONC" --max-retries 0 +rc=$? +echo "harbor_exit=$rc finished=$(date -u +%Y-%m-%dT%H:%M:%SZ)" +exit $rc diff --git a/bench/evidence/frontier/sentinel.sh b/bench/evidence/frontier/sentinel.sh new file mode 100755 index 000000000..1bcd61d58 --- /dev/null +++ b/bench/evidence/frontier/sentinel.sh @@ -0,0 +1,119 @@ +#!/bin/bash +# Two cheap trials that prove the whole path end to end before 70 expensive +# ones depend on it. +# +# stage 1 the synthetic fixture — adapter install, binary upload and SHA +# verification, tool loop, metering, verifier handoff. Identical to +# the Terminal-Bench sentinel except that it runs under *this* lane's +# newer Harbor, which is the thing that changed. +# +# stage 2 the cheapest real Frontier-Bench task — a real Dockerfile build and +# the separate-container verifier that every task in this set +# declares and that Harbor 0.6.1 would have silently ignored. +# +# The two stages are gated differently, and the difference is the point. The +# synthetic task is oracle-solvable, so anything but reward 1.0 is a broken +# harness. A real Frontier-Bench task is *not* expected to pass — the best +# published agents clear about a third of this set — so demanding reward 1.0 +# there would gate on model quality instead of on plumbing. Stage 2 asks only +# whether the trial genuinely ran: binary verified inside the container, a +# status reported, a reward produced by the verifier, and no infrastructure +# exception. A task that ran and scored 0.0 passes this gate, correctly. +# +# Both stages are excluded from every analysis. +set -uo pipefail +source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/env.sh" +fb_preflight || exit 1 + +STAMP="$(date +%Y%m%d-%H%M%S)" +JOB="${1:-fb-sentinel-$STAMP}" +test ! -e "$JOBS/$JOB" || { echo "FATAL: $JOBS/$JOB exists"; exit 1; } +echo "job=$JOB sut=$STELLA_SOURCE_COMMIT harbor=$(harbor --version)" + +cd "$TB_REPO" || exit 1 + +echo "=== stage 1: synthetic fixture (gate: reward == 1.0)" +harbor run \ + --env docker \ + --path "$TB_REPO/bench/readiness/synthetic-adapter-sentinel" \ + --agent stella_harbor:StellaAgent \ + --model "$TB_MODEL" \ + --job-name "$JOB" \ + --jobs-dir "$JOBS" \ + --n-attempts 1 --n-concurrent 1 --max-retries 0 +echo "harbor_exit=$?" + +python3 - "$JOBS/$JOB" <<'PY' || exit 1 +import glob, json, sys +rows = [json.load(open(p)) for p in glob.glob(f"{sys.argv[1]}/*/result.json")] +rows = [r for r in rows if r.get("task_name")] +if not rows: + print("STAGE 1 FAIL: no trial produced a result"); raise SystemExit(1) +r = rows[0] +reward = (r.get("verifier_result") or {}).get("rewards", {}).get("reward") +md = ((r.get("agent_result") or {}).get("metadata") or {}) +print(f"reward={reward} stella_status={md.get('stella_status')} " + f"binary_verified={md.get('stella_binary_sha256_verified_in_container')} " + f"commit_verified={md.get('stella_source_commit_verified_in_binary')}") +raise SystemExit(0 if reward == 1.0 else 1) +PY + +test -f "$FB_PLAN" || { echo "SKIP stage 2: no plan at $FB_PLAN (run fetch_dataset.sh)"; exit 0; } + +TASK="${FB_SENTINEL_TASK:-$(python3 -c " +import json +p=json.load(open('$FB_PLAN')) +runnable=set(p['runnable_tasks']) +# Cheapest by declared footprint, then by name so the choice is reproducible. +import tomllib, pathlib +best=None +for d in sorted(pathlib.Path(p['dataset_dir']).glob('*/*/task.toml')): + if d.parent.name not in runnable: continue + c=tomllib.loads(d.read_text()); e=c.get('environment',{}) or {} + key=(e.get('memory_mb',0), e.get('cpus',0), (c.get('agent',{}) or {}).get('timeout_sec',0), d.parent.name) + if best is None or key/dev/null 2>&1 || { echo "FATAL: uv is required"; exit 1; } + +uv venv "$VENV" --python 3.13 || exit 1 +uv pip install --python "$VENV/bin/python" "harbor==$FB_HARBOR_VERSION" || exit 1 + +got="$("$VENV/bin/harbor" --version)" +test "$got" = "$FB_HARBOR_VERSION" || { + echo "FATAL: installed harbor $got != $FB_HARBOR_VERSION"; exit 1; } + +# The adapter is imported from the repo (PYTHONPATH), not installed, so this is +# the moment to prove it still satisfies the newer Harbor's agent interface +# rather than discovering it one trial into a paid run. +PYTHONPATH="$TB_REPO/bench/harbor_adapter" "$VENV/bin/python" - <<'PY' || exit 1 +from stella_harbor import StellaAgent +missing = sorted(getattr(StellaAgent, "__abstractmethods__", ())) +assert not missing, f"StellaAgent does not implement: {missing}" +assert StellaAgent.name() == "stella" +print(f"adapter OK against this Harbor: {StellaAgent.name()} (ATIF={StellaAgent.SUPPORTS_ATIF})") +PY + +echo "venv ready: $VENV (harbor $got)" diff --git a/bench/evidence/frontier/tests/test_plan.py b/bench/evidence/frontier/tests/test_plan.py new file mode 100644 index 000000000..07fc7135e --- /dev/null +++ b/bench/evidence/frontier/tests/test_plan.py @@ -0,0 +1,336 @@ +"""Tests for the Frontier-Bench run planner. + +Everything here is about a *silent wrong number* rather than a crash. The +planner's whole job is to keep an infrastructure limitation from arriving as a +reward-0 row that looks exactly like Stella failing a task, so the cases worth +pinning are the ones where a mistake still produces a plausible-looking plan: +a GPU task quietly scheduled on a GPU-less host, a task admitted onto a machine +that cannot hold it, a build stage mistaken for an image to pull. + +Two of these are regressions. The headroom test exists because the argument +originally conflated storage with memory and subtracted a 20 GB storage +allowance from a 10 GB memory budget, which excluded all 74 tasks with a +negative number and a polite explanation. The GPU test exists because the +obvious Docker probe — ``{{if index .Runtimes "nvidia"}}`` — reports a GPU on +every host alive, Go returning a truthy zero-value struct for a missing key. + +``plan.py`` is a script rather than a package and imports nothing beyond the +standard library, so it loads by path and needs no Harbor to test. +""" + +from __future__ import annotations + +import importlib.util +import sys +from pathlib import Path + +import pytest + +_PLAN_PATH = Path(__file__).resolve().parent.parent / "plan.py" +_spec = importlib.util.spec_from_file_location("frontier_plan", _PLAN_PATH) +assert _spec and _spec.loader +plan = importlib.util.module_from_spec(_spec) +sys.modules["frontier_plan"] = plan +_spec.loader.exec_module(plan) + + +def _task( + root: Path, + slug: str, + *, + cpus: int = 1, + memory_mb: int = 2048, + storage_mb: int = 10240, + gpus: int = 0, + dockerfile: str = "FROM python:3.13-slim\n", + compose: str | None = None, + tests_dockerfile: str | None = None, +) -> Path: + """Write a task tree shaped like a real exported Frontier-Bench task.""" + d = root / "frontier-bench" / slug + (d / "environment").mkdir(parents=True) + (d / "task.toml").write_text( + f'[task]\nname = "frontier-bench/{slug}"\n' + f'[metadata]\ncategory = "Software"\n' + f"[agent]\ntimeout_sec = 7200.0\n" + f"[environment]\ncpus = {cpus}\nmemory_mb = {memory_mb}\n" + f"storage_mb = {storage_mb}\ngpus = {gpus}\n" + ) + (d / "environment" / "Dockerfile").write_text(dockerfile) + if compose is not None: + (d / "environment" / "docker-compose.yaml").write_text(compose) + if tests_dockerfile is not None: + (d / "tests").mkdir() + (d / "tests" / "Dockerfile").write_text(tests_dockerfile) + return d + + +@pytest.fixture +def host(monkeypatch): + """Pin the host capacity so a test never depends on the machine running it.""" + + def _set(memory_mb=16384, cpus=8, gpu=False): + monkeypatch.setattr( + plan, + "_host_capacity", + lambda: {"memory_mb": memory_mb, "cpus": cpus, "gpu": gpu, "source": "test"}, + ) + + return _set + + +class TestGpuExclusion: + def test_gpu_task_is_excluded_by_default_and_says_why(self, tmp_path, host): + host(gpu=False) + _task(tmp_path, "needs-gpu", gpus=1) + _task(tmp_path, "plain") + + result = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048) + + assert result["runnable_tasks"] == ["plain"] + (excluded,) = result["excluded"] + assert excluded["slug"] == "needs-gpu" + assert "GPU" in excluded["reason"] + + def test_allow_gpu_alone_is_not_enough_without_a_host_gpu(self, tmp_path, host): + """The operator asking for GPU tasks does not conjure a GPU. + + Honouring the flag on a GPU-less host is the exact failure this planner + exists to prevent: the container starts, the work is impossible, and the + verifier returns 0.0 indistinguishably from a genuine miss. + """ + host(gpu=False) + _task(tmp_path, "needs-gpu", gpus=1) + + result = plan.build_plan(tmp_path, allow_gpu=True, memory_headroom_mb=2048) + + assert result["runnable_tasks"] == [] + assert "nvidia" in result["excluded"][0]["reason"] + + def test_gpu_task_runs_when_the_host_actually_has_one(self, tmp_path, host): + host(gpu=True) + _task(tmp_path, "needs-gpu", gpus=1) + + result = plan.build_plan(tmp_path, allow_gpu=True, memory_headroom_mb=2048) + + assert result["runnable_tasks"] == ["needs-gpu"] + assert result["excluded"] == [] + + +class TestHeadroom: + def test_headroom_is_subtracted_from_memory_not_storage(self, tmp_path, host): + """Regression: a storage-sized headroom against memory excluded everything. + + A 20 GB allowance subtracted from a 10 GB memory budget goes negative, + and every task — including a 2 GB one — falls off the plan. + """ + host(memory_mb=10240, cpus=8) + _task(tmp_path, "small-task", memory_mb=2048, storage_mb=1024000) + + result = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048) + + assert result["runnable_tasks"] == ["small-task"] + assert result["counts"]["runnable"] == 1 + + def test_task_larger_than_the_remaining_budget_is_excluded(self, tmp_path, host): + host(memory_mb=10240, cpus=8) + _task(tmp_path, "hungry", memory_mb=12288) + + result = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048) + + assert result["runnable_tasks"] == [] + assert "12288" in result["excluded"][0]["reason"] + + def test_a_task_that_exactly_fills_the_budget_is_admitted(self, tmp_path, host): + """The comparison is strictly greater-than, and should stay that way. + + 10240 MB less a 2048 MB headroom leaves exactly 8192. Excluding a task + that fits precisely would drop real tasks for no reason — 8192 MB is the + second most common footprint in the set. + """ + host(memory_mb=10240, cpus=8) + _task(tmp_path, "exact", memory_mb=8192) + + result = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048) + + assert result["runnable_tasks"] == ["exact"] + + def test_lowering_the_headroom_admits_the_same_task(self, tmp_path, host): + host(memory_mb=10240, cpus=8) + _task(tmp_path, "hungry", memory_mb=8192) + + result = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=512) + + assert result["runnable_tasks"] == ["hungry"] + + def test_task_needing_more_cpus_than_the_host_is_excluded(self, tmp_path, host): + host(memory_mb=65536, cpus=4) + _task(tmp_path, "wide", cpus=16) + + result = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048) + + assert result["runnable_tasks"] == [] + assert "CPU" in result["excluded"][0]["reason"] + + +class TestTiering: + def test_tasks_land_in_the_tier_matching_their_memory(self, tmp_path, host): + host(memory_mb=65536, cpus=32) + _task(tmp_path, "a-tiny", memory_mb=2048) + _task(tmp_path, "b-mid", memory_mb=4096) + _task(tmp_path, "c-big", memory_mb=8192) + _task(tmp_path, "d-huge", memory_mb=32768) + + tiers = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048)["tiers"] + + assert tiers["small"]["tasks"] == ["a-tiny"] + assert tiers["medium"]["tasks"] == ["b-mid"] + assert tiers["large"]["tasks"] == ["c-big"] + assert tiers["xlarge"]["tasks"] == ["d-huge"] + + def test_concurrency_is_bounded_by_whichever_resource_runs_out_first( + self, tmp_path, host + ): + # 16 GB usable after headroom; each task wants 4 GB, so memory allows 4. + # But 5 CPUs against 2 per task allows only 2 — the smaller wins. + host(memory_mb=18432, cpus=5) + _task(tmp_path, "a", memory_mb=4096, cpus=2) + _task(tmp_path, "b", memory_mb=4096, cpus=2) + + tiers = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048)["tiers"] + + assert tiers["medium"]["concurrency"] == 2 + + def test_concurrency_never_drops_below_one(self, tmp_path, host): + """A task that only just fits still gets scheduled, serially.""" + host(memory_mb=10240, cpus=2) + _task(tmp_path, "just-fits", memory_mb=8192, cpus=2) + + tiers = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=512)["tiers"] + + assert tiers["large"]["concurrency"] == 1 + + def test_concurrency_is_capped_regardless_of_headroom(self, tmp_path, host): + host(memory_mb=1 << 20, cpus=512) + _task(tmp_path, "tiny", memory_mb=2048, cpus=1) + + tiers = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048)["tiers"] + + assert tiers["small"]["concurrency"] == plan.MAX_CONCURRENCY + + +class TestBaseImages: + def test_multi_stage_aliases_are_not_pulled(self, tmp_path, host): + """``FROM builder`` names a local stage; pulling it would fail.""" + host() + _task( + tmp_path, + "staged", + dockerfile=( + "FROM golang:1.24 AS builder\n" + "RUN go build\n" + "FROM debian:12-slim\n" + "COPY --from=builder /app /app\n" + ), + ) + + images = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048)["base_images"] + + assert images == ["debian:12-slim", "golang:1.24"] + + def test_scratch_and_arg_substituted_bases_are_skipped(self, tmp_path, host): + host() + _task( + tmp_path, + "odd", + dockerfile="ARG BASE\nFROM scratch\nFROM ${BASE}\nFROM ubuntu:24.04\n", + ) + + images = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048)["base_images"] + + assert images == ["ubuntu:24.04"] + + def test_compose_service_images_are_included(self, tmp_path, host): + host() + _task( + tmp_path, + "composed", + compose="services:\n db:\n image: postgres:16\n app:\n build: .\n", + ) + + images = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048)["base_images"] + + assert "postgres:16" in images + + def test_the_verifier_dockerfile_counts_too(self, tmp_path, host): + """Every task here declares environment_mode = "separate". + + The verifier builds its own container, so its base is just as much a + precondition of the run as the agent's. + """ + host() + _task(tmp_path, "separate-verifier", tests_dockerfile="FROM node:22-bookworm\n") + + images = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048)["base_images"] + + assert "node:22-bookworm" in images + + def test_images_are_deduplicated_across_tasks(self, tmp_path, host): + host() + _task(tmp_path, "one", dockerfile="FROM python:3.13-slim\n") + _task(tmp_path, "two", dockerfile="FROM python:3.13-slim\n") + + images = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048)["base_images"] + + assert images == ["python:3.13-slim"] + + def test_an_excluded_task_does_not_drag_in_its_images(self, tmp_path, host): + """Warming an image for a task that will not run is wasted bandwidth.""" + host(gpu=False) + _task(tmp_path, "gpu-only", gpus=1, dockerfile="FROM nvidia/cuda:12.6.3-base\n") + _task(tmp_path, "plain", dockerfile="FROM ubuntu:24.04\n") + + images = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048)["base_images"] + + assert images == ["ubuntu:24.04"] + + +class TestPlanShape: + def test_counts_reconcile(self, tmp_path, host): + host(memory_mb=10240, cpus=8, gpu=False) + _task(tmp_path, "ok") + _task(tmp_path, "gpu", gpus=1) + _task(tmp_path, "huge", memory_mb=32768) + + result = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048) + + c = result["counts"] + assert c["total"] == 3 + assert c["runnable"] + c["excluded"] == c["total"] + assert len(result["excluded"]) == c["excluded"] + + def test_an_empty_dataset_directory_is_fatal_rather_than_an_empty_plan( + self, tmp_path, host + ): + host() + with pytest.raises(SystemExit): + plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048) + + def test_unknown_host_capacity_still_produces_a_runnable_plan(self, tmp_path, monkeypatch): + """`docker info` failing must not silently empty the plan. + + Excluding every task because capacity could not be read looks identical + to a host that is too small, so the planner admits them and lets the + preflight's own docker check be the thing that fails. + """ + monkeypatch.setattr( + plan, "_host_capacity", + lambda: {"memory_mb": None, "cpus": None, "gpu": False, + "source": "unavailable", "error": "boom"}, + ) + _task(tmp_path, "a", memory_mb=32768) + + result = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048) + + assert result["runnable_tasks"] == ["a"] + assert result["tiers"]["xlarge"]["concurrency"] == 1 diff --git a/bench/evidence/frontier/warm_images.sh b/bench/evidence/frontier/warm_images.sh new file mode 100755 index 000000000..1ef4df8c2 --- /dev/null +++ b/bench/evidence/frontier/warm_images.sh @@ -0,0 +1,46 @@ +#!/bin/bash +# Pull every base image the task builds depend on, with retries, before the +# measured run. +# +# The Terminal-Bench lane calls this prepull, and pulls each task's prebuilt +# `docker_image`. No Frontier-Bench task has one: all 74 ship an +# environment/Dockerfile and build in place. So the thing to warm is the set of +# `FROM` bases (plus any compose service images), which `plan.py` collects. +# +# The reason is unchanged from the Terminal-Bench lane, and worth restating +# because it is the whole point: with --max-retries 0 a registry hiccup is a +# permanent reward-0 row, arithmetically identical to Stella failing the task. +# Image availability is a precondition of the measurement, not a term in it. +set -uo pipefail +source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/env.sh" + +LIST="${1:-$TB_ROOT/frontier-images.txt}" +FAILED="$TB_ROOT/frontier_warm_failed.txt" +test -f "$LIST" || { echo "FATAL: no image list at $LIST (run fetch_dataset.sh)"; exit 1; } +: > "$FAILED" + +total=$(grep -c . "$LIST") +i=0 +while IFS= read -r img; do + [ -n "$img" ] || continue + i=$((i+1)) + if docker image inspect "$img" >/dev/null 2>&1; then + echo "[$i/$total] have $img"; continue + fi + ok=0 + for attempt in 1 2 3 4 5; do + if timeout 1200 docker pull --quiet "$img" >/dev/null 2>&1; then ok=1; break; fi + sleep $((attempt * 10)) + done + if [ "$ok" = 1 ]; then echo "[$i/$total] pulled $img" + else echo "[$i/$total] FAILED $img"; echo "$img" >> "$FAILED"; fi +done < "$LIST" + +nfailed=$(grep -c . "$FAILED" || echo 0) +echo "WARM_DONE failed=$nfailed" +# A base image that will not pull now is a task that cannot build later. Say so +# before the run rather than letting it arrive as a reward-0 row. +test "$nfailed" = 0 || { + echo "WARNING: the tasks depending on these images will fail to build." + echo " Resolve or exclude them before starting a measured run." + exit 1; } diff --git a/bench/harbor_adapter/stella_harbor/secure_launcher.py b/bench/harbor_adapter/stella_harbor/secure_launcher.py index 5ab2a4bc6..e2d16f3e9 100644 --- a/bench/harbor_adapter/stella_harbor/secure_launcher.py +++ b/bench/harbor_adapter/stella_harbor/secure_launcher.py @@ -75,6 +75,7 @@ "bench/harbor_adapter/stella_harbor/atif.py", "bench/harbor_adapter/stella_harbor/credential_bundle.py", "bench/harbor_adapter/stella_harbor/host_attestation.py", + "bench/harbor_adapter/stella_harbor/live_feed.py", "bench/harbor_adapter/stella_harbor/portability.py", "bench/harbor_adapter/stella_harbor/posture.py", "bench/harbor_adapter/stella_harbor/secure_launcher.py", diff --git a/bench/harbor_adapter/tests/test_secure_launcher.py b/bench/harbor_adapter/tests/test_secure_launcher.py index 71add41bd..6837ac385 100644 --- a/bench/harbor_adapter/tests/test_secure_launcher.py +++ b/bench/harbor_adapter/tests/test_secure_launcher.py @@ -1096,6 +1096,32 @@ def _host_preflight_fixture( ) +def test_frozen_adapter_source_paths_match_the_package_on_disk() -> None: + """The freeze list must enumerate every .py the runtime actually hashes. + + The runtime digest rglobs ``*.py`` under the package while the public-tree + check walks this hand-written tuple, so a module added to one and not the + other makes the two disagree and everything downstream fails closed. That + is the design working — but it announces itself as thirty unrelated tests + failing with "public adapter tree hash differs from runtime identity", + which names neither the file nor the fix. + + This is what happened when ``live_feed.py`` arrived in #1027 without being + added here. One assertion that names the offending file is worth more than + thirty that describe its consequences. + """ + package_dir = Path(stella_harbor.__file__).parent + on_disk = { + f"bench/harbor_adapter/stella_harbor/{p.relative_to(package_dir)}" + for p in package_dir.rglob("*.py") + } + assert on_disk == set(launcher_module._FIXED_ADAPTER_SOURCE_PATHS), ( + "add new adapter modules to _FIXED_ADAPTER_SOURCE_PATHS in " + "secure_launcher.py (and push them publicly — the check reads the " + "published tree)" + ) + + def test_launch_receipt_controls_are_exact_nonsecret_attestation() -> None: assert LAUNCH_RECEIPT_CONTROLS == { "command": "harbor-run-only", diff --git a/scripts/file-size-baseline.txt b/scripts/file-size-baseline.txt index 28c1f1f32..f56e53b1a 100644 --- a/scripts/file-size-baseline.txt +++ b/scripts/file-size-baseline.txt @@ -8,9 +8,9 @@ # never silent — it lands as a visible diff here, to be justified in # review like any other change. 1928 bench/harbor_adapter/stella_harbor/__init__.py -4269 bench/harbor_adapter/stella_harbor/secure_launcher.py +4270 bench/harbor_adapter/stella_harbor/secure_launcher.py 1910 bench/harbor_adapter/tests/test_adapter.py -3204 bench/harbor_adapter/tests/test_secure_launcher.py +3230 bench/harbor_adapter/tests/test_secure_launcher.py 8166 bench/terminal_bench_analysis/tb21_analysis.py 1887 bench/terminal_bench_analysis/tb21_evidence_contract.py 4173 bench/terminal_bench_analysis/tests/test_tb21_analysis.py