From a1f39f1f1bf19c4018c89daf835338b87d3b37c8 Mon Sep 17 00:00:00 2001 From: Stella Test Date: Fri, 31 Jul 2026 15:04:26 -0700 Subject: [PATCH 1/7] feat(bench): a Frontier-Bench lane that keeps the adapter untouched MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Frontier-Bench is Harbor's successor to Terminal-Bench: 74 tasks over seven domains, where the best published agents clear about a third. `stella_harbor:StellaAgent` runs it unmodified, so nothing under bench/harbor_adapter/ changes — its Python tree is digest-frozen for the audited claim path, and a new benchmark has to earn its place without perturbing that. Everything specific to this one lives in the new bench/evidence/frontier/. The lane pins its own Harbor, 0.20.0, rather than the Terminal-Bench claim's audited 0.6.1. All 74 tasks declare `environment_mode = "separate"` under [verifier]; 0.6.1 has no such field and pydantic's default extra="ignore" drops it silently, so 0.6.1 would run the set with the verifier sharing the agent's container, produce rewards, and warn about nothing. Plausible numbers answering the wrong question are worse than an error. The Terminal-Bench venv stays on 0.6.1. Three more differences, each because the Terminal-Bench assumption is false here rather than merely untidy: * No task has a `docker_image` — all 74 build from an environment/Dockerfile, 12 with compose. So warm_images.sh pulls the FROM bases plan.py collects, for the prepull's original reason: with --max-retries 0 a registry hiccup is a permanent reward-0 row. * Terminal-Bench's one-line resource rule puts 58 of 74 in the serial phase. plan.py tiers by footprint against what `docker info` reports and gives each tier the concurrency the host supports. * Four tasks want a GPU. On a GPU-less host they do not error — the container starts, the work is impossible, the verifier returns 0.0, and the row is arithmetically identical to a genuine miss, quietly costing ~5.4 points. They are excluded by name with a stated reason unless FB_ALLOW_GPU=1 and Docker exposes an nvidia runtime. Dataset pinned to sha256:97fd2ba3…, the leaderboard's own DATASET_REF. Resolving by name today yields sha256:63f363a1… instead, which exports a byte-identical tree (verified by diff) — two version records over the same content — but the leaderboard compares the ref string, so a run on the equally-valid other digest is rejected while being the same run. SUBMISSION.md records what a leaderboard entry needs and the two things that block one: four tasks need real GPUs and subsetting is not allowed, and a 10 GB Docker VM admits only 48 of 74. Both mean a submission run happens on Modal, not a Mac. The intake PR upstream has not merged yet. Verified: 74/74 task specs parse under both Harbors; the adapter's suite is 132/132 under 0.6.1 and 131/132 under 0.20.0, the one difference being a test asserting the audited 0.6.1 constant. --- bench/README.md | 3 +- bench/evidence/frontier/README.md | 86 +++++++ bench/evidence/frontier/SUBMISSION.md | 96 ++++++++ bench/evidence/frontier/env.sh | 137 +++++++++++ bench/evidence/frontier/fetch_dataset.sh | 21 ++ bench/evidence/frontier/plan.py | 284 +++++++++++++++++++++++ bench/evidence/frontier/primary.sh | 72 ++++++ bench/evidence/frontier/sentinel.sh | 119 ++++++++++ bench/evidence/frontier/setup_venv.sh | 32 +++ bench/evidence/frontier/warm_images.sh | 46 ++++ 10 files changed, 895 insertions(+), 1 deletion(-) create mode 100644 bench/evidence/frontier/README.md create mode 100644 bench/evidence/frontier/SUBMISSION.md create mode 100755 bench/evidence/frontier/env.sh create mode 100755 bench/evidence/frontier/fetch_dataset.sh create mode 100755 bench/evidence/frontier/plan.py create mode 100755 bench/evidence/frontier/primary.sh create mode 100755 bench/evidence/frontier/sentinel.sh create mode 100755 bench/evidence/frontier/setup_venv.sh create mode 100755 bench/evidence/frontier/warm_images.sh diff --git a/bench/README.md b/bench/README.md index 9af409ea5..2e6578eba 100644 --- a/bench/README.md +++ b/bench/README.md @@ -5,11 +5,12 @@ with other agents, and standalone. Everything here is **BYOK** (bring your own key), makes **no phone-home**, and never hard-codes a secret. Claim runs use a secure launcher that consumes the selected credential before Harbor starts. -Four entry points: +Five entry points: | Path | What it does | Needs | |---|---|---| | [`harbor_adapter/`](harbor_adapter/) | A Harbor *installed-agent* adapter — run Stella on Terminal-Bench 2.x / SWE-bench in the same container + verifier as Claude Code, Codex CLI, Terminus, etc. | Docker, `harbor`, a provider key | +| [`evidence/frontier/`](evidence/frontier/) | **Frontier-Bench** — Harbor's successor to Terminal-Bench, 74 tasks over seven domains. Same adapter, unmodified; a separate Harbor pin, a resource-aware schedule, and GPU tasks excluded by name rather than scored as zeros. | Docker, a provider key (a *submittable* run needs GPUs — see its [SUBMISSION.md](evidence/frontier/SUBMISSION.md)) | | [`run_swebench.py`](run_swebench.py) | A standalone SWE-bench *prediction* harness — clone each instance, run Stella, emit the official predictions JSONL. No Harbor. | `git`, a provider key (Docker only for the official scoring step) | | [`loop-bench/`](loop-bench/) | A cheap **turn-loop + context-query correctness** harness: runs N tasks on a flash-tier model, budget-capped, and reports loop health (silent-death / zero-work / stuck-loop) and `project_overview`/`graph_query` adoption — the signals the pass-rate number hides. | `cargo`, Docker, `harbor`, a key | | [`smoke/smoke_test.py`](smoke/smoke_test.py) | An **offline, zero-cost** self-test of the adapter wiring for CI. | just the built `stella` binary | diff --git a/bench/evidence/frontier/README.md b/bench/evidence/frontier/README.md new file mode 100644 index 000000000..290af9ec5 --- /dev/null +++ b/bench/evidence/frontier/README.md @@ -0,0 +1,86 @@ +# Frontier-Bench (dev-baseline lane) + +Runs Stella on [Frontier-Bench](https://github.com/harbor-framework/frontier-bench) +— Harbor's successor to Terminal-Bench, 74 tasks across seven domains, where the +best published agents clear roughly a third. + +**The adapter is unchanged.** `stella_harbor:StellaAgent` runs this benchmark +exactly as it runs Terminal-Bench 2.1, with no edit to anything under +`bench/harbor_adapter/`. That was a design constraint, not a happy accident: the +adapter's Python tree is digest-frozen for the audited claim path, so a new +benchmark has to earn its place without perturbing it. Everything specific to +Frontier-Bench lives in this directory. + +## Run it + +```bash +export TB_REPO=/path/to/stella TB_ROOT=/abs/scratch OPENROUTER_API_KEY=... +bench/evidence/run/build_sut.sh # shared with the Terminal-Bench lane +bench/evidence/frontier/setup_venv.sh # this lane's Harbor pin +bench/evidence/frontier/fetch_dataset.sh # pinned dataset + resource plan +bench/evidence/frontier/warm_images.sh # pull the base images the builds need +bench/evidence/frontier/sentinel.sh # two cheap trials, end to end +bench/evidence/frontier/primary.sh small fb-small-01 +``` + +## What differs from Terminal-Bench, and why each difference exists + +**A second Harbor pin (0.20.0), not the audited 0.6.1.** Every one of the 74 +tasks declares `environment_mode = "separate"` under `[verifier]`. Harbor 0.6.1 +has no such field, and pydantic's default `extra="ignore"` drops it *silently* — +0.6.1 runs the whole set with the verifier sharing the agent's container instead +of the separate one the task asked for, produces rewards, and warns about +nothing. Plausible numbers answering the wrong question are worse than an error. +This lane gets its own venv; `bench/harbor_adapter/.venv` stays on 0.6.1 so +Terminal-Bench reruns keep measuring what they measured. + +**Base images are warmed, not task images pulled.** No Frontier-Bench task has a +`docker_image`; all 74 ship an `environment/Dockerfile` (12 also a compose file) +and build in place. `warm_images.sh` therefore pulls the `FROM` bases that +`plan.py` collects. The reasoning is the Terminal-Bench prepull's, unchanged: +with `--max-retries 0` a registry hiccup is a permanent reward-0 row. + +**Resource tiers, not an A/B split.** Terminal-Bench's one-line rule — over 4 GB +or one CPU goes serial — puts 58 of these 74 in the serial phase. Frontier-Bench +declares 1–16 CPUs, 2–32 GB memory, and 4 GB–1 TB storage. `plan.py` tiers tasks +by footprint and gives each tier the concurrency the host can actually support. + +**GPU tasks are excluded by default and named.** Four tasks (`exam-pdf-eval`, +`fp8-rmsnorm-gemm`, `jax-speedrun-gpu`, `math-eval-grader`) declare `gpus = 1`. +On a GPU-less host they do not error — the container starts, the work is +impossible, the verifier returns 0.0, and the row is arithmetically identical to +Stella genuinely failing. Silently that is ~5.4 points of pass rate. They are +excluded unless `FB_ALLOW_GPU=1` *and* Docker exposes an nvidia runtime, and the +plan prints every exclusion with its reason. + +**A higher budget default ($2.50/trial).** Declared agent timeouts run 30 minutes +to 8 hours against expert time estimates measured in days. Terminal-Bench's +$0.60 cap would truncate most of this set and record the truncation as failure. +`STELLA_BUDGET=` (explicitly empty) means no cap. + +## The sentinel's two gates + +Stage 1 runs the synthetic fixture and demands reward 1.0 — it is +oracle-solvable, so anything less is a broken harness. Stage 2 runs the cheapest +real task and demands only that the trial *ran*: binary verified in-container, a +status reported, a reward produced, no infrastructure exception. Requiring 1.0 +there would gate on model quality instead of plumbing; a real task that ran and +scored 0.0 passes stage 2, correctly. + +## Known local limits + +`plan.py` reports what the host can take. On a 10 GB / 6-CPU Docker VM (a +typical Mac) it admits **48 of 74** tasks — 4 GPU-excluded and 22 over the memory +budget. That is a real constraint, not a bug, and it is why a submittable run +needs a bigger machine: see [SUBMISSION.md](SUBMISSION.md). + +## Harbor version and the test suite + +`bench/harbor_adapter/tests/` passes 132/132 under 0.6.1 and 131/132 under +0.20.0. The single difference is +`test_hashes_exact_uploaded_binary_and_records_source_commit`, which asserts +`agent._harbor_version_value == "0.6.1"` — the Terminal-Bench claim's audited +constant, correctly failing when a different Harbor is installed. CI runs the +suite in the 0.6.1 venv and stays green. Nothing in the adapter needed changing: +every import resolves, the `name`/`install`/`run` interface is unchanged, and +`setup_venv.sh` re-proves that on every venv build. diff --git a/bench/evidence/frontier/SUBMISSION.md b/bench/evidence/frontier/SUBMISSION.md new file mode 100644 index 000000000..0a894e9d3 --- /dev/null +++ b/bench/evidence/frontier/SUBMISSION.md @@ -0,0 +1,96 @@ +# Getting Stella onto the Frontier-Bench leaderboard + +What the leaderboard requires, what this lane already satisfies, and the two +things that genuinely block a submission today. Researched 2026-07-31 against +the frontier-bench repo, its pending leaderboard PR, and the live pipelines on +the sibling benchmarks; re-check before acting, because the intake is still +moving. + +## The intake is not open yet + +Frontier-Bench has no merged self-serve submission pipeline as of 2026-07-31. +The tooling exists as [PR #1405](https://github.com/harbor-framework/frontier-bench/pull/1405) +("Add leaderboard package and PR-submit CI for Harbor Hub") and has only been +exercised end-to-end on a fork. Two "Leaderboard Submission" PRs against +upstream (#1408 Claude Fable 5, #1419 Kimi K3) were both closed by their own +author as opened against the wrong repo during that fork test. The identical +pipeline is already live for `harbor-index` and `terminal-bench-2-1`, so the +mechanics below are stable in shape even though the door is shut. + +**Consequence for us:** the run can be produced now; the PR cannot be filed +until #1405 merges. Producing the run first is still the right order — it is the +long pole by a wide margin. + +## What a submission is + +Two parts. A public job upload, then a one-file PR. + +```bash +harbor run -d frontier-bench/frontier-bench --agent -m --upload --public +cd leaderboard && uv run lb submit https://hub.harborframework.com/jobs/ [...] +``` + +`lb submit` writes one JSON per unique (agent, agent version, model, reasoning +effort) into `leaderboard/submissions/` and opens one PR per file — CI requires +exactly one added file. The PR carries only job links and metadata; CI re-derives +every trial from the uploaded job, so trajectories must be public. Verification +is audit-based, not re-execution: static analysis, then promotion into +leaderboard-owned copies, then an LLM judge that reviews *every* trajectory for +reward hacking, then merge. + +## Requirements, and where this lane stands + +| Requirement | Status | +|---|---| +| Dataset pinned to the leaderboard's `DATASET_REF` | **Done.** `env.sh` pins `sha256:97fd2ba3…`, the leaderboard's own ref | +| Harbor ≥ 0.20.0 | **Done.** This lane pins 0.20.0 | +| Default execution settings; no timeout or resource overrides | **Done.** Nothing here overrides either; tiering only chooses what runs *concurrently* | +| All 74 tasks, no subsetting | **Blocked by hardware** — see below | +| ≥ 5 trials per task | **Supported, off by default.** `FB_ATTEMPTS=5`; default 1 is the dev baseline | +| `--upload --public` | **Not wired.** Deliberate: uploading publishes trajectories, which is a decision to take explicitly, not a flag to inherit | +| Agent + model disclosure metadata | Supplied at `lb submit` time, not by this lane | + +Note the metric: **accuracy = trials with reward > 0, over all trials**. Not +reward == 1.0, and errored trials count as reward 0 rather than being dropped. +Infrastructure flakiness therefore lands directly in the score, which is exactly +why `warm_images.sh` and the preflight exist. + +## The two real blockers + +**1. GPUs.** Four tasks need one, and the benchmark authors ran them on a single +H100 each. Subsetting is not allowed for a submission, so those four cannot be +excluded the way the dev baseline excludes them — they must actually run. + +**2. Host size.** On a 10 GB / 6-CPU Docker VM, `plan.py` admits 48 of 74 tasks; +22 more are simply over the memory budget. The full set declares up to 32 GB +memory, 16 CPUs, and ~1.9 TB of storage in aggregate. + +Both point the same way: **a submission run does not happen on a Mac.** The +benchmark's own guidance is Modal (`uv tool install 'harbor[modal]'`, `--env +modal`), with Daytona as an alternative that recently added GPU support. Harbor's +repo CI defaults to `env: modal`. + +## Cost, honestly + +74 tasks × 5 trials = 370 trials, against declared agent timeouts averaging +around two hours. At this lane's $2.50/trial default that is roughly $925 in +model spend before Modal compute, and the budget cap would truncate the longer +tasks — for a leaderboard run you would want `STELLA_BUDGET=` (uncapped), which +removes the ceiling on that estimate. This is a four-figure decision, not a +weekend experiment, and it should be preregistered like the Terminal-Bench claim +was. + +## Suggested order + +1. Land this lane; run the sentinel locally to prove the plumbing on the new + Harbor pin. Cheap. +2. Run the dev baseline on the 48 locally-runnable tasks at one attempt. Cheap + enough to be worth it, and it tells you where Stella actually lands before + you spend four figures. +3. Only if that number justifies it: preregister, move to Modal with GPU access, + run all 74 at five attempts uncapped with `--upload --public`. +4. File the PR once #1405 merges. + +Steps 1 and 2 are what this directory is for. Step 3 needs a Modal environment +this lane does not yet configure — `primary.sh` hardcodes `--env docker`, and +adding a Modal path is the next piece of work, not something already done. diff --git a/bench/evidence/frontier/env.sh b/bench/evidence/frontier/env.sh new file mode 100755 index 000000000..e09385c67 --- /dev/null +++ b/bench/evidence/frontier/env.sh @@ -0,0 +1,137 @@ +#!/bin/bash +# Shared environment + preflight for a Frontier-Bench development-baseline run. +# Sourced by every other script here. Never echoes a credential. +# +# Frontier-Bench is Harbor's successor to Terminal-Bench: 74 tasks across seven +# domains, where Terminal-Bench 2.1's ceiling had already been cleared. The +# agent side is identical — the same `stella_harbor:StellaAgent` installed-agent +# runs both, unmodified — so everything specific to this benchmark lives in this +# directory and nothing under `bench/harbor_adapter/` changes. That is +# deliberate: the adapter's Python tree is digest-frozen for the audited claim +# path, and a benchmark that only differs in *which dataset and how it is +# scheduled* must not perturb it. +# +# The shared half — the glibc floor, the target triple, the SUT path, the +# portability assert, the forced amd64 platform — is sourced from the +# Terminal-Bench lane rather than copied, so the two cannot drift. The halves +# that genuinely differ are re-declared below with the reason. +set -uo pipefail + +FB_HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# The frozen dataset, pinned by content digest. An unversioned name is not a +# freeze, and `@latest` least of all: Frontier-Bench is explicitly a *moving* +# benchmark — the announcement's whole point is that tasks are added and revised +# continuously — so the name that resolved to 74 tasks today resolves to a +# different set next month. Terminal-Bench could get away with a name because it +# had stopped moving. This one cannot. +# +# This exact digest is the leaderboard's own `DATASET_REF`. Resolving the +# dataset by name today yields +# sha256:63f363a191f0a0429fd1c5b318080616bab839473ce27e39f44868d327b03a89 +# instead, which exports a *byte-identical* 74-task tree — verified by diffing +# both exports, no file differs. They are two version records over the same +# content. The leaderboard's static analysis compares the ref *string* a job +# recorded, not the tasks it contains, so a run pinned to the equally-valid +# other digest is rejected on submission while being, in substance, the same +# run. Pin the string CI is looking for. +# +# Re-resolve for a newer snapshot with: +# harbor download frontier-bench/frontier-bench --export -o # prints it +export FB_DATASET="${FB_DATASET:-frontier-bench/frontier-bench@sha256:97fd2ba3aabdda16823a1a8ea695a3875e50e800caa60b450686deedc7171763}" +export FB_TASK_PREFIX="frontier-bench" +export FB_TASK_COUNT="${FB_TASK_COUNT:-74}" + +# Trials per task. One is right for a development baseline, where the question +# is "does the harness work and roughly where do we land". It is *not* a +# leaderboard submission: that requires every task covered at five or more +# trials each, and CI counts them. Raising this multiplies cost by the same +# factor, so it is a deliberate act, not a default. +export FB_ATTEMPTS="${FB_ATTEMPTS:-1}" +export FB_SUBMISSION_MIN_ATTEMPTS=5 + +# Frontier-Bench tasks run far longer than Terminal-Bench ones: declared agent +# timeouts run from 30 minutes to 8 hours, with a 2-hour mode, against expert +# time estimates measured in whole days. Terminal-Bench's $0.60 per-trial cap +# would truncate most of this set mid-task and report the truncation as a task +# failure, which is a measurement error, not a thrifty default. `-` and not `:-` +# for the same reason as the Terminal-Bench lane: an explicitly empty value +# stays empty and means *no cap*, which is the posture a leaderboard-facing run +# has to be able to take. +export STELLA_BUDGET="${STELLA_BUDGET-2.50}" + +# What the sourced Terminal-Bench env would otherwise default. Set before the +# source so its `${TB_DATASET:-...}` sees a value and leaves it alone; nothing +# here reads TB_DATASET afterwards, but leaving it pointing at Terminal-Bench +# while this lane runs Frontier-Bench is the kind of stale-variable trap that +# ends up in a log line and then in a claim. +export TB_DATASET="$FB_DATASET" + +# shellcheck source=bench/evidence/run/env.sh +source "$FB_HERE/../run/env.sh" + +# --- Everything below overrides the Terminal-Bench lane on purpose. --- + +# Harbor 0.6.1 is the Terminal-Bench claim's audited constant. It cannot be this +# lane's constant. Every one of the 74 Frontier-Bench tasks declares +# `environment_mode = "separate"` under `[verifier]`, and 0.6.1's VerifierConfig +# has no such field — pydantic's default `extra="ignore"` drops it silently, so +# 0.6.1 runs the whole set with the verifier sharing the agent's container +# instead of the separate one the task asked for. It does not warn, and the run +# still produces rewards. Those rewards answer a different question than the +# leaderboard's, which is the worst possible failure mode: plausible numbers. +# +# So this lane pins its own Harbor, in its own virtualenv, and asserts it. The +# Terminal-Bench venv at bench/harbor_adapter/.venv is left untouched. +export FB_HARBOR_VERSION="${FB_HARBOR_VERSION:-0.20.0}" +export VENV="$FB_HERE/.venv" +export PATH="$VENV/bin:$PATH" +export PYTHONPATH="$TB_REPO/bench/harbor_adapter" + +# Distinct from the Terminal-Bench lane's, so the two can share one TB_ROOT +# without one run's dataset export, resource plan, or job tree overwriting the +# other's. +export DATASET_DIR="$TB_ROOT/dataset-frontier" +export JOBS="$TB_ROOT/jobs-frontier" +export FB_PLAN="$TB_ROOT/frontier-plan.json" +mkdir -p "$JOBS" + +# Four tasks (exam-pdf-eval, fp8-rmsnorm-gemm, jax-speedrun-gpu, +# math-eval-grader) declare `gpus = 1`. On a host without a GPU they do not +# error in any way that is distinguishable from Stella failing: the container +# comes up, the work is impossible, the verifier returns 0.0, and the row is +# arithmetically identical to a genuine miss. Terminal-Bench had no such tasks +# and so no such hazard. +# +# The default is therefore to exclude them and say so, loudly, in the score's +# denominator — not to include them and quietly absorb four guaranteed zeros. +# Set FB_ALLOW_GPU=1 only on a host that actually has one; `plan.py` checks. +export FB_ALLOW_GPU="${FB_ALLOW_GPU:-0}" + +# The declared storage footprint of the full set is ~1.9 TB, one task +# (jax-speedrun-gpu) accounting for 1 TB of it. A run that fills the disk +# halfway through does not fail cleanly — it fails as a scatter of unrelated +# task errors — so the plan refuses to schedule a bin the host cannot hold. +export FB_STORAGE_HEADROOM_MB="${FB_STORAGE_HEADROOM_MB:-20480}" + +fb_preflight() { + test -n "${OPENROUTER_API_KEY:-}" || { echo "FATAL: OPENROUTER_API_KEY unset"; return 1; } + test -x "$STELLA_BINARY" || { echo "FATAL: no SUT binary at $STELLA_BINARY (run ../run/build_sut.sh)"; return 1; } + # Reused, not reimplemented: one definition of the glibc floor for both lanes. + assert_portable_binary || return 1 + test "${#STELLA_SOURCE_COMMIT}" = 40 || { echo "FATAL: STELLA_SOURCE_COMMIT is not a full SHA"; return 1; } + test "$(command -v harbor)" = "$VENV/bin/harbor" || { + echo "FATAL: wrong harbor on PATH — expected $VENV/bin/harbor" + echo " Create this lane's venv with: bench/evidence/frontier/setup_venv.sh" + return 1; } + test "$(harbor --version)" = "$FB_HARBOR_VERSION" || { + echo "FATAL: harbor $(harbor --version) != $FB_HARBOR_VERSION" + echo " 0.6.1 silently ignores this dataset's verifier environment_mode." + return 1; } + docker info >/dev/null 2>&1 || { echo "FATAL: docker unreachable"; return 1; } +} + +# The Terminal-Bench lane's `preflight` is still defined and still asserts +# harbor 0.6.1, which is false here. Point the familiar name at this lane's +# check so a script that calls `preflight` out of habit fails safe. +preflight() { fb_preflight "$@"; } diff --git a/bench/evidence/frontier/fetch_dataset.sh b/bench/evidence/frontier/fetch_dataset.sh new file mode 100755 index 000000000..1bed48757 --- /dev/null +++ b/bench/evidence/frontier/fetch_dataset.sh @@ -0,0 +1,21 @@ +#!/bin/bash +# Fetch the pinned dataset and build the resource-aware run plan. +set -euo pipefail +source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/env.sh" + +test "$(command -v harbor)" = "$VENV/bin/harbor" || { + echo "FATAL: run setup_venv.sh first"; exit 1; } + +harbor download "$FB_DATASET" --export -o "$DATASET_DIR" + +# Shell glob, not fd/find: these scripts run on whatever bench host is handy, +# and the only tool they may assume is the shell itself. +got=0 +for f in "$DATASET_DIR"/*/*/task.toml; do [ -f "$f" ] && got=$((got+1)); done +test "$got" = "$FB_TASK_COUNT" || { + echo "FATAL: dataset exported $got tasks, expected $FB_TASK_COUNT." + echo " The digest pin and FB_TASK_COUNT disagree — one of them is stale." + exit 1; } + +python3 "$(dirname "${BASH_SOURCE[0]}")/plan.py" "$DATASET_DIR" \ + -o "$FB_PLAN" --images-out "$TB_ROOT/frontier-images.txt" diff --git a/bench/evidence/frontier/plan.py b/bench/evidence/frontier/plan.py new file mode 100755 index 000000000..9dc2b7769 --- /dev/null +++ b/bench/evidence/frontier/plan.py @@ -0,0 +1,284 @@ +#!/usr/bin/env python3 +"""Turn an exported Frontier-Bench dataset into a runnable schedule. + +Terminal-Bench's dev-baseline runner classified tasks with one line — anything +over 4 GB or one CPU went into a serial phase — because Terminal-Bench 2.1 was +homogeneous enough for that to work. Frontier-Bench is not. Its 74 tasks declare +1 to 16 CPUs, 2 to 32 GB of memory, 4 GB to 1 TB of storage, and four of them +want a GPU. Applying the Terminal-Bench rule puts 58 of 74 in the serial phase, +which is not a schedule so much as a surrender. + +Three things this file exists to prevent, each of which produces a *plausible +wrong number* rather than an error: + +1. A GPU task on a GPU-less host. The container starts, the work is impossible, + the verifier returns 0.0. That row is arithmetically identical to Stella + genuinely failing the task. Four of them silently cost ~5.4 points of pass + rate. They are excluded by default and named in the plan, so the denominator + is a stated choice instead of an accident. + +2. A task that cannot fit on the host at all. Same failure shape: not an error, + a zero. Excluded, named, and counted separately from tasks that ran. + +3. Oversubscription. Two 16 GB tasks scheduled together on a 16 GB Docker VM do + not fail cleanly — they fail as an unrelated-looking scatter of OOM kills, + build timeouts, and verifier errors spread across whichever tasks happened to + be running. Tasks are tiered by footprint and each tier gets the concurrency + its own headroom allows. + +It also collects the base images to warm. Terminal-Bench tasks named a prebuilt +``docker_image``; every Frontier-Bench task instead ships an +``environment/Dockerfile`` (12 of them a compose file too) and builds. The thing +to pull ahead of a measured run is therefore each Dockerfile's ``FROM`` bases, +not a task image — same reasoning as the Terminal-Bench prepull, which is that +with ``--max-retries 0`` a registry hiccup becomes a permanent reward-0 row, and +image availability is a precondition of the measurement rather than a term in +it. + +Usage:: + + python3 plan.py -o [--allow-gpu] [--headroom-mb N] +""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import shutil +import subprocess +import sys +from pathlib import Path + +if sys.version_info < (3, 11): # tomllib is what makes this parse rather than guess + sys.exit("FATAL: plan.py needs Python 3.11+ (source env.sh so the lane venv is on PATH)") + +import tomllib + +# Tier boundaries are memory ceilings in MB. A task lands in the first tier it +# fits. The names are load-bearing only as labels; the concurrency each tier +# gets is computed from the host, not from the name. +TIERS: tuple[tuple[str, int], ...] = ( + ("small", 2048), + ("medium", 4096), + ("large", 8192), + ("xlarge", 1 << 30), +) + +# Concurrency is capped regardless of how much headroom arithmetic allows. +# Docker's own build and network machinery, not task memory, is the next +# bottleneck past this point, and a stuck build is far more expensive to debug +# than a slightly longer run. +MAX_CONCURRENCY = 4 + +_FROM_RE = re.compile(r"^\s*FROM\s+(?:--\S+\s+)*(\S+)(?:\s+AS\s+(\S+))?", re.IGNORECASE) +_COMPOSE_IMAGE_RE = re.compile(r"^\s*image:\s*[\"']?([^\s\"']+)", re.MULTILINE) + + +# `.Path` and not the runtime entry itself: Go's `index` returns the zero value +# for a missing map key, and a zero-value struct is truthy in a template, so +# `{{if index .Runtimes "nvidia"}}` reports a GPU on every host alive. The +# registered binary's path is empty exactly when the runtime is absent. +_HOST_FORMAT = '{{.MemTotal}}|{{.NCPU}}|{{(index .Runtimes "nvidia").Path}}' + + +def _host_capacity() -> dict: + """What the Docker daemon says it can actually give a container.""" + cap = {"memory_mb": None, "cpus": None, "gpu": False, "source": "unavailable"} + try: + out = subprocess.run( + ["docker", "info", "--format", _HOST_FORMAT], + capture_output=True, text=True, timeout=60, check=True, + ).stdout.strip() + mem, cpus, nvidia_path = out.split("|", 2) + cap["memory_mb"] = int(mem) // (1024 * 1024) + cap["cpus"] = int(cpus) + # A GPU on the host that Docker cannot pass through is, for our + # purposes, no GPU at all. + cap["gpu"] = bool(nvidia_path.strip()) + cap["source"] = "docker info" + except Exception as exc: # noqa: BLE001 - any failure means "cannot verify" + cap["error"] = f"{type(exc).__name__}: {exc}" + return cap + + +def _read_task(task_dir: Path) -> dict: + cfg = tomllib.loads((task_dir / "task.toml").read_text()) + env = cfg.get("environment", {}) or {} + name = (cfg.get("task", {}) or {}).get("name") or task_dir.name + meta = cfg.get("metadata", {}) or {} + compose = [ + p for p in (task_dir / "environment").glob("docker-compose.y*ml") + ] if (task_dir / "environment").is_dir() else [] + return { + "name": name, + "slug": task_dir.name, + "cpus": int(env.get("cpus", 1)), + "memory_mb": int(env.get("memory_mb", 2048)), + "storage_mb": int(env.get("storage_mb", 0)), + "gpus": int(env.get("gpus", 0)), + "agent_timeout_sec": float((cfg.get("agent", {}) or {}).get("timeout_sec", 0) or 0), + "category": meta.get("category", "?"), + "compose": bool(compose), + "base_images": _base_images(task_dir, compose), + } + + +def _base_images(task_dir: Path, compose: list[Path]) -> list[str]: + """Every externally-pulled image this task's build depends on. + + Multi-stage builds name intermediate stages in later ``FROM`` lines + (``FROM builder``); those are local and must not be pulled. A stage name has + no registry path, tag, or digest, so anything without ``/``, ``:`` or ``@`` + is dropped — along with ``scratch``, which is not an image at all. + """ + images: set[str] = set() + stages: set[str] = set() + for dockerfile in sorted(task_dir.rglob("Dockerfile*")): + text = dockerfile.read_text(errors="replace") + for line in text.splitlines(): + m = _FROM_RE.match(line) + if not m: + continue + ref, alias = m.group(1), m.group(2) + if alias: + stages.add(alias.lower()) + images.add(ref) + for composefile in compose: + images.update(_COMPOSE_IMAGE_RE.findall(composefile.read_text(errors="replace"))) + + resolved = set() + for ref in images: + low = ref.lower() + if low in stages or low == "scratch" or ref.startswith("$"): + continue + if not any(c in ref for c in ("/", ":", "@")): + continue # a bare word with no registry coordinates is a build stage + resolved.add(ref) + return sorted(resolved) + + +def _tier(mem_mb: int) -> str: + for name, ceiling in TIERS: + if mem_mb <= ceiling: + return name + return TIERS[-1][0] + + +def build_plan(dataset_dir: Path, *, allow_gpu: bool, memory_headroom_mb: int) -> dict: + roots = sorted(p.parent for p in dataset_dir.glob("*/*/task.toml")) + if not roots: + roots = sorted(p.parent for p in dataset_dir.glob("*/task.toml")) + if not roots: + sys.exit(f"FATAL: no task.toml under {dataset_dir}") + + tasks = [_read_task(p) for p in roots] + host = _host_capacity() + host_mem = host.get("memory_mb") + host_cpus = host.get("cpus") + + usable_mem = (host_mem - memory_headroom_mb) if host_mem else None + excluded: list[dict] = [] + runnable: list[dict] = [] + + for t in tasks: + if t["gpus"] > 0 and not allow_gpu: + excluded.append({**t, "reason": "requires a GPU; FB_ALLOW_GPU is not set"}) + continue + if t["gpus"] > 0 and not host["gpu"]: + excluded.append({**t, "reason": "requires a GPU; Docker exposes no nvidia runtime"}) + continue + if usable_mem is not None and t["memory_mb"] > usable_mem: + excluded.append({ + **t, + "reason": f"needs {t['memory_mb']} MB; host offers {usable_mem} MB after headroom", + }) + continue + if host_cpus is not None and t["cpus"] > host_cpus: + excluded.append({**t, "reason": f"needs {t['cpus']} CPUs; host has {host_cpus}"}) + continue + runnable.append(t) + + tiers: dict[str, dict] = {} + for t in runnable: + tiers.setdefault(_tier(t["memory_mb"]), {"tasks": []})["tasks"].append(t["slug"]) + + for name, tier in tiers.items(): + members = [t for t in runnable if t["slug"] in set(tier["tasks"])] + peak_mem = max(t["memory_mb"] for t in members) + peak_cpus = max(t["cpus"] for t in members) + by_mem = (usable_mem // peak_mem) if usable_mem else 1 + by_cpu = (host_cpus // peak_cpus) if host_cpus else 1 + tier["concurrency"] = max(1, min(MAX_CONCURRENCY, by_mem, by_cpu)) + tier["peak_memory_mb"] = peak_mem + tier["peak_cpus"] = peak_cpus + tier["tasks"].sort() + + images = sorted({img for t in runnable for img in t["base_images"]}) + + # Storage is advisory: the declared figure is a ceiling the task may never + # approach, and on macOS the Docker VM's free space is not visible from the + # host anyway. Reporting it beats pretending to enforce it. + free_mb = shutil.disk_usage(dataset_dir).free // (1024 * 1024) + + return { + "dataset_dir": str(dataset_dir), + "host": host, + "memory_headroom_mb": memory_headroom_mb, + "allow_gpu": allow_gpu, + "counts": { + "total": len(tasks), + "runnable": len(runnable), + "excluded": len(excluded), + }, + "tiers": {k: tiers[k] for k, _ in TIERS if k in tiers}, + "excluded": sorted(excluded, key=lambda t: t["slug"]), + "base_images": images, + "declared_storage_mb": sum(t["storage_mb"] for t in runnable), + "host_free_mb_at_dataset": free_mb, + "runnable_tasks": sorted(t["slug"] for t in runnable), + } + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("dataset_dir", type=Path) + ap.add_argument("-o", "--output", type=Path, required=True) + ap.add_argument("--allow-gpu", action="store_true", + default=os.environ.get("FB_ALLOW_GPU") == "1") + # Memory, not storage. The two were one knob in the first draft of this + # file and a 20 GB *storage* headroom subtracted from a 10 GB Docker VM's + # *memory* excluded all 74 tasks with a negative budget — the plan was + # empty and said so politely. + ap.add_argument("--memory-headroom-mb", type=int, + default=int(os.environ.get("FB_MEMORY_HEADROOM_MB", "2048"))) + ap.add_argument("--images-out", type=Path, default=None) + args = ap.parse_args() + + plan = build_plan(args.dataset_dir, allow_gpu=args.allow_gpu, + memory_headroom_mb=args.memory_headroom_mb) + args.output.write_text(json.dumps(plan, indent=1) + "\n") + if args.images_out: + args.images_out.write_text("\n".join(plan["base_images"]) + "\n") + + host = plan["host"] + print(f"host: {host.get('memory_mb')} MB / {host.get('cpus')} CPUs " + f"/ gpu={host.get('gpu')} ({host.get('source')})") + for name, tier in plan["tiers"].items(): + print(f" tier {name:<7} tasks={len(tier['tasks']):<3} " + f"concurrency={tier['concurrency']} peak={tier['peak_memory_mb']}MB/" + f"{tier['peak_cpus']}cpu") + c = plan["counts"] + print(f"runnable={c['runnable']}/{c['total']} excluded={c['excluded']} " + f"images={len(plan['base_images'])}") + for t in plan["excluded"]: + print(f" EXCLUDED {t['slug']}: {t['reason']}") + if plan["excluded"]: + print("NOTE: excluded tasks are absent from the run, not scored as failures.") + print(" Any published number must state the denominator it used.") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/bench/evidence/frontier/primary.sh b/bench/evidence/frontier/primary.sh new file mode 100755 index 000000000..276d0b756 --- /dev/null +++ b/bench/evidence/frontier/primary.sh @@ -0,0 +1,72 @@ +#!/bin/bash +# The measured run for one resource tier. Publish the preregistration before +# calling this. +set -uo pipefail +source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/env.sh" +fb_preflight || exit 1 + +TIER="${1:?usage: primary.sh }" +JOB="${2:?usage: primary.sh }" +test ! -e "$JOBS/$JOB" || { echo "FATAL: $JOBS/$JOB exists — a run never resumes"; exit 1; } +test -f "$FB_PLAN" || { echo "FATAL: no plan at $FB_PLAN (run fetch_dataset.sh)"; exit 1; } + +read -r N CONC </dev/null || { echo "FATAL: no tasks for tier $TIER"; exit 1; } + +CONC="${FB_CONCURRENCY:-$CONC}" + +# macOS ships bash 3.2: no mapfile, no arrays from process substitution. +INCLUDES="" +while IFS= read -r t; do + [ -n "$t" ] || continue + INCLUDES="$INCLUDES --include-task-name $FB_TASK_PREFIX/$t" +done < "$TB_ROOT/frontier-$TIER.tasks" + +echo "tier=$TIER job=$JOB tasks=$N concurrency=$CONC attempts=$FB_ATTEMPTS" +echo "sut=$STELLA_SOURCE_COMMIT model=$TB_MODEL budget/trial=${STELLA_BUDGET:-uncapped}" +echo "dataset=$FB_DATASET" +echo "harbor=$(harbor --version) started=$(date -u +%Y-%m-%dT%H:%M:%SZ)" + +# Say plainly what this run is and is not. A tiered, GPU-excluding, single +# attempt run is a development baseline; the leaderboard wants all 74 tasks at +# five-plus attempts with no subsetting. Both are legitimate — reporting one as +# the other is not, and the distinction is easiest to lose at the moment a +# number finally appears. +if [ "$TIER" != "all" ] || [ "$FB_ATTEMPTS" -lt "$FB_SUBMISSION_MIN_ATTEMPTS" ]; then + echo "posture=development-baseline (not submittable: see SUBMISSION.md)" +else + echo "posture=submission-shaped" +fi + +cd "$TB_REPO" || exit 1 +# INCLUDES is deliberately unquoted: a pre-built flag list, one word per token, +# every task name a fixed [a-z0-9-] slug from the digest-pinned dataset. +# shellcheck disable=SC2086 +harbor run \ + --env docker \ + --dataset "$FB_DATASET" \ + $INCLUDES \ + --agent-import-path stella_harbor:StellaAgent \ + --model "$TB_MODEL" \ + --job-name "$JOB" \ + --jobs-dir "$JOBS" \ + --n-attempts "$FB_ATTEMPTS" --n-concurrent "$CONC" --max-retries 0 +rc=$? +echo "harbor_exit=$rc finished=$(date -u +%Y-%m-%dT%H:%M:%SZ)" +exit $rc diff --git a/bench/evidence/frontier/sentinel.sh b/bench/evidence/frontier/sentinel.sh new file mode 100755 index 000000000..4d40f0534 --- /dev/null +++ b/bench/evidence/frontier/sentinel.sh @@ -0,0 +1,119 @@ +#!/bin/bash +# Two cheap trials that prove the whole path end to end before 70 expensive +# ones depend on it. +# +# stage 1 the synthetic fixture — adapter install, binary upload and SHA +# verification, tool loop, metering, verifier handoff. Identical to +# the Terminal-Bench sentinel except that it runs under *this* lane's +# newer Harbor, which is the thing that changed. +# +# stage 2 the cheapest real Frontier-Bench task — a real Dockerfile build and +# the separate-container verifier that every task in this set +# declares and that Harbor 0.6.1 would have silently ignored. +# +# The two stages are gated differently, and the difference is the point. The +# synthetic task is oracle-solvable, so anything but reward 1.0 is a broken +# harness. A real Frontier-Bench task is *not* expected to pass — the best +# published agents clear about a third of this set — so demanding reward 1.0 +# there would gate on model quality instead of on plumbing. Stage 2 asks only +# whether the trial genuinely ran: binary verified inside the container, a +# status reported, a reward produced by the verifier, and no infrastructure +# exception. A task that ran and scored 0.0 passes this gate, correctly. +# +# Both stages are excluded from every analysis. +set -uo pipefail +source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/env.sh" +fb_preflight || exit 1 + +STAMP="$(date +%Y%m%d-%H%M%S)" +JOB="${1:-fb-sentinel-$STAMP}" +test ! -e "$JOBS/$JOB" || { echo "FATAL: $JOBS/$JOB exists"; exit 1; } +echo "job=$JOB sut=$STELLA_SOURCE_COMMIT harbor=$(harbor --version)" + +cd "$TB_REPO" || exit 1 + +echo "=== stage 1: synthetic fixture (gate: reward == 1.0)" +harbor run \ + --env docker \ + --path "$TB_REPO/bench/readiness/synthetic-adapter-sentinel" \ + --agent-import-path stella_harbor:StellaAgent \ + --model "$TB_MODEL" \ + --job-name "$JOB" \ + --jobs-dir "$JOBS" \ + --n-attempts 1 --n-concurrent 1 --max-retries 0 +echo "harbor_exit=$?" + +python3 - "$JOBS/$JOB" <<'PY' || exit 1 +import glob, json, sys +rows = [json.load(open(p)) for p in glob.glob(f"{sys.argv[1]}/*/result.json")] +rows = [r for r in rows if r.get("task_name")] +if not rows: + print("STAGE 1 FAIL: no trial produced a result"); raise SystemExit(1) +r = rows[0] +reward = (r.get("verifier_result") or {}).get("rewards", {}).get("reward") +md = ((r.get("agent_result") or {}).get("metadata") or {}) +print(f"reward={reward} stella_status={md.get('stella_status')} " + f"binary_verified={md.get('stella_binary_sha256_verified_in_container')} " + f"commit_verified={md.get('stella_source_commit_verified_in_binary')}") +raise SystemExit(0 if reward == 1.0 else 1) +PY + +test -f "$FB_PLAN" || { echo "SKIP stage 2: no plan at $FB_PLAN (run fetch_dataset.sh)"; exit 0; } + +TASK="${FB_SENTINEL_TASK:-$(python3 -c " +import json +p=json.load(open('$FB_PLAN')) +runnable=set(p['runnable_tasks']) +# Cheapest by declared footprint, then by name so the choice is reproducible. +import tomllib, pathlib +best=None +for d in sorted(pathlib.Path(p['dataset_dir']).glob('*/*/task.toml')): + if d.parent.name not in runnable: continue + c=tomllib.loads(d.read_text()); e=c.get('environment',{}) or {} + key=(e.get('memory_mb',0), e.get('cpus',0), (c.get('agent',{}) or {}).get('timeout_sec',0), d.parent.name) + if best is None or key/dev/null 2>&1 || { echo "FATAL: uv is required"; exit 1; } + +uv venv "$VENV" --python 3.13 || exit 1 +uv pip install --python "$VENV/bin/python" "harbor==$FB_HARBOR_VERSION" || exit 1 + +got="$("$VENV/bin/harbor" --version)" +test "$got" = "$FB_HARBOR_VERSION" || { + echo "FATAL: installed harbor $got != $FB_HARBOR_VERSION"; exit 1; } + +# The adapter is imported from the repo (PYTHONPATH), not installed, so this is +# the moment to prove it still satisfies the newer Harbor's agent interface +# rather than discovering it one trial into a paid run. +PYTHONPATH="$TB_REPO/bench/harbor_adapter" "$VENV/bin/python" - <<'PY' || exit 1 +from stella_harbor import StellaAgent +missing = sorted(getattr(StellaAgent, "__abstractmethods__", ())) +assert not missing, f"StellaAgent does not implement: {missing}" +assert StellaAgent.name() == "stella" +print(f"adapter OK against this Harbor: {StellaAgent.name()} (ATIF={StellaAgent.SUPPORTS_ATIF})") +PY + +echo "venv ready: $VENV (harbor $got)" diff --git a/bench/evidence/frontier/warm_images.sh b/bench/evidence/frontier/warm_images.sh new file mode 100755 index 000000000..1ef4df8c2 --- /dev/null +++ b/bench/evidence/frontier/warm_images.sh @@ -0,0 +1,46 @@ +#!/bin/bash +# Pull every base image the task builds depend on, with retries, before the +# measured run. +# +# The Terminal-Bench lane calls this prepull, and pulls each task's prebuilt +# `docker_image`. No Frontier-Bench task has one: all 74 ship an +# environment/Dockerfile and build in place. So the thing to warm is the set of +# `FROM` bases (plus any compose service images), which `plan.py` collects. +# +# The reason is unchanged from the Terminal-Bench lane, and worth restating +# because it is the whole point: with --max-retries 0 a registry hiccup is a +# permanent reward-0 row, arithmetically identical to Stella failing the task. +# Image availability is a precondition of the measurement, not a term in it. +set -uo pipefail +source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/env.sh" + +LIST="${1:-$TB_ROOT/frontier-images.txt}" +FAILED="$TB_ROOT/frontier_warm_failed.txt" +test -f "$LIST" || { echo "FATAL: no image list at $LIST (run fetch_dataset.sh)"; exit 1; } +: > "$FAILED" + +total=$(grep -c . "$LIST") +i=0 +while IFS= read -r img; do + [ -n "$img" ] || continue + i=$((i+1)) + if docker image inspect "$img" >/dev/null 2>&1; then + echo "[$i/$total] have $img"; continue + fi + ok=0 + for attempt in 1 2 3 4 5; do + if timeout 1200 docker pull --quiet "$img" >/dev/null 2>&1; then ok=1; break; fi + sleep $((attempt * 10)) + done + if [ "$ok" = 1 ]; then echo "[$i/$total] pulled $img" + else echo "[$i/$total] FAILED $img"; echo "$img" >> "$FAILED"; fi +done < "$LIST" + +nfailed=$(grep -c . "$FAILED" || echo 0) +echo "WARM_DONE failed=$nfailed" +# A base image that will not pull now is a task that cannot build later. Say so +# before the run rather than letting it arrive as a reward-0 row. +test "$nfailed" = 0 || { + echo "WARNING: the tasks depending on these images will fail to build." + echo " Resolve or exclude them before starting a measured run." + exit 1; } From c525c8e15b0a321b2f988066ad0e001b43f0d3d9 Mon Sep 17 00:00:00 2001 From: Stella Test Date: Fri, 31 Jul 2026 15:09:55 -0700 Subject: [PATCH 2/7] fix(bench): wire the memory-headroom knob the rename orphaned MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit plan.py's headroom argument started life conflating two unrelated quantities — a 20 GB *storage* headroom was being subtracted from a 10 GB Docker VM's *memory*, which left a negative budget and excluded all 74 tasks with a polite explanation. Splitting them fixed the arithmetic but left env.sh exporting the old FB_STORAGE_HEADROOM_MB, which nothing reads any more, while plan.py's FB_MEMORY_HEADROOM_MB fell back to its default however the operator set it. Export the name plan.py actually reads, and say why the default is conservative: memory_mb is a cap Docker accepts even on a smaller VM, so an 8 GB task on a 10 GB daemon usually starts and then gets OOM-killed partway in — a reward-0 row indistinguishable from a real failure. At the 2048 MB default this host runs 48 of 74 tasks; at 512 it runs 66. Both are defensible, neither is silent. --- bench/evidence/frontier/README.md | 7 +++++++ bench/evidence/frontier/env.sh | 15 ++++++++++----- 2 files changed, 17 insertions(+), 5 deletions(-) diff --git a/bench/evidence/frontier/README.md b/bench/evidence/frontier/README.md index 290af9ec5..d31233c8c 100644 --- a/bench/evidence/frontier/README.md +++ b/bench/evidence/frontier/README.md @@ -74,6 +74,13 @@ typical Mac) it admits **48 of 74** tasks — 4 GPU-excluded and 22 over the mem budget. That is a real constraint, not a bug, and it is why a submittable run needs a bigger machine: see [SUBMISSION.md](SUBMISSION.md). +The memory exclusion is deliberately conservative. `memory_mb` is a cap Docker +accepts even when the VM is smaller, so an 8 GB task on a 10 GB daemon usually +starts — and then swaps and gets OOM-killed partway in, arriving as a reward-0 +row that looks exactly like a genuine failure. Lower `FB_MEMORY_HEADROOM_MB` +(default 2048) to attempt them anyway; every exclusion is named with its reason, +so nothing disappears quietly either way. + ## Harbor version and the test suite `bench/harbor_adapter/tests/` passes 132/132 under 0.6.1 and 131/132 under diff --git a/bench/evidence/frontier/env.sh b/bench/evidence/frontier/env.sh index e09385c67..da3863037 100755 --- a/bench/evidence/frontier/env.sh +++ b/bench/evidence/frontier/env.sh @@ -108,11 +108,16 @@ mkdir -p "$JOBS" # Set FB_ALLOW_GPU=1 only on a host that actually has one; `plan.py` checks. export FB_ALLOW_GPU="${FB_ALLOW_GPU:-0}" -# The declared storage footprint of the full set is ~1.9 TB, one task -# (jax-speedrun-gpu) accounting for 1 TB of it. A run that fills the disk -# halfway through does not fail cleanly — it fails as a scatter of unrelated -# task errors — so the plan refuses to schedule a bin the host cannot hold. -export FB_STORAGE_HEADROOM_MB="${FB_STORAGE_HEADROOM_MB:-20480}" +# Memory the plan leaves to the Docker daemon itself rather than handing to a +# task. A task declaring more than what remains is excluded as unrunnable. +# +# This is deliberately conservative: `memory_mb` is a cap Docker will happily +# accept on a smaller VM, so an 8 GB task on a 10 GB daemon usually *starts* — +# and then competes with the daemon, swaps, and gets OOM-killed partway in, +# which arrives as a reward-0 row indistinguishable from a genuine failure. +# Excluding it and saying so is the honest version of the same outcome. Lower +# this to attempt those tasks anyway; the exclusion list names every one. +export FB_MEMORY_HEADROOM_MB="${FB_MEMORY_HEADROOM_MB:-2048}" fb_preflight() { test -n "${OPENROUTER_API_KEY:-}" || { echo "FATAL: OPENROUTER_API_KEY unset"; return 1; } From d485a2b9caf9c33de1761ad927034fd859e6b342 Mon Sep 17 00:00:00 2001 From: Stella Test Date: Fri, 31 Jul 2026 15:12:37 -0700 Subject: [PATCH 3/7] test(bench): cover the frontier planner's silent-wrong-number paths MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit plan.py decides which tasks are allowed to run, and every way it can be wrong produces a plausible plan rather than an error — a GPU task scheduled on a GPU-less host, a task admitted onto a machine that cannot hold it, a build stage mistaken for an image to pull. Two of the 21 cases are regressions already hit while writing it: the headroom that subtracted a storage allowance from a memory budget and excluded all 74 tasks, and the Docker GPU probe, where the obvious `{{if index .Runtimes "nvidia"}}` reports a GPU on every host alive because Go returns a truthy zero-value struct for a missing key. Also pins the boundary the first draft of the test got wrong: a task whose footprint exactly equals the remaining budget is admitted, not excluded. The comparison is strictly greater-than and should stay that way — 8192 MB is the second most common footprint in the set, so an off-by-one there would drop real tasks for no reason. plan.py imports nothing outside the standard library, which is what lets CI run these with `uv run --with pytest --no-project` — no project sync, no Harbor install. The piece that decides what runs should not need the benchmark harness to be testable. Extends bench.yml's scope check to bench/evidence/frontier/ so the suite actually gates. --- .github/workflows/bench.yml | 10 +- bench/evidence/frontier/tests/test_plan.py | 336 +++++++++++++++++++++ 2 files changed, 345 insertions(+), 1 deletion(-) create mode 100644 bench/evidence/frontier/tests/test_plan.py diff --git a/.github/workflows/bench.yml b/.github/workflows/bench.yml index d82ebbff0..34c21e62c 100644 --- a/.github/workflows/bench.yml +++ b/.github/workflows/bench.yml @@ -58,7 +58,7 @@ jobs: exit 0 fi if git diff --name-only "$BASE_SHA"...HEAD \ - | grep -Eq '^(bench/harbor_adapter/|bench/terminal_bench_analysis/|\.github/workflows/bench\.yml$)'; then + | grep -Eq '^(bench/harbor_adapter/|bench/terminal_bench_analysis/|bench/evidence/frontier/|\.github/workflows/bench\.yml$)'; then echo "changed=true" >> "$GITHUB_OUTPUT" else echo "changed=false" >> "$GITHUB_OUTPUT" @@ -83,3 +83,11 @@ jobs: run: | uv sync --locked --extra dev uv run --no-sync pytest -q + + # The Frontier-Bench run planner. No project to sync and no Harbor to + # install: plan.py imports nothing outside the standard library, which is + # deliberate — the piece that decides which tasks are allowed to run + # should not need the benchmark harness to be testable. + - name: frontier planner — pytest + if: steps.scope.outputs.changed == 'true' + run: uv run --with pytest --no-project pytest -q bench/evidence/frontier/tests diff --git a/bench/evidence/frontier/tests/test_plan.py b/bench/evidence/frontier/tests/test_plan.py new file mode 100644 index 000000000..07fc7135e --- /dev/null +++ b/bench/evidence/frontier/tests/test_plan.py @@ -0,0 +1,336 @@ +"""Tests for the Frontier-Bench run planner. + +Everything here is about a *silent wrong number* rather than a crash. The +planner's whole job is to keep an infrastructure limitation from arriving as a +reward-0 row that looks exactly like Stella failing a task, so the cases worth +pinning are the ones where a mistake still produces a plausible-looking plan: +a GPU task quietly scheduled on a GPU-less host, a task admitted onto a machine +that cannot hold it, a build stage mistaken for an image to pull. + +Two of these are regressions. The headroom test exists because the argument +originally conflated storage with memory and subtracted a 20 GB storage +allowance from a 10 GB memory budget, which excluded all 74 tasks with a +negative number and a polite explanation. The GPU test exists because the +obvious Docker probe — ``{{if index .Runtimes "nvidia"}}`` — reports a GPU on +every host alive, Go returning a truthy zero-value struct for a missing key. + +``plan.py`` is a script rather than a package and imports nothing beyond the +standard library, so it loads by path and needs no Harbor to test. +""" + +from __future__ import annotations + +import importlib.util +import sys +from pathlib import Path + +import pytest + +_PLAN_PATH = Path(__file__).resolve().parent.parent / "plan.py" +_spec = importlib.util.spec_from_file_location("frontier_plan", _PLAN_PATH) +assert _spec and _spec.loader +plan = importlib.util.module_from_spec(_spec) +sys.modules["frontier_plan"] = plan +_spec.loader.exec_module(plan) + + +def _task( + root: Path, + slug: str, + *, + cpus: int = 1, + memory_mb: int = 2048, + storage_mb: int = 10240, + gpus: int = 0, + dockerfile: str = "FROM python:3.13-slim\n", + compose: str | None = None, + tests_dockerfile: str | None = None, +) -> Path: + """Write a task tree shaped like a real exported Frontier-Bench task.""" + d = root / "frontier-bench" / slug + (d / "environment").mkdir(parents=True) + (d / "task.toml").write_text( + f'[task]\nname = "frontier-bench/{slug}"\n' + f'[metadata]\ncategory = "Software"\n' + f"[agent]\ntimeout_sec = 7200.0\n" + f"[environment]\ncpus = {cpus}\nmemory_mb = {memory_mb}\n" + f"storage_mb = {storage_mb}\ngpus = {gpus}\n" + ) + (d / "environment" / "Dockerfile").write_text(dockerfile) + if compose is not None: + (d / "environment" / "docker-compose.yaml").write_text(compose) + if tests_dockerfile is not None: + (d / "tests").mkdir() + (d / "tests" / "Dockerfile").write_text(tests_dockerfile) + return d + + +@pytest.fixture +def host(monkeypatch): + """Pin the host capacity so a test never depends on the machine running it.""" + + def _set(memory_mb=16384, cpus=8, gpu=False): + monkeypatch.setattr( + plan, + "_host_capacity", + lambda: {"memory_mb": memory_mb, "cpus": cpus, "gpu": gpu, "source": "test"}, + ) + + return _set + + +class TestGpuExclusion: + def test_gpu_task_is_excluded_by_default_and_says_why(self, tmp_path, host): + host(gpu=False) + _task(tmp_path, "needs-gpu", gpus=1) + _task(tmp_path, "plain") + + result = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048) + + assert result["runnable_tasks"] == ["plain"] + (excluded,) = result["excluded"] + assert excluded["slug"] == "needs-gpu" + assert "GPU" in excluded["reason"] + + def test_allow_gpu_alone_is_not_enough_without_a_host_gpu(self, tmp_path, host): + """The operator asking for GPU tasks does not conjure a GPU. + + Honouring the flag on a GPU-less host is the exact failure this planner + exists to prevent: the container starts, the work is impossible, and the + verifier returns 0.0 indistinguishably from a genuine miss. + """ + host(gpu=False) + _task(tmp_path, "needs-gpu", gpus=1) + + result = plan.build_plan(tmp_path, allow_gpu=True, memory_headroom_mb=2048) + + assert result["runnable_tasks"] == [] + assert "nvidia" in result["excluded"][0]["reason"] + + def test_gpu_task_runs_when_the_host_actually_has_one(self, tmp_path, host): + host(gpu=True) + _task(tmp_path, "needs-gpu", gpus=1) + + result = plan.build_plan(tmp_path, allow_gpu=True, memory_headroom_mb=2048) + + assert result["runnable_tasks"] == ["needs-gpu"] + assert result["excluded"] == [] + + +class TestHeadroom: + def test_headroom_is_subtracted_from_memory_not_storage(self, tmp_path, host): + """Regression: a storage-sized headroom against memory excluded everything. + + A 20 GB allowance subtracted from a 10 GB memory budget goes negative, + and every task — including a 2 GB one — falls off the plan. + """ + host(memory_mb=10240, cpus=8) + _task(tmp_path, "small-task", memory_mb=2048, storage_mb=1024000) + + result = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048) + + assert result["runnable_tasks"] == ["small-task"] + assert result["counts"]["runnable"] == 1 + + def test_task_larger_than_the_remaining_budget_is_excluded(self, tmp_path, host): + host(memory_mb=10240, cpus=8) + _task(tmp_path, "hungry", memory_mb=12288) + + result = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048) + + assert result["runnable_tasks"] == [] + assert "12288" in result["excluded"][0]["reason"] + + def test_a_task_that_exactly_fills_the_budget_is_admitted(self, tmp_path, host): + """The comparison is strictly greater-than, and should stay that way. + + 10240 MB less a 2048 MB headroom leaves exactly 8192. Excluding a task + that fits precisely would drop real tasks for no reason — 8192 MB is the + second most common footprint in the set. + """ + host(memory_mb=10240, cpus=8) + _task(tmp_path, "exact", memory_mb=8192) + + result = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048) + + assert result["runnable_tasks"] == ["exact"] + + def test_lowering_the_headroom_admits_the_same_task(self, tmp_path, host): + host(memory_mb=10240, cpus=8) + _task(tmp_path, "hungry", memory_mb=8192) + + result = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=512) + + assert result["runnable_tasks"] == ["hungry"] + + def test_task_needing_more_cpus_than_the_host_is_excluded(self, tmp_path, host): + host(memory_mb=65536, cpus=4) + _task(tmp_path, "wide", cpus=16) + + result = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048) + + assert result["runnable_tasks"] == [] + assert "CPU" in result["excluded"][0]["reason"] + + +class TestTiering: + def test_tasks_land_in_the_tier_matching_their_memory(self, tmp_path, host): + host(memory_mb=65536, cpus=32) + _task(tmp_path, "a-tiny", memory_mb=2048) + _task(tmp_path, "b-mid", memory_mb=4096) + _task(tmp_path, "c-big", memory_mb=8192) + _task(tmp_path, "d-huge", memory_mb=32768) + + tiers = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048)["tiers"] + + assert tiers["small"]["tasks"] == ["a-tiny"] + assert tiers["medium"]["tasks"] == ["b-mid"] + assert tiers["large"]["tasks"] == ["c-big"] + assert tiers["xlarge"]["tasks"] == ["d-huge"] + + def test_concurrency_is_bounded_by_whichever_resource_runs_out_first( + self, tmp_path, host + ): + # 16 GB usable after headroom; each task wants 4 GB, so memory allows 4. + # But 5 CPUs against 2 per task allows only 2 — the smaller wins. + host(memory_mb=18432, cpus=5) + _task(tmp_path, "a", memory_mb=4096, cpus=2) + _task(tmp_path, "b", memory_mb=4096, cpus=2) + + tiers = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048)["tiers"] + + assert tiers["medium"]["concurrency"] == 2 + + def test_concurrency_never_drops_below_one(self, tmp_path, host): + """A task that only just fits still gets scheduled, serially.""" + host(memory_mb=10240, cpus=2) + _task(tmp_path, "just-fits", memory_mb=8192, cpus=2) + + tiers = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=512)["tiers"] + + assert tiers["large"]["concurrency"] == 1 + + def test_concurrency_is_capped_regardless_of_headroom(self, tmp_path, host): + host(memory_mb=1 << 20, cpus=512) + _task(tmp_path, "tiny", memory_mb=2048, cpus=1) + + tiers = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048)["tiers"] + + assert tiers["small"]["concurrency"] == plan.MAX_CONCURRENCY + + +class TestBaseImages: + def test_multi_stage_aliases_are_not_pulled(self, tmp_path, host): + """``FROM builder`` names a local stage; pulling it would fail.""" + host() + _task( + tmp_path, + "staged", + dockerfile=( + "FROM golang:1.24 AS builder\n" + "RUN go build\n" + "FROM debian:12-slim\n" + "COPY --from=builder /app /app\n" + ), + ) + + images = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048)["base_images"] + + assert images == ["debian:12-slim", "golang:1.24"] + + def test_scratch_and_arg_substituted_bases_are_skipped(self, tmp_path, host): + host() + _task( + tmp_path, + "odd", + dockerfile="ARG BASE\nFROM scratch\nFROM ${BASE}\nFROM ubuntu:24.04\n", + ) + + images = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048)["base_images"] + + assert images == ["ubuntu:24.04"] + + def test_compose_service_images_are_included(self, tmp_path, host): + host() + _task( + tmp_path, + "composed", + compose="services:\n db:\n image: postgres:16\n app:\n build: .\n", + ) + + images = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048)["base_images"] + + assert "postgres:16" in images + + def test_the_verifier_dockerfile_counts_too(self, tmp_path, host): + """Every task here declares environment_mode = "separate". + + The verifier builds its own container, so its base is just as much a + precondition of the run as the agent's. + """ + host() + _task(tmp_path, "separate-verifier", tests_dockerfile="FROM node:22-bookworm\n") + + images = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048)["base_images"] + + assert "node:22-bookworm" in images + + def test_images_are_deduplicated_across_tasks(self, tmp_path, host): + host() + _task(tmp_path, "one", dockerfile="FROM python:3.13-slim\n") + _task(tmp_path, "two", dockerfile="FROM python:3.13-slim\n") + + images = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048)["base_images"] + + assert images == ["python:3.13-slim"] + + def test_an_excluded_task_does_not_drag_in_its_images(self, tmp_path, host): + """Warming an image for a task that will not run is wasted bandwidth.""" + host(gpu=False) + _task(tmp_path, "gpu-only", gpus=1, dockerfile="FROM nvidia/cuda:12.6.3-base\n") + _task(tmp_path, "plain", dockerfile="FROM ubuntu:24.04\n") + + images = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048)["base_images"] + + assert images == ["ubuntu:24.04"] + + +class TestPlanShape: + def test_counts_reconcile(self, tmp_path, host): + host(memory_mb=10240, cpus=8, gpu=False) + _task(tmp_path, "ok") + _task(tmp_path, "gpu", gpus=1) + _task(tmp_path, "huge", memory_mb=32768) + + result = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048) + + c = result["counts"] + assert c["total"] == 3 + assert c["runnable"] + c["excluded"] == c["total"] + assert len(result["excluded"]) == c["excluded"] + + def test_an_empty_dataset_directory_is_fatal_rather_than_an_empty_plan( + self, tmp_path, host + ): + host() + with pytest.raises(SystemExit): + plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048) + + def test_unknown_host_capacity_still_produces_a_runnable_plan(self, tmp_path, monkeypatch): + """`docker info` failing must not silently empty the plan. + + Excluding every task because capacity could not be read looks identical + to a host that is too small, so the planner admits them and lets the + preflight's own docker check be the thing that fails. + """ + monkeypatch.setattr( + plan, "_host_capacity", + lambda: {"memory_mb": None, "cpus": None, "gpu": False, + "source": "unavailable", "error": "boom"}, + ) + _task(tmp_path, "a", memory_mb=32768) + + result = plan.build_plan(tmp_path, allow_gpu=False, memory_headroom_mb=2048) + + assert result["runnable_tasks"] == ["a"] + assert result["tiers"]["xlarge"]["concurrency"] == 1 From 7889e1d157f0571b73f483e02805ea05d7828f47 Mon Sep 17 00:00:00 2001 From: Stella Test Date: Fri, 31 Jul 2026 15:15:54 -0700 Subject: [PATCH 4/7] fix(bench): the frontier lane passed a flag Harbor 0.20.0 removed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `--agent-import-path` is 0.6.1 spelling. By 0.20.0 it is folded into `--agent`, which takes either a built-in name or an import path. Both primary.sh and sentinel.sh passed the old one, so every run in this lane would have died on `No such option` at the first trial — after the venv build, the dataset download, the image warm, and the preflight had all passed. The Terminal-Bench lane keeps the old spelling and is still right to: it is pinned to 0.6.1, where that is the flag. Add fb_assert_cli_flags to the preflight so the next such rename costs one message instead of a run. Running two Harbor majors in one repo makes the CLI a moving contract, and it has now moved once. The check scrubs ANSI before matching, which is not incidental: Harbor renders help through Rich, which styles each option individually and leaves escape sequences inside the flag names. A literal search for `--env` over the raw 42 KB finds nothing, so the unscrubbed first draft reported all eleven flags missing and failed closed on a working CLI. --- bench/evidence/frontier/env.sh | 30 +++++++++++++++++++++++++++++ bench/evidence/frontier/primary.sh | 2 +- bench/evidence/frontier/sentinel.sh | 4 ++-- 3 files changed, 33 insertions(+), 3 deletions(-) diff --git a/bench/evidence/frontier/env.sh b/bench/evidence/frontier/env.sh index da3863037..3c4751726 100755 --- a/bench/evidence/frontier/env.sh +++ b/bench/evidence/frontier/env.sh @@ -119,6 +119,35 @@ export FB_ALLOW_GPU="${FB_ALLOW_GPU:-0}" # this to attempt those tasks anyway; the exclusion list names every one. export FB_MEMORY_HEADROOM_MB="${FB_MEMORY_HEADROOM_MB:-2048}" +# Assert `harbor run` still advertises every flag these scripts pass it. +# +# Running two Harbor major versions in one repo means the CLI is a moving +# contract, and it has already moved once: 0.6.1's `--agent-import-path` was +# folded into `--agent` by 0.20.0, which accepts either a built-in name or an +# import path. The Terminal-Bench lane still passes the old spelling and is +# still correct to, because it is pinned to 0.6.1. This lane would have died on +# `No such option` at the first trial of the first run. +# +# Checking the help text costs one subprocess and turns that into one message. +fb_assert_cli_flags() { + local help missing="" + # Scrub ANSI first. Harbor renders help through Rich, which styles each + # option individually and leaves escape sequences *inside* the flag names — + # a literal search for `--env` over the raw 42 KB finds nothing, so an + # unscrubbed version of this check reports every flag missing and fails + # closed on a perfectly good CLI. + help="$(harbor run --help 2>&1 | LC_ALL=C sed $'s/\033\[[0-9;?]*[a-zA-Z]//g')" || + { echo "FATAL: \`harbor run --help\` failed"; return 1; } + for flag in --env --dataset --path --agent --model --job-name --jobs-dir \ + --include-task-name --n-attempts --n-concurrent --max-retries; do + case "$help" in *"$flag"*) ;; *) missing="$missing $flag" ;; esac + done + test -z "$missing" || { + echo "FATAL: harbor $(harbor --version) does not advertise:$missing" + echo " The CLI contract moved. Update this lane's scripts before running." + return 1; } +} + fb_preflight() { test -n "${OPENROUTER_API_KEY:-}" || { echo "FATAL: OPENROUTER_API_KEY unset"; return 1; } test -x "$STELLA_BINARY" || { echo "FATAL: no SUT binary at $STELLA_BINARY (run ../run/build_sut.sh)"; return 1; } @@ -133,6 +162,7 @@ fb_preflight() { echo "FATAL: harbor $(harbor --version) != $FB_HARBOR_VERSION" echo " 0.6.1 silently ignores this dataset's verifier environment_mode." return 1; } + fb_assert_cli_flags || return 1 docker info >/dev/null 2>&1 || { echo "FATAL: docker unreachable"; return 1; } } diff --git a/bench/evidence/frontier/primary.sh b/bench/evidence/frontier/primary.sh index 276d0b756..40042d4e4 100755 --- a/bench/evidence/frontier/primary.sh +++ b/bench/evidence/frontier/primary.sh @@ -62,7 +62,7 @@ harbor run \ --env docker \ --dataset "$FB_DATASET" \ $INCLUDES \ - --agent-import-path stella_harbor:StellaAgent \ + --agent stella_harbor:StellaAgent \ --model "$TB_MODEL" \ --job-name "$JOB" \ --jobs-dir "$JOBS" \ diff --git a/bench/evidence/frontier/sentinel.sh b/bench/evidence/frontier/sentinel.sh index 4d40f0534..1bcd61d58 100755 --- a/bench/evidence/frontier/sentinel.sh +++ b/bench/evidence/frontier/sentinel.sh @@ -36,7 +36,7 @@ echo "=== stage 1: synthetic fixture (gate: reward == 1.0)" harbor run \ --env docker \ --path "$TB_REPO/bench/readiness/synthetic-adapter-sentinel" \ - --agent-import-path stella_harbor:StellaAgent \ + --agent stella_harbor:StellaAgent \ --model "$TB_MODEL" \ --job-name "$JOB" \ --jobs-dir "$JOBS" \ @@ -82,7 +82,7 @@ harbor run \ --env docker \ --dataset "$FB_DATASET" \ --include-task-name "$FB_TASK_PREFIX/$TASK" \ - --agent-import-path stella_harbor:StellaAgent \ + --agent stella_harbor:StellaAgent \ --model "$TB_MODEL" \ --job-name "$JOB2" \ --jobs-dir "$JOBS" \ From 4f7cedeb6d2fb27106e6cd312dc2763bacf66dab Mon Sep 17 00:00:00 2001 From: Stella Test Date: Fri, 31 Jul 2026 15:17:27 -0700 Subject: [PATCH 5/7] docs(bench): record the cross-version CLI gotcha in the frontier README MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The flag rename is the kind of thing that reads as a mistake in the diff unless the reason is written down: this lane and the Terminal-Bench lane pass *different* agent flags on purpose, because they are pinned to different Harbor majors and each is correct for its own. Also notes what did not change — the trial result schema the sentinel's gates parse is identical in both, so those gates read the same fields either way. --- bench/evidence/frontier/README.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/bench/evidence/frontier/README.md b/bench/evidence/frontier/README.md index d31233c8c..a33dcb6ba 100644 --- a/bench/evidence/frontier/README.md +++ b/bench/evidence/frontier/README.md @@ -81,6 +81,20 @@ row that looks exactly like a genuine failure. Lower `FB_MEMORY_HEADROOM_MB` (default 2048) to attempt them anyway; every exclusion is named with its reason, so nothing disappears quietly either way. +## Two Harbor majors means the CLI is a moving contract + +`--agent-import-path` is 0.6.1 spelling; 0.20.0 folds it into `--agent`, which +takes either a built-in name or an import path. This lane uses `--agent`, the +Terminal-Bench lane keeps the old flag, and both are correct for their pin. The +preflight's `fb_assert_cli_flags` asserts every flag these scripts pass is still +advertised, so the next rename costs one message rather than a run — it would +otherwise surface as `No such option` at the first trial, after the venv build, +dataset download, image warm and preflight had all passed. + +The trial result schema, by contrast, is unchanged: `task_name`, +`agent_result`, `verifier_result.rewards`, and `exception_info` are identical in +both versions, so the sentinel's gates read the same fields either way. + ## Harbor version and the test suite `bench/harbor_adapter/tests/` passes 132/132 under 0.6.1 and 131/132 under From c7f68a7e6742af6fdc684218749fd764b088126b Mon Sep 17 00:00:00 2001 From: Stella Test Date: Fri, 31 Jul 2026 15:36:37 -0700 Subject: [PATCH 6/7] fix(bench): register live_feed.py in the frozen adapter source list MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #1027 added stella_harbor/live_feed.py without adding it to _FIXED_ADAPTER_SOURCE_PATHS. The runtime digest rglobs *.py under the package, so it hashed the new module; the public-tree check walks the hand-written tuple, so it did not. The two disagreed and every path through the secure launcher failed closed — 30 tests, all reporting "public adapter tree hash differs from runtime identity". That is the freeze mechanism working exactly as designed. It was already red on origin/main before this branch existed; this PR is simply the first since #1027 to touch a path that makes the bench job run, and the job is not a required check, so nothing blocked the merge that broke it. Also adds the assertion that should have caught it. Thirty tests failing with a message that names neither the file nor the fix is a poor way to learn a module is unregistered, so one test now compares the tuple against the package on disk and names the missing path. Verified it fails with "live_feed.py" when the entry is removed again. harbor_adapter 311 passed / 1 skipped, terminal_bench_analysis 240 passed, frontier planner 21 passed. --- .../stella_harbor/secure_launcher.py | 1 + .../tests/test_secure_launcher.py | 26 +++++++++++++++++++ 2 files changed, 27 insertions(+) diff --git a/bench/harbor_adapter/stella_harbor/secure_launcher.py b/bench/harbor_adapter/stella_harbor/secure_launcher.py index 5ab2a4bc6..e2d16f3e9 100644 --- a/bench/harbor_adapter/stella_harbor/secure_launcher.py +++ b/bench/harbor_adapter/stella_harbor/secure_launcher.py @@ -75,6 +75,7 @@ "bench/harbor_adapter/stella_harbor/atif.py", "bench/harbor_adapter/stella_harbor/credential_bundle.py", "bench/harbor_adapter/stella_harbor/host_attestation.py", + "bench/harbor_adapter/stella_harbor/live_feed.py", "bench/harbor_adapter/stella_harbor/portability.py", "bench/harbor_adapter/stella_harbor/posture.py", "bench/harbor_adapter/stella_harbor/secure_launcher.py", diff --git a/bench/harbor_adapter/tests/test_secure_launcher.py b/bench/harbor_adapter/tests/test_secure_launcher.py index 71add41bd..6837ac385 100644 --- a/bench/harbor_adapter/tests/test_secure_launcher.py +++ b/bench/harbor_adapter/tests/test_secure_launcher.py @@ -1096,6 +1096,32 @@ def _host_preflight_fixture( ) +def test_frozen_adapter_source_paths_match_the_package_on_disk() -> None: + """The freeze list must enumerate every .py the runtime actually hashes. + + The runtime digest rglobs ``*.py`` under the package while the public-tree + check walks this hand-written tuple, so a module added to one and not the + other makes the two disagree and everything downstream fails closed. That + is the design working — but it announces itself as thirty unrelated tests + failing with "public adapter tree hash differs from runtime identity", + which names neither the file nor the fix. + + This is what happened when ``live_feed.py`` arrived in #1027 without being + added here. One assertion that names the offending file is worth more than + thirty that describe its consequences. + """ + package_dir = Path(stella_harbor.__file__).parent + on_disk = { + f"bench/harbor_adapter/stella_harbor/{p.relative_to(package_dir)}" + for p in package_dir.rglob("*.py") + } + assert on_disk == set(launcher_module._FIXED_ADAPTER_SOURCE_PATHS), ( + "add new adapter modules to _FIXED_ADAPTER_SOURCE_PATHS in " + "secure_launcher.py (and push them publicly — the check reads the " + "published tree)" + ) + + def test_launch_receipt_controls_are_exact_nonsecret_attestation() -> None: assert LAUNCH_RECEIPT_CONTROLS == { "command": "harbor-run-only", From fa6c2b538cf3b19f9ea25a1de096c8544c7ca34e Mon Sep 17 00:00:00 2001 From: Stella Test Date: Fri, 31 Jul 2026 15:38:04 -0700 Subject: [PATCH 7/7] chore(bench): raise the file-size baseline for the two files that grew Both increases are irreducible in the sense the ratchet cares about. _FIXED_ADAPTER_SOURCE_PATHS is the freeze list, so a registered module is one line in secure_launcher.py and cannot live anywhere else; the test that guards it asserts on that constant, so it belongs in that constant's test file. Splitting either to dodge the ceiling would move code away from the thing it describes. secure_launcher.py 4269 -> 4270, test_secure_launcher.py 3204 -> 3230. --- scripts/file-size-baseline.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/file-size-baseline.txt b/scripts/file-size-baseline.txt index 28c1f1f32..f56e53b1a 100644 --- a/scripts/file-size-baseline.txt +++ b/scripts/file-size-baseline.txt @@ -8,9 +8,9 @@ # never silent — it lands as a visible diff here, to be justified in # review like any other change. 1928 bench/harbor_adapter/stella_harbor/__init__.py -4269 bench/harbor_adapter/stella_harbor/secure_launcher.py +4270 bench/harbor_adapter/stella_harbor/secure_launcher.py 1910 bench/harbor_adapter/tests/test_adapter.py -3204 bench/harbor_adapter/tests/test_secure_launcher.py +3230 bench/harbor_adapter/tests/test_secure_launcher.py 8166 bench/terminal_bench_analysis/tb21_analysis.py 1887 bench/terminal_bench_analysis/tb21_evidence_contract.py 4173 bench/terminal_bench_analysis/tests/test_tb21_analysis.py