feat(bench): a Frontier-Bench lane that keeps the adapter untouched by macanderson · Pull Request #1031 · macanderson/stella

macanderson · 2026-07-31T22:19:37Z

Harbor released Frontier-Bench — the successor to Terminal-Bench, 74 tasks over seven domains, where the best published agents clear about a third. This adds a lane to run Stella on it.

stella_harbor:StellaAgent runs it unmodified. Nothing under bench/harbor_adapter/ changes. That was the design constraint, not a lucky outcome: the adapter's Python tree is digest-frozen for the audited claim path, so a new benchmark has to earn its place without perturbing it. Everything Frontier-Bench-specific lives in the new bench/evidence/frontier/.

The one that would have been expensive to find later

All 74 tasks declare environment_mode = "separate" under [verifier]. Harbor 0.6.1 — the Terminal-Bench claim's audited pin — has no such field, and pydantic's default extra="ignore" drops it silently. 0.6.1 will happily run the whole set with the verifier sharing the agent's container instead of the separate one the task asked for, produce rewards, and warn about nothing. Plausible numbers answering the wrong question are worse than an error.

So this lane pins its own Harbor (0.20.0) in its own venv. bench/harbor_adapter/.venv stays on 0.6.1, so Terminal-Bench reruns keep measuring what they measured.

What else genuinely differs

	Terminal-Bench 2.1	Frontier-Bench
Images	prebuilt `docker_image` per task	0 of 74 have one; all build from a Dockerfile, 12 with compose
Resources	homogeneous	1–16 CPUs, 2–32 GB memory, 4 GB–1 TB storage
GPUs	none	4 tasks need one
Agent timeout	shorter	30 min – 8 h, 2 h mode

Each of those becomes a silent wrong number if handled the Terminal-Bench way:

warm_images.sh pulls FROM bases, not task images — same reason as the existing prepull, that with --max-retries 0 a registry hiccup is a permanent reward-0 row.
plan.py tiers by footprint against what docker info reports. Terminal-Bench's one-line rule would put 58 of 74 in the serial phase.
GPU tasks are excluded by name with a stated reason. On a GPU-less host they don't error — the container starts, the work is impossible, the verifier returns 0.0, and the row is arithmetically identical to a genuine miss. Silently that's ~5.4 points of pass rate.
Budget default raised to $2.50/trial. The $0.60 cap would truncate most of this set and record the truncation as failure. STELLA_BUDGET= (empty) means uncapped.

Dataset pin

Pinned to sha256:97fd2ba3…, the leaderboard's own DATASET_REF. Resolving the dataset by name today yields sha256:63f363a1… instead, which exports a byte-identical 74-task tree — verified by diff, no file differs; they're two version records over the same content. But the leaderboard compares the ref string, so a run pinned to the equally-valid other digest is rejected on submission while being, in substance, the same run.

On getting to the leaderboard

SUBMISSION.md has the detail. Short version: the run can be produced now, the PR cannot be filed yet (the intake tooling is PR #1405, unmerged — the two "submissions" against upstream were closed by their own author as fork-test mistakes).

Two things block a submittable run regardless, and both point the same way — it doesn't happen on a Mac:

Four tasks need real GPUs, and subsetting isn't allowed for a submission.
On a 10 GB / 6-CPU Docker VM plan.py admits 48 of 74; 22 more are over the memory budget.

Also worth knowing before anyone commits to it: the metric is reward > 0 over all trials, errored trials count as 0 rather than being dropped, and 74 tasks × 5 required trials is a four-figure decision that should be preregistered like the Terminal-Bench claim was. primary.sh hardcodes --env docker; a Modal path is the next piece of work, not something already done here.

Also fixes a pre-existing break on main

The bench suite was already red before this branch existed. #1027 added stella_harbor/live_feed.py without adding it to _FIXED_ADAPTER_SOURCE_PATHS. The runtime digest rglobs *.py under the package so it hashed the new module; the public-tree check walks the hand-written tuple so it didn't. The two disagreed and every path through the secure launcher failed closed — 30 tests, all reporting public adapter tree hash differs from runtime identity.

That's the freeze mechanism working as designed. What let it land is that the bench job only runs when a PR touches its filtered paths, and it is not a required check — the same failure mode bench.yml's own header documents from #659. This PR is simply the first since #1027 to touch a path that makes the job run.

Fixed by registering the module, plus the assertion that should have caught it: one test now compares the tuple against the package on disk and names the missing path, instead of thirty tests describing its consequences. Verified it fails with live_feed.py when the entry is removed again.

Worth considering separately: making this check required, since it has now silently drifted red twice.

Verification

74/74 task specs parse under both Harbor versions.
Full bench job, all three steps, run locally as CI runs them: harbor_adapter 311 passed / 1 skipped, terminal_bench_analysis 240 passed, frontier planner 21 passed.
Adapter suite (the four Harbor-version-sensitive files): 132/132 under 0.6.1, 131/132 under 0.20.0 — the one difference is a test asserting _harbor_version_value == "0.6.1", the audited constant, correctly failing under a different Harbor. CI runs it in the 0.6.1 venv and stays green.
New plan.py suite: 21 tests, wired into bench.yml. Two cases are regressions hit while writing it — a headroom that subtracted a storage allowance from a memory budget and excluded all 74 tasks, and the Docker GPU probe, where the obvious {{if index .Runtimes "nvidia"}} reports a GPU on every host alive because Go returns a truthy zero-value struct for a missing key.
Caught before it cost anything: --agent-import-path doesn't exist in 0.20.0 (folded into --agent). Every run in this lane would have died on No such option at the first trial, after the venv build, dataset download, image warm and preflight had all passed. fb_assert_cli_flags now asserts the whole flag list in preflight.

Not done

No paid trial has been run. The sentinel is written and syntax-checked but unexecuted — running it costs money and is the natural first step after this merges.

Frontier-Bench is Harbor's successor to Terminal-Bench: 74 tasks over seven domains, where the best published agents clear about a third. `stella_harbor:StellaAgent` runs it unmodified, so nothing under bench/harbor_adapter/ changes — its Python tree is digest-frozen for the audited claim path, and a new benchmark has to earn its place without perturbing that. Everything specific to this one lives in the new bench/evidence/frontier/. The lane pins its own Harbor, 0.20.0, rather than the Terminal-Bench claim's audited 0.6.1. All 74 tasks declare `environment_mode = "separate"` under [verifier]; 0.6.1 has no such field and pydantic's default extra="ignore" drops it silently, so 0.6.1 would run the set with the verifier sharing the agent's container, produce rewards, and warn about nothing. Plausible numbers answering the wrong question are worse than an error. The Terminal-Bench venv stays on 0.6.1. Three more differences, each because the Terminal-Bench assumption is false here rather than merely untidy: * No task has a `docker_image` — all 74 build from an environment/Dockerfile, 12 with compose. So warm_images.sh pulls the FROM bases plan.py collects, for the prepull's original reason: with --max-retries 0 a registry hiccup is a permanent reward-0 row. * Terminal-Bench's one-line resource rule puts 58 of 74 in the serial phase. plan.py tiers by footprint against what `docker info` reports and gives each tier the concurrency the host supports. * Four tasks want a GPU. On a GPU-less host they do not error — the container starts, the work is impossible, the verifier returns 0.0, and the row is arithmetically identical to a genuine miss, quietly costing ~5.4 points. They are excluded by name with a stated reason unless FB_ALLOW_GPU=1 and Docker exposes an nvidia runtime. Dataset pinned to sha256:97fd2ba3…, the leaderboard's own DATASET_REF. Resolving by name today yields sha256:63f363a1… instead, which exports a byte-identical tree (verified by diff) — two version records over the same content — but the leaderboard compares the ref string, so a run on the equally-valid other digest is rejected while being the same run. SUBMISSION.md records what a leaderboard entry needs and the two things that block one: four tasks need real GPUs and subsetting is not allowed, and a 10 GB Docker VM admits only 48 of 74. Both mean a submission run happens on Modal, not a Mac. The intake PR upstream has not merged yet. Verified: 74/74 task specs parse under both Harbors; the adapter's suite is 132/132 under 0.6.1 and 131/132 under 0.20.0, the one difference being a test asserting the audited 0.6.1 constant.

plan.py's headroom argument started life conflating two unrelated quantities — a 20 GB *storage* headroom was being subtracted from a 10 GB Docker VM's *memory*, which left a negative budget and excluded all 74 tasks with a polite explanation. Splitting them fixed the arithmetic but left env.sh exporting the old FB_STORAGE_HEADROOM_MB, which nothing reads any more, while plan.py's FB_MEMORY_HEADROOM_MB fell back to its default however the operator set it. Export the name plan.py actually reads, and say why the default is conservative: memory_mb is a cap Docker accepts even on a smaller VM, so an 8 GB task on a 10 GB daemon usually starts and then gets OOM-killed partway in — a reward-0 row indistinguishable from a real failure. At the 2048 MB default this host runs 48 of 74 tasks; at 512 it runs 66. Both are defensible, neither is silent.

plan.py decides which tasks are allowed to run, and every way it can be wrong produces a plausible plan rather than an error — a GPU task scheduled on a GPU-less host, a task admitted onto a machine that cannot hold it, a build stage mistaken for an image to pull. Two of the 21 cases are regressions already hit while writing it: the headroom that subtracted a storage allowance from a memory budget and excluded all 74 tasks, and the Docker GPU probe, where the obvious `{{if index .Runtimes "nvidia"}}` reports a GPU on every host alive because Go returns a truthy zero-value struct for a missing key. Also pins the boundary the first draft of the test got wrong: a task whose footprint exactly equals the remaining budget is admitted, not excluded. The comparison is strictly greater-than and should stay that way — 8192 MB is the second most common footprint in the set, so an off-by-one there would drop real tasks for no reason. plan.py imports nothing outside the standard library, which is what lets CI run these with `uv run --with pytest --no-project` — no project sync, no Harbor install. The piece that decides what runs should not need the benchmark harness to be testable. Extends bench.yml's scope check to bench/evidence/frontier/ so the suite actually gates.

`--agent-import-path` is 0.6.1 spelling. By 0.20.0 it is folded into `--agent`, which takes either a built-in name or an import path. Both primary.sh and sentinel.sh passed the old one, so every run in this lane would have died on `No such option` at the first trial — after the venv build, the dataset download, the image warm, and the preflight had all passed. The Terminal-Bench lane keeps the old spelling and is still right to: it is pinned to 0.6.1, where that is the flag. Add fb_assert_cli_flags to the preflight so the next such rename costs one message instead of a run. Running two Harbor majors in one repo makes the CLI a moving contract, and it has now moved once. The check scrubs ANSI before matching, which is not incidental: Harbor renders help through Rich, which styles each option individually and leaves escape sequences inside the flag names. A literal search for `--env` over the raw 42 KB finds nothing, so the unscrubbed first draft reported all eleven flags missing and failed closed on a working CLI.

The flag rename is the kind of thing that reads as a mistake in the diff unless the reason is written down: this lane and the Terminal-Bench lane pass *different* agent flags on purpose, because they are pinned to different Harbor majors and each is correct for its own. Also notes what did not change — the trial result schema the sentinel's gates parse is identical in both, so those gates read the same fields either way.

vercel · 2026-07-31T22:19:43Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
stella-cli-docs	Ignored	Preview	Jul 31, 2026 10:40pm

sourcery-ai · 2026-07-31T22:20:21Z

Reviewer's Guide

Adds a new Frontier-Bench development-baseline lane that runs Stella through the existing Harbor adapter without modifying it, by introducing a separate Harbor pin, dataset pin, resource-aware planner, warmup tooling, and CI tests under bench/evidence/frontier/ while keeping the Terminal-Bench lane and adapter behavior unchanged.

Sequence diagram for Frontier-Bench primary run planning and execution

sequenceDiagram
  actor user
  participant env_sh as env_sh
  participant plan_py as plan.py
  participant primary_sh as primary.sh
  participant harbor_cli as harbor run

  user->>primary_sh: run primary.sh <tier> <job-name>
  primary_sh->>env_sh: source env.sh
  env_sh->>env_sh: fb_preflight()
  env_sh-->>primary_sh: preflight OK (Harbor 0.20.0, venv, dataset pin)

  primary_sh->>primary_sh: read FB_PLAN
  alt plan missing
    primary_sh-->>user: FATAL: no plan at $FB_PLAN
  else plan exists
    note over primary_sh,plan_py: FB_PLAN created earlier by fetch_dataset.sh
    primary_sh->>plan_py: compute tasks, concurrency for tier
    plan_py-->>primary_sh: N tasks, tier concurrency

    primary_sh->>harbor_cli: harbor run --env docker --dataset $FB_DATASET 
    primary_sh->>harbor_cli: --include-task-name FB_TASK_PREFIX/<slug>...
    primary_sh->>harbor_cli: --agent stella_harbor:StellaAgent
    primary_sh->>harbor_cli: --n-attempts $FB_ATTEMPTS --n-concurrent $CONC
    harbor_cli-->>primary_sh: job results
    primary_sh-->>user: harbor_exit, finished timestamp
  end

File-Level Changes

Change	Details	Files
Introduce a standalone Frontier-Bench lane with its own Harbor pin, dataset pin, environment wiring, and run scripts that reuse the existing stella_harbor adapter unmodified.	Add env.sh to define Frontier-specific environment variables, dataset digest pin, Harbor version 0.20.0 venv, resource headroom, GPU task policy, and preflight checks including CLI flag assertions and Harbor version validation. Add setup_venv.sh to create a dedicated virtualenv for Harbor 0.20.0, install Harbor, and sanity-check that StellaAgent still satisfies Harbor’s agent interface via PYTHONPATH imports. Add fetch_dataset.sh to download the pinned Frontier-Bench dataset, verify the expected task count by task.toml files, and invoke plan.py to produce a run plan JSON plus a base-image list. Add warm_images.sh to pre-pull all base images referenced in the plan (Dockerfile FROMs and compose service images) with retries, writing failures to a file and failing fast if any images cannot be warmed. Add primary.sh to drive the measured runs per resource tier, reading tier tasks and concurrency from the plan, materializing include-task-name flags, enforcing job-directory uniqueness, and marking runs as dev-baseline vs submission-shaped based on tier and attempt count. Add sentinel.sh to run a two-stage sanity check (synthetic sentinel task and cheapest real Frontier task) that validates harness plumbing, binary verification, and separate-container verifier behavior before expensive trials.	`bench/evidence/frontier/env.sh` `bench/evidence/frontier/setup_venv.sh` `bench/evidence/frontier/fetch_dataset.sh` `bench/evidence/frontier/warm_images.sh` `bench/evidence/frontier/primary.sh` `bench/evidence/frontier/sentinel.sh`
Implement a Frontier-Bench run planner that is independent of Harbor, computes a resource-aware schedule, enforces GPU and memory/CPU constraints, and collects base images for warmup.	Add plan.py as a standalone Python 3.11+ script that parses Frontier-Bench task.toml files via tomllib, reads environment declarations, and produces a JSON plan describing runnable vs excluded tasks, tier-specific concurrency, and base images to pull. Implement _host_capacity to query docker info for memory, CPU count, and presence of an nvidia runtime, handling failures by recording an error but still allowing a runnable plan when capacity is unknown. Implement _base_images to walk task Dockerfiles and compose files, collecting FROM bases and compose service images while skipping multi-stage aliases, scratch, and arg-substituted or coordinate-less references, and deduplicating images across tasks. Define memory-based tiers (small/medium/large/xlarge) and compute per-tier concurrency bounded by both memory and CPU availability with a MAX_CONCURRENCY cap, ensuring concurrency never drops below one. Encode exclusion reasons for GPU-only tasks when GPUs are disallowed or unavailable, tasks exceeding usable memory after headroom, and tasks requiring more CPUs than the host, and surface these in the plan for transparency. Expose a CLI in main() to generate the plan JSON and an optional images list file, including host capacity and tier summary printing, and a note about excluded tasks affecting the score denominator.	`bench/evidence/frontier/plan.py`
Add unit tests for the Frontier planner to lock in behavior around GPU exclusion, memory/CPU headroom, tiering, base-image extraction, and host-capacity edge cases, and wire them into CI.	Create test_plan.py that loads plan.py as a module by path, constructs synthetic Frontier-like task directories with task.toml, environment Dockerfiles, optional compose files, and verifier Dockerfiles, and exercises build_plan under various host configurations. Test GPU behavior including default exclusion with clear reasons, honoring allow_gpu only when a host GPU is available, and avoiding false positives from Docker runtime probes. Test memory and CPU headroom handling: subtracting headroom from memory not storage, excluding tasks that exceed the remaining budget, admitting tasks that exactly fit, and enforcing CPU-based exclusions. Test tier assignment, concurrency calculations bounded by available resources and MAX_CONCURRENCY, and minimum concurrency of one for just-fitting tasks. Test base image collection logic for multi-stage builds, scratch and ARG substitutions, compose service images, verifier Dockerfiles, deduplication, and ensuring excluded tasks do not contribute their images. Test plan shape invariants including count reconciliation, treating empty dataset directories as fatal, and producing runnable plans when host capacity is unavailable.	`bench/evidence/frontier/tests/test_plan.py` `.github/workflows/bench.yml`
Document the Frontier-Bench lane, its differences from Terminal-Bench, and submission requirements and limitations, and expose it in the top-level bench README.	Update bench/README.md to add evidence/frontier/ as a fifth entry point, describing it as a Frontier-Bench lane that uses the existing Harbor adapter unchanged but with separate Harbor pin, resource-aware schedule, and GPU exclusions by name. Add Frontier-specific README.md explaining how to run the lane, how it differs from the Terminal-Bench lane, the rationale for a separate Harbor pin and base-image warming, GPU task exclusion, budget defaults, sentinel gates, host limitations, and Harbor version/test interactions. Add SUBMISSION.md detailing what a leaderboard submission entails (dataset ref, Harbor version, full-task coverage, trial counts, upload/public requirements), current intake status, hardware blockers (GPU and host size), cost considerations, and a recommended order of operations from dev baseline to potential submission. Ensure documentation clearly states that Terminal-Bench’s Harbor pin and adapter remain unchanged and that Frontier-specific logic is confined to bench/evidence/frontier/.	`bench/README.md` `bench/evidence/frontier/README.md` `bench/evidence/frontier/SUBMISSION.md`

Tips and commands

Interacting with Sourcery

Trigger a new review: Comment @sourcery-ai review on the pull request.
Continue discussions: Reply directly to Sourcery's review comments.
Generate a GitHub issue from a review comment: Ask Sourcery to create an
issue from a review comment by replying to it. You can also reply to a
review comment with @sourcery-ai issue to create an issue from it.
Generate a pull request title: Write @sourcery-ai anywhere in the pull
request title to generate a title at any time. You can also comment
@sourcery-ai title on the pull request to (re-)generate the title at any time.
Generate a pull request summary: Write @sourcery-ai summary anywhere in
the pull request body to generate a PR summary at any time exactly where you
want it. You can also comment @sourcery-ai summary on the pull request to
(re-)generate the summary at any time.
Generate reviewer's guide: Comment @sourcery-ai guide on the pull
request to (re-)generate the reviewer's guide at any time.
Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
pull request to resolve all Sourcery comments. Useful if you've already
addressed all the comments and don't want to see them anymore.
Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
request to dismiss all existing Sourcery reviews. Especially useful if you
want to start fresh with a new review - don't forget to comment
@sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

Enable or disable review features such as the Sourcery-generated pull request
summary, the reviewer's guide, and others.
Change the review language.
Add, remove or edit custom review instructions.
Adjust other review settings.

Getting Help

Contact our support team for questions or feedback.
Visit our documentation for detailed guides and information.
Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

#1027 added stella_harbor/live_feed.py without adding it to _FIXED_ADAPTER_SOURCE_PATHS. The runtime digest rglobs *.py under the package, so it hashed the new module; the public-tree check walks the hand-written tuple, so it did not. The two disagreed and every path through the secure launcher failed closed — 30 tests, all reporting "public adapter tree hash differs from runtime identity". That is the freeze mechanism working exactly as designed. It was already red on origin/main before this branch existed; this PR is simply the first since #1027 to touch a path that makes the bench job run, and the job is not a required check, so nothing blocked the merge that broke it. Also adds the assertion that should have caught it. Thirty tests failing with a message that names neither the file nor the fix is a poor way to learn a module is unregistered, so one test now compares the tuple against the package on disk and names the missing path. Verified it fails with "live_feed.py" when the entry is removed again. harbor_adapter 311 passed / 1 skipped, terminal_bench_analysis 240 passed, frontier planner 21 passed.

Both increases are irreducible in the sense the ratchet cares about. _FIXED_ADAPTER_SOURCE_PATHS is the freeze list, so a registered module is one line in secure_launcher.py and cannot live anywhere else; the test that guards it asserts on that constant, so it belongs in that constant's test file. Splitting either to dodge the ceiling would move code away from the thing it describes. secure_launcher.py 4269 -> 4270, test_secure_launcher.py 3204 -> 3230.

vercel · 2026-07-31T22:41:42Z

+  else echo "[$i/$total] FAILED  $img"; echo "$img" >> "$FAILED"; fi
+done < "$LIST"
+
+nfailed=$(grep -c . "$FAILED" || echo 0)


Suggested change

nfailed=$(grep -c . "$FAILED" || echo 0)

nfailed=$(grep -c . "$FAILED" || true)

grep -c . "$FAILED" || echo 0 produces multi-line "0\n0" on the success path, making the script falsely report failure and exit 1 even when all images pulled successfully.

sourcery-ai

Sorry @macanderson, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

Follow-on to #1031. That PR landed the Frontier-Bench lane but it could only ever produce a *development baseline* — on this laptop's 10 GB / 6-CPU Docker VM the plan is **48 of 74**. A submittable run must cover all 74 tasks, four of which want a GPU, without overriding any task's declared resources. No amount of scheduling fixes that; it needs a machine that can hand a container 32 GB and an H100 on demand. `FB_ENV` now selects the backend. **On `modal` the plan is 74/74 with zero exclusions.** ```bash FB_ENV=modal FB_CONCURRENCY=32 bench/evidence/frontier/fetch_dataset.sh ``` ## Three things change with the backend Each because a docker assumption is *false* there, not merely cautious: | | why | |---|---| | **Nothing excluded for capacity** | Modal builds a sandbox per trial from that trial's own declared resources, so "does this task fit" is the wrong question. Carrying the laptop's exclusions over would produce a 48-task result that looks exactly like a complete one. | | **GPU tasks default to on** | Harbor passes each task's `gpus`/`gpu_types` straight through. Defaulting them off would silently drop the four tasks a submission is *required* to cover — the same plausible-wrong-number failure this lane exists to prevent, inverted. `--no-allow-gpu` still wins when said explicitly, which is why the flag is tri-state rather than `store_true`. | | **`warm_images.sh` skips** | The builds happen in the cloud; a local pull warms a cache no trial reads. | **What does not change:** the glibc-2.17 portability assert runs on both. Modal is a different scheduler, not a different libc — a binary that cannot exec would fail there the same way it did on 2026-07-31, as an ordinary reward-0 row. ## Upload, opt-in `--upload` is now wired behind `FB_UPLOAD=1`, private by default. The leaderboard requires publicly readable trials — CI re-derives every trial from the uploaded job and a maintainer's LLM judge reads every trajectory — so `FB_UPLOAD_VISIBILITY=public` publishes the agent's full reasoning and every command it ran, permanently. That is a deliberate act, and it must never be inherited from a shell that had it set for an earlier run. ## Posture reporting `primary.sh` now names **every** reason a run would be refused as a submission, not the first: ``` posture=development-baseline — not submittable: tier=small(needs-all) attempts=1(needs-5) excluded=26(needs-0) upload=off visibility=private(needs-public) posture=submission-shaped ``` An operator who fixes one gap and reruns only to be told about the next has paid for a full run to learn something one line could have said. ## Also `setup_venv.sh` installs the `modal` extra and passes `--clear`, so it is idempotent — it previously failed the moment a venv existed, which is exactly when you want to run it (after changing the Harbor pin or adding an extra). Installing the extra unconditionally means switching backends is one variable rather than a rebuild, and lets the preflight tell "modal is not authenticated" apart from "the SDK was never installed" — different problems, different fixes. ## Verified - Full Modal chain end to end: dataset fetch → plan **74/74, 0 excluded** → `warm_images.sh` correctly skips. - Docker path unchanged: **48/74**, same exclusions and reasons as before. - `--no-allow-gpu` on Modal → 70/74, so the operator override genuinely wins. - Modal auth confirmed live (`~/.modal.toml`, workspace reachable), and the preflight distinguishes missing-SDK from unauthenticated. - **27 planner tests** (6 new for the backend), shell syntax + shellcheck clean. - Nothing under `bench/harbor_adapter/` is touched — the digest-frozen tree stays untouched, same as #1031. ## Not done No paid trial has run. The sentinel now works against either backend but is still unexecuted; that is the next step and it costs money. A real submission is ~370 multi-hour trials with no spend cap — a four-figure decision that deserves the same preregistration the Terminal-Bench claim got. The upstream intake ([PR #1405](harbor-framework/frontier-bench#1405)) is also still unmerged. ## Summary by Sourcery Add a selectable backend for Frontier-Bench runs (docker vs Modal), enabling full 74-task planning on Modal while keeping docker behavior unchanged, and wire upload posture and reporting around this. New Features: - Introduce a Modal backend that provisions capacity per task, allowing plans that include all 74 Frontier-Bench tasks without host-based exclusions. - Expose backend selection, concurrency, and GPU allowance via CLI flags and FB_ENV/FB_CONCURRENCY/FB_ALLOW_GPU environment variables, with sensible defaults per backend. - Add opt-in upload controls (FB_UPLOAD and FB_UPLOAD_VISIBILITY) so runs can be explicitly published to Harbor Hub as public or kept private. - Record backend and upload posture in run output, and report overall submission posture including all reasons a run is or is not submittable. Enhancements: - Make the planning logic support explicit concurrency overrides across backends and propagate the chosen backend into the plan metadata. - Refine preflight checks to validate the selected backend, including distinguishing missing Modal SDK from unauthenticated Modal credentials. - Update scripts (primary.sh, sentinel.sh, warm_images.sh) to respect the chosen backend, skipping local image warming and using the correct Harbor env as needed. - Improve documentation in README and SUBMISSION.md to describe backend choice, Modal workflow, hardware constraints, and submission posture. - Make setup_venv.sh idempotent by clearing the venv and always installing the harbor[modal] extra, simplifying backend switching. Tests: - Add Modal backend planner tests to ensure Docker capacity is ignored, GPU tasks are runnable, explicit concurrency overrides are honored, and backend is recorded in plans. Co-authored-by: Stella Test <test@stella.local>

Stella Test added 5 commits July 31, 2026 15:04

Stella Test added 2 commits July 31, 2026 15:36

vercel Bot reviewed Jul 31, 2026

View reviewed changes

macanderson marked this pull request as ready for review July 31, 2026 23:48

sourcery-ai Bot reviewed Jul 31, 2026

View reviewed changes

macanderson merged commit 7302fef into main Jul 31, 2026
10 checks passed

macanderson deleted the worktree-frontier-bench-adapter branch July 31, 2026 23:48

macanderson mentioned this pull request Aug 1, 2026

feat(bench): run Frontier-Bench on Modal, where all 74 tasks fit #1048

Merged

Navigation Menu

Uh oh!

feat(bench): a Frontier-Bench lane that keeps the adapter untouched - #1031

feat(bench): a Frontier-Bench lane that keeps the adapter untouched#1031
macanderson merged 7 commits into
mainfrom
worktree-frontier-bench-adapter

macanderson commented Jul 31, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Jul 31, 2026 •

edited

Loading

Uh oh!

sourcery-ai Bot commented Jul 31, 2026

Interacting with Sourcery

Customizing Your Experience

Getting Help

Uh oh!

vercel Bot Jul 31, 2026

Uh oh!

sourcery-ai Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	nfailed=$(grep -c . "$FAILED" \|\| echo 0)
	nfailed=$(grep -c . "$FAILED" \|\| true)

Uh oh!

Conversation

macanderson commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The one that would have been expensive to find later

What else genuinely differs

Dataset pin

On getting to the leaderboard

Also fixes a pre-existing break on main

Verification

Not done

Uh oh!

vercel Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sourcery-ai Bot commented Jul 31, 2026

Reviewer's Guide

Sequence diagram for Frontier-Bench primary run planning and execution

File-Level Changes

Interacting with Sourcery

Customizing Your Experience

Getting Help

Uh oh!

vercel Bot Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

sourcery-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

macanderson commented Jul 31, 2026 •

edited

Loading

vercel Bot commented Jul 31, 2026 •

edited

Loading