From 6b3dfcbc6f22d5bb15062f38f96318f449745cb4 Mon Sep 17 00:00:00 2001 From: Stella Test Date: Fri, 31 Jul 2026 17:20:34 -0700 Subject: [PATCH] feat(bench): run Frontier-Bench on Modal, where all 74 tasks fit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The lane could only ever produce a development baseline. A submittable run must cover all 74 tasks, four of which want a GPU, without overriding any task's declared resources — so it needs a machine that can hand a container 32 GB and an H100 on demand. This laptop's Docker VM is 10 GB with no GPU, and no amount of scheduling fixes that: the plan was 48/74. FB_ENV now selects the backend. On modal the plan is 74/74. Three things change with the backend, each because a docker assumption is false there rather than merely cautious: * Nothing is excluded for capacity. Modal builds a sandbox per trial from that trial's own declared resources, so "does this task fit" is the wrong question. Carrying the laptop's exclusions over would produce a 48-task result that looks exactly like a complete one. * GPU tasks default to on. Harbor passes each task's gpus/gpu_types straight through. Defaulting them off there would silently drop the four tasks a submission is required to cover — the same plausible-wrong-number failure this lane exists to prevent, inverted. --no-allow-gpu still wins when said explicitly, which is why the flag is tri-state rather than store_true. * warm_images.sh skips. The builds happen in the cloud; a local pull warms a cache no trial reads. What does not change: the glibc-2.17 assert runs on both. Modal is a different scheduler, not a different libc, and a binary that cannot exec would fail there the same way it did on 2026-07-31 — as a reward-0 row. Also wires --upload, opt-in behind FB_UPLOAD with private visibility by default. The leaderboard requires publicly readable trials, so this publishes every trajectory and every command the agent ran, permanently. That is a deliberate act and must never be inherited from a shell that had it set for an earlier run. primary.sh now names *every* reason a run would be refused as a submission — tier, attempts, any excluded task, upload off, visibility private — rather than the first. An operator who fixes one and reruns only to be told about the next has paid for a full run to learn something one line could have said. setup_venv.sh installs the modal extra and passes --clear, so it is idempotent; it previously failed the moment a venv existed, which is exactly when you want to run it. 27 planner tests (6 new for the backend), shell syntax and shellcheck clean. Nothing under bench/harbor_adapter/ is touched. --- bench/evidence/frontier/README.md | 32 ++++++++- bench/evidence/frontier/SUBMISSION.md | 69 ++++++++++++++---- bench/evidence/frontier/env.sh | 72 ++++++++++++++++++- bench/evidence/frontier/fetch_dataset.sh | 3 +- bench/evidence/frontier/plan.py | 74 ++++++++++++++++--- bench/evidence/frontier/primary.sh | 37 ++++++++-- bench/evidence/frontier/sentinel.sh | 4 +- bench/evidence/frontier/setup_venv.sh | 13 +++- bench/evidence/frontier/tests/test_plan.py | 82 ++++++++++++++++++++++ bench/evidence/frontier/warm_images.sh | 9 +++ 10 files changed, 356 insertions(+), 39 deletions(-) diff --git a/bench/evidence/frontier/README.md b/bench/evidence/frontier/README.md index a33dcb6ba..14b8f4030 100644 --- a/bench/evidence/frontier/README.md +++ b/bench/evidence/frontier/README.md @@ -23,6 +23,30 @@ bench/evidence/frontier/sentinel.sh # two cheap trials, end to end bench/evidence/frontier/primary.sh small fb-small-01 ``` +## Pick a backend + +`FB_ENV` selects where trials run — `docker` (this host, the default) or +`modal` (a sandbox provisioned per task, which is what the benchmark's own +authors used). + +```bash +FB_ENV=modal FB_CONCURRENCY=32 bench/evidence/frontier/fetch_dataset.sh +``` + +This is not a preference. A submittable run must cover all 74 tasks — four of +which want a GPU — without overriding any task's declared resources, so it needs +a machine that can hand a container 32 GB and an H100 on demand. On `docker` the +plan reports what this host can take and names what it cannot; on `modal` the +host's shape stops being a term in the measurement, and the plan is 74/74. + +Modal needs `modal token new` (or `MODAL_TOKEN_ID`/`MODAL_TOKEN_SECRET`); the +preflight checks for it and distinguishes "not authenticated" from "SDK not +installed", because they have different fixes. `warm_images.sh` skips on Modal — +the builds happen in the cloud, so a local pull warms a cache no trial reads. + +The glibc-2.17 assert runs on both. Modal is a different scheduler, not a +different libc. + ## What differs from Terminal-Bench, and why each difference exists **A second Harbor pin (0.20.0), not the audited 0.6.1.** Every one of the 74 @@ -72,7 +96,13 @@ scored 0.0 passes stage 2, correctly. `plan.py` reports what the host can take. On a 10 GB / 6-CPU Docker VM (a typical Mac) it admits **48 of 74** tasks — 4 GPU-excluded and 22 over the memory budget. That is a real constraint, not a bug, and it is why a submittable run -needs a bigger machine: see [SUBMISSION.md](SUBMISSION.md). +uses `FB_ENV=modal`, where the same plan is 74/74: see +[SUBMISSION.md](SUBMISSION.md). + +Worth knowing before you conclude the laptop is hopeless: 18 of those 22 want +exactly 8192 MB, which needs roughly a 10.2 GB VM once headroom is taken. Raising +Docker Desktop's memory allocation moves local coverage from 48 to 66 without +touching anything here. The memory exclusion is deliberately conservative. `memory_mb` is a cap Docker accepts even when the VM is smaller, so an 8 GB task on a 10 GB daemon usually diff --git a/bench/evidence/frontier/SUBMISSION.md b/bench/evidence/frontier/SUBMISSION.md index 0a894e9d3..96c37d390 100644 --- a/bench/evidence/frontier/SUBMISSION.md +++ b/bench/evidence/frontier/SUBMISSION.md @@ -45,9 +45,9 @@ reward hacking, then merge. | Dataset pinned to the leaderboard's `DATASET_REF` | **Done.** `env.sh` pins `sha256:97fd2ba3…`, the leaderboard's own ref | | Harbor ≥ 0.20.0 | **Done.** This lane pins 0.20.0 | | Default execution settings; no timeout or resource overrides | **Done.** Nothing here overrides either; tiering only chooses what runs *concurrently* | -| All 74 tasks, no subsetting | **Blocked by hardware** — see below | +| All 74 tasks, no subsetting | **Done on Modal.** `FB_ENV=modal` plans 74/74 with zero exclusions; `FB_ENV=docker` on a laptop plans 48 | | ≥ 5 trials per task | **Supported, off by default.** `FB_ATTEMPTS=5`; default 1 is the dev baseline | -| `--upload --public` | **Not wired.** Deliberate: uploading publishes trajectories, which is a decision to take explicitly, not a flag to inherit | +| `--upload --public` | **Wired, opt-in.** `FB_UPLOAD=1 FB_UPLOAD_VISIBILITY=public`. Off by default because uploading publishes every trajectory — a decision to take explicitly, never inherited from a shell | | Agent + model disclosure metadata | Supplied at `lb submit` time, not by this lane | Note the metric: **accuracy = trials with reward > 0, over all trials**. Not @@ -55,20 +55,43 @@ reward == 1.0, and errored trials count as reward 0 rather than being dropped. Infrastructure flakiness therefore lands directly in the score, which is exactly why `warm_images.sh` and the preflight exist. -## The two real blockers +## The two hardware blockers, and how the lane clears them **1. GPUs.** Four tasks need one, and the benchmark authors ran them on a single -H100 each. Subsetting is not allowed for a submission, so those four cannot be -excluded the way the dev baseline excludes them — they must actually run. +H100 each. Subsetting is not allowed, so they must actually run. -**2. Host size.** On a 10 GB / 6-CPU Docker VM, `plan.py` admits 48 of 74 tasks; -22 more are simply over the memory budget. The full set declares up to 32 GB -memory, 16 CPUs, and ~1.9 TB of storage in aggregate. +**2. Host size.** On a 10 GB / 6-CPU Docker VM `plan.py` admits 48 of 74; 22 more +are over the memory budget. The full set declares up to 32 GB memory, 16 CPUs, +and ~1.9 TB of storage in aggregate. -Both point the same way: **a submission run does not happen on a Mac.** The -benchmark's own guidance is Modal (`uv tool install 'harbor[modal]'`, `--env -modal`), with Daytona as an alternative that recently added GPU support. Harbor's -repo CI defaults to `env: modal`. +Both point the same way: **a submission run does not happen on a Mac.** So the +lane takes a backend: + +```bash +FB_ENV=modal bench/evidence/frontier/fetch_dataset.sh # plans 74/74 +FB_ENV=modal FB_CONCURRENCY=32 bench/evidence/frontier/sentinel.sh +``` + +`FB_ENV=modal` changes three things, each because a docker assumption is simply +false there rather than merely cautious: + +- **Nothing is excluded for capacity.** Modal builds a sandbox per trial from + that trial's own declared resources, so "does this task fit" is the wrong + question. Carrying the laptop's exclusions over would produce a 48-task + result that looks like a complete one. +- **GPU tasks default to *on*.** Harbor passes each task's `gpus`/`gpu_types` + straight through to Modal. Defaulting them off there would silently drop the + four tasks a submission is required to cover. `--no-allow-gpu` still wins if + you say it. +- **`warm_images.sh` skips.** The builds happen in the cloud; a local pull warms + a cache no trial reads. + +What does *not* change: the glibc-2.17 portability assert still runs. Modal is a +different scheduler, not a different libc — the SUT still has to exec inside a +Linux container. + +Daytona is the documented alternative that recently added GPU support; Harbor's +own repo CI defaults to `env: modal`. ## Cost, honestly @@ -91,6 +114,22 @@ was. run all 74 at five attempts uncapped with `--upload --public`. 4. File the PR once #1405 merges. -Steps 1 and 2 are what this directory is for. Step 3 needs a Modal environment -this lane does not yet configure — `primary.sh` hardcodes `--env docker`, and -adding a Modal path is the next piece of work, not something already done. +Step 3 is now a matter of setting variables rather than writing code: + +```bash +export FB_ENV=modal FB_ATTEMPTS=5 FB_CONCURRENCY=32 +export STELLA_BUDGET= # explicitly empty = no per-trial cap +export FB_UPLOAD=1 FB_UPLOAD_VISIBILITY=public +bench/evidence/frontier/fetch_dataset.sh +bench/evidence/frontier/primary.sh all fb-submission-01 +``` + +`primary.sh` prints `posture=` on every run and names every reason a run would +be refused — wrong tier, too few attempts, any excluded task, upload off, +visibility private — so a baseline cannot be mistaken for a submission at the +moment a number appears. It says `posture=submission-shaped` only when all five +hold. + +Do not skip the preregistration. This is ~370 trials of multi-hour work with no +spend cap; that is a four-figure decision and it deserves the same treatment the +Terminal-Bench claim got. diff --git a/bench/evidence/frontier/env.sh b/bench/evidence/frontier/env.sh index 3c4751726..9efead2aa 100755 --- a/bench/evidence/frontier/env.sh +++ b/bench/evidence/frontier/env.sh @@ -50,6 +50,20 @@ export FB_TASK_COUNT="${FB_TASK_COUNT:-74}" export FB_ATTEMPTS="${FB_ATTEMPTS:-1}" export FB_SUBMISSION_MIN_ATTEMPTS=5 +# Upload the finished job to Harbor Hub. Off by default and opt-in per run, +# because it is not a transport detail: the leaderboard requires the trials to +# be *publicly readable* — CI re-derives every trial from the uploaded job and a +# maintainer's judge reads every trajectory — so setting this publishes the +# agent's full reasoning and every command it ran, permanently and to everyone. +# That is a deliberate act. It should never be inherited from a shell that +# happened to have it set for some earlier run. +export FB_UPLOAD="${FB_UPLOAD:-0}" +export FB_UPLOAD_VISIBILITY="${FB_UPLOAD_VISIBILITY:-private}" +case "$FB_UPLOAD_VISIBILITY" in + public|private) ;; + *) echo "FATAL: FB_UPLOAD_VISIBILITY must be 'public' or 'private'"; return 1 2>/dev/null || exit 1 ;; +esac + # Frontier-Bench tasks run far longer than Terminal-Bench ones: declared agent # timeouts run from 30 minutes to 8 hours, with a 2-hour mode, against expert # time estimates measured in whole days. Terminal-Bench's $0.60 per-trial cap @@ -84,6 +98,23 @@ source "$FB_HERE/../run/env.sh" # So this lane pins its own Harbor, in its own virtualenv, and asserts it. The # Terminal-Bench venv at bench/harbor_adapter/.venv is left untouched. export FB_HARBOR_VERSION="${FB_HARBOR_VERSION:-0.20.0}" + +# Where trials actually run. `docker` is this host; `modal` is the cloud +# sandbox the benchmark's own authors used. +# +# The distinction is not a preference. A submittable run must cover all 74 +# tasks, four of which want a GPU, and must not override any task's declared +# resources — so it needs a machine that can hand a container 32 GB and an +# H100 on demand. This laptop cannot, and no amount of scheduling makes it. +# On `docker` the plan honestly reports what the host can take and excludes +# the rest by name; on `modal` each sandbox is provisioned per task, so the +# host's shape stops being a term in the measurement at all. +export FB_ENV="${FB_ENV:-docker}" +case "$FB_ENV" in + docker|modal) ;; + *) echo "FATAL: FB_ENV must be 'docker' or 'modal' (got '$FB_ENV')"; return 1 2>/dev/null || exit 1 ;; +esac + export VENV="$FB_HERE/.venv" export PATH="$VENV/bin:$PATH" export PYTHONPATH="$TB_REPO/bench/harbor_adapter" @@ -106,7 +137,16 @@ mkdir -p "$JOBS" # The default is therefore to exclude them and say so, loudly, in the score's # denominator — not to include them and quietly absorb four guaranteed zeros. # Set FB_ALLOW_GPU=1 only on a host that actually has one; `plan.py` checks. -export FB_ALLOW_GPU="${FB_ALLOW_GPU:-0}" +# +# On Modal the hazard does not exist: sandboxes are provisioned per task and +# Harbor passes each task's declared `gpus`/`gpu_types` straight through, so +# the four run for real. Defaulting them off there would silently drop the +# tasks a submission is required to cover. +if [ "$FB_ENV" = modal ]; then + export FB_ALLOW_GPU="${FB_ALLOW_GPU:-1}" +else + export FB_ALLOW_GPU="${FB_ALLOW_GPU:-0}" +fi # Memory the plan leaves to the Docker daemon itself rather than handing to a # task. A task declaring more than what remains is excluded as unrunnable. @@ -148,6 +188,34 @@ fb_assert_cli_flags() { return 1; } } +# Whatever FB_ENV names must actually be reachable before a run starts. +# +# Note what is *not* branched here: the glibc portability assert runs for both. +# Modal is a different scheduler, not a different libc — the SUT still has to +# exec inside a Linux container there, and a binary that cannot would fail the +# same way it did on 2026-07-31, as an ordinary reward-0 row. +fb_assert_backend() { + case "$FB_ENV" in + docker) + docker info >/dev/null 2>&1 || { echo "FATAL: docker unreachable"; return 1; } + ;; + modal) + "$VENV/bin/python" -c 'import modal' 2>/dev/null || { + echo "FATAL: the modal SDK is not installed in $VENV" + echo " Rebuild the venv: bench/evidence/frontier/setup_venv.sh" + return 1; } + # Harbor's own ModalEnvironment.preflight accepts either form; mirror it + # exactly rather than inventing a stricter rule it would then contradict. + if [ ! -f "$HOME/.modal.toml" ] && + { [ -z "${MODAL_TOKEN_ID:-}" ] || [ -z "${MODAL_TOKEN_SECRET:-}" ]; }; then + echo "FATAL: Modal is not authenticated." + echo " Run 'modal token new', or set MODAL_TOKEN_ID and MODAL_TOKEN_SECRET." + return 1 + fi + ;; + esac +} + fb_preflight() { test -n "${OPENROUTER_API_KEY:-}" || { echo "FATAL: OPENROUTER_API_KEY unset"; return 1; } test -x "$STELLA_BINARY" || { echo "FATAL: no SUT binary at $STELLA_BINARY (run ../run/build_sut.sh)"; return 1; } @@ -163,7 +231,7 @@ fb_preflight() { echo " 0.6.1 silently ignores this dataset's verifier environment_mode." return 1; } fb_assert_cli_flags || return 1 - docker info >/dev/null 2>&1 || { echo "FATAL: docker unreachable"; return 1; } + fb_assert_backend || return 1 } # The Terminal-Bench lane's `preflight` is still defined and still asserts diff --git a/bench/evidence/frontier/fetch_dataset.sh b/bench/evidence/frontier/fetch_dataset.sh index 1bed48757..72cf94666 100755 --- a/bench/evidence/frontier/fetch_dataset.sh +++ b/bench/evidence/frontier/fetch_dataset.sh @@ -18,4 +18,5 @@ test "$got" = "$FB_TASK_COUNT" || { exit 1; } python3 "$(dirname "${BASH_SOURCE[0]}")/plan.py" "$DATASET_DIR" \ - -o "$FB_PLAN" --images-out "$TB_ROOT/frontier-images.txt" + -o "$FB_PLAN" --images-out "$TB_ROOT/frontier-images.txt" \ + --backend "$FB_ENV" ${FB_CONCURRENCY:+--concurrency "$FB_CONCURRENCY"} diff --git a/bench/evidence/frontier/plan.py b/bench/evidence/frontier/plan.py index 9dc2b7769..6de0bc6e0 100755 --- a/bench/evidence/frontier/plan.py +++ b/bench/evidence/frontier/plan.py @@ -83,6 +83,18 @@ _HOST_FORMAT = '{{.MemTotal}}|{{.NCPU}}|{{(index .Runtimes "nvidia").Path}}' +def _elastic_capacity() -> dict: + """Capacity for a backend that provisions per task rather than sharing a host. + + Modal builds a sandbox per trial from that trial's own declared resources, + so there is no fixed pool to divide and nothing to exclude — asking "does + this task fit" is the wrong question there. Returning unbounded capacity + makes every downstream check trivially true, which is the honest encoding: + the constraint genuinely does not exist rather than being ignored. + """ + return {"memory_mb": None, "cpus": None, "gpu": True, "source": "modal (per-task sandbox)"} + + def _host_capacity() -> dict: """What the Docker daemon says it can actually give a container.""" cap = {"memory_mb": None, "cpus": None, "gpu": False, "source": "unavailable"} @@ -166,7 +178,14 @@ def _tier(mem_mb: int) -> str: return TIERS[-1][0] -def build_plan(dataset_dir: Path, *, allow_gpu: bool, memory_headroom_mb: int) -> dict: +def build_plan( + dataset_dir: Path, + *, + allow_gpu: bool, + memory_headroom_mb: int, + backend: str = "docker", + concurrency: int | None = None, +) -> dict: roots = sorted(p.parent for p in dataset_dir.glob("*/*/task.toml")) if not roots: roots = sorted(p.parent for p in dataset_dir.glob("*/task.toml")) @@ -174,7 +193,7 @@ def build_plan(dataset_dir: Path, *, allow_gpu: bool, memory_headroom_mb: int) - sys.exit(f"FATAL: no task.toml under {dataset_dir}") tasks = [_read_task(p) for p in roots] - host = _host_capacity() + host = _elastic_capacity() if backend == "modal" else _host_capacity() host_mem = host.get("memory_mb") host_cpus = host.get("cpus") @@ -208,9 +227,16 @@ def build_plan(dataset_dir: Path, *, allow_gpu: bool, memory_headroom_mb: int) - members = [t for t in runnable if t["slug"] in set(tier["tasks"])] peak_mem = max(t["memory_mb"] for t in members) peak_cpus = max(t["cpus"] for t in members) - by_mem = (usable_mem // peak_mem) if usable_mem else 1 - by_cpu = (host_cpus // peak_cpus) if host_cpus else 1 - tier["concurrency"] = max(1, min(MAX_CONCURRENCY, by_mem, by_cpu)) + if concurrency is not None: + # An explicit ceiling replaces the arithmetic entirely. On a + # per-task backend there is no shared pool to divide, so deriving + # concurrency from capacity would compute 1 from two unknowns and + # serialise a run that has no reason to be serial. + tier["concurrency"] = max(1, concurrency) + else: + by_mem = (usable_mem // peak_mem) if usable_mem else 1 + by_cpu = (host_cpus // peak_cpus) if host_cpus else 1 + tier["concurrency"] = max(1, min(MAX_CONCURRENCY, by_mem, by_cpu)) tier["peak_memory_mb"] = peak_mem tier["peak_cpus"] = peak_cpus tier["tasks"].sort() @@ -224,6 +250,7 @@ def build_plan(dataset_dir: Path, *, allow_gpu: bool, memory_headroom_mb: int) - return { "dataset_dir": str(dataset_dir), + "backend": backend, "host": host, "memory_headroom_mb": memory_headroom_mb, "allow_gpu": allow_gpu, @@ -245,8 +272,13 @@ def main() -> int: ap = argparse.ArgumentParser(description=__doc__) ap.add_argument("dataset_dir", type=Path) ap.add_argument("-o", "--output", type=Path, required=True) - ap.add_argument("--allow-gpu", action="store_true", - default=os.environ.get("FB_ALLOW_GPU") == "1") + # Tri-state on purpose. The default has to depend on the backend — a Modal + # run that quietly drops the four GPU tasks is unsubmittable and looks + # exactly like a complete one — but an operator saying "no GPUs" must still + # win over that. `None` distinguishes "unset" from "explicitly off", which + # `store_true` alone cannot. + ap.add_argument("--allow-gpu", action="store_true", default=None) + ap.add_argument("--no-allow-gpu", dest="allow_gpu", action="store_false") # Memory, not storage. The two were one knob in the first draft of this # file and a 20 GB *storage* headroom subtracted from a 10 GB Docker VM's # *memory* excluded all 74 tasks with a negative budget — the plan was @@ -254,17 +286,37 @@ def main() -> int: ap.add_argument("--memory-headroom-mb", type=int, default=int(os.environ.get("FB_MEMORY_HEADROOM_MB", "2048"))) ap.add_argument("--images-out", type=Path, default=None) + ap.add_argument("--backend", choices=("docker", "modal"), + default=os.environ.get("FB_ENV", "docker"), + help="docker: schedule against this host. modal: per-task " + "sandboxes, so nothing is excluded for capacity.") + ap.add_argument("--concurrency", type=int, default=None, + help="Override the derived per-tier concurrency. Required " + "in spirit for modal, where there is no host pool to " + "divide.") args = ap.parse_args() - plan = build_plan(args.dataset_dir, allow_gpu=args.allow_gpu, - memory_headroom_mb=args.memory_headroom_mb) + concurrency = args.concurrency + if concurrency is None and args.backend == "modal": + concurrency = int(os.environ.get("FB_CONCURRENCY", "16")) + + allow_gpu = args.allow_gpu + if allow_gpu is None: + allow_gpu = args.backend == "modal" or os.environ.get("FB_ALLOW_GPU") == "1" + + plan = build_plan(args.dataset_dir, allow_gpu=allow_gpu, + memory_headroom_mb=args.memory_headroom_mb, + backend=args.backend, concurrency=concurrency) args.output.write_text(json.dumps(plan, indent=1) + "\n") if args.images_out: args.images_out.write_text("\n".join(plan["base_images"]) + "\n") host = plan["host"] - print(f"host: {host.get('memory_mb')} MB / {host.get('cpus')} CPUs " - f"/ gpu={host.get('gpu')} ({host.get('source')})") + if plan["backend"] == "modal": + print(f"backend: modal — {host.get('source')}; host capacity is not a constraint") + else: + print(f"host: {host.get('memory_mb')} MB / {host.get('cpus')} CPUs " + f"/ gpu={host.get('gpu')} ({host.get('source')})") for name, tier in plan["tiers"].items(): print(f" tier {name:<7} tasks={len(tier['tasks']):<3} " f"concurrency={tier['concurrency']} peak={tier['peak_memory_mb']}MB/" diff --git a/bench/evidence/frontier/primary.sh b/bench/evidence/frontier/primary.sh index 40042d4e4..aa62eb066 100755 --- a/bench/evidence/frontier/primary.sh +++ b/bench/evidence/frontier/primary.sh @@ -38,7 +38,20 @@ while IFS= read -r t; do INCLUDES="$INCLUDES --include-task-name $FB_TASK_PREFIX/$t" done < "$TB_ROOT/frontier-$TIER.tasks" +# Uploading publishes every trajectory. Build the flags once, here, so the +# invocation below cannot half-apply them — `--public` without `--upload` is a +# Harbor error, and `--upload` without meaning to is a disclosure. +UPLOAD_FLAGS="" +if [ "$FB_UPLOAD" = 1 ]; then + UPLOAD_FLAGS="--upload --$FB_UPLOAD_VISIBILITY" + if [ "$FB_UPLOAD_VISIBILITY" = public ]; then + echo "NOTE: this run will be uploaded to Harbor Hub as PUBLIC — every" + echo " trajectory and command becomes permanently world-readable." + fi +fi + echo "tier=$TIER job=$JOB tasks=$N concurrency=$CONC attempts=$FB_ATTEMPTS" +echo "backend=$FB_ENV upload=${FB_UPLOAD}${FB_UPLOAD:+/$FB_UPLOAD_VISIBILITY}" echo "sut=$STELLA_SOURCE_COMMIT model=$TB_MODEL budget/trial=${STELLA_BUDGET:-uncapped}" echo "dataset=$FB_DATASET" echo "harbor=$(harbor --version) started=$(date -u +%Y-%m-%dT%H:%M:%SZ)" @@ -48,24 +61,38 @@ echo "harbor=$(harbor --version) started=$(date -u +%Y-%m-%dT%H:%M:%SZ)" # five-plus attempts with no subsetting. Both are legitimate — reporting one as # the other is not, and the distinction is easiest to lose at the moment a # number finally appears. -if [ "$TIER" != "all" ] || [ "$FB_ATTEMPTS" -lt "$FB_SUBMISSION_MIN_ATTEMPTS" ]; then - echo "posture=development-baseline (not submittable: see SUBMISSION.md)" +# Name every reason this run would be refused, not just the first. An operator +# who fixes one and reruns only to be told about the next has paid for a full +# run to learn something this loop could have said in one line. +EXCLUDED_N=$(python3 -c "import json;print(len(json.load(open('$FB_PLAN'))['excluded']))") +GAPS="" +[ "$TIER" = "all" ] || GAPS="$GAPS tier=$TIER(needs-all)" +[ "$FB_ATTEMPTS" -ge "$FB_SUBMISSION_MIN_ATTEMPTS" ] || GAPS="$GAPS attempts=$FB_ATTEMPTS(needs-$FB_SUBMISSION_MIN_ATTEMPTS)" +[ "$EXCLUDED_N" = 0 ] || GAPS="$GAPS excluded=$EXCLUDED_N(needs-0;-subsetting-is-not-allowed)" +[ "$FB_UPLOAD" = 1 ] || GAPS="$GAPS upload=off" +[ "$FB_UPLOAD_VISIBILITY" = public ] || GAPS="$GAPS visibility=$FB_UPLOAD_VISIBILITY(needs-public)" +if [ -n "$GAPS" ]; then + echo "posture=development-baseline — not submittable:$GAPS" + echo " (see SUBMISSION.md; a baseline is a legitimate thing to run," + echo " it just must not be reported as a leaderboard number)" else echo "posture=submission-shaped" fi cd "$TB_REPO" || exit 1 -# INCLUDES is deliberately unquoted: a pre-built flag list, one word per token, -# every task name a fixed [a-z0-9-] slug from the digest-pinned dataset. +# INCLUDES and UPLOAD_FLAGS are deliberately unquoted: pre-built flag lists, +# one word per token, every task name a fixed [a-z0-9-] slug from the +# digest-pinned dataset and every upload flag a literal from this script. # shellcheck disable=SC2086 harbor run \ - --env docker \ + --env "$FB_ENV" \ --dataset "$FB_DATASET" \ $INCLUDES \ --agent stella_harbor:StellaAgent \ --model "$TB_MODEL" \ --job-name "$JOB" \ --jobs-dir "$JOBS" \ + $UPLOAD_FLAGS \ --n-attempts "$FB_ATTEMPTS" --n-concurrent "$CONC" --max-retries 0 rc=$? echo "harbor_exit=$rc finished=$(date -u +%Y-%m-%dT%H:%M:%SZ)" diff --git a/bench/evidence/frontier/sentinel.sh b/bench/evidence/frontier/sentinel.sh index 1bcd61d58..b2409208c 100755 --- a/bench/evidence/frontier/sentinel.sh +++ b/bench/evidence/frontier/sentinel.sh @@ -34,7 +34,7 @@ cd "$TB_REPO" || exit 1 echo "=== stage 1: synthetic fixture (gate: reward == 1.0)" harbor run \ - --env docker \ + --env "$FB_ENV" \ --path "$TB_REPO/bench/readiness/synthetic-adapter-sentinel" \ --agent stella_harbor:StellaAgent \ --model "$TB_MODEL" \ @@ -79,7 +79,7 @@ test -n "$TASK" || { echo "SKIP stage 2: no runnable task in the plan"; exit 0; JOB2="$JOB-real" echo "=== stage 2: real task $TASK (gate: the trial ran; any reward is acceptable)" harbor run \ - --env docker \ + --env "$FB_ENV" \ --dataset "$FB_DATASET" \ --include-task-name "$FB_TASK_PREFIX/$TASK" \ --agent stella_harbor:StellaAgent \ diff --git a/bench/evidence/frontier/setup_venv.sh b/bench/evidence/frontier/setup_venv.sh index 9dc337789..9f56f3f8c 100755 --- a/bench/evidence/frontier/setup_venv.sh +++ b/bench/evidence/frontier/setup_venv.sh @@ -11,8 +11,17 @@ source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/env.sh" command -v uv >/dev/null 2>&1 || { echo "FATAL: uv is required"; exit 1; } -uv venv "$VENV" --python 3.13 || exit 1 -uv pip install --python "$VENV/bin/python" "harbor==$FB_HARBOR_VERSION" || exit 1 +# --clear so this is idempotent. Without it the script fails the moment a venv +# exists, which is precisely when you most want to run it: after changing the +# Harbor pin or adding the modal extra. The venv is a derived, gitignored +# artifact rebuilt from two pinned inputs, so replacing it loses nothing. +uv venv "$VENV" --python 3.13 --clear || exit 1 +# The `modal` extra unconditionally, not only when FB_ENV=modal. Harbor loads +# vendor SDKs lazily, so a docker-only run never imports it — but installing it +# up front means switching backends is one variable rather than a rebuild, and +# the preflight can then tell "modal is not authenticated" apart from "the SDK +# was never installed". Two different problems deserve two different messages. +uv pip install --python "$VENV/bin/python" "harbor[modal]==$FB_HARBOR_VERSION" || exit 1 got="$("$VENV/bin/harbor" --version)" test "$got" = "$FB_HARBOR_VERSION" || { diff --git a/bench/evidence/frontier/tests/test_plan.py b/bench/evidence/frontier/tests/test_plan.py index 07fc7135e..bf7ccd031 100644 --- a/bench/evidence/frontier/tests/test_plan.py +++ b/bench/evidence/frontier/tests/test_plan.py @@ -295,6 +295,88 @@ def test_an_excluded_task_does_not_drag_in_its_images(self, tmp_path, host): assert images == ["ubuntu:24.04"] +class TestModalBackend: + """On a per-task backend the host stops being a term in the measurement. + + These matter because the docker path's caution becomes a bug when carried + over unchanged: a Modal run that excludes tasks for *this laptop's* memory + would silently produce a 48-task result while looking like a complete one, + and a submission may not subset. + """ + + def test_a_task_far_larger_than_this_host_still_runs(self, tmp_path, host): + host(memory_mb=8192, cpus=2) # deliberately smaller than the task + _task(tmp_path, "huge", memory_mb=32768, cpus=16) + + result = plan.build_plan( + tmp_path, allow_gpu=True, memory_headroom_mb=2048, + backend="modal", concurrency=32, + ) + + assert result["runnable_tasks"] == ["huge"] + assert result["excluded"] == [] + + def test_gpu_tasks_are_runnable(self, tmp_path, host): + host(memory_mb=8192, cpus=2, gpu=False) # the *host* has no GPU; Modal does + _task(tmp_path, "gpu-task", gpus=1) + + result = plan.build_plan( + tmp_path, allow_gpu=True, memory_headroom_mb=2048, + backend="modal", concurrency=32, + ) + + assert result["runnable_tasks"] == ["gpu-task"] + + def test_docker_probe_is_not_consulted_at_all(self, tmp_path, monkeypatch): + """A Modal plan must not fail because Docker is absent.""" + + def _explode(): + raise AssertionError("_host_capacity must not run for backend=modal") + + monkeypatch.setattr(plan, "_host_capacity", _explode) + _task(tmp_path, "a", memory_mb=32768) + + result = plan.build_plan( + tmp_path, allow_gpu=True, memory_headroom_mb=2048, + backend="modal", concurrency=8, + ) + + assert result["backend"] == "modal" + assert result["runnable_tasks"] == ["a"] + + def test_explicit_concurrency_beats_the_capacity_arithmetic(self, tmp_path, host): + """Unknown capacity would otherwise derive 1 and serialise the run.""" + host() + _task(tmp_path, "a", memory_mb=16384, cpus=16) + + tiers = plan.build_plan( + tmp_path, allow_gpu=True, memory_headroom_mb=2048, + backend="modal", concurrency=64, + )["tiers"] + + assert tiers["xlarge"]["concurrency"] == 64 + + def test_explicit_concurrency_also_overrides_on_docker(self, tmp_path, host): + """The override is a property of the flag, not of the backend.""" + host(memory_mb=65536, cpus=64) + _task(tmp_path, "a", memory_mb=2048, cpus=1) + + tiers = plan.build_plan( + tmp_path, allow_gpu=False, memory_headroom_mb=2048, + backend="docker", concurrency=2, + )["tiers"] + + # 2, not MAX_CONCURRENCY, which the arithmetic alone would have chosen. + assert tiers["small"]["concurrency"] == 2 + + def test_backend_is_recorded_in_the_plan(self, tmp_path, host): + host() + _task(tmp_path, "a") + assert plan.build_plan( + tmp_path, allow_gpu=False, memory_headroom_mb=2048, + )["backend"] == "docker" + + class TestPlanShape: def test_counts_reconcile(self, tmp_path, host): host(memory_mb=10240, cpus=8, gpu=False) diff --git a/bench/evidence/frontier/warm_images.sh b/bench/evidence/frontier/warm_images.sh index 1ef4df8c2..0fd2c5dec 100755 --- a/bench/evidence/frontier/warm_images.sh +++ b/bench/evidence/frontier/warm_images.sh @@ -14,6 +14,15 @@ set -uo pipefail source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/env.sh" +# On Modal the builds happen in the cloud, against Modal's own registry path. +# Pulling those bases onto this laptop would warm a cache no trial ever reads — +# minutes and gigabytes spent to change nothing. Skipping is the correct +# behaviour, and saying so beats appearing to have prepared something. +if [ "$FB_ENV" = modal ]; then + echo "SKIP: FB_ENV=modal builds task images in the cloud; a local pull warms nothing." + exit 0 +fi + LIST="${1:-$TB_ROOT/frontier-images.txt}" FAILED="$TB_ROOT/frontier_warm_failed.txt" test -f "$LIST" || { echo "FATAL: no image list at $LIST (run fetch_dataset.sh)"; exit 1; }