bench: the runbook's STELLA_BINARY path is 291 commits stale and passes the pinning check

[macanderson]

Problem

~/.arenabench/sut/stella — the path every runbook and launch script sets STELLA_BINARY to — is exactly 291 commits stale, and nothing refuses it.

~/.arenabench/sut/stella                      sha256 4cf1c1da…   Aug 4 19:22
~/.arenabench/sut/sut_commit.txt              47d587a37da0e1013010f4479295b33878760b7d

~/.arenabench/sut/6c3455326adfb04c0b3dcb35aad192c0f78b6e33/stella
                                              sha256 1c658620…   Aug 6 20:49

$ git rev-list --count 47d587a3..6c345532
291
$ git rev-list --count 6c345532..origin/main
12

This is the same 291-commit staleness that already invalidated every match run Aug 4–7. PRs #2016/#2020 fixed the measurement half — the SUT is now pinned by commit and written to sut/<commit>/stella, and an unpinned run refuses — but the legacy sut/stella path was never repointed or removed, and it is still what the operator-facing launch instructions name:

ARENABENCH_STELLA_ADAPTER=/Users/macanderson/Projects/stella/bench/harbor_adapter \
STELLA_BINARY=/Users/macanderson/.arenabench/sut/stella \
nohup uv run arenabench serve > ~/.arenabench/server.log 2>&1 &

A bench cycle launched that way silently measures a 291-commit-old agent and the result gets attributed to current code. That is the precise failure mode the commit pinning was introduced to end; it survives because the pinning added a new correct path instead of invalidating the old wrong one.

Why it is not caught

The refusal added in #2016/#2020 triggers on an unpinned run. sut/stella is accompanied by its own sut_commit.txt (47d587a3), so it looks pinned and passes — it is pinned, just to a commit from three days and 291 changes ago. Staleness and pinning are different properties and only one is checked.

What to do

Pick one, in rough order of preference:

1. Delete ~/.arenabench/sut/stella and make the bare path an error. Force STELLA_BINARY to resolve through sut/<commit>/stella, so naming a binary without naming its commit is impossible.
2. Make sut/stella a symlink to the newest pinned build, refreshed by whatever writes sut/<commit>/. Keeps every existing script working and cannot go stale.
3. Add a staleness refusal beside the pinning one: compare the resolved SUT commit against origin/main and refuse (or require an explicit override) past some distance. This is the only option that also catches a pinned-but-ancient build reached by a path nobody anticipated.

Whichever is chosen, update the operator runbook and ~/.arenabench/serve-both-benchmarks.sh in the same change, since the stale path is copied by hand between sessions.

Verify

$ readlink -f "$STELLA_BINARY"
$ cat "$(dirname "$(readlink -f "$STELLA_BINARY")")/sut_commit.txt"
$ git rev-list --count <that-commit>..origin/main

Done when a run launched from the documented runbook either uses a binary within a stated distance of main, or refuses with a message naming the commit and the distance. A witness: point STELLA_BINARY at sut/stella (47d587a) and confirm the launch refuses, where today it proceeds and produces scoreable trials.

Constraints

• bench/harbor_adapter/stella_harbor/portability.py and host_attestation.py are where SUT identity is currently attested; the check belongs beside them, not in the adapter root (which is at its file-size ceiling).
• Do not "fix" this by regenerating the stale binary in place — the point is that the path cannot express staleness, not that this particular build is old.

Related: #2016, #2020 (the commit pinning this completes).

[macanderson]

Rig-side mitigation applied: ~/.arenabench/sut/stella is now a symlink to the commit-pinned build (option 2 above), so the runbook path can no longer go stale.

Rebuilt from origin/main via bench/evidence/run/build_sut.sh:

• commit 6a41889a66ef7000f71c22d277ce2c45c645d484
• sha256 af34f686e2ca38103cd8fbe792304d09b54dd0f83b479b136c6bbd19df1134be
• assert_portable_binary passes — satisfies the glibc floor for every task image

The already-running arena server picked it up with no restart, because it resolves STELLA_BINARY per trial.

One extra finding for whoever takes this. build_sut.sh runs its own git fetch origin main and then refuses on any Rust drift between the working tree and the freshly-fetched origin/main. If main moves between your checkout and the script's fetch, the build fails with a drift list that looks like local contamination but is not — it happened on the first attempt here (44f4c63a → 9b502f1c). The fix is to fetch and check out inside the same wrapper that invokes the build. Worth either pinning the SUT commit as an argument or making the refusal message name the race, since the current message sends you hunting for local edits that do not exist.

The repo-side fix in this issue still stands — the symlink is one machine's mitigation, not the guard.

[macanderson]

Correction — do NOT use option 2 (symlink) on this rig. I applied it, and it broke every trial:

Trial ... failed: Command failed (exit 1): sha256sum /tmp/stella-upload && cp /tmp/stella-upload /usr/local/bin/stella && ...
stdout: sha256sum: /tmp/stella-upload: Permission denied

The adapter's binary upload ships the symlink rather than the bytes it points at, so the in-container /tmp/stella-upload is unusable and all five trials die in harbor's _setup_agent before the agent ever starts. Scored as an operational abort (void), not a loss.

Working mitigation is a real copy, refreshed by the build:

rm -f ~/.arenabench/sut/stella
cp ~/.arenabench/sut/$SUT_COMMIT/stella ~/.arenabench/sut/stella
chmod 0755 ~/.arenabench/sut/stella

Verified: shasum -a 256 of the copy matches the pinned build (af34f686…), and a relaunched match produces stella-events.jsonl normally.

So the surviving options are 1 (delete the bare path, force resolution through sut/<commit>/stella) or 3 (a staleness refusal). Option 1 is now clearly the better of the two, since it makes the wrong path unusable rather than making it correct-but-fragile — and this failure shows the bare path has a second requirement nobody had written down: it must be a regular file, not a link.

[macanderson]

Repo-side guard shipped in #2052, taking option 3 (the staleness refusal), plus the build_sut.sh race you flagged.

The guard reads the artifact, not the paperwork. sut_commit.txt is a claim about a file — the thing that went stale here, and the thing a symlink can silently repoint. crates/stella-cli/build.rs stamps STELLA_BUILD_GIT_SHA into a NUL-delimited literal, and build_info.rs's doc comment says outright that the delimiting exists so the identity can be attested without executing the binary. New bench/harbor_adapter/stella_harbor/freshness.py reads that and measures the distance from origin/main, so a repointed path or a hand-refreshed sidecar cannot make an old build look new — and a sidecar disagreeing with the binary is itself a refusal. Reading the ELF rather than running stella --version is also what lets it run on a macOS host that cannot exec the cross-build at all.

Wired at both launch paths, with deliberately different postures:

• Evidence runbook (preflight() in bench/evidence/run/env.sh) — fails closed. An unidentifiable binary is fatal, because this path publishes numbers.
• Unpinned arena match (sut.unpinned_problem()) — refuses only on positive evidence of staleness. A plain cargo build --release carries no stamp, cannot be dated, and must not block the local loop.

The limit is 25 commits, which I measured rather than guessed: origin/main moved 654 commits in the seven days to 2026-08-07 (~93/day), so 25 is about six hours of drift — it tolerates a binary staged at the start of a long run and refuses the 291-commit artifact by a factor of twelve.

Your build_sut.sh finding. Fixed as both of the things you suggested. It takes an optional commit argument, so a wrapper that fetches and checks out can build exactly what it prepared. Unargued, it now records origin/main before the fetch as well and separates the two causes: if the tree is byte-identical to where the branch started, the refusal says so explicitly, names both SHAs and the distance, and suggests build_sut.sh <that-commit> — instead of handing you a drift list that reads as local contamination. It also verifies its own output now (--reference "$SUT" --max-behind 0 is an equality assertion on the compile-time stamp), so a build that silently stamped something else is caught at the build rather than becoming the next thing that measures the wrong code.

Your symlink mitigation is pinned by a test. read_identity resolves the path before reading the sidecar, with a regression test for exactly your layout — a stale sut_commit.txt at the legacy location next to a symlink pointing into sut/<commit>/. A guard reading the link's directory would report the stale claim; that one is now a test failure rather than a hazard.

The witness is the scenario from the issue: an unpinned Stella seat with STELLA_BINARY at a far-behind binary refuses, naming the commit and the distance. Verified behaviourally by reverting only the dispatch line — on the old code it returns None and the launch proceeds.

Three follow-ups filed rather than folded in: #2049 (make shellcheck does not cover bench/evidence/run/*.sh, so my edits to two of those scripts passed the gate ungated), #2050 (two copies of the embedded-commit reader now exist inside stella_harbor), #2051 (a caller who drives Harbor directly without env.sh is still unguarded — the residual half of option 3).

One thing worth your call: I did not take option 1. sut_ref="" is an existing documented affordance, and turning it into a hard error felt like a behaviour decision to surface rather than make silently. Also flagged in the PR: the arena still cannot catch a stale unstamped binary, which is the price of not breaking the local development loop. Happy to make it fail closed there too if you would rather.