test(integration): per-section timing, section selection, model pin, image warm-up

[rappdw]

The integration suite took ~60 minutes with no way to see where the time went, no way to re-run one section, and no protection against cold image builds landing inside a 300s per-test timeout. Five changes, all confined to the harness.

1. Per-section timing (the one that made the rest decidable)

There was zero instrumentation — no SECONDS, no durations. Every optimization was guesswork. A section "N. Title" helper replaces the 24 info headers and times each via bash's SECONDS (BSD date has no %N; bash 3.2 has no EPOCHREALTIME). Prints a slowest-first table plus total wall-clock, and still emits it on the abort path.

2. Section selection

SANDY_INTEG_ONLY=13,19 / SANDY_INTEG_SKIP=19,20,21. Exact-token matching, so ONLY=7 does not select 7b and ONLY=12 does not select 12b/12c; SKIP wins over ONLY; whitespace stripped so a natural ONLY=7, 13 works rather than silently running nothing. Unknown tokens warn.

Chasing one failure no longer costs a full run — the codex investigation that prompted this would have been SANDY_INTEG_ONLY=2, about two minutes.

3. Fast-model pin (claude only)

Nearly every assertion here is about sandy's plumbing — launch, mounts, proxy allow/deny, credentials, sandbox layout — not model quality. Claude runs now use Haiku instead of the default Opus.

• Section 7 deliberately opts back out (SANDY_MODEL= → sandy's own default), so one canonical test still proves the default-model path end to end. This works because sandy uses ${SANDY_MODEL:-…} — colon-dash — at both sites, so empty is treated as unset. Verified in source before relying on it.
• SANDY_INTEG_NO_MODEL_PIN=1 restores previous behavior; an explicitly-set SANDY_MODEL always wins.
• gemini/codex/opencode/grok are NOT pinned. No fast model id for them is documented anywhere in this repo, and inventing one would fail the suite. For opencode it is also structurally unsafe — the right provider/model depends on which credential the runner has.

4. Image warm-up preflight — and the real bug it fixes

The per-test timeout bounds an agent turn, not a multi-minute build. But any base-Dockerfile change invalidates every image, so the next run rebuilds the whole stack inside whichever test fires first.

That is exactly how the bookworm→trixie migration produced three unrelated-looking failures: §15b died with SIGALRM mid apt-get install, and the codex launch blew its 300s budget building base+proxy+codex — returning no answer (→ one failure) and leaving no sandbox dir (→ a second failure). None of it was an agent bug, and a naive reading blamed codex.

Warm-up builds credentialed agents' images up front under SANDY_INTEG_BUILD_TIMEOUT (1800s), restoring the per-test budget afterwards. ensure_image_built short-circuits on an existing image, so this is a no-op on a warm host. Recorded as pseudo-section 0 so build cost is visible but deliberately not selectable.

5. Fixed the last remaining failure

The corrupt-credentials test points SANDY_HOME at a throwaway dir, so sandy found no .base_build_hash (sandy:6013) and rebuilt the base with --no-cache --pull (6016) before reaching the credential check ~2,000 lines later (8136). It timed out rather than asserting anything.

Seeding the throwaway SANDY_HOME with the real Dockerfiles and build hashes skips every build: §15 drops from 306s to seconds and the test finally exercises what it is actually about.

The data (first timed run, before fix 5)

Total 1607s.

Bucket	Time	Share
Acceptance harnesses (19+20+21)	733s	46%
§15 timeout (fixed here)	306s	19%
gemini 157 + grok 139 + opencode 116	412s	26%
Other 18 sections combined	~156s	10%

This says parallelism is not the next move. The three acceptance harnesses cannot run concurrently (host-wide daemon state; §20's --rebuild deliberately marks other sessions stale) and dominate what remains. SANDY_INTEG_SKIP=19,20,21 alone takes a routine run to ~9.5 minutes. Next best targets are §20's 434s (it runs --rebuild) and fast models for the three unpinned agents.

Portability (this repo's recurring failure mode)

SECONDS (never assigned — an assignment would reset it); parallel indexed arrays + scalar count (no bash-4 associative arrays); POSIX case token matching (no awk, avoiding the BWK \$ trap); ${arr[@]+"${arr[@]}"} for possibly-empty arrays under set -u; sort -rn (no -V). No process substitution, no python3 -c "…", no set -E traps — the exact classes that caused three macOS-only failures earlier today.

Verification (static — Docker unavailable in the authoring environment)

• bash -n; zero new shellcheck warnings vs HEAD
• 24 headers / 24 guards / 24 closers correctly interleaved, with per-section if/fi/case/do balance — bash -n cannot catch misnesting
• No cross-section variable reads — a set -u abort risk if a skipped section owned a later section's variable. The plan claimed "no dependencies"; verified directly rather than trusted
• All run-tests.sh assertions about this file still pass, including the §80(b) awk range still containing PIPESTATUS/RESULT:
• Selection semantics unit-tested against the extracted functions
• Timeout save/restore verified for both the set and previously-unset cases

Independent of #134 (different files); either can merge first.

🤖 Generated with Claude Code