[Bug]: one sandbox call that never returns stalls cb run and the Tinker trainer indefinitely #5193

Primary area

CLI

Summary

If one call into a variant's sandbox never returns (for example a shell command that hangs in a healthy sandbox), cb run never finishes: BatchRunner has no bound on a variant's task setup, agent and evaluation, so the batch's gather never returns and summary.json is never written. The Tinker GRPO loop (cua_bench_rl/trainer/off_policy/tinker/rollout.py) waits on cb run with no timeout, so training stops with no error, and the variants of that run that did finish never reach the trainer. Separately, if cb run is killed, rollout.py polls the session store for up to timeout (3600 s by default) for sessions that can no longer finish.

Reproduction

Local sandbox, no model calls (a scripted agent):

1. A dataset with two variants of a task like example_tasks/hello_file_env, whose agent calls await session.run_command("sleep 100000") in one variant (a command that never returns; the sandbox stays healthy).
2. Collect rollouts the way the Tinker loop does: rollout.run_rollouts(tasks_path, agent, model, max_steps, num_rollouts=2) (it runs cb run dataset ...).
3. The other variant finishes and writes its result.json; cb run never exits; run_rollouts never returns.

Also: cb run killed mid-run (SIGKILL): run_rollouts keeps polling for up to timeout (3600 s), then raises TimeoutError.

Observed in 3/3 runs on a real local container (ghcr.io/trycua/linux:24.04, runc on Colima) and 3/3 with cua-bench's in-memory test sandbox; the cb run kill case 2/2 on the real container and 3/3 in-memory (numbers below).

Expected behavior

A stuck variant fails after a bound and releases its sandbox; the batch finishes; the trainer gets the variants that finished. A cb run that died is not waited on.

Actual behavior

• Hung command in one variant: the trainer made no further progress, with no error, in every run (3/3, real container, 5-minute windows); 1 evaluated episode of that run never reached the trainer; the hung variant's sandbox stays up for as long as cb waits.
• cb run SIGKILLed mid-run: no progress for the rest of the window (128 s+ of 180 s; by the code, up to 3600 s, then TimeoutError); 2 local sandboxes left running.
• For comparison, a sandbox frozen with docker pause mid-episode is handled: the native transport fails the call (CuaError.Transport: ... http2 error (operation timed out)) and the run continues (recovered in 43.5 s, 1 episode lost). The stall needs a call that never errors.

Environment

• cua-bench / cua-bench-rl at main 8be7052; cua 0.4.1, cua-sandbox 0.9.0 (PyPI wheels)
• macOS 26 (Apple silicon), Python 3.12.15, Colima (runc), image ghcr.io/trycua/linux:24.04 (arm64)
• Local only; outbound network denied for the run (sandbox-exec), no model calls

Evidence

Fault-injection study (deterministic, local; harness drives the real cb CLI, BatchRunner, run_episode, RemoteDesktopSession and the Tinker loop's run_rollouts + traces.load_runs):

Real container (ghcr.io/trycua/linux:24.04), each epoch = 2 rollouts of a 2-variant dataset, 3 seeds × 5 min. "Fixed" passes variant_timeout=90 (scaled to these ~20–50 s episodes):

cua-bench's in-memory test sandbox (8 variants × 2 rollouts per epoch; "fixed" passes variant_timeout=30), 3 seeds × 5 min:

The deadlines above are scaled to the short test episodes; with the proposed 3600 s default the same recovery comes an hour after the hang instead of never. The fixed runs also included a separate file_exists fix that does not touch these paths.

Workaround and additional context

Workaround: run cb run under an external watchdog and kill it; finished variants are then on disk but rollout.py still waits on their sessions.

Proposed fix (I have it on a branch with tests and can open a PR once the option and default are agreed):

• cb run --variant-timeout SECONDS (and BatchRunner(variant_timeout_s=...)): a variant whose task setup, agent and evaluation (not sandbox start) run longer is cancelled the same way Ctrl-C cancels it (its sandbox is released by the existing async with), recorded as failed ("variant timed out after N s"), and not retried.
• Default: 3600 s proposed (above the 1800 s CUA_BENCH_HARNESS_TIMEOUT a harness agent gets, which the deadline must not cut off), 0 = no limit. Decision for you: this changes behaviour for anyone with a variant that legitimately runs longer than the default; alternatives are a larger default or opt-in (0 by default). Migration note: such users pass --variant-timeout 0 (or a larger value).
• rollout.py: pass an optional variant_timeout through to cb; an optional run_timeout for the whole cb run (SIGTERM, then SIGKILL after 60 s; the run directory is returned so finished variants are kept); do not wait on the sessions of a cb that exited non-zero.

Related: #4929 (draft) also edits cli/main.py and cli/commands/run.py. Not a duplicate: I searched issues and PRs for cb run hangs, variant/rollout timeouts and the Tinker rollout.

Submission checks

• I searched for duplicate issues and active pull requests.
• This issue describes a single problem.
• I removed credentials, private data, sensitive screenshots, and vulnerability details from this public report.