[TanishGudise]

Summary

• Problem this solves: on a cua-sandbox session, file_exists() and directory_exists() caught every exception and returned False, so a sandbox that froze or crashed read as "the file is missing". A task that scores a missing answer file as 0 (the cb task create starter template, example_tasks/hello_file_env) then recorded a reward of 0 for an episode it never scored, and the variant was marked completed. In RL use that reward reaches the trainer as a genuine failure. Reproduced on a real local container: docker pause on the sandbox as evaluation started; the agent had solved the task, the evaluation recorded [0.0], the variant completed, and the Tinker loop's traces.load_runs delivered it as reward 0.
• What changed: SandboxInterface.file_exists/directory_exists (compat/legacy_interface.py) let errors that mean the sandbox did not answer propagate: TimeoutError, ConnectionError, cua_sandbox.SpacesdNotAvailable, and the native SDK's CuaError.Transport, Timeout, Closed, SpacesdNotAvailable and DaemonNotRunning (looked up lazily, skipped if not installed). A path that does not exist still returns False (cua-sandbox's env transport already answers NotFound with False), and other errors still read as missing. The RemoteDesktopSession docstrings say so.

Related work

Refs #4929 (fail-closed benchmark scoring, draft) and #5163 (fail closed on missing/invalid evaluator rewards in run_single_task): same direction, different code path. I found no issue or PR about file_exists swallowing errors.

RFC (required for a public SDK, CLI, MCP, protocol, compatibility, permission, or cross-component contract change): not needed in my view: no new API; the methods keep their signatures and still return booleans whenever the sandbox answers.

Compatibility and risk

• User-visible, API/CLI/MCP, migration, permission, or platform impact: when the sandbox does not answer, file_exists/directory_exists now raise instead of returning False. The variant then fails (and is retryable under --retries) instead of completing with a 0. Two callers change with it: write_bytes(append=True) raises RuntimeError("Failed to write file: ...") instead of overwriting the file when the existence check fails, and the legacy _send_command("file_exists") dispatch returns {"success": False, "error": ...} instead of {"success": True, "exists": False}. Nothing changes when the sandbox answers.
• Risk and rollback: limited to the two methods' error handling; rollback is a revert.

Validation

• Focused tests and checks: cua_bench/tests/test_sandbox_exists_errors.py:
  • an unreachable sandbox (TimeoutError, ConnectionResetError, cua_sandbox.SpacesdNotAvailable) raises from both methods: all 3 fail on main (they read as missing);
  • with the native SDK installed, CuaError.Transport/Timeout raise through file_exists and CuaError.NotFound reads as missing: fails on main; skipped where the native library is not built (it passes under cua 0.4.1);
  • present/missing paths and other errors (PermissionError) answer as before (pass on both).
  • pytest -q cua_bench/tests --ignore=cua_bench/tests/live: 370 passed, 3 skipped, also with -W error::RuntimeWarning -W error::pytest.PytestUnhandledThreadExceptionWarning (main: 365 passed, 2 skipped). Run with cua-bench-rl's locked environment, which installs this checkout's cua-bench; cua-bench's own lock needs cua-fleet from wheels.cua.ai, which I could not fetch offline.
  • black 25.9.0, isort 7.0.0 and ruff 0.14.1 clean on the changed files.
• Manual or platform evidence: real local container (ghcr.io/trycua/linux:24.04, runc on Colima, cua 0.4.1), the container paused at the start of evaluation of a solved task, 3 seeds each:

Code	Solved episodes delivered to the trainer with reward 0	Episodes reaching the trainer (5 min)
main	1 per run (3/3)	44–48
this PR	0 (3/3): the frozen episode fails and is not delivered	39–47

The same with cua-bench's in-memory test sandbox (a freeze that times out after 20 s): main 1 per run (3/3), this PR 0 (3/3). Both "this PR" rows also included a separate per-variant deadline fix (proposed in #5193; variant_timeout=90 / 30), which does not touch evaluation scoring.

• Known gaps or CI still required: these still record 0 when the sandbox does not answer, because they catch errors themselves: example_tasks/test_notepad_env (its own except Exception: return [0.0]), the WinArena adapter (controller_adapter.py, evaluator.py, getters_async/file.py), and legacy computer-server sessions (LegacyInterface.file_exists reads .get("exists", False) from a failed command). A Fleet gateway answering Unavailable/DeadlineExceeded surfaces as CuaError.Env("rpc failed: ... (Unavailable)"), which is not matched (that would need message matching). CI: cua-bench tests.

Contributor and release checks

• The PR is focused and the description matches the final diff.
• This change does not require an RFC, or the accepted RFC is linked above.
• Tests, documentation, and platform evidence are included or the gap is explained.
• The PR title is a Conventional Commit describing the production change.
• External contributor authorship is preserved, or no external contribution is included.
• If release-tracked files changed but this is intentionally non-releasing, the no-release label is applied. (Not applicable: this is a releasing fix to cua-bench.)

AI assistance

AI tools helped with tracing and tests; I reviewed the diff and can explain every change.

[TanishGudise]

Thanks for the quick review and merge! If you have a moment, #4898 is the companion fix on the worker-pool side (the timeout-checker deadlock), and it's still waiting on first-time-contributor CI approval.

[f-trycua]

Thanks, @TanishGudise. #4898 is on our list; we'll reply there.