Skip to content

docs(sandbox): note exec output-limit truncation is provider-dependent#4506

Open
MattFisher wants to merge 3 commits into
UKGovernmentBEIS:mainfrom
MattFisher:docs/exec-output-truncation-note
Open

docs(sandbox): note exec output-limit truncation is provider-dependent#4506
MattFisher wants to merge 3 commits into
UKGovernmentBEIS:mainfrom
MattFisher:docs/exec-output-truncation-note

Conversation

@MattFisher

Copy link
Copy Markdown
Contributor

Summary

SandboxEnvironment.exec's docstring states that when an output stream exceeds INSPECT_SANDBOX_MAX_EXEC_OUTPUT_SIZE (default 10 MiB), "an OutputLimitExceededError will be raised." That isn't what happens for the built-in providers: the Docker (and local) exec path captures stdout/stderr through a CircularByteBuffer that keeps only the trailing bytes within the limit and returns the truncated output. SandboxEnvironmentProxy.verify_exec_result_size does raise, but only when a provider's own exec returns un-truncated output over the limit — for Docker it receives content already trimmed to within the limit, so it never fires.

Net effect for callers: a command that emits more than the limit doesn't error — the scorer/solver silently receives the last N bytes with the beginning discarded. Code that parses exec output (e.g. json.loads(result.stdout)) then fails on an invalid tail, and the failure looks unrelated to output size.

This is a docs-only clarification (no behaviour change). It documents the provider-dependent behaviour on exec and points callers at read_file() for large payloads, which checks size up front and always raises OutputLimitExceededError.

Motivation

We hit this in inspect_evals: a scorer that POSTed a proof-of-concept and parsed the executor's JSON response via curl stdout crashed with JSONDecodeError on large program output, because the 10 MiB exec limit truncated the JSON tail rather than raising (inspect_evals#1644). The exec-vs-read_file distinction (truncate vs raise) is subtle and currently the docstring points the wrong way, so it's easy to write a guard against the wrong failure mode.

Changes

  • SandboxEnvironment.exec docstring: replace "an OutputLimitExceededError will be raised" with a note that the over-limit behaviour is provider-dependent — some providers (including Docker) pre-truncate to the trailing bytes and return them, others raise — and steer large payloads to read_file().
  • Raises: entry qualified to match.
  • One Docs: bullet in ## Unreleased (drop if you'd rather keep the changelog behaviour-only).

🤖 Generated with Claude Code

SandboxEnvironment.exec claimed an over-limit output stream raises
OutputLimitExceededError, but some providers (including the built-in
Docker provider) pre-truncate each stream to the trailing bytes and
return the truncated output instead of raising. Callers parsing large
exec output (e.g. JSON) can therefore receive a silently truncated tail
that fails to parse. Document the provider-dependent behaviour and point
at read_file() for large payloads.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@MattFisher

Copy link
Copy Markdown
Contributor Author

Removed the changelog entry since there were no other "Docs:" entries I could see.

Comment thread src/inspect_ai/util/_sandbox/environment.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants