Skip to content

docs(sandbox): note exec output-limit truncation is provider-dependent #4506

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
MattFisher wants to merge 3 commits into
base: main
Choose a base branch
from
Open

docs(sandbox): note exec output-limit truncation is provider-dependent #4506

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3674b05
docs(sandbox): note exec output-limit truncation is provider-dependent
MattFisher Jul 15, 2026
86d1331
Merge branch 'main' into docs/exec-output-truncation-note
MattFisher Jul 16, 2026
13e726d
Clarify raise condition
MattFisher Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions src/inspect_ai/util/_sandbox/environment.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,9 @@ async def exec(
The current working directory for execution will be the per-sample
filesystem context.

By default, each output stream (stdout and stderr) is limited to 10 MiB. You can override this by setting the `INSPECT_SANDBOX_MAX_EXEC_OUTPUT_SIZE` environment variable (specified in bytes). If exceeded, an `OutputLimitExceededError` will be raised.
By default, each output stream (stdout and stderr) is limited to 10 MiB. You can override this by setting the `INSPECT_SANDBOX_MAX_EXEC_OUTPUT_SIZE` environment variable (specified in bytes).

The behaviour when a stream exceeds the limit is **provider-dependent**. Some providers (including the built-in Docker provider) pre-truncate each stream to the trailing bytes within the limit and return the truncated output rather than raising — so the caller receives the *last* N bytes with the beginning discarded. Other providers raise `OutputLimitExceededError`. Consequently, do not rely on an over-limit stream surfacing as an error: if you parse `exec` output that may be large (e.g. JSON), a silently truncated tail may reach your code and fail to parse. For large payloads, prefer writing to a file and reading it with `read_file()`, which checks the size up front and always raises `OutputLimitExceededError` when the limit is exceeded.

@craig-meridian craig-meridian Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense and will hopefully save users being met with the same confusion you likely met. Couple of thoughts:

  1. I don't love the coupling of documenting DockerSandboxEnvironment-specific functionality here. I'm assuming when the front-buffer functionality was added, we forgot to update the docs here, so I worry this could happen again.
  2. SandboxEnvironment.exec() .qmd docs instruct implementers to front-truncate output. So, it might be reasonable that more providers will start to follow this direction. Not sure why this isn't documented in the Python docstring, but that's unrelated to this PR.
  3. The Docker provider can still raise a OutputLimitExceededError if the trim bisects a UTF-8 character, so we should soften the claim.
  4. “read_file() checks the size up front and always raises” is also slightly too strong: Docker copies the file before checking its size (arguably not "up front"), and custom providers are responsible for enforcing the contract themselves.

I therefore suggest we soften/generalise the claims slightly

Behaviour above this limit depends on the sandbox provider. A provider may
raise `OutputLimitExceededError`, or return only the trailing portion of
the output with the beginning discarded. Callers should therefore not
assume that returned output is complete or rely on an exception to detect
overflow. This is particularly important when parsing structured output
such as JSON. For large output, write to a file and use `read_file()`,
which always raises `OutputLimitExceededError` when the limit is exceeded.
...
OutputLimitExceededError: May be raised if an output stream exceeds
    the limit.


Args:
cmd: Command or command and arguments to execute.
Expand All @@ -145,8 +147,8 @@ async def exec(
characters which cannot be decoded.
PermissionError: If the user does not have
permission to execute the command.
OutputLimitExceededError: If an output stream
exceeds the limit.
OutputLimitExceededError: If an output stream exceeds the limit
and the sandbox provider does not pre-truncate it.
"""
...

Expand Down
Loading