From 3674b05c1c6934a097f69ea228efcb7765c8bf11 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Wed, 15 Jul 2026 16:02:59 +1000 Subject: [PATCH 1/3] docs(sandbox): note exec output-limit truncation is provider-dependent SandboxEnvironment.exec claimed an over-limit output stream raises OutputLimitExceededError, but some providers (including the built-in Docker provider) pre-truncate each stream to the trailing bytes and return the truncated output instead of raising. Callers parsing large exec output (e.g. JSON) can therefore receive a silently truncated tail that fails to parse. Document the provider-dependent behaviour and point at read_file() for large payloads. Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 1 + src/inspect_ai/util/_sandbox/environment.py | 10 +++++++--- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index eacd708ad9..b1938cf68d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,6 @@ ## Unreleased +- Docs: `SandboxEnvironment.exec` docstring now notes that some providers (including Docker) pre-truncate over-limit output streams to the trailing bytes and return them, rather than raising `OutputLimitExceededError` — so callers parsing large `exec` output should not assume the limit surfaces as an error. - Eval: `EvalSample` and `EvalSampleSummary` now record `turn_count` and the sample's token limit (`token_limit`, `token_limit_type`, and metered `token_limit_usage`). - Analysis: `samples_df` gains default `turn_count` and `token_limit_usage` columns, and `evals_df` configuration columns gain `token_limit_type`. - Control Channel: `inspect ctl sample list` now shows per-sample turn count and, when a token limit is configured, its computed usage and configured ceiling. diff --git a/src/inspect_ai/util/_sandbox/environment.py b/src/inspect_ai/util/_sandbox/environment.py index ba32bb4502..b36a2c5de9 100644 --- a/src/inspect_ai/util/_sandbox/environment.py +++ b/src/inspect_ai/util/_sandbox/environment.py @@ -118,7 +118,9 @@ async def exec( The current working directory for execution will be the per-sample filesystem context. - By default, each output stream (stdout and stderr) is limited to 10 MiB. You can override this by setting the `INSPECT_SANDBOX_MAX_EXEC_OUTPUT_SIZE` environment variable (specified in bytes). If exceeded, an `OutputLimitExceededError` will be raised. + By default, each output stream (stdout and stderr) is limited to 10 MiB. You can override this by setting the `INSPECT_SANDBOX_MAX_EXEC_OUTPUT_SIZE` environment variable (specified in bytes). + + The behaviour when a stream exceeds the limit is **provider-dependent**. Some providers (including the built-in Docker provider) pre-truncate each stream to the trailing bytes within the limit and return the truncated output rather than raising — so the caller receives the *last* N bytes with the beginning discarded. Other providers raise `OutputLimitExceededError`. Consequently, do not rely on an over-limit stream surfacing as an error: if you parse `exec` output that may be large (e.g. JSON), a silently truncated tail may reach your code and fail to parse. For large payloads, prefer writing to a file and reading it with `read_file()`, which checks the size up front and always raises `OutputLimitExceededError` when the limit is exceeded. Args: cmd: Command or command and arguments to execute. @@ -145,8 +147,10 @@ async def exec( characters which cannot be decoded. PermissionError: If the user does not have permission to execute the command. - OutputLimitExceededError: If an output stream - exceeds the limit. + OutputLimitExceededError: If an output stream exceeds the limit + *and* the sandbox provider raises rather than pre-truncating + (see the note above — some providers, e.g. Docker, return the + truncated trailing output instead of raising). """ ... From 13e726df9e776e7354ead5a23525fe09c7d2f359 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Thu, 16 Jul 2026 10:12:41 +1000 Subject: [PATCH 2/3] Clarify raise condition --- src/inspect_ai/util/_sandbox/environment.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/inspect_ai/util/_sandbox/environment.py b/src/inspect_ai/util/_sandbox/environment.py index b36a2c5de9..4a0f6a76bf 100644 --- a/src/inspect_ai/util/_sandbox/environment.py +++ b/src/inspect_ai/util/_sandbox/environment.py @@ -148,9 +148,7 @@ async def exec( PermissionError: If the user does not have permission to execute the command. OutputLimitExceededError: If an output stream exceeds the limit - *and* the sandbox provider raises rather than pre-truncating - (see the note above — some providers, e.g. Docker, return the - truncated trailing output instead of raising). + and the sandbox provider does not pre-truncate it. """ ... From 0e84107f7252d70b5be55be42fca8ea0fedd393e Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Wed, 22 Jul 2026 10:37:36 +1000 Subject: [PATCH 3/3] Apply suggestions from code review Co-authored-by: Matt Fisher --- src/inspect_ai/util/_sandbox/environment.py | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/src/inspect_ai/util/_sandbox/environment.py b/src/inspect_ai/util/_sandbox/environment.py index 4a0f6a76bf..25746d58b1 100644 --- a/src/inspect_ai/util/_sandbox/environment.py +++ b/src/inspect_ai/util/_sandbox/environment.py @@ -120,7 +120,13 @@ async def exec( By default, each output stream (stdout and stderr) is limited to 10 MiB. You can override this by setting the `INSPECT_SANDBOX_MAX_EXEC_OUTPUT_SIZE` environment variable (specified in bytes). - The behaviour when a stream exceeds the limit is **provider-dependent**. Some providers (including the built-in Docker provider) pre-truncate each stream to the trailing bytes within the limit and return the truncated output rather than raising — so the caller receives the *last* N bytes with the beginning discarded. Other providers raise `OutputLimitExceededError`. Consequently, do not rely on an over-limit stream surfacing as an error: if you parse `exec` output that may be large (e.g. JSON), a silently truncated tail may reach your code and fail to parse. For large payloads, prefer writing to a file and reading it with `read_file()`, which checks the size up front and always raises `OutputLimitExceededError` when the limit is exceeded. + Behaviour above this limit depends on the sandbox provider. A provider may + raise `OutputLimitExceededError`, or return only the trailing portion of + the output with the beginning discarded. Callers should therefore not + assume that returned output is complete or rely on an exception to detect + overflow. This is particularly important when parsing structured output + such as JSON. For large output, write to a file and use `read_file()`, + which always raises `OutputLimitExceededError` when the limit is exceeded. Args: cmd: Command or command and arguments to execute. @@ -147,8 +153,8 @@ async def exec( characters which cannot be decoded. PermissionError: If the user does not have permission to execute the command. - OutputLimitExceededError: If an output stream exceeds the limit - and the sandbox provider does not pre-truncate it. + OutputLimitExceededError: May be raised if an output stream + exceeds the limit. """ ...