fix(run): report input guardrail results when a tripwire aborts the run - #4071
Merged
seratch merged 1 commit intoJul 31, 2026
Merged
Conversation
Runner.run() and Runner.run_sync() raised InputGuardrailTripwireTriggered with an empty RunErrorDetails.input_guardrail_results, while Runner.run_streamed() reported every completed result. run_input_guardrails() accumulated results locally and raised before run.py could merge them into the run-level list. Record results into a caller-owned accumulator as each guardrail completes, including the tripping result, so all three entry points expose the same observable guardrail state.
hsusul
force-pushed
the
fix/4068-input-guardrail-results-non-streamed
branch
from
July 31, 2026 19:32
881ff9e to
a15d063
Compare
4 tasks
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When an input guardrail tripwire aborts a run,
InputGuardrailTripwireTriggered.run_data.input_guardrail_resultswas empty forRunner.run()andRunner.run_sync(), whileRunner.run_streamed()reported every completed result. This makes the same error handler observe different run state depending on which entry point was used.Affected component:
src/agents/run_internal/guardrails.py::run_input_guardrailsand its three call sites insrc/agents/run.py(non-streaming input-guardrail path). The streamed path usesrun_input_guardrails_with_queueand is unchanged.Problem.
run_input_guardrails()accumulated results in a local list and raisedInputGuardrailTripwireTriggered(result)as soon as a tripwire fired, discarding the accumulation.run.pymerged those results into the run-levelinput_guardrail_resultsonly on the success path (input_guardrail_results.extend(sequential_results)), so theRunErrorDetailsbuilt by the outer failure handler saw an empty list.run_input_guardrails_with_queue()assignsstreamed_result.input_guardrail_resultsbefore propagating, which is why streaming was already correct.Minimal reproduction (no API key, no network, no paid model call):
Current behavior.
Runner.run/Runner.run_sync→[].Runner.run_streamed→["passes", "trips"].Corrected behavior. All three entry points report
["passes", "trips"]— the guardrails that completed before the run was aborted, including the tripping one.exc.guardrail_resultis unchanged.Root cause. Results were owned by the callee and only published on a normal return, so the failure path had nothing to attach.
Implementation.
run_input_guardrails()takes an optionalresults_sinklist and records each result into it as the guardrail completes — including the tripping result, immediately before raising.run.pypasses its existinginput_guardrail_resultslist at all three call sites (sandbox pre-run sequential, normal sequential, and the parallel guardrail task) and the two now-redundantextend()calls are removed.Why this is minimal. The change publishes results the runner already collected, at the point they become known. It reuses the run-level accumulator that was already the destination on the success path rather than adding new state, and it removes two lines rather than adding a parallel bookkeeping path. There is no public API change:
run_input_guardrailslives inrun_internal, the new parameter is optional and appended last, andInputGuardrailTripwireTriggered,RunErrorDetails, andRunResult.input_guardrail_resultskeep their existing shapes. Success-path content and ordering (sequential results before parallel results) are unchanged.Regression tests (all in
tests/test_guardrails.py):test_input_guardrail_tripwire_reports_results[False|True]—Runner.run, blocking and parallel guardrails.test_input_guardrail_tripwire_reports_results_streamed[False|True]— streamed parity, asserting bothrun_dataandRunResultStreaming.input_guardrail_results.test_input_guardrail_tripwire_reports_results_sync—Runner.run_sync.test_input_guardrail_results_reported_on_success— passing guardrails still land on the successful result exactly once, in sequential-then-parallel order (guards against double-counting from the removedextend()calls).test_input_guardrail_exception_reports_completed_results— a guardrail raising a non-tripwire error still preserves earlier results.Completion order is fixed with an
asyncio.Eventbarrier rather than sleeps, so the assertions are deterministic. The four tests coveringRunner.run,Runner.run_sync, and the direct helper fail onupstream/main(assert [] == ['passes', 'trips']); the streamed and success tests pass before and after and act as the no-regression baseline.Execution modes covered:
Runner.run,Runner.run_sync,Runner.run_streamed; blocking (run_in_parallel=False) and parallel input guardrails; tripwire and non-tripwire guardrail failures; the success path.Cleanup and lifecycle. Sibling-task cancellation and draining in
run_input_guardrailsare untouched: the tripwire branch still cancels andgathers siblings, and theexcept BaseExceptioncleanup block is unchanged. Recording the tripping result happens before cancellation and does not alter which tasks are awaited.make tests-asyncio-stabilitywas run to confirm no teardown regressions.Test plan
Run from the repository root on
fix/4068-input-guardrail-results-non-streamed(Python 3.12.13, macOS 15.7.3):make format842 files left unchanged;ruff check --fix→All checks passed!make lintAll checks passed!make typecheckSuccess: no issues found in 833 source files; pyright0 errors, 0 warnings, 0 informationsmake tests5965 passed, 3 skipped, 2 warnings(parallel) and45 passed, 4 skipped, 5968 deselected(serial)make tests-asyncio-stabilitygit diff --checkuv run pytest tests/test_guardrails.py -q52 passed(run 5× consecutively, no flakes)uv run pytest tests/test_guardrails.py tests/test_agent_runner.py tests/test_agent_runner_streamed.py tests/test_run_state.py tests/test_tracing_errors.py tests/test_tracing_errors_streamed.py -q522 passed(run 5× consecutively)UV_PROJECT_ENVIRONMENT=.venv_310 uv run --python 3.10 -m pytest tests/test_guardrails.py -q52 passedPre-fix baseline on the same branch with only
src/agents/run.pyandsrc/agents/run_internal/guardrails.pyreverted:4 failed, 3 passed— theRunner.run,Runner.run_sync, and direct-helper tests fail, the streamed and success tests pass.No OpenAI API key, network access, or paid model call was used; the tests rely on
tests/fake_model.py::FakeModel.Not run:
make integration-tests*(requires live provider credentials and external services) andmake build-docs(no documentation files changed). No inline snapshots were added or modified. No lockfile or dependency changes.Compatibility. No public API, exception type, or serialized-state change. Behavior changes only on the failure path, where the reported list goes from empty to populated; the success path is byte-identical in content and order. Callers that assumed
run_data.input_guardrail_resultswas always empty on a tripwire would see the populated list, which is the documented intent.Non-goals. Output-guardrail tripwire results (empty on both paths today — consistent, so out of scope here), guardrail cancellation semantics, and the
run_input_guardrails_with_queuestreamed implementation.Issue number
Closes #4068
Checks
.agents/skills/code-change-verification/scripts/run.sh/reviewbefore submitting this PR