Skip to content

fix(run): report input guardrail results when a tripwire aborts the run - #4071

Merged
seratch merged 1 commit into
openai:mainfrom
hsusul:fix/4068-input-guardrail-results-non-streamed
Jul 31, 2026
Merged

fix(run): report input guardrail results when a tripwire aborts the run#4071
seratch merged 1 commit into
openai:mainfrom
hsusul:fix/4068-input-guardrail-results-non-streamed

Conversation

@hsusul

@hsusul hsusul commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

When an input guardrail tripwire aborts a run, InputGuardrailTripwireTriggered.run_data.input_guardrail_results was empty for Runner.run() and Runner.run_sync(), while Runner.run_streamed() reported every completed result. This makes the same error handler observe different run state depending on which entry point was used.

Affected component: src/agents/run_internal/guardrails.py::run_input_guardrails and its three call sites in src/agents/run.py (non-streaming input-guardrail path). The streamed path uses run_input_guardrails_with_queue and is unchanged.

Problem. run_input_guardrails() accumulated results in a local list and raised InputGuardrailTripwireTriggered(result) as soon as a tripwire fired, discarding the accumulation. run.py merged those results into the run-level input_guardrail_results only on the success path (input_guardrail_results.extend(sequential_results)), so the RunErrorDetails built by the outer failure handler saw an empty list. run_input_guardrails_with_queue() assigns streamed_result.input_guardrail_results before propagating, which is why streaming was already correct.

Minimal reproduction (no API key, no network, no paid model call):

def _guardrail(name: str, trip: bool, delay: float = 0.0) -> InputGuardrail[Any]:
    async def fn(ctx, agent, inp) -> GuardrailFunctionOutput:
        if delay:
            await asyncio.sleep(delay)
        return GuardrailFunctionOutput(output_info=name, tripwire_triggered=trip)

    return InputGuardrail(guardrail_function=fn, name=name)


agent = Agent(
    name="A",
    model=FakeModel(),
    input_guardrails=[
        _guardrail("passes", trip=False),
        _guardrail("trips", trip=True, delay=0.02),
    ],
)

with pytest.raises(InputGuardrailTripwireTriggered) as exc_info:
    await Runner.run(agent, input="hi")

# streamed reports ["passes", "trips"]; non-streamed reports []
[r.guardrail.get_name() for r in exc_info.value.run_data.input_guardrail_results]

Current behavior. Runner.run / Runner.run_sync[]. Runner.run_streamed["passes", "trips"].

Corrected behavior. All three entry points report ["passes", "trips"] — the guardrails that completed before the run was aborted, including the tripping one. exc.guardrail_result is unchanged.

Root cause. Results were owned by the callee and only published on a normal return, so the failure path had nothing to attach.

Implementation. run_input_guardrails() takes an optional results_sink list and records each result into it as the guardrail completes — including the tripping result, immediately before raising. run.py passes its existing input_guardrail_results list at all three call sites (sandbox pre-run sequential, normal sequential, and the parallel guardrail task) and the two now-redundant extend() calls are removed.

Why this is minimal. The change publishes results the runner already collected, at the point they become known. It reuses the run-level accumulator that was already the destination on the success path rather than adding new state, and it removes two lines rather than adding a parallel bookkeeping path. There is no public API change: run_input_guardrails lives in run_internal, the new parameter is optional and appended last, and InputGuardrailTripwireTriggered, RunErrorDetails, and RunResult.input_guardrail_results keep their existing shapes. Success-path content and ordering (sequential results before parallel results) are unchanged.

Regression tests (all in tests/test_guardrails.py):

  • test_input_guardrail_tripwire_reports_results[False|True]Runner.run, blocking and parallel guardrails.
  • test_input_guardrail_tripwire_reports_results_streamed[False|True] — streamed parity, asserting both run_data and RunResultStreaming.input_guardrail_results.
  • test_input_guardrail_tripwire_reports_results_syncRunner.run_sync.
  • test_input_guardrail_results_reported_on_success — passing guardrails still land on the successful result exactly once, in sequential-then-parallel order (guards against double-counting from the removed extend() calls).
  • test_input_guardrail_exception_reports_completed_results — a guardrail raising a non-tripwire error still preserves earlier results.

Completion order is fixed with an asyncio.Event barrier rather than sleeps, so the assertions are deterministic. The four tests covering Runner.run, Runner.run_sync, and the direct helper fail on upstream/main (assert [] == ['passes', 'trips']); the streamed and success tests pass before and after and act as the no-regression baseline.

Execution modes covered: Runner.run, Runner.run_sync, Runner.run_streamed; blocking (run_in_parallel=False) and parallel input guardrails; tripwire and non-tripwire guardrail failures; the success path.

Cleanup and lifecycle. Sibling-task cancellation and draining in run_input_guardrails are untouched: the tripwire branch still cancels and gathers siblings, and the except BaseException cleanup block is unchanged. Recording the tripping result happens before cancellation and does not alter which tasks are awaited. make tests-asyncio-stability was run to confirm no teardown regressions.

Test plan

Run from the repository root on fix/4068-input-guardrail-results-non-streamed (Python 3.12.13, macOS 15.7.3):

Command Result
make format 842 files left unchanged; ruff check --fixAll checks passed!
make lint All checks passed!
make typecheck mypy Success: no issues found in 833 source files; pyright 0 errors, 0 warnings, 0 informations
make tests 5965 passed, 3 skipped, 2 warnings (parallel) and 45 passed, 4 skipped, 5968 deselected (serial)
make tests-asyncio-stability 5/5 runs passed
git diff --check clean
uv run pytest tests/test_guardrails.py -q 52 passed (run 5× consecutively, no flakes)
uv run pytest tests/test_guardrails.py tests/test_agent_runner.py tests/test_agent_runner_streamed.py tests/test_run_state.py tests/test_tracing_errors.py tests/test_tracing_errors_streamed.py -q 522 passed (run 5× consecutively)
UV_PROJECT_ENVIRONMENT=.venv_310 uv run --python 3.10 -m pytest tests/test_guardrails.py -q 52 passed

Pre-fix baseline on the same branch with only src/agents/run.py and src/agents/run_internal/guardrails.py reverted: 4 failed, 3 passed — the Runner.run, Runner.run_sync, and direct-helper tests fail, the streamed and success tests pass.

No OpenAI API key, network access, or paid model call was used; the tests rely on tests/fake_model.py::FakeModel.

Not run: make integration-tests* (requires live provider credentials and external services) and make build-docs (no documentation files changed). No inline snapshots were added or modified. No lockfile or dependency changes.

Compatibility. No public API, exception type, or serialized-state change. Behavior changes only on the failure path, where the reported list goes from empty to populated; the success path is byte-identical in content and order. Callers that assumed run_data.input_guardrail_results was always empty on a tripwire would see the populated list, which is the documented intent.

Non-goals. Output-guardrail tripwire results (empty on both paths today — consistent, so out of scope here), guardrail cancellation semantics, and the run_input_guardrails_with_queue streamed implementation.

Issue number

Closes #4068

Checks

  • I've added new tests, if relevant
  • I've run .agents/skills/code-change-verification/scripts/run.sh
  • I've confirmed all verification steps pass
  • If using Codex, I've run /review before submitting this PR

Note on the unchecked boxes: I ran the documented verification stack directly (make format, make lint, make typecheck, make tests, plus make tests-asyncio-stability and a Python 3.10 run) rather than invoking the skill script wrapper, and I did not use Codex, so /review does not apply.

Runner.run() and Runner.run_sync() raised InputGuardrailTripwireTriggered
with an empty RunErrorDetails.input_guardrail_results, while
Runner.run_streamed() reported every completed result. run_input_guardrails()
accumulated results locally and raised before run.py could merge them into
the run-level list.

Record results into a caller-owned accumulator as each guardrail completes,
including the tripping result, so all three entry points expose the same
observable guardrail state.
@hsusul
hsusul force-pushed the fix/4068-input-guardrail-results-non-streamed branch from 881ff9e to a15d063 Compare July 31, 2026 19:32
@seratch seratch added this to the 0.19.x milestone Jul 31, 2026
@seratch
seratch merged commit 7872542 into openai:main Jul 31, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Input guardrail tripwire loses input_guardrail_results in non-streaming runs

2 participants