Skip to content

fix(rails): distinguish check execution failures from policy blocks - #2485

Draft
Pouyanpi wants to merge 20 commits into
developfrom
pouyanpi/2458-check-failure-followup
Draft

Pouyanpi wants to merge 20 commits into
developfrom
pouyanpi/2458-check-failure-followup

Conversation

@Pouyanpi

@Pouyanpi Pouyanpi commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up to #2458: rail execution failures

Origin: review discussion on #2458.

Why

The server already distinguishes a policy block from an inspection failure through its ContentChecker contract. Its adapter could not reliably use that distinction because check_async() returned some internally handled execution failures as ordinary BLOCKED results.

Keeping the server contract independent of the evolving core API follows dependency inversion. This follow-up supplies the missing structured failure information and maps it in the adapter, without relying on error-message text.

Commit order

  1. IORails: expose the existing failure flag through RailsResult, preserving BLOCKED status for compatibility.
  2. LLMRails: report failures from current-check action events and typed rail outcomes.
  3. Server: map failures to ContentCheckFailed and follow the library's NEMO_GUARDRAILS_IORAILS_ENGINE switch, including LLMRails fallback. Reject unsupported Colang 2.x checks at startup.

…er boundaries

Resolve configuration after projection validation and before checking or dispatch. Preserve original request bytes, select response encoding from the resolved inspection policy, and compose fresh streaming adapters through the reviewed bindings.

Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Parse configuration files off the event loop while keeping engine construction and startup on their application loop. Preserve lazy selection and shared startup ownership when an individual caller is cancelled.

Reject new work during shutdown, cancel unfinished startup, and independently bound and shield each checker cleanup. Retain incomplete checker cleanup for retry and always attempt separately bounded HTTP cleanup without masking primary failures. Cover stalled startup, close failures, cancellation, concurrent close, and event-loop responsiveness.

Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Initialize every entry in the fixed trusted catalog during lifespan startup. Requests now perform synchronous lookup of already-started checkers, removing request-owned startup tasks and cached lazy failures.

Fail readiness if any configuration cannot initialize, roll back previously started checkers, and close the shared HTTP client. Retain off-loop file parsing and bounded cleanup, and clear runtime state on shutdown. Document the startup/resource tradeoff and test full-catalog readiness, partial failure, cancellation, and repeated lookup.

Validation: 1,773 server tests passed with 10 skipped; 10 replay-only recorded/inventory tests passed; pre-commit passed.
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Count non-empty per-tool flow lists when rejecting tool rails that the
reference proxy cannot execute. Empty per-tool entries remain unconfigured,
matching the real core's configuration semantics.

Add real-RailsConfig regressions for buffered and streaming rejection before
upstream dispatch, including mixed empty and non-empty tool entries.

Validation: all 11 configuration-resolution tests passed; pre-commit passed.
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Exercise repository cleanup with the real Guardrails facade, IORails engine,
worker queue, and HTTP client. Interrupt shutdown during unfinished queue
work and require a subsequent close to stop every worker before dropping
the checker.

This regression intentionally fails until the separate core lifecycle fix
in #2459 is incorporated. It passes with those core modules loaded.

Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Own each checker before awaiting startup and retain the application runtime when Rails cleanup remains incomplete. Retry pending cleanup before creating a replacement runtime while keeping request lookups disabled after failure.

Cover failed and cancelled startup with real Guardrails and HTTPX owners, including cleanup errors, timeouts, unavailable readiness, and successful retry.

Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Assert that the task is cancelled instead of depending on cancellation-message propagation added in Python 3.11. Preserve the CancelledError expectation and all startup ownership and cleanup retry checks.

Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Starting a Guardrails engine did not prove that its configuration was supported by the proxy adapter. A catalog containing retrieval or tool rails could become ready and then reject every request selecting that policy.

Validate the adapter inspection policy before engine startup, retaining the acquired engine for rollback. Unsupported entries now prevent readiness even when the resolver would never select them. Request-dependent operation and streaming checks remain on the request path.

Add regression coverage using parsed RailsConfig values for retrieval, canonical and legacy tool names, and per-tool declarations. Verify rollback of the earlier engine, the rejected engine, and outbound HTTP ownership. Update lifecycle test doubles to carry real configurations.

Validation: confirmed the new readiness regression failed before the fix; all 78 repository, startup-cleanup, and configuration-resolution tests pass with network access blocked. Pre-commit covers all changed files.
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Preloading removes engine startup from requests, but a trusted resolver may still await an external authorization or tenant service indefinitely. Add config_resolution_timeout to private application assembly and the runtime, with a five-second default and finite positive validation before resource allocation.

Use a cooperative deadline around resolver execution, including synchronous factories returning awaitables. Timeouts use the existing provider-native config_resolution_failed response before upstream dispatch; caller cancellation still propagates. No weaker fallback policy is selected.

Document that synchronous resolvers run on the application loop and must remain short and non-blocking. External lookups require asynchronous cancellation-aware clients and their own I/O timeouts. The deadline does not forcibly stop blocking or cancellation-suppressing code.

Validation: 112 configuration-resolution, repository, real-core startup-cleanup, and application-transport tests pass with network access blocked. New cases cover buffered and streaming timeouts, awaitable factories, cancellation cleanup, and invalid deadline values. Pre-commit runs on all changed files.
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
The application lifespan previously reached through ServerRuntime into the checker repository to initialize policies and decide whether cleanup remained pending. That coupled ASGI lifecycle code to the repository implementation.

Expose startup and pending engine-cleanup status on ServerRuntime. The application now coordinates runtime start, publication, close, and retry without inspecting the repository. Keep existing ownership, HTTP cleanup budgets, and restart behavior; this does not introduce a lifecycle framework or change config-ID sharing.

Add runtime-boundary coverage for failed and timed-out engine shutdown, independent HTTP closure, successful retry, and idempotent subsequent close. Existing real-core tests continue to verify that failed startup cannot publish a runtime and that restart waits for retained engine cleanup.

Validation: all 114 focused configuration, repository, startup-cleanup, and application-transport tests pass with network access blocked. Pre-commit passed on every changed file.
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Show an async resolver awaiting an application-owned policy registry using authenticated tenant context. Wire the function into application assembly with a two-second resolution deadline and explain that returned IDs must belong to the declared catalog.

Make registry lifecycle ownership and client I/O timeouts explicit. Explain how to attach the authentication middleware, pass the resolver function without invoking it, and preserve fail-closed behavior on exceptions or deadlines.

Validation: pre-commit passed and all five Python examples in the guide parse successfully. Documentation-only change; runtime tests were not rerun.
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Populate the request body cache after bounded reading and framing validation so custom Rails resolvers can read JSON or bytes without consuming the request stream again. Preserve the original bytes for upstream dispatch.

Cover body-based selection for buffered and streaming responses, repeated body access, per-request policy isolation, and rejection of oversized fixed-length and chunked requests before resolution.

Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Propagate the existing RailResult failure flag into RailsResult while preserving BLOCKED status for compatibility. Cover sync and async IORails propagation and result serialization, and characterize the legacy default pending LLMRails support.

Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Read action completion events and typed rail outcomes from the current check to distinguish failures from policy blocks. Preserve BLOCKED status and return a generic failure message without interpreting refusal text or action metadata.

Cover sync and async input/output checks, parallel execution, missing actions, failure recovery, and unchanged exception propagation.

Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
Map reported rail execution failures to ContentCheckFailed before handling policy blocks, including after streaming response startup. Follow the library environment-selected engine through the Guardrails facade and retain startup and cleanup ownership.

Validate both engines, fallback, HTTP and SSE decisions, and lifecycle behavior. Reject Colang 2.x configurations whose check API requires unsupported logging options.

Signed-off-by: Pouyanpi <13303554+Pouyanpi@users.noreply.github.com>
@github-actions github-actions Bot added size: M status: needs triage New issues that have not yet been reviewed or categorized. labels Oct 22, 2026
Base automatically changed from pouyanpi/phase-d-final-20261013-d5-rails to develop October 22, 2026 05:18
@codecov

codecov Bot commented Oct 22, 2026

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size: M status: needs triage New issues that have not yet been reviewed or categorized.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant