feat(server): integrate OpenAI Chat streaming

[Pouyanpi]

Summary

Integrates the preceding OpenAI Chat stream modules and HTTP bridge with the existing guarded endpoint. Buffered and streaming requests share one input projection and check.

What Changed

• Selects buffered or streaming response handling after one shared request preparation and input check. stream: true selects streaming; false, null, or omission selects buffered, and the request bytes are forwarded unchanged.
• Binds the OpenAI stream adapter through the endpoint and preserves request target state across preparation and dispatch.
• Exports the actual bound buffered and streaming declarations without invoking hooks or constructing runtime components.
• Rejects exports that would misdescribe runtime behavior: custom stream validators, a hand-written response-mode property, a non-strict or factory-defaulted stream selector, and schema keywords the format cannot carry. The format gains minimum and maximum.
• Extends the OpenAI error mapping for routing failures, pre-header errors, in-stream failures, and OpenAPI documentation.
• Covers native route/framing rejection, unsafe policies, hidden blocked content, original-text inspection through the route, source closure, and buffered regressions.
• Documents both JSON and SSE success responses, and states that opaque metadata is relayed without inspection.

Review Notes

Review target-state continuity, one input check, request-local hooks, error disclosure, and buffered regressions. Known limits: JSON Schema cannot express Python strictness, and endpoints whose response mode is fixed by the route cannot be exported yet. The stream projection, classifier, and hooks are supplied by part 2; the HTTP bridge is supplied by part 3. Concrete outbound HTTP, application configuration, live recordings, and replacement execution remain separate work.

AI Assistance

• No AI tools were used.
• AI tools were used (Codex and Claude Code assisted with implementation, regression tests, review probes, and PR drafting). Human review of the new diff is pending.

Checklist

• I've read the CONTRIBUTING guidelines.
• This is maintainer-directed work.
• The PR title follows the project commit convention.
• Applicable declarations and tests are included.
• Generated changelog files were not edited manually.
• Automated and human review comments are addressed or answered.
• The responsible reviewer or team is mentioned.

Stack Position

Part 4 of 4.

• Previous: feat(server): bind guarded streams to HTTP #2436
• Next: none, final stack PR

Stack Context

Adds OpenAI Chat streaming above the merged buffered implementation on develop in four parts: provider-neutral execution, typed OpenAI stream declarations and handwritten hooks, injected HTTP lifecycle handling, and integration with the buffered request pipeline.

The open chain is develop → #2434 → #2435 → #2436 → #2437. The buffered foundation is merged; #2412 is closed.

Python declarations own field and event policy. Hooks are handwritten and created for each stream by the integration. The final PR exports the bound buffered and streaming policy for inspection; export never constructs runtime components. Concrete outbound HTTP, deployment configuration, and replacement execution land separately.

Review each PR against its listed base branch.

Order	PR	Branch	Base
1	#2434	pouyanpi/transparent-proxy-streaming-kernel-1	develop
2	#2435	pouyanpi/openai-chat-streaming-contract-2	pouyanpi/transparent-proxy-streaming-kernel-1
3	#2436	pouyanpi/transparent-proxy-streaming-http-3	pouyanpi/openai-chat-streaming-contract-2
4	#2437	pouyanpi/openai-chat-streaming-integration-4	pouyanpi/transparent-proxy-streaming-http-3

Summary by CodeRabbit

• New Features
  • Added streaming support for OpenAI Chat Completions, including request and response inspection, with streaming routes documented in the API.
  • Streaming requests can use an optional streaming dispatcher; requests that require unsupported output inspection are rejected before dispatch.
• Bug Fixes
  • Unsupported stream inspection and upstream stream failures now return clear, provider-compatible error responses instead of uncaught errors.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Confidence Score: 5/5

[High impact] The PR appears safe to merge; no new actionable defect was found.

Summary

Adds OpenAI Chat streaming after one shared request preparation and input check.

• OpenAI Chat requests choose a response mode after one input check.
• The exported guard contract describes the endpoint’s bound stream rules.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Request] --> B[Prepare and project input]
  B --> C[Run enabled input check]
  C --> D{Input allowed?}
  D -- No --> E[Return OpenAI error]
  D -- Yes --> F{Streaming selected?}
  F -- No --> G[Buffered dispatch and output check]
  F -- Yes --> H[Check stream configuration]
  H --> I[Create fresh hooks and dispatch]
  I --> J[Check pending text before release]
  G --> K[Return response]
  J --> K

Loading

_{Reviews (11) · Last reviewed commit: "fix(server): name the stream outcomes th..." · Reviewed by Greptile}

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA-NeMo/Guardrails/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 049cda59-b0d1-4fc1-a476-6524dbdc815c

---

📥 Commits

Reviewing files that changed from the base of the PR and between fe7408e and 6269303.

---

📒 Files selected for processing (13)

• nemoguardrails/server/experimental/_buffered_kernel.py
• nemoguardrails/server/experimental/_guarded_proxy.py
• nemoguardrails/server/experimental/_guarded_stream.py
• nemoguardrails/server/experimental/_http_kernel.py
• nemoguardrails/server/experimental/_streaming_http.py
• nemoguardrails/server/experimental/provider/endpoint.py
• nemoguardrails/server/experimental/provider/errors.py
• nemoguardrails/server/experimental/providers/openai/chat_completions/README.md
• nemoguardrails/server/experimental/providers/openai/chat_completions/endpoint.py
• nemoguardrails/server/experimental/providers/openai/errors.py
• nemoguardrails/server/experimental/providers/openai/integration.py
• tests/server/experimental/test_openai_chat_streaming_integration.py
• tests/server/experimental/test_streaming_http.py

---

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.

---

---

📝 Walkthrough

Walkthrough

The guarded HTTP pipeline now separates request preparation from execution. OpenAI Chat Completions can route requests through streaming dispatch when configured, while retaining buffered dispatch for non-streaming requests.

Changes

Guarded streaming HTTP

Layer / File(s)	Summary
Shared request preparation and execution nemoguardrails/server/experimental/_buffered_kernel.py, nemoguardrails/server/experimental/_http_kernel.py	Input preparation now returns reusable prepared input. The HTTP handler can pass it to a prepared-request handler or continue with buffered execution.
Streaming adapter and guarded execution nemoguardrails/server/experimental/provider/endpoint.py, nemoguardrails/server/experimental/_guarded_stream.py, nemoguardrails/server/experimental/_guarded_proxy.py, nemoguardrails/server/experimental/_streaming_http.py, nemoguardrails/server/experimental/provider/errors.py	Endpoints can create and validate stream adapters. The guarded proxy routes prepared streaming requests to streaming execution when configured. Unsupported stream inspection is rendered as an outcome.
OpenAI Chat Completions streaming wiring nemoguardrails/server/experimental/providers/openai/chat_completions/endpoint.py, nemoguardrails/server/experimental/providers/openai/errors.py, nemoguardrails/server/experimental/providers/openai/integration.py, nemoguardrails/server/experimental/providers/openai/chat_completions/README.md	The endpoint binds a stream adapter, and the router accepts optional stream dispatch. The error renderer maps stream outcomes to OpenAI-compatible errors. The README describes runtime support and the buffered-only contract export.
Streaming behavior and routing tests tests/server/experimental/test_openai_chat_streaming_integration.py, tests/server/experimental/test_streaming_http.py	Tests cover dispatch selection, stream inspection, error handling, route and request-framing validation, adapter creation, and OpenAPI response media types.

Priority: ⬇️ Low

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant GuardedHttpOperation
  participant prepare_operation_input
  participant prepared_request_handler
  participant execute_streaming_http
  participant StreamingHttpDispatch
  Client->>GuardedHttpOperation: Send request
  GuardedHttpOperation->>prepare_operation_input: Project and check input
  prepare_operation_input-->>GuardedHttpOperation: Return prepared input
  GuardedHttpOperation->>prepared_request_handler: Select response mode
  prepared_request_handler->>execute_streaming_http: Execute streaming request
  execute_streaming_http->>StreamingHttpDispatch: Dispatch provider request

Loading

---

Merge Risk: ⚪ Minimal · up to 62693

No identified streaming integration issue blocks merging after normal checks.

Pre-merge checks | 5 | 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 68 functions across 12 files. (1 skipped:…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Test Results For Major Changes		The PR is a major feature change, but its description includes testing information. It states that coverage includes route/framing rejection, unsafe policies, hidden blocked content, source closure, a…
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main change: integrating OpenAI Chat streaming into the server.

---

Full details: Docstring Coverage

Explanation

Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 68 functions across 12 files. (1 skipped: 1 unsupported.)

---

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

---

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[tgasser-nv]

Looks godo, just a couple of things to address before merging:

1. Could you add a local end-to-end test to show this works with a build.nvidia.com - hosted model?
2. Can you add test coverage to get it up to 100% ?

[Pouyanpi]

Thanks Tim. Added tests across the stack, including the kernel features Chat doesn't use yet. The end-to-end test against build.nvidia.com is deferred to the transport work.