Skip to content

fix(proxy): handle upstream HTTP connection closure - #5335

Open
shiju-nv wants to merge 5 commits into
NVIDIA:mainfrom
shiju-nv:fix/5322-upstream-idle-close/shiju-nv
Open

shiju-nv wants to merge 5 commits into
NVIDIA:mainfrom
shiju-nv:fix/5322-upstream-idle-close/shiju-nv

Conversation

@shiju-nv

@shiju-nv shiju-nv commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Retire HTTP connections promptly when the upstream closes while idle, including JSON-RPC and MCP. For relay-owned upstream failures, return a correlated 502 Bad Gateway with Connection: close before any response bytes are written to the client; close without appending another status once a response has started. Apply the same error handling to plain forward HTTP and CONNECT tunnels.

Related Issue

Closes #5322

Scope follows the issue discussion: automatic reconnect/replay is deliberately excluded. The original replay acceptance criterion still needs the issue author's correction to reflect the agreed idle-close and explicit-error scope.

Changes

  • Watch idle upstream connections in all five HTTP request loops, including JSON-RPC/MCP, and reject unsolicited bytes even when a client request is already queued.
  • Track response bytes accepted by the client and preserve the original upstream error cause. Distinguish incomplete framing from normal body completion followed by policy or middleware errors.
  • Share correlated gateway errors, upstream shutdown, and failure telemetry with forward HTTP. Record a 502 only after its write succeeds, and end affected WebSocket middleware sessions as upstream failures.
  • Extend existing test matrices for inspected requests, protocol coverage, truncated bodies, resets, and policy reload; add a full forward-handler regression. Existing response-middleware-owned failures retain their own responses and events.

Release Notes

HTTP clients receive idle upstream closure promptly. Relay-owned failures before a response starts return an explicit gateway error with a request ID.

Testing

  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable)

Local validation used Rust 1.95.0 on macOS. Full Linux sandbox E2E and the reporter's EKS/Istio/Bun deployment were not run; hosted CI owns broad verification after this push.

  • Six relay regressions passed, including real TCP keep-alive for passthrough, REST, JSON-RPC and MCP, queued failures, inspected 502s, reset causes, and unsolicited upstream data.
  • Full forward-handler EOF regression passed with 502, request ID and allowed activity.
  • Two response-policy-reload tests passed through the adapter owner with both fixed-length and close-delimited responses; two keep-alive policy-generation tests passed.
  • Existing endpoint-observation, JSON-RPC and unchanged response-middleware focused receipts remain valid for their covered inputs.
  • cargo clippy --offline --locked -p openshell-supervisor-network --lib --tests -- -D warnings: passed. Rust formatting, diff whitespace and SPDX checks passed.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

Signed-off-by: Shiju <shiju@nvidia.com>
Consolidate overlapping forwarding cases through the complete relay and retain distinct failure boundaries. Remove relay implementation details from the network policy guide.

Signed-off-by: Shiju <shiju@nvidia.com>

@EmilienM EmilienM left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another iteration needed: 1 blocking, 4 major, 7 minor (inline, 5 below)

The tunnel REST/GraphQL/passthrough loops are covered and tested, but the JSON-RPC/MCP loop and the forward-proxy entry path don't get the fix, and response_started can still suppress the 502 with nothing delivered to the client.

Outside the diff:

  • Blocking: the PR has no area label, so Review Readiness fails; area:supervisor (or area:runtime) fits.
  • Major: crates/openshell-supervisor-network/src/proxy.rs:6836, the forward-proxy error boundary downcasts the sibling typed relay errors but has no arm for the new UpstreamHttpError, so plain forward requests still close silently with no 502 and the allow-event for an allowed request is now skipped.
  • Major: issue #5322's first acceptance criterion (replaying a request sent on a reused connection when the upstream closed before any response byte) isn't implemented; since the PR closes the issue, either implement it for idempotent methods or update the issue so the descope is recorded.
  • Minor: relay.rs:2355, an upstream failure now ends the websocket middleware session as Cancellation while the sibling arms use UpstreamFailure.
  • Minor: the PR body says the connection behavior is documented, but the second commit removed the docs paragraph; restore a trimmed note or drop the claim.
How this was reviewed
  • Commit e5fe0357, round 1
  • Standard review, because the change is code in crates/openshell-supervisor-network/src/l7/relay.rs, l7/rest.rs, and l7/rest/http_response.rs
  • Passes: architecture, performance, security, scalability, rust, supply-chain, then correctness
  • Independent skeptics checked each candidate finding (2 per blocking or major, 1 per minor); 11 of 11 held up, 2 of them after re-checking the code
  • Repo rules read from main: AGENTS.md, CLAUDE.md, CONTRIBUTING.md, docs/CONTRIBUTING.mdx
  • Not covered: nothing skipped

Cover JSON-RPC/MCP idle closure and forward HTTP failures. Track response commitment at client writes, retain upstream causes, and distinguish truncated framing from normal completion and policy reload. Preserve request correlation and upstream/session cleanup.

Signed-off-by: Shiju <shiju@nvidia.com>
@shiju-nv shiju-nv added area:supervisor Proxy and routing-path work area:runtime and removed area:supervisor Proxy and routing-path work labels Nov 6, 2026
@github-actions
github-actions Bot requested review from TaylorMutch, drew and purp November 6, 2026 17:21
Integrate audit trace correlation while preserving upstream HTTP failure handling.

Signed-off-by: Shiju <shiju@nvidia.com>
@shiju-nv

shiju-nv commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

All five HTTP loops now watch idle upstream connections, including JSON-RPC/MCP. Forward HTTP uses the same upstream-error handling and preserves allowed-request telemetry. Response commitment tracks successful client writes, so an incomplete buffered header can still produce a 502. Original I/O causes are preserved, and truncated bodies are distinguished from policy or middleware failures after normal body completion.

The shared handler preserves the request ID, shuts down upstream, and records a 502 only after delivery succeeds. Session cleanup uses UpstreamFailure. The tests now cover inspected REST, resets and the forward-handler path. The stale docs claim is removed, and area:runtime is set.

Record the response request ID in the HTTP upstream-failure event so
operators can match client errors to structured and shorthand logs.
Cover supplied and absent IDs in the existing event schema test.

Signed-off-by: Shiju <shiju@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(proxy): a reused keep-alive connection whose upstream closed while idle is dropped without a response

2 participants