Problem
For a Kafka-backed subscription that performs subgraph hydration, a request deadline does not unblock the event-processing path if a wrapped RoundTripper or origin hook ignores request-context cancellation. Later events can remain queued behind that operation, even though the configured request timeout has expired.
Reproduction and tested behavior
This integration test models the condition with 100 ms request and subscription-fetch timeouts. One hydration operation intentionally ignores cancellation; the test requires an error for that event and a later Kafka event over the same WebSocket.
A proposed patch makes that scenario pass by supervising subscription hydration requests. It bounds abandoned operations per subgraph, cleans up late responses, adds metrics, and makes retry backoff context-aware. Unit, race, and repeated Kafka integration tests pass on the patched branch.
Expected behavior
A failed hydration should emit an error for that event without terminating the WebSocket subscription, and later events should continue to be processed.
Production observation (not confirmed as the same cause)
We observed approximately ten minutes without subscription data around resolver pod rolls. The reproduced blocking mechanism is consistent with that symptom, but we do not have a trace proving it caused that incident or explaining the exact ten-minute duration; infrastructure and network timeouts may also contribute.
Problem
For a Kafka-backed subscription that performs subgraph hydration, a request deadline does not unblock the event-processing path if a wrapped
RoundTripperor origin hook ignores request-context cancellation. Later events can remain queued behind that operation, even though the configured request timeout has expired.Reproduction and tested behavior
This integration test models the condition with 100 ms request and subscription-fetch timeouts. One hydration operation intentionally ignores cancellation; the test requires an error for that event and a later Kafka event over the same WebSocket.
A proposed patch makes that scenario pass by supervising subscription hydration requests. It bounds abandoned operations per subgraph, cleans up late responses, adds metrics, and makes retry backoff context-aware. Unit, race, and repeated Kafka integration tests pass on the patched branch.
Expected behavior
A failed hydration should emit an error for that event without terminating the WebSocket subscription, and later events should continue to be processed.
Production observation (not confirmed as the same cause)
We observed approximately ten minutes without subscription data around resolver pod rolls. The reproduced blocking mechanism is consistent with that symptom, but we do not have a trace proving it caused that incident or explaining the exact ten-minute duration; infrastructure and network timeouts may also contribute.