Skip to content

fix: surface streaming errors faithfully (mid-stream on errCh + pre-first-message status) - #14

Merged
yhxlele merged 2 commits into
mainfrom
fix/streaming-terminal-error
Jul 8, 2026
Merged

fix: surface streaming errors faithfully (mid-stream on errCh + pre-first-message status)#14
yhxlele merged 2 commits into
mainfrom
fix/streaming-terminal-error

Conversation

@yhxlele

@yhxlele yhxlele commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Two related fixes so streaming errors surface faithfully instead of being lost.
Both are in DoStreamingRequest's error handling; neither requires a go.mod
change (codes/status come from the already-required google.golang.org/grpc),
and the full suite passes under the repo's CI Go 1.20 toolchain.

1. Mid-stream errors are swallowed (returned as silent success)

DoStreamingRequest's decode loop only looks for the result key:

rawResult, ok := res[streamingResponseResultKey] // "result"
if !ok { continue }

grpc-gateway frames a server-streaming RPC that fails after emitting one or
more results as a terminal data: {"error": <google.rpc.Status>} event. That
event has no result key, so the loop continues past it, the next Decode()
returns io.EOF, and the goroutine close(resCh)s — so the caller sees a clean,
successful end-of-stream and silently receives a truncated result. The
streamingResponseErrorKey constant was defined but never consulted in the loop.

Fix: check for the error key before the result key; if present, unmarshal
the google.rpc.Status and deliver it on errCh. resCh is intentionally left
un-closed on this path so a consumer's select deterministically observes the
error rather than an EOF.

2. Pre-first-message errors collapse to codes.Unknown

When a streaming handler fails before its first message, grpc-gateway sets the
HTTP error status and writes the error through the SSE marshaller, so the body is
framed as data: {"error": ...}. wrapStreamingResponseError called
json.Unmarshal on the raw body, which fails on the data: prefix and returns a
generic error — so the real gRPC status (NotFound, PermissionDenied, …)
collapses to codes.Unknown.

Fix: strip the SSE data: framing before parsing. This is a no-op for
non-streaming (plain JSON) error bodies such as routing 404s, which start with
{ rather than data:.

Tests

  • TestDoStreamingRequest_ErrorAfterResultsresult, result, error: asserts both
    results are delivered and then the Internal status arrives on errCh (not a
    clean EOF).
  • TestDoStreamingRequest_ErrorBeforeResults — SSE-framed pre-first-message error:
    asserts the returned error carries the real NotFound status, not Unknown.

Both were verified RED before their respective fix and GREEN after. The test
service's TrackInvitation gained two id-keyed hooks (fail-after-events,
fail-before-events); the existing streaming test is unchanged.

Compatibility

errCh was already the loop's channel for decode/unmarshal errors, so callers
that drain it (the canonical select { case <-resCh; case <-errCh }) simply now
receive the real error instead of a truncated success. Fix (2) only changes the
initial error for SSE-framed bodies (Unknown → real code); it is a no-op for the
plain-JSON routing-404 case. No new contract for callers.

@yhxlele
yhxlele requested review from hanxiaop and jiachengxu July 7, 2026 09:08
@yhxlele yhxlele changed the title fix: surface terminal stream errors on errCh instead of ending the stream silently fix: surface streaming errors faithfully (mid-stream on errCh + pre-first-message status) Jul 8, 2026
@yhxlele
yhxlele merged commit 18de0d8 into main Jul 8, 2026
5 checks passed
@yhxlele
yhxlele deleted the fix/streaming-terminal-error branch July 8, 2026 14:22
jiachengxu added a commit that referenced this pull request Jul 23, 2026
…t errors (#15)

This PR fixes two silent-truncation defects in `DoStreamingRequest`'s
SSE decoding path:

1. **Events larger than 64KiB were silently dropped.** The
`alevinval/sse` decoder reads
lines with a default `bufio.Scanner`, which caps a line at
`bufio.MaxScanTokenSize`
(64KiB). Each streamed message arrives as a single `data: {json}` line,
so any message
   whose JSON exceeds ~64KiB made `Scan()` fail with `bufio.ErrTooLong`.
2. **Every scanner failure was flattened into `io.EOF`.** The decoder's
read loop never
checks `scanner.Err()`; on any failure — the 64KiB overflow above, or a
transport error
(connection reset) mid-stream — it returns `io.EOF`, which
`DoStreamingRequest` treats
   as successful completion (`close(resCh)`).

Combined effect: a stream carrying one oversized message, or a stream
cut mid-flight,
ends **exit-0 with partial data**. The caller cannot distinguish it from
a complete,
successful stream. #14 fixed the case where the *server* terminates the
stream with an
`{"error": ...}` event; these two failures happen below that layer, on
the client's own
read path, so no error event is ever seen.

## How

Replace the `alevinval/sse` decoder with a small internal
`bufio.Reader`-based SSE event
decoder (`sseEventDecoder` in `pkg/grpc/gateway/request.go`):

- **No line-length limit** — `ReadString('\n')` grows as needed.
- **Errors propagate** — read errors surface on `errCh` instead of
closing `resCh`.
- **Truncation is never success** — an EOF that interrupts a
partially-read event returns
`io.ErrUnexpectedEOF`; only an EOF at a clean event boundary is a normal
end-of-stream.
- SSE framing behavior is preserved: multiple `data:` lines of one event
join with `\n`,
a single leading space after the colon is trimmed, comment/heartbeat
lines (`: ...`)
and non-data fields are ignored, and both `\n` and `\r\n` line endings
are accepted
  (the alevinval decoder handled these the same way).

Scope: only the SSE decode path used by server-streaming RPC responses.
`DoRequest`
(unary), `doHTTPStreamingRequest` (`google.api.HttpBody` streams, which
use `io.Copy`),
and `wrapStreamingResponseError` are untouched. `alevinval/sse` remains
a test-only
dependency (`marshaller_test.go` uses it to decode the marshaller's
output).

## Testing

New regression tests in `pkg/grpc/gateway/request_test.go`:

| Test | Covers |
|---|---|
| `TestDoStreamingRequest_LargeEvents` | 3 × 300KiB messages through the
real gRPC → grpc-gateway → SSE pipeline; each must arrive intact |
| `TestDoStreamingRequest_TruncatedStreamSurfacesError` | connection cut
mid-event: the already-complete first event is delivered, then an
**error** (never a clean close) |
| `TestDoStreamingRequest_SSEFraming` | comment/heartbeat lines, CRLF
endings, one event split across multiple `data:` lines |

All three fail against the previous decoder and pass with the fix:
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants