http: stop writing errors back to disconnected metrics clients - #280
Merged
Conversation
The metrics handler called errorResponse after streaming of the response had already begun. Once a scrape client (for example Prometheus) closes the connection mid-response, the HTTP/2 server returns "client disconnected" on the next write, so trying to send an error status and body back to that client is futile and just fails again. The write loop also lacked a return, so a single disconnect logged one error line per remaining metric family, making one benign disconnect look like many failures. Stop calling errorResponse from the streaming path. On a write failure to the client, log at debug (a disconnect is an expected event) and return after the first error. The registry.Gather() failure path still uses errorResponse, since it fails before any body is written and a 500 there is legitimate. While here, handle two previously ignored errors on the --prometheus proxy path: metricsWriter.Copy's return value in the handler (logged at error level, since a Copy failure usually means the upstream source is unreachable or returned malformed metrics) and translator.Translate's return value inside Writer.Copy. Extract the handler into metricsHandler so it can be unit tested, and add tests for a normal scrape, a client disconnect, and a Copy failure. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
sravotto
marked this pull request as ready for review
July 14, 2026 15:12
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Customers on v1.3.1 see repeated
Error gathering metrics: client disconnectedat error level in the logs. Theclient disconnectedtext iserrClientDisconnectedfromgolang.org/x/net/http2, returned when visus writes to the HTTP/2 response after a scrape client (Prometheus) has already closed the connection, typically a scrape timeout or cancel.The metrics handler mishandled this:
errorResponsefrom inside the streaming loop, after response bytes had already been written. Once the client is gone, sending a status and body back is futile and just fails again.return, so a single disconnect logged one error line per remaining metric family, making one benign disconnect look like many failures.Changes
errorResponsefrom the streaming path. On a write failure to the client, log at debug and return after the first error. Theregistry.Gather()failure path still useserrorResponse, since it fails before any body is written and a 500 there is legitimate.--prometheusproxy path:metricsWriter.Copy's return value in the handler, logged at error level since aCopyfailure usually means the upstream source is unreachable or returned malformed metrics.translator.Translate's return value insideWriter.Copy.metricsHandlerso it can be unit tested.Note on log levels: the client-write loop logs at debug because a disconnect there is expected and benign (this is the path that produced the customer's noise). The
Copypath logs at error because a failure there is usually an actionable upstream-source problem. We deliberately do not string-match the disconnect error, since http2'serrClientDisconnectedis unexported.Tests
Added handler tests in
internal/http/server_test.go:TestMetricsHandler: a normal scrape returns 200 with the gathered metric.TestMetricsHandlerClientDisconnect: on a failing writer, the handler returns without panicking and never callsWriteHeader, confirming it no longer tries to send an error status to a disconnected client.TestMetricsHandlerCopyError: a bad upstream source makesCopyfail and the handler returns before writing gathered metrics.