Skip to content

Fix goroutine leak in /probe multi-target endpoint - #1226

Open
danlucioprada wants to merge 1 commit into
prometheus-community:masterfrom
danlucioprada:fix/probe-goroutine-leak
Open

Fix goroutine leak in /probe multi-target endpoint#1226
danlucioprada wants to merge 1 commit into
prometheus-community:masterfrom
danlucioprada:fix/probe-goroutine-leak

Conversation

@danlucioprada

@danlucioprada danlucioprada commented Jul 30, 2026

Copy link
Copy Markdown

Fixes #1225

Problem

The Shards and Indices collectors start a background cluster-info receive loop in their constructor that only exits when their update channel is closed:

go func() {
    for ci := range shards.clusterInfoCh { ... } // blocks until channel closed
}()

In single-target mode a long-lived clusterinfo.Retriever owns that channel for the process lifetime, so this is fine. But the /probe handler builds a fresh Shards and Indices per scrape and never registers them with a retriever, so nothing ever closes the channel. Each /probe scrape leaks two goroutines, and each leaked goroutine retains the collector, its per-probe HTTP client and registry, so resident memory grows unbounded until the process is OOMKilled.

Reproduced on v1.11.0: go_goroutines grows by exactly 2 per /probe request and never drops (full analysis and pprof dump in #1225).

Fix

  • Add an idempotent Close() method (guarded by sync.Once) to Shards and Indices that closes clusterInfoCh, letting the receive-loop goroutine return.
  • Call Close() via defer in the /probe handler so both goroutines exit when the handler returns.

Single-target mode is unchanged, it never calls Close(), so the retriever keeps owning the channel.

Verification

Before/after, hammering /probe against a dead target (the goroutines are started at construction, so a live ES is not needed):

after 60 probes after 120 probes
before 23 143 (~ +2 / probe)
after 16 14 (flat)

Added a regression test (collector/probe_leak_test.go) that creates and closes 50 of each collector and asserts the goroutine count returns to baseline. go test ./... passes; gofmt/go vet clean.

Note

I went with the minimal, lifecycle-based fix that mirrors the existing consumer model. An alternative would be to skip starting the receive-loop goroutine entirely when no retriever is attached (i.e. in /probe mode).

The Shards and Indices collectors start a background cluster info receive
loop in their constructors that only exits when the update channel is
closed. In single-target mode a long-lived clusterinfo.Retriever owns that
channel for the process lifetime. The /probe handler, however, builds a
fresh Shards and Indices per scrape and never registers them with a
retriever, so nothing ever closes their channel. Each /probe scrape
therefore leaks two goroutines (one per collector). The leaked goroutines
retain the collector, its HTTP client and registry, so resident memory
grows unbounded until the process is OOMKilled.

Add a Close method to both collectors that closes the channel (guarded by
sync.Once so it is idempotent) and call it via defer in the /probe handler
so the receive-loop goroutines exit when the handler returns. Single-target
mode is unchanged and never calls Close.

Add a regression test that creates and closes many collectors and asserts
the goroutine count returns to its baseline.

Signed-off-by: Dan Lucio Prada <dan.prada@effecti.com.br>
@sysadmind

Copy link
Copy Markdown
Contributor

Thanks for the issue and PR. We are actually in the process of deprecating the cluster info package and the channel logic all together. The new way to get the info is through the cluster provider. Check out the new cluster module for details.

@danlucioprada

Copy link
Copy Markdown
Author

So, is something likely already being done for a future version? In that case, I can close the PR

@sysadmind

Copy link
Copy Markdown
Contributor

Yes, in a future version, we will finish removing the old cluster info package and this bug should go away with that change.

@ArthurSens

Copy link
Copy Markdown
Contributor

@sysadmind could you clarify our plans for this one? Should we close this PR since we're removing this part of the codebase? Should we fix the leak while the code is still here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Goroutine leak in /probe (multi-target) endpoint causes unbounded memory growth / OOM

4 participants