Skip to content

fix(clustering): iterate a snapshot of targets in dispatchTargetsOnce - #1055

Open
moutyque wants to merge 1 commit into
openconfig:mainfrom
moutyque:fix/dispatch-targets-race
Open

moutyque wants to merge 1 commit into
openconfig:mainfrom
moutyque:fix/dispatch-targets-race

Conversation

@moutyque

Copy link
Copy Markdown

Problem

In clustered mode, the leader can crash with:

fatal error: concurrent map iteration and map write

goroutine ... [running]:
github.com/openconfig/gnmic/pkg/app.(*App).dispatchTargetsOnce(...)
        pkg/app/clustering.go:288
github.com/openconfig/gnmic/pkg/app.(*App).dispatchTargets(...)
        pkg/app/clustering.go:273
created by github.com/openconfig/gnmic/pkg/app.(*App).startCluster.func1
        pkg/app/clustering.go:161

dispatchTargetsOnce ranges over a.Config.Targets while holding only dispatchLock.
The loader writes the same map under configLock (startLoader, a.Config.Targets[add.Name] = add).
Since the loader releases configLock before it dispatches (v0.49.1), it writes to the map more often while the dispatch loop runs.

We hit it in production on v0.49.1 with a 6-instance cluster and about 6000 targets from the HTTP loader: the leader crashed during the initial load after a rollout, leadership moved to another instance, which crashed the same way. It settled once the loader stopped adding targets.

Fix

Copy the targets under configLock.RLock() and iterate over the copy. The lock is held only for the copy, not across dispatchTarget, so the loader and the API handlers are never blocked by a dispatch pass.

Test

TestDispatchTargetsOnceSafeAgainstConcurrentLoaderWrites runs dispatchTargetsOnce while another goroutine adds targets under configLock.

  • Before the fix: go test -race reports DATA RACE and the run dies with fatal error: concurrent map iteration and map write.
  • After the fix: go test -race ./pkg/app/ passes.

The loader writes Config.Targets under configLock while the leader's
dispatch loop ranges over the same map with only dispatchLock held.
With a few thousand targets and the loader adding targets during
startup, the leader crashes with 'fatal error: concurrent map iteration
and map write' and leadership moves to another instance, which hits
the same race.

Copy the targets under configLock.RLock and iterate the copy.
@google-cla

google-cla Bot commented Oct 26, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant