fix(server): prevent dual-active - #226
Merged
Merged
Conversation
…nd InitiateJoin operations
…n tests - Acquire clusterInitMu at the top of HandleNodeJoin so the tokenless first-node branch is serialized with CreateCluster and InitiateJoin, closing the same TOCTOU class via the peer-driven Join RPC path - Log WARN on the CreateCluster and InitiateJoin rejection paths so a racing setup script is visible in logs instead of being silently absorbed as an idempotent retry - Bound InitiateJoin's outbound Join RPC with a 30s context timeout derived from the request context so a hung target cannot hold clusterInitMu indefinitely and block concurrent CreateCluster or HandleNodeJoin callers - Add tests/unit/cluster_init_race_test.go with two tests: concurrent CreateCluster + InitiateJoin against a configured cluster (asserts documented rejection messages) and a timing-based regression test that verifies HandleNodeJoin acquires clusterInitMu by observing it blocks while CreateCluster holds the lock
- Run go mod tidy to add missing go.sum entries for transitive deps (go.opentelemetry.io/otel, golang.org/x/sys, golang.org/x/net) that CI started failing on after a Makefile-driven 'go get -u' resolved upstream tools to newer versions - Drop google.golang.org/grpc/cmd/protoc-gen-go-grpc from go.mod require block; it is a build-time tool fetched separately by the Makefile, not a runtime dependency
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Root cause: TOCTOU race between CreateCluster and InitiateJoin gRPC handlers running as concurrent goroutines on node 2.
Your setup process sent both pulsectl cluster create and pulsectl cluster join to node 2 within the same second. Both handlers check ClusterCheck() (len(config.Nodes) > 0) at the start — but both saw 0 nodes because neither had finished
writing yet. Both passed the guard and proceeded. The join correctly set node 2 to Maintenance, but CreateCluster ran after and unconditionally forced Status = Active at server.go:3326, overriding it.
The log evidence was clear:
Fix: Added clusterInitMu sync.Mutex to Server. Both CreateCluster (line 3201) and InitiateJoin (line 4549) acquire it before their ClusterCheck() call. Whichever runs first completes normally; the second acquires the mutex and now sees
nodes > 0, returns "cluster is already configured" / "leave first", and exits without touching state.