Fix active active - #225
Closed
matthewcooper wants to merge 472 commits into
Closed
Conversation
Moved official plugins into core repo
resolved FIP not coming up
renamed internal to src to allow for referencing
…interface is now pulsectl
The pulseha service is now called pulseha and the pulse command line …
… structure, and plugin failover
END-1615: Added nil check to the exported methods of IPMonitor so tha…
…rity and isolation
…ification before failover promotion
…-in-pulseha GRPC config, GRPC over unix socket and maintenance mode
…ng logic - Introduced a `Maintenance` flag in the configuration for nodes to persist state across reboots. - Improved safeguards to prevent peers from overriding local node maintenance state. - Refactored maintenance mode handling for local and remote nodes, including quorum checks and failover logic. - Updated gRPC and CLI to better support maintenance mode operations.
Fix maintenance mode
- Added a periodic gRPC-based deep check for cluster membership validation to detect split-brain scenarios. - Enhanced the health check mechanism to differentiate between TCP connectivity checks and comprehensive membership validation. - Updated health check logic to isolate nodes failing the deep check until they pass subsequent validations. - Added cluster token validation to the gRPC health check handler.
…-on-active-node-does-not-update-if Introduce deep cluster membership validation in health checks
Fix maintenance
Two follow-up fixes for the maintenance mode / Unix socket changes introduced in PR #217: - setMaintenanceLocal: the MakePassive demotion result was discarded. A transport error was logged at Warn level and the flow continued; a soft failure (Success=false response) was ignored entirely. The node would be marked maintenance while still holding its active IPs, leaving the cluster with no eligible active node. Abort maintenance entry and surface the error to the operator instead. - Server.Start: the CLI Unix socket was removed unconditionally before Listen, which silently steals the socket out from under another pulseha daemon running on the same host. Dial-probe first; if a process is already listening, refuse to start.
…ocket-safety Fail-fast on maintenance demote errors and protect CLI socket from steal
…nforce local config settings
Fix maintenance
…tdout only as a fallback
Logs and unknown
…stener
Two concurrent ConfigSync callbacks could each spawn a Reconfigure() in
their own goroutine. Each goroutine: snapshot grpcServer pointer briefly
under lock, GracefulStop the old server, create a new one, and then call
net.Listen("tcp", IP:9083). With no serialization between them and no
SO_REUSEADDR on the listener, the bind races and one or both attempts
fail with "address already in use", leaving the cluster listener torn
down and the node effectively unreachable from peers. We see this in
journals as repeated "Async reconfigure failed after ConfigSync" errors
followed by the joining node disappearing from the cluster mid-test.
- Add Server.reconfigureMu and acquire it for the duration of Reconfigure().
Concurrent ConfigSync callbacks now run serially; the second sees the
listener already at the desired address and the bind succeeds normally.
- Replace net.Listen with a small helper that sets SO_REUSEADDR (and
SO_REUSEPORT on Linux) on the listening socket so a re-bind succeeds
even if the previous socket is still in TIME_WAIT in the kernel.
- Add a Linux unit test that closes a listener and immediately re-binds
the same address, covering the failure mode.
- Replace ineffective rebind smoke test with a deterministic getsockopt assertion that SO_REUSEADDR is actually set on the bound socket; SO_REUSEPORT is verified best-effort to tolerate legacy kernels and seccomp-restricted sandboxes. - Relax the test build tag from linux-only to !windows so it runs on macOS dev machines too. - Add tests/unit/reconfigure_concurrent_test.go covering the primary fix: 8 goroutines fire Reconfigure() through a barrier channel after CreateCluster brings up the listener on an ephemeral port; asserts all return nil (without reconfigureMu, one would EADDRINUSE). - Capture s.grpcServer into a local pointer before spawning the Serve goroutine in startClusterListener so a concurrent Reconfigure swap doesn't race the goroutine's read (caught by -race in the new test). - Rewrite drifting comments (reconfigureMu field, listenTCPReuseAddr, SO_REUSEPORT inline, Reconfigure doc) as present-tense invariants and drop PR-narrative phrasing; document reconfigureMu -> s.RWMutex lock ordering and the caller constraint. - Log SO_REUSEPORT setsockopt failures at debug level instead of silently discarding errno, so legacy-kernel / seccomp issues remain traceable. - Switch listenCtx cleanup to defer cancelListen() and wrap the listen error with %w for errors.Is compatibility.
fix(server): serialize Reconfigure and SO_REUSEADDR the cluster listener
…nd InitiateJoin operations
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Root cause: TOCTOU race between CreateCluster and InitiateJoin gRPC handlers running as concurrent goroutines on node 2.
Your setup process sent both pulsectl cluster create and pulsectl cluster join to node 2 within the same second. Both handlers check ClusterCheck() (len(config.Nodes) > 0) at the start — but both saw 0 nodes because neither had finished
writing yet. Both passed the guard and proceeded. The join correctly set node 2 to Maintenance, but CreateCluster ran after and unconditionally forced Status = Active at server.go:3326, overriding it.
The log evidence was clear:
Fix: Added clusterInitMu sync.Mutex to Server. Both CreateCluster (line 3201) and InitiateJoin (line 4549) acquire it before their ClusterCheck() call. Whichever runs first completes normally; the second acquires the mutex and now sees
nodes > 0, returns "cluster is already configured" / "leave first", and exits without touching state.