fix(controller): don't lose the client disconnect when it fails - #575
Open
krangerich wants to merge 1 commit into
Open
fix(controller): don't lose the client disconnect when it fails#575krangerich wants to merge 1 commit into
krangerich wants to merge 1 commit into
Conversation
A timeout on CLIENT LIST left the old master's clients connected for good: the error was only logged, and the next reconcile finds a correctly configured replica, so replicaOf never runs for that pod again. Mark the pod before demoting it and clear the mark once the disconnect went through. CLIENT LIST plus one kill per client becomes a single CLIENT KILL LADDR. Signed-off-by: Michael J. <7890659+krangerich@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
During a failover, the dragonfly operator sends a
SLAVE OF <new master>to the old master, sets the label torole: replicaand then trys to disconnect clients. But disconnecting clients isn't robust and might fail.A timeout on
CLIENT LISTleft the old master's clients connected for good: the error was only logged, and the next reconcile finds a correctly configured replica, so replicaOf never runs for that pod again.Mark the pod before demoting it and clear the mark once the disconnect went through. Disconnecting the clients is now retried by a reconciliation loop.
CLIENT LISTplus one kill per client becomes a singleCLIENT KILL LADDR.CLIENT KILL LADDRwas added in Dragonfly v1.14.0, together with the previously usedCLIENT KILL.Fixes #324
Related: #238