Fix group coordinator rediscovery after broker decommission - #5512
Open
morgando wants to merge 1 commit into
Open
Fix group coordinator rediscovery after broker decommission#5512morgando wants to merge 1 commit into
morgando wants to merge 1 commit into
Conversation
When rd_kafka_coord_req_fsm() finds a cached group coordinator that is not UP, it returns expecting a future broker state change to re-enter the FSM. That works when the cached coordinator is merely DOWN. The broker thread remains alive and continues cycling through states while trying to reconnect. Those state changes re-enter rd_kafka_coord_req_fsm(), giving the stale coordinator cache entry a chance to age out and allowing the FSM to send a FindCoordinator request. Since f7c4273, brokers that disappear from metadata responses are decommissioned, which terminates their broker thread. However, a decommissioned broker can remain cached as group coordinator. The bug occurs when the cached coordinator has been decommissioned. The FSM still returns expecting a future broker state change, but the decommissioned broker's thread has been terminated, so no state change from that broker will occur. As a result, the FSM may not be re-entered until some other broker changes state, possibly much later. The changes in this commit: 1. Remove decommissioned brokers from the coordinator cache, so the next call to rd_kafka_coord_req_fsm() sends a new FindCoordinator request instead of continuing to wait for the stale coordinator. 2. Trigger any waiters on the decommissioned broker, causing them to re-enter rd_kafka_coord_req_fsm() and send a new FindCoordinator request.
|
🎉 All Contributor License Agreements have been signed. Ready to merge. |
Open
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5511 by adding these two steps when decommissioning a broker:
rd_kafka_coord_req_fsm()sends a FindCoordinator request instead of waiting for the stale, decommissioned coordinator to come up.rd_kafka_coord_req_t) are waiting for a connection to the broker viarkb_persistconn.coord, wake them up so that they re-enterrd_kafka_coord_req_fsm(), see that the cache has been cleared, and send a new FindCoordinator request.