Report ALL_BROKERS_DOWN only once per outage with metadata.recovery.strategy=rebootstrap

[Marko Lahma (lahma)]

Fixes #5546 (first part: repeated ALL_BROKERS_DOWN events).

Problem

With metadata.recovery.strategy=rebootstrap (the default since v2.10.0 / KIP-899), a client
whose brokers are all unreachable for a sustained period reports
RD_KAFKA_RESP_ERR__ALL_BROKERS_DOWN to the application on every re-bootstrap cycle
(roughly every reconnect interval) instead of once per outage.

Mechanism:

• rd_kafka_broker_set_state() raises the error under the /* Only 0 -> 1 */
rkb_down_reported guard when the last non-logical broker goes down, and starts a
  re-bootstrap in the same block (src/rdkafka_broker.c).
• The re-bootstrap timer callback calls rd_kafka_reset_any_broker_down_reported(), which
  resets every broker's rkb_down_reported so that all brokers are re-tried before the
  all-down condition can be reached again (intent of Avoid returning an all brokers down error on planned disconnections #5126).
• During a sustained outage every broker keeps failing, so the guard re-arms and the error
  re-fires on every cycle. Nothing rate-limits this
  (metadata.recovery.rebootstrap.trigger.ms only applies to the separate metadata-timeout
  trigger). The Avoid returning an all brokers down error on planned disconnections #5126 commit message acknowledges this side effect ("after retrying on
  re-bootstrap a new ALL_BROKERS_DOWN error is issued").

Reproduced with a mock cluster closing every connection during ApiVersion negotiation:
17 error events in 10 seconds on an idle consumer.

Fix

Add an rk_all_brokers_down_reported latch:

• The error is reported only when the all-brokers-down condition is reached and the latch
  transitions 0 → 1; subsequent cycles of the same outage log a BROKER debug message
  instead.
• The latch is cleared when any non-logical broker enters the UP state, so a genuine
  recovery followed by a new outage reports the error again.
• Re-bootstrap behavior is unchanged: it still runs on every all-down condition, and the
  Avoid returning an all brokers down error on planned disconnections #5126 semantics (reset rkb_down_reported so all brokers are re-tried) are untouched —
  only the application-visible event is de-duplicated.

Tests

Three new mock-cluster subtests in tests/0152-rebootstrap.c (local, no real cluster
needed), using ApiVersion + RD_KAFKA_RESP_ERR__TRANSPORT request-error injection to make
the mock broker close every connection during the handshake (the exact failure mode from
#5546):

1. Sustained outage (3 brokers, 10 s): exactly 1 ALL_BROKERS_DOWN error while
   re-bootstrap keeps cycling (≥ 2 sequences; observed ~19). Fails with 17 errors
   before the fix.
2. Second outage after recovery: exactly one more error (latch re-arms on broker UP).
3. Reconnect backoff regression guard: during a sustained outage the backoff keeps growing
   until capped by reconnect.backoff.max.ms, pacing connection attempts (3-6 attempts in
   10 s with 1000/5000 ms settings) — guards the investigated-and-disproved hypothesis from
   Repeated ALL_BROKERS_DOWN error on every re-bootstrap cycle during sustained outage (metadata.recovery.strategy=rebootstrap) #5546 that re-bootstrap resets the reconnect backoff.

0000 (unit tests), 0095, 0151 and the local quick suite pass.

Note: the error de-duplication relies on rd_atomic32_set() returning the previous value,
which is broken in CMake builds before #5136 (shipped in v2.15.0) — this PR targets master
where that fix is present.

[confluent-cla-assistant]

🎉 All Contributor License Agreements have been signed. Ready to merge.
✅ lahma
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

[Copilot]

Pull request overview

This PR de-duplicates application-visible RD_KAFKA_RESP_ERR__ALL_BROKERS_DOWN error events during sustained outages when metadata.recovery.strategy=rebootstrap is enabled, while keeping the existing re-bootstrap behavior unchanged.

Changes:

• Add a client-level latch (rk_all_brokers_down_reported) to report ALL_BROKERS_DOWN only once per outage and re-arm it when any non-logical broker becomes UP.
• Update broker state handling to emit a debug log (instead of re-raising the error) on subsequent all-down cycles within the same outage.
• Add mock-cluster regression tests covering sustained outage de-duplication, re-arming after recovery, and reconnect backoff growth behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/0152-rebootstrap.c	Adds mock-cluster subtests and callbacks to verify ALL_BROKERS_DOWN is emitted once per outage and backoff continues to grow during sustained failure.
src/rdkafka.c	Initializes the new rk_all_brokers_down_reported latch in rd_kafka_new().
src/rdkafka_int.h	Defines and documents the new rk_all_brokers_down_reported atomic latch in struct rd_kafka_s.
src/rdkafka_broker.c	Gates ALL_BROKERS_DOWN emission behind the new latch and clears it when any non-logical broker transitions to UP.
CHANGELOG.md	Documents the behavioral change and links it to #5546 / regression since #5126.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.