Repeated ALL_BROKERS_DOWN error on every re-bootstrap cycle during sustained outage (metadata.recovery.strategy=rebootstrap)

[lahma]

Description

With metadata.recovery.strategy=rebootstrap (the default since v2.10.0 / KIP-899), a client
whose brokers are all unreachable for a sustained period receives
RD_KAFKA_RESP_ERR__ALL_BROKERS_DOWN (Local_AllBrokersDown, -187) on every re-bootstrap
cycle — roughly once per reconnect interval, for the whole duration of the outage — instead
of once per outage. First observed in production as a consumer that never got a single
successful bootstrap connection (the peer closed the connection during APIVERSION_QUERY on
every attempt) and logged the error every few seconds for minutes.

Mechanism (traced against master / v2.15.0; identical since v2.11.1)

1. rd_kafka_broker_set_state() (src/rdkafka_broker.c) raises ALL_BROKERS_DOWN under the
   /* Only 0 -> 1 */ rkb_down_reported guard when the last non-logical broker goes down,
   and calls rd_kafka_rebootstrap() in the same all-brokers-down block.
2. The re-bootstrap runs on an immediate one-shot timer whose callback (src/rdkafka.c)
   calls rd_kafka_reset_any_broker_down_reported(), clearing every broker's
   rkb_down_reported. This is intentional per Avoid returning an all brokers down error on planned disconnections #5126 (v2.11.1): all brokers must be re-tried
   before the all-down condition may be declared again.
3. During a sustained outage every broker keeps failing, so each broker's next DOWN
   transition passes the 0 → 1 guard again, the all-down threshold is crossed again, and the
   error is reported again — once per cycle, indefinitely.

Nothing rate-limits the loop: metadata.recovery.rebootstrap.trigger.ms only applies to the
separate metadata-timeout trigger, and rk_rebootstrap_in_progress is cleared at the end of
each timer callback. The #5126 commit message acknowledges the repeat ("given the down
reported state is reset, after retrying on re-bootstrap a new ALL_BROKERS_DOWN error is
issued"). With metadata.recovery.strategy=none the error fires once per outage, as the
guard intends.

Expected behavior

ALL_BROKERS_DOWN should be reported once per distinct outage and re-armed when some broker
connection comes up again, independent of the recovery strategy.

How to reproduce (deterministic, mock cluster)

Make every connection fail during the ApiVersion handshake — the mock broker closes the
connection when returning RD_KAFKA_RESP_ERR__TRANSPORT:

rd_kafka_mock_push_request_errors_array(mcluster, RD_KAFKAP_ApiVersion,
                                        cnt, errs /* __TRANSPORT * cnt */);

An idle client connected to a 3-broker mock cluster set up this way receives one -187 error
event per re-bootstrap cycle: 17 events in 10 s with reconnect.backoff.ms=100 /
reconnect.backoff.max.ms=500.

Note on reconnect backoff

The reconnect backoff itself was initially suspected of not growing, but tracing and mock
reproduction ruled that out: during such an outage the backoff grows exponentially and caps
at reconnect.backoff.max.ms, and the steady-state cycle cadence equals the capped backoff.
Re-bootstrap does not reset any backoff state (re-added bootstrap brokers dedupe to no-ops
for existing RD_KAFKA_CONFIGURED brokers).

Related issues

• Re-bootstrap loop in case of ALL_BROKERS_DOWN #5088 — re-bootstrap loop from bootstrap-vs-learned broker connection preference; distinct,
  our case never reaches a learned-brokers state at all.
• librdkafka v2.13.0 Stops Rebootstrap and Clears Brokers on "0 brokers but topic>0" MetadataResponse, Causing Permanent Client Disconnection #5301 — permanent disconnection after a 0 brokers, topic_cnt > 0 metadata response;
  opposite failure mode (stops retrying entirely).

Environment

• librdkafka v2.14.0 (via confluent-kafka-dotnet / Confluent.Kafka 2.14.0); behavior verified
  unchanged on master (v2.15.0).
• Consumer client, security.protocol=SASL_SSL, sasl.mechanism=PLAIN, Confluent Cloud.
• metadata.recovery.strategy at its default (rebootstrap).

Fix

Proposed in #5547: report the error only on the 0 → 1 transition of a per-instance
all-brokers-down latch, cleared when any non-logical broker enters the UP state.
Re-bootstrap behavior and the #5126 down-reported reset semantics are unchanged.