mcp-kafka: reset_offsets hangs forever when target consumer group state is Dead

[jfallows]

Found while smoke-testing the Kafka MCP toolkit against develop-SNAPSHOT, confirmed via zilla dump.

Symptom

kafka__reset_offsets against a consumer group that has never had any members (broker reports it as Dead, e.g. one that was never created by an active consumer) hangs indefinitely — no response, no error, no progress — until the MCP client's own idle timeout eventually gives up (300s in our case).

Repro

1. Ensure a consumer group has never existed / has no committed state (e.g. orders-analytics, confirmed via kafka-consumer-groups.sh --describe --group orders-analytics → GroupIdNotFoundException; the MCP describe_consumer_group tool reports it as {"group_id":"orders-analytics","state":"Dead","members":[]}, which is the normal/expected shape for a nonexistent group).

2. Call kafka__reset_offsets with group_id=orders-analytics, topic=orders, partition=0, offset=0.

3. The call never returns. Repeated with a fresh zilla dump capture on example.south_mcp_kafka_client0 running throughout: the wire trace shows exactly two successful Kafka round trips —

   • FindCoordinator (v3) → Error: No Error (0), coordinator = node 1
   • DescribeGroups (v5) → Error: No Error (0), group state Dead

   — and then nothing. No third Kafka request is ever sent, and no HTTP response is ever sent back to the MCP client, for the remainder of the capture window.

Implicated code

runtime/binding-mcp-kafka/.../stream/McpKafkaProxyFactory.java, KafkaApiResetOffsetsClient (~line 8730 onward). This is a 3-stage flow: STAGE_FIND_COORDINATOR → STAGE_DESCRIBE_GROUPS → STAGE_OFFSET_COMMIT. RESETTABLE_GROUP_STATES (line 257) explicitly includes both "Empty" and "Dead", and completeDescribeGroups (line 8960) correctly takes the "proceed" branch for a Dead group, calling advanceToOffsetCommit(traceId) (line 8993), which ends the DescribeGroups stream and opens a new Kafka stream carrying an offsetCommit begin-extension (built with groupId/memberId("")/instanceId("")/coordinatorHost/coordinatorPort, line 9029-9037) — the actual bare-commit request payload is only sent later, from sendOffsetCommit, once a WINDOW arrives for that new stream (onKafkaWindow, line 9069-9097, default case).

Per the pcap, that new stream's BEGIN is either never actually dispatched south, or never receives a WINDOW back — either way, sendOffsetCommit is never reached, so the flow stalls forever in STAGE_OFFSET_COMMIT with no request sent and no error surfaced to the MCP client.

Impact

Any reset_offsets call against an inactive/nonexistent consumer group — which per the code's own RESETTABLE_GROUP_STATES comment is meant to be the normal, supported case (mirroring AdminClient.alterConsumerGroupOffsets() against an inactive group) — hangs the calling client indefinitely instead of completing or failing fast.

Suggested fix

Investigate why the third (STAGE_OFFSET_COMMIT) stream's BEGIN/WINDOW handshake never completes when advancing from DescribeGroups, and add a bounded timeout so this flow fails fast with a clear error instead of hanging forever.

[jfallows]

Re-traced this against a newer develop-SNAPSHOT build with a fresh zilla dump capture on example.south_mcp_kafka_client0, and the actual mechanism this time is different (and more fundamental) than what I found before — worth recording since it changes where the fix needs to go.

New finding: the pooled south Kafka connection silently dies from idle, and the client keeps writing into the dead stream

Tracing by zilla stream id (kafka_client0 → tcp_client0), the SAME stream id (0x1e9/reply 0x1e8) is reused across many independent Kafka API round trips over time — this is a persistent pooled connection, as expected. The timeline:

t=286.5s .. t=327.7s   normal DATA/WINDOW traffic (earlier, unrelated tool calls)
t=628.177s             RESET  (kafka_client0 → tcp_client0, same stream)
t=628.178s             ABORT  (same stream)
t=628.181s             RESET  (companion stream)
                       ... 173s of total silence on this connection ...
t=801.514s             DATA — FindCoordinator v3 Request  (our reset_offsets call, SAME stream id 0x1e9 reused)
                       — no response ever arrives; capture ends with nothing further —

628.177 - 327.721 ≈ 300.46s — the pooled connection was reset almost exactly 300 seconds after its last activity (the binding's own zilla.binding.kafka.client.max.idle.ms defaults to 60000ms in the source I checked, so this specific value may come from somewhere else in the pool/cleanup path, but the ~300s idle-then-reset pattern is exact and repeatable).

The critical bug: after that RESET/ABORT tore down the pooled connection, the next request to use it (our reset_offsets call's FindCoordinator, stage 1) was sent as a DATA frame reusing the same, already-dead stream id — it was never preceded by a fresh BEGIN to re-establish a new connection. Since the stream no longer exists, the write goes nowhere and no response (success or error) is ever generated. The MCP client hangs forever with zero feedback, which is exactly the symptom in this issue.

This looks like a connection-pool bug in mcp-kafka's (or the underlying kafka client binding's) south connection reuse: it doesn't detect that a pooled connection was reset while idle and doesn't transparently open a replacement connection for the next request — it just retries the write on the stale stream id.

Note this is a different defect from what I originally diagnosed on the pre-rebuild build (where FindCoordinator→DescribeGroups both succeeded and the stall was specifically in advanceToOffsetCommit's third stream never sending). It's possible that stall was this same connection-pool-reuse issue manifesting one stage later, rather than something specific to KafkaApiResetOffsetsClient. Either way, the practical impact remains: reset_offsets (and potentially any kafka__* tool call made after the south Kafka connection has been idle for several minutes) can hang indefinitely with no error.

Suggested fix: when a pooled south connection receives RESET/ABORT while idle, remove it from the pool immediately so the next request opens a fresh connection instead of writing into the dead stream id; and/or surface a clear error to the MCP client if a write is attempted on a connection that's already been torn down.

[jfallows]

Confirmed: zilla.binding.kafka.client.max.idle.ms now defaults to 300000 (5 min) in this build, not the 60000 I found in the older source — matches the measured ~300.46s gap exactly.

[jfallows]

Reopening — the previous fix (#2540) addressed the stale-END misattribution in KafkaApiResetOffsetsClient, but per the latest comment above, the underlying mechanism for the hang is actually a stale pooled south Kafka connection being reused after an idle RESET/ABORT instead of a fresh connection being opened. Investigating that connection-pool eviction path now.

---

Generated by Claude Code

[jfallows]

Retested against a build that reportedly tightened up the Kafka client connection state machine — confirmed the connection-pool-reuse-after-reset theory from my last comment is not what's happening here. This reproduces on a fresh restart, first call, no idle period at all.

New, more precise finding: the flow goes silent immediately after a clean DescribeGroups response — no third request is ever sent, no reply ever reaches the client

Traced two separate reset_offsets attempts (both against orders-analytics, confirmed Dead/resettable) via zilla dump on example.south_mcp_kafka_client0 and example.north_mcp_server0. Both show the identical, clean sequence:

FindCoordinator v3 Request  → Response: Error: No Error (0)
DescribeGroups v5 Request   → Response: Error: No Error (0)
WINDOW (credit granted on the pooled connection)
                             ... nothing further, ever ...

This time the underlying connection is demonstrably healthy — streams 0x7c/0x7d are a long-lived pooled pair that had already been used successfully for dozens of unrelated DescribeGroups/other calls earlier in the same capture (first BEGIN well before either of our attempts, still alive and getting fresh WINDOW credit right after our DescribeGroups response). So this isn't a stale/reset connection being reused — it's a live, working connection that simply never gets a third write.

On the client-facing side, I also traced the internal north_mcp_server → kafka_client0 application stream for our specific tool call (DATA+END carrying the request body, e.g. frame 1490-1491 in this capture) — and there is no corresponding reply ever written back on that stream. The MCP client-facing side is left with an open request and literally nothing coming back — not even an error — which is exactly why it hangs until the client's own idle-abort kicks in (300s in our harness).

So, narrowed down precisely: once the group is confirmed to be in a resettable state (Dead, no error) via DescribeGroups, whatever should happen next — building/sending the bare OffsetCommit request and eventually replying to the MCP client — never executes. Given the connection itself is healthy and has credit available, this points at application-level logic after the DescribeGroups response handler (deciding the group is resettable and proceeding to the offset-commit stage) rather than anything connection/transport-related.