fix(binding-kafka): derive the merged produce window from the least budget offered - #2546
Merged
jfallows merged 2 commits intoSep 4, 2026
Merged
Conversation
…udget offered doMergedInitialWindow aggregated the produce partitions by taking the largest unacknowledged byte count across them and, separately, the smallest maximum. A partition carrying traffic holds its acknowledge and grows its maximum, while idle partitions keep the maximum they were opened with, so those two reductions select different partitions: the unacknowledged bytes of the busiest one get paired with the maximum of an idle one. The merged window then shrinks by every byte written until it reaches zero and the stream stalls for good. With a single partition both reductions select the same stream, which is why this only appears on topics with more than one partition. Reduce over the budget each partition actually offers instead, and add the unacknowledged bytes back to express it as a maximum. For a single partition this is arithmetically identical to the previous computation. Adds a merged window trace under the existing produce debug property, since this aggregation was not observable from a running gateway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tition merged.produce.message.values.partition.idle opens two produce partitions, sends all traffic to one and leaves the other idle. Reducing the maximum across partitions independently of the unacknowledged bytes pins the merged window to the idle partition's maximum while the busy partition's unacknowledged bytes keep growing, so the stream stalls once that maximum is reached; the scenario asserts a further write still succeeds beyond it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2516
doMergedInitialWindowaggregated the produce partitions by taking the largest unacknowledged byte count across them and, separately, the smallest maximum. A partition carrying traffic holds its acknowledge and grows its maximum, while idle partitions keep the maximum they were opened with, so those two reductions select different partitions: the unacknowledged bytes of the busiest one get paired with the maximum of an idle one. The merged window then shrinks by every byte written until it reaches zero and the stream stalls for good. With a single partition both reductions select the same stream, which is why this only appears on topics with more than one partition.This is the primary defect behind the original bug report this was found in (published/forwarded 200/60 in a live 5-partition topic test before the fix, 600/600 after) — an MQTT client publishing small QoS 0 messages through an
mqtt-kafkaproxy to a multi-partition Kafka topic stopped being forwarded after roughly 60 messages, with Zilla closing the connection itself and no exception or event logged.Fix reduces over the budget each partition actually offers instead (
initialMax - initialNoAck), and adds the unacknowledged bytes back to express it as a maximum. For a single partition this is arithmetically identical to the previous computation. Also adds a merged-window trace under the existing produce debug property, since this aggregation was not observable from a running gateway.Credit to community contributor @sfr-oc, who diagnosed and fixed this as part of the combined #2523. This PR cherry-picks that fix's commit standalone (unmodified,
ccc296db), since #2523 bundles seven independently-scoped defects across three bindings into one PR — each deserves its own focused review, and this one is ready on its own.Test coverage
Also cherry-picks the companion test commit (unmodified,
1a4495b7) from the same original PR, which added the actual regression coverage for this scenario (the fix commit alone had none):merged.produce.message.values.partition.idleopens two produce partitions, sends all traffic to one and leaves the other idle, and asserts a write past the idle partition's maximum still succeeds.merged/unmergedk3po scripts, matching this binding's existing convention for every othermerged.produce.*scenarioCacheMergedIT#shouldProduceMergedMessageValuesPartitionIdleChecked the new scenario's naming against this binding's existing
merged.produce.message.values.*family (.dynamic,.null,.producer.id,.partition.id) —.partition.idlefits the established vocabulary directly, no rename needed.Verification
TestTimedOutExceptionafter exactly 10 of the 11 expected writes succeed, matching "the merged window shrinks by every byte written until it reaches zero and the stream stalls for good" — then restored the fix and confirmed the test passes🤖 Generated with Claude Code
https://claude.ai/code/session_015YVNaqKvEXZVzmg3HoGVnt
Generated by Claude Code