Downgrade node-rdkafka to 3.0.1 - #2773
Merged
bert-e merged 2 commits intoJul 17, 2026
Merged
Conversation
librdkafka >= 2.5.0 cannot consume messages stored in the legacy v0/v1 MessageSet format: when the FetchResponse uses a flexible version (v12+), the legacy reader parses the Key/Value length prefixes as compact varints instead of fixed int32, so every message is delivered with NULL key and value and the consumer logs 'Unsupported Message(Set) MagicByte' errors. S3C federation pins log.message.format.version=0.10.2.2 on the backbeat-queue brokers, so all messages are stored and served in the legacy v1 format: backbeat >= 9.4 (node-rdkafka ^3.6.0 / librdkafka 2.12.0) cannot process any kafka entry on S3C, and replication stays PENDING. This blocks the S3C 10 component bump (Federation PR 6993). Pin node-rdkafka to exactly 3.0.1, the newest release bundling librdkafka 2.3.0 (the version backbeat ran until 9.3.x): 3.0.0 is 2.18.0 with only EOL Node versions dropped, and 3.0.1 only adds an OAuthBearer setToken API. The app-side fixes from the original bump (bootstrap chained-setTimeout, partition.assignment.strategy pin) are kept. The pin must not be raised past 3.0.1 until node-rdkafka bundles a librdkafka release containing the upstream fix: confluentinc/librdkafka#5550 Note that Node >= 24 support will require node-rdkafka >= 3.6.1. Reported upstream: confluentinc/librdkafka#5549 Issue: BB-806
Contributor
Hello delthas,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
delthas
requested review from
a team,
SylvainSenechal,
benzekrimaha and
leif-scality
July 16, 2026 15:46
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
@@ Coverage Diff @@
## development/9.4 #2773 +/- ##
===================================================
+ Coverage 74.81% 74.85% +0.04%
===================================================
Files 201 201
Lines 13747 13747
===================================================
+ Hits 10285 10291 +6
+ Misses 3452 3446 -6
Partials 10 10
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
delthas
requested review from
francoisferrand
and removed request for
benzekrimaha
July 16, 2026 15:46
Contributor
Author
|
Requested @francoisferrand in place of Maha Benzekri, who is currently on PTO. |
Contributor
Request integration branchesWaiting for integration branch creation to be requested by the user. To request integration branches, please comment on this pull request with the following command: Alternatively, the |
francoisferrand
approved these changes
Jul 16, 2026
francoisferrand
left a comment
francoisferrand
left a comment
Contributor
There was a problem hiding this comment.
(note: description is a bit hard to follow, kind of mixing librdkafka version with node-rdkafka versions)
leif-scality
approved these changes
Jul 17, 2026
|
Can you bump the package.json so we can create a new preview directly |
Contributor
Author
|
/approve |
Contributor
|
I have successfully merged the changeset of this pull request
The following branches have NOT changed:
This pull request did not target the following hotfix branch(es) so they
Please check the status of the associated issue BB-806. Goodbye delthas. The following options are set: approve |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Downgrades
node-rdkafkafrom^3.6.0to exactly3.0.1.Why
librdkafka >= 2.5.0 cannot consume messages stored in the legacy v0/v1 MessageSet format: when the FetchResponse uses a flexible version (v12+), the legacy reader parses the Key/Value length prefixes as compact varints instead of fixed int32, so every message is delivered with NULL key and value (plus
Unsupported Message(Set) MagicByteerrors). Reported upstream with a fix: confluentinc/librdkafka#5549 / confluentinc/librdkafka#5550.S3C federation pins
log.message.format.version=0.10.2.2on the backbeat-queue brokers, so all messages are stored/served in the legacy v1 format: since #2728 (node-rdkafka^3.6.0/ librdkafka 2.12.0), backbeat cannot process any kafka entry on S3C — every entry fails withmalformed JSON in kafka entry — Cannot read properties of null (reading 'bootstrapId')and replication staysPENDING. This currently blocks the S3C 10 component bump (scality/Federation#6993).Why 3.0.1
range,roundrobin— Node 23/24 support).partition.assignment.strategypin) are kept.log.message.format.version=0.10.2.2: 3.0.1 consumes legacy v1 topics with intact key/value; 3.1.0+ does not.The pin must not be raised past 3.0.1 until node-rdkafka bundles a librdkafka release containing the fix from confluentinc/librdkafka#5550. Note: Node >= 24 support will require node-rdkafka >= 3.6.1 (nan bump), so the re-bump must precede or accompany any Node 24 migration.
Issue: BB-806