Repository navigation
fix: Update fluent-bit to cap pushes and refused request log - #671
Merged
Merged
Conversation
Contributor
 [Medium impact] The changes stay within Pilot’s Fluent Bit log-shipping workflow on configured hosts. Add Lua behavior tests and remove the repeated comment before merging to satisfy repository requirements. Reviews (3) · Last reviewed commit: "fix(fluent-bit): Keep a complete last ch..." · Reviewed by Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Flush 30 did not cap requests: each tail input keeps its own chunks and the http output sends each chunk as its own request, so six busy inputs could spend Datum's 12 requests a minute before any retry. A rewrite_tag filter after the Lua normalizer now re-emits every line under datum.logs through one emitter, and the output matches only that tag, so a flush sends one chunk. Retry_Limit drops to 3 and the scheduler spaces retries 10-60s apart: 2 chunks a minute, each sent at most 4 times, is at most 8 requests a minute even while Datum refuses all of them. Checked against Fluent Bit 5.1.4 with a refusing endpoint and busy inputs: worst 60s window was 9 requests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cap_message stripped any trailing UTF-8 sequence after the 8 KiB cut, so a character that fit whole was dropped too: 8190 ASCII bytes and "éX" lost the "é". It now steps back to the last character's lead byte and drops it only when fewer bytes remain than that byte announces. Checked in Fluent Bit 5.1.4 with 2, 3 and 4 byte characters landing just inside and just across the cut. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ref: frappe/datum#19