feat: add config example validation script and workflow - #2627
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdded configuration extraction and dry-run validation scripts. Integrated changed-file validation into pull-request checks. Corrected configuration examples across installation and pipeline documentation. ChangesConfiguration validation
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant PullRequest
participant GitHubActions
participant validate-changed-files.sh
participant test-config.sh
participant FluentBitContainer
PullRequest->>GitHubActions: change Markdown files
GitHubActions->>validate-changed-files.sh: pass base SHA and HEAD
validate-changed-files.sh->>test-config.sh: validate each changed file
test-config.sh->>FluentBitContainer: run configuration dry-run
FluentBitContainer-->>test-config.sh: return validation status
test-config.sh-->>GitHubActions: report failures or success
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
ba1f42c to
56630f1
Compare
eschabell
left a comment
There was a problem hiding this comment.
@patrick-stephens did some cleanup work but it now is good to go, thanks for this! Please merge this when you are ready?
|
Once merged we can see how things go and look to extend it in the future with checks for case, etc. |
There was a problem hiding this comment.
Actionable comments posted: 9
🧹 Nitpick comments (3)
scripts/test-config.sh (2)
119-128: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winAvoid rerunning the container just to capture failure output.
On failure, the container is run twice: once with output discarded (Line 120) to check the exit status, and again (Line 125) purely to surface output on stderr. Capture combined output on the first run instead, so a failing validation doesn't double the container startup/dry-run cost.
♻️ Proposed fix
- if ! $CONTAINER_RUNTIME run --rm -t -v "$OUTPUT_FILE":"$OUTPUT_FILE":ro "$VALIDATION_IMAGE" fluent-bit --dry-run --config="$OUTPUT_FILE" &>/dev/null; then + VALIDATION_OUTPUT=$($CONTAINER_RUNTIME run --rm -v "$OUTPUT_FILE":"$OUTPUT_FILE":ro "$VALIDATION_IMAGE" fluent-bit --dry-run --config="$OUTPUT_FILE" 2>&1) && VALIDATION_STATUS=0 || VALIDATION_STATUS=$? + if [ "$VALIDATION_STATUS" -ne 0 ]; then FAILED_VALIDATIONS+=("$LANGUAGE example $EXAMPLE_INDEX") - # Provide the configuration and failure output for debugging purposes on stderr echo "ERROR: Validation failed for $LANGUAGE example $EXAMPLE_INDEX in $FILE" >&2 cat "$OUTPUT_FILE" >&2 - $CONTAINER_RUNTIME run --rm -t -v "$OUTPUT_FILE":"$OUTPUT_FILE":ro "$VALIDATION_IMAGE" fluent-bit --dry-run --config="$OUTPUT_FILE" >&2 || true + echo "$VALIDATION_OUTPUT" >&2 else🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/test-config.sh` around lines 119 - 128, Update the validation command in the configuration-checking flow to capture its combined output during the initial container run while preserving the exit status. In the failure branch, print the captured output instead of invoking $CONTAINER_RUNTIME a second time; keep the existing configuration and failure-context diagnostics unchanged.
73-81: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winImage is pulled once per invocation, i.e., once per changed file per PR run.
The workflow invokes this script once per changed Markdown file (see
pr-example-validation.yaml, Line 58-61). Each invocation independently pulls$VALIDATION_IMAGE(Line 78), adding a network round-trip per file even when the image is already present locally. For PRs touching several documentation files, this adds up.Consider checking for a local image and only pulling when absent, or moving the pull into a one-time setup step.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/test-config.sh` around lines 73 - 81, Update the image setup around CONTAINER_RUNTIME and VALIDATION_IMAGE so the script checks whether VALIDATION_IMAGE is already available locally before pulling it. Invoke the runtime’s image-inspection command, pull only when the image is absent, and preserve the existing error message and exit behavior when a required pull fails.scripts/extract-config.sh (1)
146-171: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winSilent success when the tab title is never found.
In the
ENDblock, the error branch only fires whenfound_tab && !found_fence(Line 166). Ifwanted_titlenever matches anywhere in the file,found_tabstays0, so neither the success branch nor the error branch runs. The script exits0with empty output.
test-config.shmasks this because count mode already returns0and the caller skips extraction in that case. Butscripts/README.mddocuments this script for direct standalone use ("can also be called directly"). A typo'd tab title passed directly would silently succeed with no output instead of reporting an error.♻️ Proposed fix
} else if (found_tab && !found_fence) { printf "ERROR: %s code fence #%d not found in tab: %s\n", wanted_language, target_index, wanted_title > "/dev/stderr" exit 1 + } else if (!found_tab) { + printf "ERROR: tab not found: %s\n", wanted_title > "/dev/stderr" + exit 1 }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/extract-config.sh` around lines 146 - 171, Update the extract-mode validation in the END block so a missing tab title reports an error and exits nonzero instead of silently succeeding. Extend the existing found_tab/found_fence handling around wanted_title, while preserving count mode and the current missing-fence and missing-code-fence errors.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/pr-example-validation.yaml:
- Line 43: Update the changed_md grep filter in the workflow to use a literal
dot for the filename/path match, ensuring Markdown files at the repository root
and in subdirectories are detected while retaining the existing .md suffix
requirement.
- Line 43: Update the changed_md command in the PR validation workflow to use
git diff with the existing base and HEAD revision range, or first compute the
merge base before invoking git diff-tree. Preserve the current name and
diff-filter options so only added, modified, copied, or renamed Markdown files
are selected.
- Around line 42-46: Update the output-writing logic in the Markdown
change-detection step so the newline-separated value from changed_md uses GitHub
Actions’ multiline output delimiter syntax instead of echoing it as a plain
list= entry. Preserve the existing output name list and ensure the
delimiter-wrapped value is written safely to GITHUB_OUTPUT for multiple Markdown
files.
In `@pipeline/filters/parser.md`:
- Line 30: Update scripts/test-config.sh to discover each per-file tab title and
pass that title to extract-config.sh, rather than counting only
fluent-bit.yaml/fluent-bit.conf; ensure every renamed example is validated.
Apply this discovery behavior to the tab declarations at
pipeline/filters/parser.md:30-30 and :40-40, and
pipeline/outputs/kafka.md:110-110 and :157-157, using each declaration’s title
as the corresponding configuration name.
In `@pipeline/inputs/kafka.md`:
- Around line 145-156: Update the introductory paragraph to state that each
message is processed by the inline modify_kafka_message function rather than
kafka.lua, and revise the sentence structure so it clearly describes sending the
result back to the fb-sink topic on the same broker.
In `@pipeline/parsers.md`:
- Around line 68-72: Make both standalone YAML examples self-contained: in
pipeline/parsers.md, define custom_parser1 inline or reference a concrete
parser-file fixture before it is used; in pipeline/filters/kubernetes.md,
replace the legacy undefined parsers_file reference with the existing inline
custom-tag definition or another concrete parser-file fixture. Ensure each
extracted YAML tab validates independently.
In `@pipeline/router.md`:
- Line 210: Update the routing.yaml tab title in the routing documentation
around the routing.yaml example so scripts/test-config.sh recognizes and
extracts the routes configuration snippet. Use a supported extractor tab title
while preserving the existing routes syntax example.
In `@scripts/test-config.sh`:
- Around line 56-71: In the suppressed-file loop, replace the three redundant
conditions with one path-anchored match that accepts only an exact path or a
file beneath the suppressed entry as a directory. Preserve the informational
message and exit behavior, while ensuring names embedded in unrelated filenames
or extensions do not suppress validation.
- Around line 96-102: Update the Stage 1 counting flow around extract-config.sh
so genuine command failures remain visible and cause the validation to fail,
rather than being converted into EXAMPLE_COUNT=0. Preserve the legitimate
zero-example path that continues without validation, but capture and report the
extraction error separately using the existing script error-handling
conventions.
---
Nitpick comments:
In `@scripts/extract-config.sh`:
- Around line 146-171: Update the extract-mode validation in the END block so a
missing tab title reports an error and exits nonzero instead of silently
succeeding. Extend the existing found_tab/found_fence handling around
wanted_title, while preserving count mode and the current missing-fence and
missing-code-fence errors.
In `@scripts/test-config.sh`:
- Around line 119-128: Update the validation command in the
configuration-checking flow to capture its combined output during the initial
container run while preserving the exit status. In the failure branch, print the
captured output instead of invoking $CONTAINER_RUNTIME a second time; keep the
existing configuration and failure-context diagnostics unchanged.
- Around line 73-81: Update the image setup around CONTAINER_RUNTIME and
VALIDATION_IMAGE so the script checks whether VALIDATION_IMAGE is already
available locally before pulling it. Invoke the runtime’s image-inspection
command, pull only when the image is absent, and preserve the existing error
message and exit behavior when a required pull fails.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 526f19af-3dd1-4b14-80b6-0c240eb98fc1
📒 Files selected for processing (22)
.github/workflows/pr-example-validation.yamlinstallation/downloads/docker.mdpipeline/buffering.mdpipeline/filters/geoip2-filter.mdpipeline/filters/kubernetes.mdpipeline/filters/parser.mdpipeline/inputs/blob.mdpipeline/inputs/cpu-metrics.mdpipeline/inputs/kafka.mdpipeline/inputs/tail.mdpipeline/outputs/dynatrace.mdpipeline/outputs/gelf.mdpipeline/outputs/kafka.mdpipeline/outputs/loki.mdpipeline/outputs/s3.mdpipeline/parsers.mdpipeline/processors/conditional-processing.mdpipeline/processors/sql.mdpipeline/router.mdscripts/README.mdscripts/extract-config.shscripts/test-config.sh
💤 Files with no reviewable changes (2)
- pipeline/processors/conditional-processing.md
- pipeline/outputs/gelf.md
7e2c8c7 to
9fda3df
Compare
|
@patrick-stephens fixed a bunch of things that came up on the review of the review. See what you think? |
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (4)
pipeline/outputs/s3.md (4)
154-182: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winValidate Parquet examples with an Arrow-enabled image or suppress them.
scripts/test-config.shusesfluent/fluent-bit:latestby default, and the official Fluent Bit image does not include Apache Arrow/Parquet support. The discoverablefluent-bit.yamlandfluent-bit.confexamples withformat: parquetcan fail duringfluent-bit --dry-run, so run them against an image with Parquet support, such asamazon/aws-for-fluent-bit, or addpipeline/outputs/s3.mdto the validator suppression list with an explicit reason.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pipeline/outputs/s3.md` around lines 154 - 182, Update the Parquet examples in the S3 output documentation so validation does not run them against the default image lacking Arrow/Parquet support. Either configure their validation to use an Arrow-enabled image such as amazon/aws-for-fluent-bit, or add pipeline/outputs/s3.md to the validator suppression list with an explicit reason, while preserving the examples themselves.Source: Learnings
171-182: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winParquet PutObject examples need explicit
total_file_size.These examples rely on the 100M default: the YAML example at 171-182, plus the migratory YAML and classic config examples around 844-882. Add
total_file_size: 50Mor another valid PutObject value so the examples document a supported explicit size.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pipeline/outputs/s3.md` around lines 171 - 182, Add an explicit valid total_file_size setting, such as 50M, to the Parquet PutObject examples in the shown YAML configuration and the related migratory YAML and classic configuration examples. Preserve the existing use_put_object settings and formatting while ensuring each relevant example documents the size explicitly.
49-54: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winClarify Arrow output and unset compression behavior.
compression: arrowis legacy compatibility syntax, not a normal codec. Useformat: arrowfor Arrow/Feather output, withcompression: zstdor unset for no Arrow compression. Addnone/unset to the S3compressiontable, and show Parquet files as uncompressed whenformat: parquetomitscompression.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pipeline/outputs/s3.md` around lines 49 - 54, The S3 documentation must clarify that Arrow output uses format: arrow, while compression: arrow is legacy syntax; update the compression table in pipeline/outputs/s3.md lines 49-54 to document none/unset, and revise pipeline/outputs/s3.md line 778 to show Parquet files are uncompressed when format: parquet omits compression, with Arrow compression using zstd or unset for no compression.
194-199: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winAlign the migration example with the documented PutObject requirement.
The Parquet section says
format parquetrequiresuse_put_object On, but lines 194-199 omit that setting. Adduse_put_object: onor update this block to state thatformat parquetenables it automatically.Proposed fix
**After (recommended):** ```yaml +use_put_object: on format: parquet compression: snappy🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pipeline/outputs/s3.md` around lines 194 - 199, Update the Parquet migration YAML example to include the documented use_put_object: on setting alongside format: parquet and compression: snappy, unless the surrounding documentation explicitly establishes that Parquet enables it automatically.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/pr-example-validation.yaml:
- Around line 65-68: Update the changed-file iteration around CHANGED_MD_FILES
to read one line at a time, preserving each newline-delimited path as a single
argument to ./scripts/test-config.sh. Replace the unquoted for loop while
retaining the existing processing message and error_count increment behavior.
- Line 43: Update the changed-file collection around changed_md so git diff runs
separately and its failure causes the validation step to exit nonzero; only
after a successful diff should its output be filtered for .md files, preserving
the existing no-Markdown-files behavior for an empty result.
In `@scripts/README.md`:
- Around line 41-49: Update the all-files validation example to track whether
any invocation of test-config.sh fails, while continuing to process every
Markdown file; after the loop completes, exit with the tracked failure status so
automation receives a non-zero result when any file fails.
---
Outside diff comments:
In `@pipeline/outputs/s3.md`:
- Around line 154-182: Update the Parquet examples in the S3 output
documentation so validation does not run them against the default image lacking
Arrow/Parquet support. Either configure their validation to use an Arrow-enabled
image such as amazon/aws-for-fluent-bit, or add pipeline/outputs/s3.md to the
validator suppression list with an explicit reason, while preserving the
examples themselves.
- Around line 171-182: Add an explicit valid total_file_size setting, such as
50M, to the Parquet PutObject examples in the shown YAML configuration and the
related migratory YAML and classic configuration examples. Preserve the existing
use_put_object settings and formatting while ensuring each relevant example
documents the size explicitly.
- Around line 49-54: The S3 documentation must clarify that Arrow output uses
format: arrow, while compression: arrow is legacy syntax; update the compression
table in pipeline/outputs/s3.md lines 49-54 to document none/unset, and revise
pipeline/outputs/s3.md line 778 to show Parquet files are uncompressed when
format: parquet omits compression, with Arrow compression using zstd or unset
for no compression.
- Around line 194-199: Update the Parquet migration YAML example to include the
documented use_put_object: on setting alongside format: parquet and compression:
snappy, unless the surrounding documentation explicitly establishes that Parquet
enables it automatically.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 0818df98-a13a-4d95-b0be-3d1bad914208
📒 Files selected for processing (22)
.github/workflows/pr-example-validation.yamlinstallation/downloads/docker.mdpipeline/buffering.mdpipeline/filters/geoip2-filter.mdpipeline/filters/kubernetes.mdpipeline/filters/parser.mdpipeline/inputs/blob.mdpipeline/inputs/cpu-metrics.mdpipeline/inputs/kafka.mdpipeline/inputs/tail.mdpipeline/outputs/dynatrace.mdpipeline/outputs/gelf.mdpipeline/outputs/kafka.mdpipeline/outputs/loki.mdpipeline/outputs/s3.mdpipeline/parsers.mdpipeline/processors/conditional-processing.mdpipeline/processors/sql.mdpipeline/router.mdscripts/README.mdscripts/extract-config.shscripts/test-config.sh
💤 Files with no reviewable changes (2)
- pipeline/outputs/gelf.md
- pipeline/processors/conditional-processing.md
🚧 Files skipped from review as they are similar to previous changes (17)
- pipeline/filters/kubernetes.md
- installation/downloads/docker.md
- pipeline/inputs/blob.md
- pipeline/processors/sql.md
- pipeline/filters/geoip2-filter.md
- pipeline/buffering.md
- scripts/extract-config.sh
- pipeline/outputs/loki.md
- pipeline/inputs/cpu-metrics.md
- pipeline/parsers.md
- pipeline/filters/parser.md
- pipeline/inputs/tail.md
- pipeline/outputs/dynatrace.md
- pipeline/inputs/kafka.md
- pipeline/router.md
- scripts/test-config.sh
- pipeline/outputs/kafka.md
Cheers, yeah I've not run it on macOS and it's "special" shell so thanks for that. I'll re-run on Ubuntu to confirm all ok for CI and look to resolve that workflow issue which is outstanding. It may be we'll need to iterate a little after landing as testing workflows is unnecessarily difficult but using local scripts hopefully mitigates most of it. |
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pipeline/filters/multiline-stacktrace.md`:
- Around line 75-76: Enable the parsers_file setting in the example
configuration and the corresponding Parsers_File setting in the alternate
configuration, using parsers_multiline.yaml and parsers_multiline.conf
respectively, so multiline-regex-test resolves correctly.
In `@pipeline/filters/parser.md`:
- Around line 58-60: Activate the parser-file settings for the examples that
reference custom parsers: update pipeline/filters/parser.md lines 58-60,
pipeline/inputs/exec-wasi.md lines 70-71, pipeline/inputs/standard-input.md
lines 207-209, and pipeline/parsers/multiline-parsing.md lines 122-123 to use
valid paths in their YAML or classic configurations. Alternatively, remove the
custom parser references and related claims from each affected example.
In `@scripts/get-changed-files.sh`:
- Around line 47-52: Update the argument parsing and endpoint validation around
BASE_REF and HEAD_REF so a one-argument symmetric range is split into its two
revisions before git rev-parse validation. Preserve support for git diff’s A...B
form, validating each endpoint independently, or remove the documented
single-argument mode consistently.
In `@scripts/README.md`:
- Around line 110-111: Update the scripts README documentation to use
origin/master instead of origin/main for the base_ref default and every affected
explicit example, including the referenced sections, so all documented Git
defaults match the current scripts.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: b9cd93fa-5815-4cf2-88d8-a6d511e72057
📒 Files selected for processing (21)
.github/workflows/pr-example-validation.yamlCONTRIBUTING.mdadministration/configuring-fluent-bit/yaml/parsers-section.mdinstallation/downloads/docker.mdlocal-testing/validating-your-data-and-structure.mdpipeline/buffering.mdpipeline/filters/grep.mdpipeline/filters/multiline-stacktrace.mdpipeline/filters/parser.mdpipeline/inputs/exec-wasi.mdpipeline/inputs/standard-input.mdpipeline/inputs/syslog.mdpipeline/inputs/systemd.mdpipeline/inputs/tail.mdpipeline/outputs/kafka.mdpipeline/outputs/s3.mdpipeline/parsers/decoders.mdpipeline/parsers/multiline-parsing.mdscripts/README.mdscripts/get-changed-files.shscripts/validate-changed-files.sh
💤 Files with no reviewable changes (4)
- pipeline/filters/grep.md
- pipeline/inputs/systemd.md
- local-testing/validating-your-data-and-structure.md
- pipeline/inputs/syslog.md
🚧 Files skipped from review as they are similar to previous changes (5)
- pipeline/outputs/s3.md
- pipeline/inputs/tail.md
- pipeline/buffering.md
- installation/downloads/docker.md
- pipeline/outputs/kafka.md
Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
…ass (#2645) Add the powersupplyclass collector, mark filesystem as available on macOS, and note the TcpExt and IpExt counters exposed by netstat. Correct the metrics default, which was stale on Linux and undocumented on macOS, and mark the timex collector as unreleased pending fluent-bit#11718. Also, clarify per-OS metric sources. Note, update for code changes without corresponding docs PR. Signed-off-by: Eric D. Schabell <eric@schabell.org> Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
#2658) A rewrite_tag rule whose match pattern also matches the tags it emits creates a self-referential emitter. Fluent Bit now rejects that emission and logs an error once per emitter instead of recursing until the process crashes on pause or shutdown. Add an Avoid emitter cycles subsection covering the error message, the fact that the triggering record is kept under its original tag regardless of the rule's KEEP value, and how to fix the configuration. Signed-off-by: Eric D. Schabell <eric@schabell.org> Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
…2656) The http, splunk, elasticsearch, opentelemetry, and prometheus_remote_write inputs now implement pause and resume callbacks that pause the shared HTTP listener. Previously these plugins had no pause callback. - backpressure.md: add a section describing what clients observe while one of these inputs is paused. The listener stays open but each incoming connection is accepted and immediately closed, keep-alive is disabled, and in-flight requests are dropped, so senders must retry. - pipeline-section.md: cross-reference the new section from the shared HTTP listener settings. - backpressure.md: add alt text to the tracking pixel to clear a markdownlint MD045 warning on the page. Signed-off-by: Eric D. Schabell <eric@schabell.org> Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
- Add extensions to the list of supported top-level YAML sections. - Add an "extensions section" section describing it as a place to keep settings for tooling around Fluent Bit, which Fluent Bit parses and retains but the data pipeline never reads. - Note that extensions is the only section accepting nested maps, and that nesting elsewhere fails with "variant values are only valid in the extensions section". Note, update for code changes without corresponding docs PR. Signed-off-by: Eric D. Schabell <eric@schabell.org> Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
Two merged source changes alter existing behavior in ways users can hit on upgrade. - pipeline/outputs/s3.md: note in the log_key row that the record key name must match exactly, linking to the new section below. - pipeline/outputs/s3.md: add a "Key matching for log_key" section describing the prefix matching used before 5.1, where a record key that was a prefix of the configured value matched (log_key log_level selecting a log field instead), and the "Could not find log_key" error that identifies an affected configuration after upgrading. - administration/monitoring.md: document that a monitoring HTTP server that cannot bind now fails startup with "could not start HTTP server", where earlier versions continued running without the endpoint. Note, update for code changes without corresponding docs PR. Signed-off-by: Eric D. Schabell <eric@schabell.org> Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
…#2647) Document that Fluent Bit reloads TLS certificates from disk when it creates a new session, covering the files it watches, how it detects changes including inode swaps from Kubernetes secret updates, that established sessions keep their existing context, and what happens when a reload fails. Also clarify tls.ca_path reload detection. Note, updates for code changes without corresponding docs PR. Signed-off-by: Eric D. Schabell <eric@schabell.org> Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
…ias (#2648) Note that the es output plugin also loads under the name elasticsearch in Fluent Bit 5.1 and later, and that the alias applies only to output plugins so the elasticsearch input plugin is unaffected. Note, update for code changes without corresponding docs PR. Signed-off-by: Eric D. Schabell <eric@schabell.org> Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
) * docs: administration: document logs tag records telemetry metrics Document the service-level telemetry.metrics.logs.tag_records block, its per-input enabled override, and the two metrics it exposes: - service-section.md: add a Telemetry configuration section covering enabled, max_series, and max_tag_length, and note that the block is YAML-only with no dotted-key or classic-mode form. - pipeline-section.md: document the per-input enabled override and that the limits are service-level only. - monitoring.md: add fluentbit_input_logs_tag_records_total and fluentbit_input_logs_tag_records_untracked_total, including the max_series, tag_length_limit, and error reason labels. Signed-off-by: Eric D. Schabell <eric@schabell.org> * docs: administration: yaml: document tag_records limit validation rules The max_series and max_tag_length rows documented the 0 sentinel but not the accepted types or what happens on a bad value. - Note that both limits accept an integer or an integer-only string, that strings are environment-variable expanded, and that the value must fit in a signed 32-bit integer. - Document that unparsable or out-of-range values fail at startup. - Correct the sentinel: 0 or less removes the limit. Neither limit enforces a minimum, so negative values are accepted rather than rejected. Signed-off-by: Eric D. Schabell <eric@schabell.org> --------- Signed-off-by: Eric D. Schabell <eric@schabell.org> Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
Document record accessor support in the path and file parameters along with max_dynamic_files, on_missing_field, on_limit_reached, fallback_path, and fallback_file. Add a section covering the static prefix requirement, the mandatory fallback_file, destination safety validation, and the available actions. Note, this covers code merged without corresponding docs PR Signed-off-by: Eric D. Schabell <eric@schabell.org> Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
* docs: administration: http-proxy: document tls.proxy.* options Fluent Bit 5.1 adds tls.proxy.ca_file, tls.proxy.ca_path, tls.proxy.verify and tls.proxy.verify_hostname, which configure certificate verification for the proxy leg when an output connects through an HTTPS proxy (HTTP_PROXY using the https:// scheme), independent from the destination's own tls.* settings. - administration/http-proxy.md: explain the https:// proxy scheme and add a new "TLS to the proxy" section with the properties table and a config example. - administration/transport-security.md: add the same properties to the shared tls.* reference table. - pipeline/outputs/s3.md, opentelemetry.md, azure_kusto.md: add the properties to each page's own inline tls.* table. - administration/networking.md and the three plugin pages above: fix net.proxy_env_ignore's description, which referenced a non-existent HTTPS_PROXY environment variable instead of the https:// scheme inside HTTP_PROXY/http_proxy. Signed-off-by: Antônio Franco <13881523+antoniomrfranco@users.noreply.github.com> * docs: docs: administration: http-proxy: transport-security: fix formatting and lint - http-proxy: add missing lead-in sentence for the HTTP_PROXY example that followed the config tabs with no introduction. - http-proxy: use Title_Case Tls in the classic config example to match repo convention. Dotted keys such as tls.proxy.ca_file stay lowercase. - http-proxy: drop italics from the "Supported in v5.1 or later." version note so it matches transport-security, s3, opentelemetry, and azure_kusto. - transport-security: convert the tls.* properties table to compact style. The long tls.proxy.* rows broke the table's aligned style, adding 12 markdownlint MD060 errors. Aligning instead would require padding every row to over 330 characters. Also clears 12 pre-existing MD060 errors. Signed-off-by: Eric D. Schabell <eric@schabell.org> --------- Signed-off-by: Antônio Franco <13881523+antoniomrfranco@users.noreply.github.com> Signed-off-by: Eric D. Schabell <eric@schabell.org> Co-authored-by: Eric D. Schabell <eric@schabell.org> Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: edsiper <369718+edsiper@users.noreply.github.com> Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
…ge (#2666) * docs: installation: consolidate what's new pages into a rolling v5 page - Rename whats-new-in-fluent-bit-v5.0.md to whats-new-in-fluent-bit-v5.md and restructure as a rolling page covering the v5 line, newest release first - Add a v5.1 section covering multi-worker ingestion, input rate gate, FIPS mode, TLS cert reload, tls.proxy.* HTTPS proxy, DTLS syslog output, ETW input, NVIDIA/NVML GPU metrics, node exporter metrics additions, GCS output, Kafka Schema Registry, file output rotation, elasticsearch alias, Windows Server 2025/Nano Server images, and the 64-bit timestamp fix - Update SUMMARY.md to a single "What's new in Fluent Bit v5" entry - Add a .gitbook.yaml redirect from the old v5.0 page URL - Add a Fluent Bit v5.1 section to upgrade-notes.md covering the Debian/Ubuntu restart-on-upgrade behavior, syslog mode: tls/dtls, and the 64-bit timestamp fix - Fix markdownlint errors and dangling links from v5 consolidation Signed-off-by: Eric D. Schabell <eric@schabell.org> * docs: installation: whats-new-in-fluent-bit-v5: fix vale lint failures Reword sentences and headings flagged by Vale on PR-added lines (sentence length, directional wording, contractions, a topology/ placement false positive, and heading capitalization). Extend the shared spelling, acronym, and heading exception lists for legitimate technical terms (msgpack, NVML, MIG, DTLS, Nano/Windows Server, Confluent Schema Registry, codec/codecs) that the generic rules don't recognize. Signed-off-by: Eric D. Schabell <eric@schabell.org> --------- Signed-off-by: Eric D. Schabell <eric@schabell.org> Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
* docs: pipeline: outputs: syslog: document DTLS transport mode Replace the statement that DTLS isn't supported with the four available transport modes and their tls requirements, including the startup validation rules for mode=dtls and mode=udp. Note that mode=tls alone does not secure the connection, and add a DTLS configuration example. Also, clarified that TLS is not a transport. Note, covers code changes without corresponding docs PR. Signed-off-by: Eric D. Schabell <eric@schabell.org> * docs: docs: pipeline: outputs: syslog: correct TLS mode behavior Upstream commit 40641ddb (out_syslog: Handle TLS mode automatically) changed how mode=tls and mode=dtls interact with the tls parameter. Both modes now enable TLS themselves, and the mode=dtls requires tls=on startup check was removed. Update the docs to match: - Intro: list UDP, TCP, TLS, and DTLS as four transports, and drop the claim that TLS isn't a transport of its own. - Transport modes table: change the tls setting for tls and dtls from required to optional and enabled automatically, and drop "behaves the same as tcp" from the tls row. - Replace the paragraph stating that mode=tls doesn't secure the connection on its own with the automatic TLS behavior. - Startup validation: remove the mode=dtls requires tls=on bullet, which no longer exists in the source, and add a bullet for builds compiled without TLS support failing with "TLS support is unavailable". - DTLS example: drop the now-redundant tls: on from both the YAML and classic configuration tabs. Signed-off-by: Eric D. Schabell <eric@schabell.org> --------- Signed-off-by: Eric D. Schabell <eric@schabell.org> Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
The monthly link checker reported 3 errors, 1 timeout, and 27 redirects.
Fix the two genuine 404s, correct a citation that silently redirected to
the wrong API, resolve the redirects worth resolving, and stop LinkedIn
from generating a false error every month.
Broken links (404):
- pipeline/outputs/azure_kusto.md: repoint "Authorize the app in your
database" at learn.microsoft.com/en-us/kusto/management/reference-
security-principals. The old URL 404s, and Microsoft's own redirect
from it lands on another 404. The new page carries the
aadapp=ApplicationId;TenantId syntax the step refers to. Also fixed
broken Eventhouse link.
- pipeline/outputs/s3.md: drop the link on the `timer` callback. It
pointed at the AWS IoT Events Data API Timer data type, which is
unrelated to S3 or to Fluent Bit. The S3 upload timer is the internal
cb_s3_upload scheduler callback and has no external documentation, so
there is no correct replacement URL. Keep `timer` as inline code.
Stale citation (redirected to unrelated content):
- pipeline/outputs/bigquery.md: the data deduplication and template
tables bullets cited the legacy streaming page, which Google now
redirects to the Storage Write API. That page documents neither
insertId nor templateSuffix. Cite the tabledata.insertAll REST
reference instead, which documents both.
Redirects resolved:
- CONTRIBUTING.md: vale.sh/docs to docs.vale.sh, drop the trailing slash
on docs.fluentbit.io/manual, and update the Microsoft smart quotes
support URL.
- MAINTAINERS.md: telemetryforge.io to www.telemetryforge.io, and
re-pad the table cell so the pipe alignment still holds.
- README.md: drop the trailing slash on the LinkedIn profile URL.
- development/external-libraries.md, development/wasm-filter-plugins.md:
wasm-micro-runtime moved out of the bytecodealliance org to its own.
The project is still a Bytecode Alliance project, so only the URL
changes.
- pipeline/filters/tensorflow.md: ai.google.dev/edge/litert to
developers.google.com/edge/litert (2 links).
- pipeline/outputs/bigquery.md, pipeline/outputs/chronicle.md:
cloud.google.com to docs.cloud.google.com.
- pipeline/outputs/forward.md: fluentd.org to www.fluentd.org (3 links).
- pipeline/outputs/influxdb.md: add the trailing slash on the InfluxDB
product URL.
- pipeline/outputs/azure_kusto.md: update moved Eventhouse blog link.
Link checker configuration:
- .github/workflows/linkcheck.yaml: accept HTTP 999. LinkedIn returns it
to non-browser clients, so the README profile link is reported as an
error every run even though it resolves fine. Accepting 999 is
narrower than excluding linkedin.com and keeps the URL checked for
DNS and host failures.
Deliberately unchanged: the asciinema link in pipeline/inputs/http.md,
which timed out in CI but resolves normally; the Dynatrace shortlink,
where expanding it to the resolved deep path would be more fragile; the
three fluent-bit issue template links, which only redirect because the
checker is unauthenticated; and the RFC Editor, GitHub codeload, Datadog,
Treasure Data, Docker, and ECR redirects, which are correct as written.
Fixes #2652
Signed-off-by: Eric D. Schabell <eric@schabell.org>
Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
* docs: pipeline: outputs: kafka: document schema registry support
Document the schema_registry_* parameters for resolving Avro schemas
from a Confluent Schema Registry, including subject and ID based
resolution, basic and bearer authentication, multiple registry
endpoints with failover, and the accepted dotted key spellings.
Note, covers code merges without corresponding docs PR.
Signed-off-by: Eric D. Schabell <eric@schabell.org>
* docs: pipeline: outputs: kafka: note Avro encoder build requirement for schema registry
Signed-off-by: Eric D. Schabell <eric@schabell.org>
* docs: pipeline: outputs: kafka: fix schema registry doc accuracy and security
- Scope endpoint failover to the initial schema fetch only; the schema
is cached for the plugin's lifetime and the registry isn't re-contacted
- Use https:// in the authenticated Schema Registry example so basic
auth credentials aren't shown going out over plain HTTP
Signed-off-by: Eric D. Schabell <eric@schabell.org>
---------
Signed-off-by: Eric D. Schabell <eric@schabell.org>
Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
…nd Nano Server images (#2665) * docs: installation: downloads: docker: document Windows Server 2025 and Nano Server images - Replace stale Windows Server 2019/2022 claim with a tag table covering windows-2022, windows-2025, and windows-nano-2025 (new in 5.1) - Note Nano Server's PowerShell/exec-plugin limitations and its default Forward-to-stdout configuration - Drop the Windows Server 2019 claim, deprecated upstream well before this release and no longer built Signed-off-by: Eric D. Schabell <eric@schabell.org> * docs: vale: add Nano to spelling exceptions Fixes CI failure on PR #2665 where 'Nano' (Windows Nano Server) was flagged by FluentBit.Spelling on added lines in docker.md. Signed-off-by: Eric D. Schabell <eric@schabell.org> --------- Signed-off-by: Eric D. Schabell <eric@schabell.org> Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
…2664) - Add rate_gate, rate_gate.backpressure, rate_gate.max_bytes, rate_gate.max_records, rate_gate.resume_ratio, and rate_window to the per-input settings table (new in 5.1) - Add a YAML/classic configuration example enabling the rate gate on an input plugin - Clarify rate_gate resume behavior for both limits Signed-off-by: Eric D. Schabell <eric@schabell.org> Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
…eleased in 5.1 (#2663) - Update timex collector version from "Unreleased" to "5.1" - Remove dead "Collectors marked Unreleased" caveat paragraph (no other collector carries that flag) - Add timex to the Linux default metrics list and note its adjtimex(2) dependency Note, update for code changes without corresponding docs PR. Signed-off-by: Eric D. Schabell <eric@schabell.org> Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
Signed-off-by: Patrick Stephens <pat@telemetryforge.io>
0ca5d8f to
aba5558
Compare
Signed-off-by: Pat <pat@fluent.do>
Resolves #2459 by providing a simple AWK based approach to validating configuration examples:
--dry-runthe configuration and report success/failureA local script is provided that can be run for any file in the repo or all of them:
A workflow is provided to run this for any files changed in a PR so if an update is made to documentation it should check it is valid.
There is a basic suppression approach by file as there are some valid reasons for this:
execare not part of the container imageAs part of these changes we also found failures in existing files that were resolved.
Some tweaks were also required, e.g. parser definition must be in a separate file for legacy TOML config so it was updated to be a comment for the examples (which would be rejected anyway otherwise).
There are options to use something more complex like markdown-tree or similar to build an AST from the Markdown file to then pull out the bits we need but this may not work with the specific Gitbook format anyway and requires a whole load of extra dependencies.
Currently there is an upstream failure with certain plugins triggering a segmentation fault for
--dry-run: fluent/fluent-bit#12113This is resolved by plugins: fix dry-run segmentation faults fluent-bit#12114 so waiting on that to merge.
Summary by CodeRabbit
Documentation
Validation