diff --git a/.github/workflows/pr-example-validation.yaml b/.github/workflows/pr-example-validation.yaml new file mode 100644 index 000000000..76ed67ab7 --- /dev/null +++ b/.github/workflows/pr-example-validation.yaml @@ -0,0 +1,31 @@ +name: Validate example configurations in PRs + +on: + pull_request: + paths: + - '**/*.md' + +permissions: {} + +# Avoid multiple commits on the same PR racing to update the comment. +concurrency: + group: ${{ github.workflow }}-${{ github.event.pull_request.number }} + cancel-in-progress: true + +jobs: + validate-configs: + name: Validate example configurations in changed Markdown files + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - name: Check out code + uses: actions/checkout@v7 + with: + fetch-depth: 0 + persist-credentials: false + + - name: Validate example configurations in changed files + run: | + ./scripts/validate-changed-files.sh "${{ github.event.pull_request.base.sha }}" HEAD + shell: bash diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ba69b99c7..665f58bb1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -141,6 +141,10 @@ Every plugin page should include both a YAML and a classic configuration example {% endtabs %} ``` +We have some checks in CI to validate these examples so ensure the `scripts/validate-changed-files.sh` script passes for your branch vs the origin. +This uses the Fluent Bit linux container so may not always pass, e.g. for Windows plugins. +Suppressions can be defined in the `scripts/test-config.sh` script with a comment explaining why. + ### Key casing in examples Key casing differs between the two configuration formats: diff --git a/administration/configuring-fluent-bit/yaml/parsers-section.md b/administration/configuring-fluent-bit/yaml/parsers-section.md index 4aec29143..71e0bd6c1 100644 --- a/administration/configuring-fluent-bit/yaml/parsers-section.md +++ b/administration/configuring-fluent-bit/yaml/parsers-section.md @@ -75,7 +75,7 @@ parsers: 1. Update the `parsers_file` parameter in the `service` section of your YAML configuration file: {% tabs %} -{% tab title="fluent-bit.yaml" %} +{% tab title="custom-parsers-fluent-bit.yaml" %} ```yaml service: diff --git a/installation/downloads/docker.md b/installation/downloads/docker.md index 9a837d7f2..99caaa96e 100644 --- a/installation/downloads/docker.md +++ b/installation/downloads/docker.md @@ -15,7 +15,7 @@ docker run -ti cr.fluentbit.io/fluent/fluent-bit Use the following command to start Fluent Bit while using a configuration file: {% tabs %} -{% tab title="fluent-bit.conf" %} +{% tab title="Legacy Configuration: fluent-bit.conf" %} ```shell docker run -ti -v ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf \ @@ -24,7 +24,7 @@ docker run -ti -v ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf \ {% endtab %} -{% tab title="fluent-bit.yaml" %} +{% tab title="YAML configuration: fluent-bit.yaml" %} ```shell docker run -ti -v ./fluent-bit.yaml:/fluent-bit/etc/fluent-bit.yaml \ @@ -317,7 +317,7 @@ The reasons for using distroless are well covered in - Reduces false positives on scans (and reduces resources required for scanning). - Reduces supply chain security requirements to only what you need. - Helps prevent unauthorised processes or users interacting with the container. -- Less need to harden the container (and container runtime, K8s, and so on). +- Less need to harden the container (and container runtime, Kubernetes, and so on). - Faster CI/CD processes. With any choice, there are downsides: diff --git a/local-testing/validating-your-data-and-structure.md b/local-testing/validating-your-data-and-structure.md index 81c1132cc..e61b2f0ff 100644 --- a/local-testing/validating-your-data-and-structure.md +++ b/local-testing/validating-your-data-and-structure.md @@ -83,7 +83,6 @@ The following is the Fluent Bit YAML configuration file: service: flush: 1 log_level: info - parsers_file: parsers.yaml pipeline: inputs: @@ -128,7 +127,6 @@ The following is the Fluent Bit classic configuration file: [SERVICE] flush 1 log_level info - parsers_file parsers.conf [INPUT] name tail @@ -177,7 +175,6 @@ The following is the Fluent Bit YAML configuration file: service: flush: 1 log_level: info - parsers_file: parsers.yaml pipeline: inputs: @@ -230,7 +227,6 @@ pipeline: [SERVICE] flush 1 log_level info - parsers_file parsers.conf [INPUT] name tail diff --git a/pipeline/buffering.md b/pipeline/buffering.md index 1796bbb7c..87637a9ed 100644 --- a/pipeline/buffering.md +++ b/pipeline/buffering.md @@ -102,7 +102,8 @@ pipeline: # Inherits storage.type: filesystem from service - name: mem - storage.type: memory # Overrides the inherited default + # Overrides the inherited default + storage.type: memory ``` {% endtab %} @@ -124,7 +125,8 @@ pipeline: [INPUT] Name mem - Storage.Type memory # Overrides the inherited default + # Overrides the inherited default + Storage.Type memory ``` {% endtab %} diff --git a/pipeline/filters/geoip2-filter.md b/pipeline/filters/geoip2-filter.md index bbd981580..5e4f969c0 100644 --- a/pipeline/filters/geoip2-filter.md +++ b/pipeline/filters/geoip2-filter.md @@ -33,7 +33,7 @@ The following configuration processes the incoming `remote_addr` and appends cou pipeline: inputs: - name: dummy - dummy: {"remote_addr": "8.8.8.8"} + dummy: '{"remote_addr": "8.8.8.8"}' filters: - name: geoip2 diff --git a/pipeline/filters/grep.md b/pipeline/filters/grep.md index 1232efc63..2d8ff2376 100644 --- a/pipeline/filters/grep.md +++ b/pipeline/filters/grep.md @@ -55,9 +55,6 @@ fluent-bit -i tail -p 'path=lines.txt' -F grep -p 'regex=log aa' -m '*' -o stdou {% tab title="fluent-bit.yaml" %} ```yaml -service: - parsers_file: /path/to/parsers.conf - pipeline: inputs: - name: tail @@ -78,9 +75,6 @@ pipeline: {% tab title="fluent-bit.conf" %} ```text -[SERVICE] - Parsers_File /path/to/parsers.conf - [INPUT] Name tail Path lines.txt diff --git a/pipeline/filters/kubernetes.md b/pipeline/filters/kubernetes.md index 028952caf..5501e90fc 100644 --- a/pipeline/filters/kubernetes.md +++ b/pipeline/filters/kubernetes.md @@ -360,10 +360,11 @@ pipeline: {% tab title="fluent-bit.conf" %} ```text -[PARSER] - Name custom-tag - Format regex - Regex ^(?[^_]+)\.(?[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)\.(?.+)\.(?[a-z0-9]{64}) +# Ensure this is added to parsers_file loaded in the SERVICE section +# [PARSER] +# Name custom-tag +# Format regex +# Regex ^(?[^_]+)\.(?[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)\.(?.+)\.(?[a-z0-9]{64}) [INPUT] Name tail @@ -488,7 +489,7 @@ pipeline: parser: docker docker_mode: on mem_buf_limit: 50MB - skip_login_lines: on + skip_long_lines: on refresh_interval: 10 filters: diff --git a/pipeline/filters/multiline-stacktrace.md b/pipeline/filters/multiline-stacktrace.md index d45324a20..a4c4cd9d9 100644 --- a/pipeline/filters/multiline-stacktrace.md +++ b/pipeline/filters/multiline-stacktrace.md @@ -72,7 +72,8 @@ This is the primary Fluent Bit YAML configuration file. It includes the `parsers service: flush: 1 log_level: info - parsers_file: parsers_multiline.yaml + # Configure this to load your custom parser if required + # parsers_file: parsers_multiline.yaml pipeline: inputs: @@ -100,7 +101,8 @@ This is the primary Fluent Bit classic configuration file. It includes the `pars [SERVICE] Flush 1 Log_Level info - Parsers_File parsers_multiline.conf + # Configure this to load your custom parser if required + # Parsers_File parsers_multiline.conf [INPUT] Name tail diff --git a/pipeline/filters/parser.md b/pipeline/filters/parser.md index 1db516612..d47bd196a 100644 --- a/pipeline/filters/parser.md +++ b/pipeline/filters/parser.md @@ -16,7 +16,7 @@ The plugin supports the following configuration parameters: | `parser` | Specify the parser name to interpret the field. Multiple parser entries are allowed (one per line). | _none_ | | `preserve_key` | Keep the original `key_name` field in the parsed result. If false, the field will be removed. | `false` | | `reserve_data` | Keep all other original fields in the parsed result. If false, all other original fields will be removed. | `false` | -| `Unescape_key` | Deprecated. This option is retained only for backward compatibility and should not be used in new configurations. | _deprecated_ | +| `Unescape_key` | Deprecated. This option is retained only for backward compatibility and shouldn't be used in new configurations. | _deprecated_ | ## Get started @@ -27,7 +27,7 @@ The plugin needs a parser file which defines how to parse each field. This is an example of parsing a record `{"data":"100 0.5 true This is example"}`. {% tabs %} -{% tab title="fluent-bit.yaml" %} +{% tab title="parser.yaml" %} ```yaml parsers: @@ -37,7 +37,7 @@ parsers: ``` {% endtab %} -{% tab title="fluent-bit.conf" %} +{% tab title="parser.conf" %} ```text [PARSER] @@ -55,8 +55,9 @@ The path of the parser file should be written in configuration file under the `[ {% tab title="fluent-bit.yaml" %} ```yaml -service: - parsers_file: /path/to/parsers.yaml +# service: + # Provide the actual path here + # parsers_file: /path/to/parsers.yaml pipeline: inputs: @@ -80,7 +81,7 @@ pipeline: ```text [SERVICE] - Parsers_File /path/to/parsers.conf + # Parsers_File /path/to/parsers.conf [INPUT] Name dummy @@ -154,8 +155,9 @@ Now add `reserve_data` to the filter section of the corresponding configuration {% tab title="fluent-bit.yaml" %} ```yaml -service: - parsers_file: /path/to/parsers.yaml +# service: + # Provide the actual path here + # parsers_file: /path/to/parsers.yaml pipeline: inputs: @@ -180,7 +182,7 @@ pipeline: ```text [SERVICE] - Parsers_File /path/to/parsers.conf + # Parsers_File /path/to/parsers.conf [INPUT] Name dummy @@ -248,8 +250,9 @@ Now add `reserve_data` and `preserve_key` to the filter section of the correspon {% tab title="fluent-bit.yaml" %} ```yaml -service: - parsers_file: /path/to/parsers.yaml +# service: + # Provide the actual path here + # parsers_file: /path/to/parsers.yaml pipeline: inputs: @@ -275,7 +278,7 @@ pipeline: ```text [SERVICE] - Parsers_File /path/to/parsers.conf + # Parsers_File /path/to/parsers.conf [INPUT] Name dummy diff --git a/pipeline/inputs/blob.md b/pipeline/inputs/blob.md index aa393a36f..d02704ca2 100644 --- a/pipeline/inputs/blob.md +++ b/pipeline/inputs/blob.md @@ -177,8 +177,8 @@ This example excludes certain file patterns and uses filesystem storage for bett pipeline: inputs: - name: blob - path: /data/artifacts/**/* - exclude_pattern: *.tmp,*.bak,*.old + path: '/data/artifacts/**/*' + exclude_pattern: '*.tmp,*.bak,*.old' storage.type: filesystem storage.pause_on_chunks_overlimit: true mem_buf_limit: 50M diff --git a/pipeline/inputs/cpu-metrics.md b/pipeline/inputs/cpu-metrics.md index 7546e7f5b..669ab7702 100644 --- a/pipeline/inputs/cpu-metrics.md +++ b/pipeline/inputs/cpu-metrics.md @@ -82,7 +82,7 @@ pipeline: {% endtab %} {% tab title="fluent-bit.conf" %} -```shell +```text [INPUT] Name cpu Tag my_cpu diff --git a/pipeline/inputs/exec-wasi.md b/pipeline/inputs/exec-wasi.md index 1af674f36..b0e0de697 100644 --- a/pipeline/inputs/exec-wasi.md +++ b/pipeline/inputs/exec-wasi.md @@ -67,7 +67,8 @@ Then, you can specify the `parsers.conf` in the main Fluent Bit configuration: service: flush: 1 daemon: off - parsers_file: parsers.yaml + # Configure this to load your custom parser if required + # parsers_file: parsers.yaml log_level: info http_server: off http_listen: 0.0.0.0 @@ -94,7 +95,8 @@ pipeline: [SERVICE] Flush 1 Daemon Off - Parsers_File parsers.conf + # Configure this to load your custom parser if required + # Parsers_File parsers.conf Log_Level info HTTP_Server Off HTTP_Listen 0.0.0.0 diff --git a/pipeline/inputs/kafka.md b/pipeline/inputs/kafka.md index 9b460ec5f..404d06e9c 100644 --- a/pipeline/inputs/kafka.md +++ b/pipeline/inputs/kafka.md @@ -95,8 +95,16 @@ pipeline: filters: - name: lua match: '*' - script: kafka.lua call: modify_kafka_message + code: | + local count = 0 + function modify_kafka_message(tag, timestamp, record) + count = count + 1 + local payload = record.payload + payload.topic = record.topic + payload.status = 'processed by fluent-bit, total records: '..tostring(count) + return 1, timestamp, payload + end outputs: - name: kafka @@ -118,8 +126,8 @@ pipeline: [FILTER] Name lua Match * - Script kafka.lua Call modify_kafka_message + Code local count = 0 function modify_kafka_message(tag, timestamp, record) count = count + 1 local payload = record.payload payload.topic = record.topic payload.status = 'processed by fluent-bit, total records: '..tostring(count) return 1, timestamp, payload end [OUTPUT] Name kafka @@ -134,7 +142,18 @@ The previous example will connect to the broker listening on `kafka-broker:9092` Since the payload will be in JSON format, the plugin is configured to parse the payload with `format json`. -Every message received is then processed with `kafka.lua` and sent back to the `fb-sink` topic of the same broker. +Every message received is then processed with `kafka.lua` and sent back to the `fb-sink` topic of the same broker, this is shown inline for each configuration but is as follows: + +```lua +local count = 0 +function modify_kafka_message(tag, timestamp, record) + count = count + 1 + local payload = record.payload + payload.topic = record.topic + payload.status = 'processed by fluent-bit, total records: '..tostring(count) + return 1, timestamp, payload +end +``` The example can be executed locally with `make start` in the `examples/kafka_filter` directory (`docker/compose` is used). diff --git a/pipeline/inputs/standard-input.md b/pipeline/inputs/standard-input.md index a8bf2ca1d..ec267ff28 100644 --- a/pipeline/inputs/standard-input.md +++ b/pipeline/inputs/standard-input.md @@ -204,8 +204,9 @@ You can then use the parsers file in a `stdin` plugin in the main Fluent Bit con {% tab title="fluent-bit.yaml" %} ```yaml -service: - parsers_file: parsers.yaml +# service: + # Configure this to load your custom parser if required + # parsers_file: parsers.yaml pipeline: inputs: @@ -223,7 +224,8 @@ pipeline: ```text [SERVICE] - Parsers_File parsers.conf + # Configure this to load your custom parser if required + # Parsers_File parsers.conf [INPUT] Name stdin diff --git a/pipeline/inputs/syslog.md b/pipeline/inputs/syslog.md index e5faa5cb6..b29139468 100644 --- a/pipeline/inputs/syslog.md +++ b/pipeline/inputs/syslog.md @@ -60,7 +60,6 @@ In your main configuration file append the following sections: service: flush: 1 log_level: info - parsers_file: parsers.yaml pipeline: inputs: @@ -82,7 +81,6 @@ pipeline: [SERVICE] Flush 1 Log_Level info - Parsers_File parsers.conf [INPUT] Name syslog @@ -141,7 +139,6 @@ Put the following content in your configuration file: ```yaml service: flush: 1 - parsers_file: parsers.yaml pipeline: inputs: @@ -162,7 +159,6 @@ pipeline: ```text [SERVICE] Flush 1 - Parsers_File parsers.conf [INPUT] Name syslog @@ -207,7 +203,6 @@ Put the following content in your Fluent Bit configuration: ```yaml service: flush: 1 - parsers_file: parsers.yaml pipeline: inputs: @@ -228,7 +223,6 @@ pipeline: ```text [SERVICE] Flush 1 - Parsers_File parsers.conf [INPUT] Name syslog diff --git a/pipeline/inputs/systemd.md b/pipeline/inputs/systemd.md index 783db3420..0e521442b 100644 --- a/pipeline/inputs/systemd.md +++ b/pipeline/inputs/systemd.md @@ -53,7 +53,6 @@ In your main configuration file append the following sections: service: flush: 1 log_level: info - parsers_file: parsers.yaml pipeline: inputs: @@ -72,7 +71,6 @@ pipeline: [SERVICE] Flush 1 Log_Level info - Parsers_File parsers.conf [INPUT] Name systemd diff --git a/pipeline/inputs/tail.md b/pipeline/inputs/tail.md index ddb9ab636..6118106fe 100644 --- a/pipeline/inputs/tail.md +++ b/pipeline/inputs/tail.md @@ -168,7 +168,7 @@ systemctl edit fluent-bit.service When the Linux inotify event queue overflows (indicated by an `IN_Q_OVERFLOW` event), Fluent Bit automatically reconciles all monitored files. During reconciliation it: -- Detects rotated files by comparing inodes and file names and re-registers watches as needed. +- Detects rotated files by comparing `inode` numbers and file names and re-registers watches as needed. - Resets offset tracking for files that have been truncated. - Retries watch registration for files that previously failed, for example due to `ENOSPC` when `fs.inotify.max_user_watches` is exhausted. @@ -273,8 +273,8 @@ pipeline: path: /var/log/syslog outputs: - - stdout: - match: * + - name: stdout + match: '*' ``` {% endtab %} @@ -369,13 +369,13 @@ pipeline: {% tab title="fluent-bit.conf" %} ```text -# Note this is generally added to parsers.conf and referenced in [SERVICE] -[PARSER] - Name multiline - Format regex - Regex /(?