feat: add config example validation script and workflow

.github/workflows/pr-example-validation.yaml

@@ -0,0 +1,31 @@
+name: Validate example configurations in PRs
+
+on:
+  pull_request:
+    paths:
+      - '**/*.md'
+
+permissions: {}
+
+# Avoid multiple commits on the same PR racing to update the comment.
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
+  cancel-in-progress: true
+
+jobs:
+  validate-configs:
+    name: Validate example configurations in changed Markdown files
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - name: Check out code
+        uses: actions/checkout@v7
+        with:
+          fetch-depth: 0
+          persist-credentials: false
+
+      - name: Validate example configurations in changed files
+        run: |
+          ./scripts/validate-changed-files.sh "${{ github.event.pull_request.base.sha }}" HEAD
+        shell: bash

CONTRIBUTING.md

@@ -141,6 +141,10 @@ Every plugin page should include both a YAML and a classic configuration example
 {% endtabs %}
 ```
 
+We have some checks in CI to validate these examples so ensure the `scripts/validate-changed-files.sh` script passes for your branch vs the origin.
+This uses the Fluent Bit linux container so may not always pass, e.g. for Windows plugins.
+Suppressions can be defined in the `scripts/test-config.sh` script with a comment explaining why.
+
 ### Key casing in examples
 
 Key casing differs between the two configuration formats:

administration/configuring-fluent-bit/yaml/parsers-section.md

@@ -75,7 +75,7 @@ parsers:
 1. Update the `parsers_file` parameter in the `service` section of your YAML configuration file:
 
 {% tabs %}
-{% tab title="fluent-bit.yaml" %}
+{% tab title="custom-parsers-fluent-bit.yaml" %}
 
 ```yaml
 service:

installation/downloads/docker.md

@@ -15,7 +15,7 @@ docker run -ti cr.fluentbit.io/fluent/fluent-bit
 Use the following command to start Fluent Bit while using a configuration file:
 
 {% tabs %}
-{% tab title="fluent-bit.conf" %}
+{% tab title="Legacy Configuration: fluent-bit.conf" %}
 
 ```shell
 docker run -ti -v ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf \
@@ -24,7 +24,7 @@ docker run -ti -v ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf \
 
 {% endtab %}
 
-{% tab title="fluent-bit.yaml" %}
+{% tab title="YAML configuration: fluent-bit.yaml" %}
 
 ```shell
 docker run -ti -v ./fluent-bit.yaml:/fluent-bit/etc/fluent-bit.yaml \
@@ -317,7 +317,7 @@ The reasons for using distroless are well covered in
 - Reduces false positives on scans (and reduces resources required for scanning).
 - Reduces supply chain security requirements to only what you need.
 - Helps prevent unauthorised processes or users interacting with the container.
-- Less need to harden the container (and container runtime, K8s, and so on).
+- Less need to harden the container (and container runtime, Kubernetes, and so on).
 - Faster CI/CD processes.
 
 With any choice, there are downsides:

local-testing/validating-your-data-and-structure.md

@@ -83,7 +83,6 @@ The following is the Fluent Bit YAML configuration file:
 service:
   flush: 1
   log_level: info
-  parsers_file: parsers.yaml
 
 pipeline:
   inputs:
@@ -128,7 +127,6 @@ The following is the Fluent Bit classic configuration file:
 [SERVICE]
     flush        1
     log_level    info
-    parsers_file parsers.conf
 
 [INPUT]
     name        tail
@@ -177,7 +175,6 @@ The following is the Fluent Bit YAML configuration file:
 service:
   flush: 1
   log_level: info
-  parsers_file: parsers.yaml
 
 pipeline:
   inputs:
@@ -230,7 +227,6 @@ pipeline:
 [SERVICE]
     flush        1
     log_level    info
-    parsers_file parsers.conf
 
 [INPUT]
     name         tail

pipeline/buffering.md

@@ -102,7 +102,8 @@ pipeline:
       # Inherits storage.type: filesystem from service
 
     - name: mem
-      storage.type: memory  # Overrides the inherited default
+      # Overrides the inherited default
+      storage.type: memory
 ```
 
 {% endtab %}
@@ -124,7 +125,8 @@ pipeline:
 
 [INPUT]
   Name          mem
-  Storage.Type  memory  # Overrides the inherited default
+  # Overrides the inherited default
+  Storage.Type  memory
 ```
 
 {% endtab %}

pipeline/filters/geoip2-filter.md

@@ -33,7 +33,7 @@ The following configuration processes the incoming `remote_addr` and appends cou
 pipeline:
   inputs:
     - name: dummy
-      dummy: {"remote_addr": "8.8.8.8"}
+      dummy: '{"remote_addr": "8.8.8.8"}'
 
   filters:
     - name: geoip2

pipeline/filters/grep.md

@@ -55,9 +55,6 @@ fluent-bit -i tail -p 'path=lines.txt' -F grep -p 'regex=log aa' -m '*' -o stdou
 {% tab title="fluent-bit.yaml" %}
 
 ```yaml
-service:
-  parsers_file: /path/to/parsers.conf
-
 pipeline:
   inputs:
     - name: tail
@@ -78,9 +75,6 @@ pipeline:
 {% tab title="fluent-bit.conf" %}
 
 ```text
-[SERVICE]
-  Parsers_File /path/to/parsers.conf
-
 [INPUT]
   Name   tail
   Path   lines.txt

pipeline/filters/kubernetes.md

@@ -360,10 +360,11 @@ pipeline:
 {% tab title="fluent-bit.conf" %}
 
 ```text
-[PARSER]
-  Name    custom-tag
-  Format  regex
-  Regex   ^(?<namespace_name>[^_]+)\.(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)\.(?<container_name>.+)\.(?<container_id>[a-z0-9]{64})
+# Ensure this is added to parsers_file loaded in the SERVICE section
+# [PARSER]
+#   Name    custom-tag
+#   Format  regex
+#   Regex   ^(?<namespace_name>[^_]+)\.(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)\.(?<container_name>.+)\.(?<container_id>[a-z0-9]{64})
 
 [INPUT]
   Name              tail
@@ -488,7 +489,7 @@ pipeline:
       parser: docker
       docker_mode: on
       mem_buf_limit: 50MB
-      skip_login_lines: on
+      skip_long_lines: on
       refresh_interval: 10
 
   filters:

pipeline/filters/multiline-stacktrace.md

@@ -72,7 +72,8 @@ This is the primary Fluent Bit YAML configuration file. It includes the `parsers
 service:
   flush: 1
   log_level: info
-  parsers_file: parsers_multiline.yaml
+  # Configure this to load your custom parser if required
+  # parsers_file: parsers_multiline.yaml
 
 pipeline:
   inputs:
@@ -100,7 +101,8 @@ This is the primary Fluent Bit classic configuration file. It includes the `pars
 [SERVICE]
   Flush                 1
   Log_Level             info
-  Parsers_File          parsers_multiline.conf
+  # Configure this to load your custom parser if required
+  # Parsers_File          parsers_multiline.conf
 
 [INPUT]
   Name                  tail

pipeline/filters/parser.md

@@ -16,7 +16,7 @@ The plugin supports the following configuration parameters:
 | `parser` | Specify the parser name to interpret the field. Multiple parser entries are allowed (one per line). | _none_ |
 | `preserve_key` | Keep the original `key_name` field in the parsed result. If false, the field will be removed. | `false` |
 | `reserve_data` | Keep all other original fields in the parsed result. If false, all other original fields will be removed. | `false` |
-| `Unescape_key` | Deprecated. This option is retained only for backward compatibility and should not be used in new configurations. | _deprecated_ |
+| `Unescape_key` | Deprecated. This option is retained only for backward compatibility and shouldn't be used in new configurations. | _deprecated_ |
 
 ## Get started
 
@@ -27,7 +27,7 @@ The plugin needs a parser file which defines how to parse each field.
 This is an example of parsing a record `{"data":"100 0.5 true This is example"}`.
 
 {% tabs %}
-{% tab title="fluent-bit.yaml" %}
+{% tab title="parser.yaml" %}
 
 ```yaml
 parsers:
@@ -37,7 +37,7 @@ parsers:
 ```
 
 {% endtab %}
-{% tab title="fluent-bit.conf" %}
+{% tab title="parser.conf" %}
 
 ```text
 [PARSER]
@@ -55,8 +55,9 @@ The path of the parser file should be written in configuration file under the `[
 {% tab title="fluent-bit.yaml" %}
 
 ```yaml
-service:
-  parsers_file: /path/to/parsers.yaml
+# service:
+  # Provide the actual path here
+  # parsers_file: /path/to/parsers.yaml
 
 pipeline:
   inputs:
@@ -80,7 +81,7 @@ pipeline:
 
 ```text
 [SERVICE]
-  Parsers_File /path/to/parsers.conf
+  # Parsers_File /path/to/parsers.conf
 
 [INPUT]
   Name dummy
@@ -154,8 +155,9 @@ Now add `reserve_data` to the filter section of the corresponding configuration
 {% tab title="fluent-bit.yaml" %}
 
 ```yaml
-service:
-  parsers_file: /path/to/parsers.yaml
+# service:
+  # Provide the actual path here
+  # parsers_file: /path/to/parsers.yaml
 
 pipeline:
   inputs:
@@ -180,7 +182,7 @@ pipeline:
 
 ```text
 [SERVICE]
-  Parsers_File /path/to/parsers.conf
+  # Parsers_File /path/to/parsers.conf
 
 [INPUT]
   Name dummy
@@ -248,8 +250,9 @@ Now add `reserve_data` and `preserve_key` to the filter section of the correspon
 {% tab title="fluent-bit.yaml" %}
 
 ```yaml
-service:
-  parsers_file: /path/to/parsers.yaml
+# service:
+  # Provide the actual path here
+  # parsers_file: /path/to/parsers.yaml
 
 pipeline:
   inputs:
@@ -275,7 +278,7 @@ pipeline:
 
 ```text
 [SERVICE]
-  Parsers_File /path/to/parsers.conf
+  # Parsers_File /path/to/parsers.conf
 
 [INPUT]
   Name dummy

pipeline/inputs/blob.md

@@ -177,8 +177,8 @@ This example excludes certain file patterns and uses filesystem storage for bett
 pipeline:
   inputs:
     - name: blob
-      path: /data/artifacts/**/*
-      exclude_pattern: *.tmp,*.bak,*.old
+      path: '/data/artifacts/**/*'
+      exclude_pattern: '*.tmp,*.bak,*.old'
       storage.type: filesystem
       storage.pause_on_chunks_overlimit: true
       mem_buf_limit: 50M

pipeline/inputs/cpu-metrics.md

@@ -82,7 +82,7 @@ pipeline:
 {% endtab %}
 {% tab title="fluent-bit.conf" %}
 
-```shell
+```text
 [INPUT]
   Name cpu
   Tag  my_cpu

pipeline/inputs/exec-wasi.md

@@ -67,7 +67,8 @@ Then, you can specify the `parsers.conf` in the main Fluent Bit configuration:
 service:
   flush: 1
   daemon: off
-  parsers_file: parsers.yaml
+  # Configure this to load your custom parser if required
+  # parsers_file: parsers.yaml
   log_level: info
   http_server: off
   http_listen: 0.0.0.0
@@ -94,7 +95,8 @@ pipeline:
 [SERVICE]
   Flush        1
   Daemon       Off
-  Parsers_File parsers.conf
+  # Configure this to load your custom parser if required
+  # Parsers_File parsers.conf
   Log_Level    info
   HTTP_Server  Off
   HTTP_Listen  0.0.0.0

pipeline/inputs/kafka.md

@@ -95,8 +95,16 @@ pipeline:
   filters:
     - name: lua
       match: '*'
-      script: kafka.lua
       call: modify_kafka_message
+      code: |
+        local count = 0
+        function modify_kafka_message(tag, timestamp, record)
+            count = count + 1
+            local payload = record.payload
+            payload.topic = record.topic
+            payload.status = 'processed by fluent-bit, total records: '..tostring(count)
+            return 1, timestamp, payload
+        end
 
   outputs:
     - name: kafka
@@ -118,8 +126,8 @@ pipeline:
 [FILTER]
   Name    lua
   Match   *
-  Script  kafka.lua
   Call    modify_kafka_message
+  Code    local count = 0 function modify_kafka_message(tag, timestamp, record) count = count + 1 local payload = record.payload payload.topic = record.topic payload.status = 'processed by fluent-bit, total records: '..tostring(count) return 1, timestamp, payload end
 
 [OUTPUT]
   Name    kafka
@@ -134,7 +142,18 @@ The previous example will connect to the broker listening on `kafka-broker:9092`
 
 Since the payload will be in JSON format, the plugin is configured to parse the payload with `format json`.
 
-Every message received is then processed with `kafka.lua` and sent back to the `fb-sink` topic of the same broker.
+Every message received is then processed with `kafka.lua` and sent back to the `fb-sink` topic of the same broker, this is shown inline for each configuration but is as follows:
+
+```lua
+local count = 0
+function modify_kafka_message(tag, timestamp, record)
+    count = count + 1
+    local payload = record.payload
+    payload.topic = record.topic
+    payload.status = 'processed by fluent-bit, total records: '..tostring(count)
+    return 1, timestamp, payload
+end
+```
 
 The example can be executed locally with `make start` in the `examples/kafka_filter` directory (`docker/compose` is used).