From 3bfd06cdfaf88f7d3589b10a32e591cb9bc91b85 Mon Sep 17 00:00:00 2001 From: Alexa Kreizinger Date: Mon, 7 Jul 2025 16:56:41 -0700 Subject: [PATCH 1/3] pipeline: parsers: decoders: general cleanup Signed-off-by: Alexa Kreizinger --- pipeline/parsers/decoders.md | 39 ++++++++++++++---------------------- 1 file changed, 15 insertions(+), 24 deletions(-) diff --git a/pipeline/parsers/decoders.md b/pipeline/parsers/decoders.md index 585f1c0ea..8f7114a43 100644 --- a/pipeline/parsers/decoders.md +++ b/pipeline/parsers/decoders.md @@ -1,8 +1,6 @@ # Decoders -There are cases where the log messages being parsed contain encoded data. A typical -use case can be found in containerized environments with Docker. Docker logs its -data in JSON format, which uses escaped strings. +There are cases where the log messages that you want to parse contain encoded data. A typical use case can be found in containerized environments with Docker. Docker logs its data in JSON format, which uses escaped strings. Consider the following message generated by the application: @@ -16,20 +14,18 @@ The Docker log message encapsulates something like this: {"log":"{\"status\": \"up and running\"}\r\n","stream":"stdout","time":"2018-03-09T01:01:44.851160855Z"} ``` -The original message is handled as an escaped string. Fluent Bit wants to use the -original structured message and not a string. +The original message is handled as an escaped string. Fluent Bit wants to use the original structured message and not a string. -## Getting Started +## Get started -Decoders are a built-in feature available through the Parsers file. Each parser -definition can optionally set one or more decoders. There are two types of decoders: +Decoders are a built-in feature available through the `parsers.conf` file. Each parser definition can optionally set one or more decoders. There are two types of decoders: - `Decode_Field`: If the content can be decoded in a structured message, append the structured message (keys and values) to the original log message. - `Decode_Field_As`: Any decoded content (unstructured or structured) will be replaced in the same key/value, and no extra keys are added. -Our pre-defined Docker parser has the following definition: +For example, the predefined Docker parser has the following definition: {% tabs %} {% tab title="parsers.yaml" %} @@ -64,34 +60,29 @@ parsers: {% endtab %} {% endtabs %} -Each line in the parser with a key `Decode_Field` instructs the parser to apply -a specific decoder on a given field. Optionally, it offers the option to take an -extra action if the decoder doesn't succeed. +Each line in the parser with a key `Decode_Field` instructs the parser to apply a specific decoder on a given field. Optionally, it offers the option to take an extra action if the decoder doesn't succeed. ### Decoder options | Name | Description | | -------------- | ----------- | -| `json` | Handle the field content as a JSON map. If it finds a JSON map, it replaces the content with a structured map. | +| `json` | Handle the field content as a JSON map. If the decoder finds a JSON map, it replaces the content with a structured map. | | `escaped` | Decode an escaped string. | | `escaped_utf8` | Decode a UTF8 escaped string. | -### Optional Actions +### Optional actions -If a decoder fails to decode the field or, you want to try another decoder, you can -define an optional action. Available actions are: +If a decoder fails to decode the field, or if you want to try another decoder, you can define an optional action. Available actions are: | Name | Description | | -----| ----------- | -| `try_next` | if the decoder failed, apply the next decoder in the list for the same field. | -| `do_next` | if the decoder succeeded or failed, apply the next decoder in the list for the same field. | +| `try_next` | If the decoder failed, apply the next decoder in the list for the same field. | +| `do_next` | If the decoder succeeded or failed, apply the next decoder in the list for the same field. | Actions are affected by some restrictions: -- `Decode_Field_As`: If successful, another decoder of the same type and the same - field can be applied only if the data continues being an unstructured message (raw text). -- `Decode_Field`: If successful, can only be applied once for the same field. - `Decode`_Field` is intended to decode a structured message. +- `Decode_Field_As`: If successful, another decoder of the same type and the same field can be applied only if the data continues being an unstructured message (raw text). +- `Decode_Field`: If successful, can only be applied once for the same field. `Decode_Field` is intended to decode a structured message. ### Examples @@ -124,7 +115,7 @@ Decoder example Fluent Bit configuration files: ```yaml service: parsers_file: parsers.yaml - + pipeline: inputs: - name: tail @@ -183,4 +174,4 @@ parsers: ``` {% endtab %} -{% endtabs %} \ No newline at end of file +{% endtabs %} From 876b40f2dd392be27cbac2f613e2ea1617d0bfc1 Mon Sep 17 00:00:00 2001 From: Alexa Kreizinger Date: Mon, 7 Jul 2025 17:26:48 -0700 Subject: [PATCH 2/3] Update pipeline/parsers/decoders.md Signed-off-by: Alexa Kreizinger --- pipeline/parsers/decoders.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/parsers/decoders.md b/pipeline/parsers/decoders.md index 8f7114a43..418e45c9c 100644 --- a/pipeline/parsers/decoders.md +++ b/pipeline/parsers/decoders.md @@ -18,7 +18,7 @@ The original message is handled as an escaped string. Fluent Bit wants to use th ## Get started -Decoders are a built-in feature available through the `parsers.conf` file. Each parser definition can optionally set one or more decoders. There are two types of decoders: +Decoders are a built-in feature of parsers in Fluent Bit. Each parser definition can optionally set one or more decoders. There are two types of decoders: - `Decode_Field`: If the content can be decoded in a structured message, append the structured message (keys and values) to the original log message. From 6ae1cf5acc2107c53d60be3b9a38455ffafe075c Mon Sep 17 00:00:00 2001 From: Alexa Kreizinger Date: Wed, 9 Jul 2025 12:07:44 -0700 Subject: [PATCH 3/3] Apply suggestions from code review Co-authored-by: Craig Norris <112565517+cnorris-cs@users.noreply.github.com> Signed-off-by: Alexa Kreizinger --- pipeline/parsers/decoders.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/pipeline/parsers/decoders.md b/pipeline/parsers/decoders.md index 418e45c9c..cefa4c480 100644 --- a/pipeline/parsers/decoders.md +++ b/pipeline/parsers/decoders.md @@ -1,6 +1,6 @@ # Decoders -There are cases where the log messages that you want to parse contain encoded data. A typical use case can be found in containerized environments with Docker. Docker logs its data in JSON format, which uses escaped strings. +There are cases where the log messages you want to parse contain encoded data. A typical use case can be found in containerized environments with Docker. Docker logs its data in JSON format, which uses escaped strings. Consider the following message generated by the application: @@ -14,11 +14,11 @@ The Docker log message encapsulates something like this: {"log":"{\"status\": \"up and running\"}\r\n","stream":"stdout","time":"2018-03-09T01:01:44.851160855Z"} ``` -The original message is handled as an escaped string. Fluent Bit wants to use the original structured message and not a string. +The original message is handled as an escaped string. Fluent Bit will use the original structured message, and not a string. ## Get started -Decoders are a built-in feature of parsers in Fluent Bit. Each parser definition can optionally set one or more decoders. There are two types of decoders: +Decoders are a built-in feature of parsers in Fluent Bit. Each parser definition can optionally set one or more decoders. Select from one of these decoder types: - `Decode_Field`: If the content can be decoded in a structured message, append the structured message (keys and values) to the original log message. @@ -82,7 +82,7 @@ If a decoder fails to decode the field, or if you want to try another decoder, y Actions are affected by some restrictions: - `Decode_Field_As`: If successful, another decoder of the same type and the same field can be applied only if the data continues being an unstructured message (raw text). -- `Decode_Field`: If successful, can only be applied once for the same field. `Decode_Field` is intended to decode a structured message. +- `Decode_Field`: If successful, can be applied only once for the same field. `Decode_Field` is intended to decode a structured message. ### Examples