From 13595da9f3c682949352a7404fbf8c9a10fd79ae Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Fri, 11 Oct 2024 15:17:32 +0900 Subject: [PATCH 1/2] in_tail: Add a description and note for Unicode.Encoding parameter Signed-off-by: Hiroshi Hatake --- pipeline/inputs/tail.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/pipeline/inputs/tail.md b/pipeline/inputs/tail.md index 9d56320b7..a9f348309 100644 --- a/pipeline/inputs/tail.md +++ b/pipeline/inputs/tail.md @@ -37,6 +37,7 @@ The plugin supports the following configuration parameters: | `static_batch_size` | Set the maximum number of bytes to process per iteration for the monitored static files (files that already exist upon Fluent Bit start). | `50M` | | `file_cache_advise` | Set the `posix_fadvise` in `POSIX_FADV_DONTNEED` mode. This reduces the usage of the kernel file cache. This option is ignored if not running on Linux. | `on` | | `threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | +| `Unicode.Encoding` | Set the Unicode character encoding of the file data. This parameter requests two-byte aligned chunk and buffer sizes. If data is not aligned for two bytes, Fluent Bit will use two-byte alignment automatically to avoid character breakages on consuming boundaries. Supported values: `UTF-16LE`, `UTF-16BE`, and `auto`. | `none` | ## Buffers and memory management @@ -77,6 +78,17 @@ If no database file is present, positioning behavior depends on the value of `re The database file essentially stores `inode=offset` so it should be unique per instance of the plugin, for example if you have two tail inputs then use two separate `db` files for each. That way each tail input can independently track its own state. +{% hint style="info" %} +Note that `Unicode.Encoding` depends on simdutf library which is written in C++11 or above. +So, the older platforms are not supported for this feature. +In addition, `Unicode.Encoding auto` is not covered for the all of the usages. +This is because sometimes this auto-detecting for character encodings makes a mistake to guess the correct encoding. + +We recommend to use `UTF-16LE` or `UTF-16BE` if the target file encoding is pre-determined or known beforehand. +In details, this parameter requests to use 2-bytes aligned chunk and buffer sizes. +If they are not aligned for 2 bytes, Fluent Bit will use 2-bytes alignments automatically to avoid character breakages on consuming boundaries. +{% endhint %} + ## Monitor a large number of files To monitor a large number of files, you can increase the `inotify` settings in your Linux environment by modifying the following `sysctl` parameters: From cbfe1e919f0b6def4a82940c7bb2585f6be81b90 Mon Sep 17 00:00:00 2001 From: Hiroshi Hatake Date: Tue, 8 Jul 2025 12:06:21 +0900 Subject: [PATCH 2/2] Update pipeline/inputs/tail.md Co-authored-by: Alexa Kreizinger Signed-off-by: Hiroshi Hatake --- pipeline/inputs/tail.md | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/pipeline/inputs/tail.md b/pipeline/inputs/tail.md index a9f348309..3a39949b6 100644 --- a/pipeline/inputs/tail.md +++ b/pipeline/inputs/tail.md @@ -79,14 +79,9 @@ If no database file is present, positioning behavior depends on the value of `re The database file essentially stores `inode=offset` so it should be unique per instance of the plugin, for example if you have two tail inputs then use two separate `db` files for each. That way each tail input can independently track its own state. {% hint style="info" %} -Note that `Unicode.Encoding` depends on simdutf library which is written in C++11 or above. -So, the older platforms are not supported for this feature. -In addition, `Unicode.Encoding auto` is not covered for the all of the usages. -This is because sometimes this auto-detecting for character encodings makes a mistake to guess the correct encoding. - -We recommend to use `UTF-16LE` or `UTF-16BE` if the target file encoding is pre-determined or known beforehand. -In details, this parameter requests to use 2-bytes aligned chunk and buffer sizes. -If they are not aligned for 2 bytes, Fluent Bit will use 2-bytes alignments automatically to avoid character breakages on consuming boundaries. +The `Unicode.Encoding` parameter is dependent on the simdutf library, which is itself dependent on C++ version 11 or later. In environments that use earlier versions of C++, the `Unicode.Encoding` parameter will fail. + +Additionally, the `auto` setting for `Unicode.Encoding` isn't supported in all cases, and can make mistakes when it tries to guess the correct encoding. For best results, use either the `UTF-16LE` or `UTF-16BE` setting if you know the encoding type of the target file. {% endhint %} ## Monitor a large number of files