Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions administration/configuring-fluent-bit.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,15 @@ As of Fluent Bit 4.2, `--dry-run` performs full property validation in addition
```
[error] [config] dummy: unknown configuration property 'invalid_property_that_does_not_exist'.
```

### Require FIPS mode with `--enable-fips`

Available in Fluent Bit version 5.1 and greater.

Use the `--enable-fips` flag to require the OpenSSL FIPS provider at startup:

```shell
fluent-bit --enable-fips -c /path/to/fluent-bit.yaml
```

This flag sets the `security.fips_mode` service property. If the FIPS provider isn't available, Fluent Bit exits instead of starting. For the requirements and the effect on plugin behavior, see [FIPS mode](configuring-fluent-bit/yaml/service-section.md#fips-mode).
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,14 @@ The `Service` section defines global properties of the service. The following ke
| daemon | Boolean. Determines whether Fluent Bit should run as a Daemon (background). Allowed values are: `yes`, `no`, `on`, and `off`. Don't enable when using a Systemd based unit, such as the one provided in Fluent Bit packages.  | `Off` |
| `dns.mode` | Set the primary transport layer protocol used by the asynchronous DNS resolver. It can be overridden on a per-plugin basis. | `UDP` |
| `flush` | Set the flush time in `seconds.nanoseconds`. The engine loop uses a Flush timeout to define when it's required to flush the records ingested by input plugins through the defined output plugins. | `1` |
| `flush.adaptive` | If enabled, the flush interval is adjusted at runtime based on chunk backpressure. See [Adaptive flush intervals](../yaml/service-section.md#adaptive-flush-intervals). | `Off` |
| `flush.adaptive.min_interval` | Lower bound in seconds for the adaptive flush interval. | `0.5` |
| `flush.adaptive.max_interval` | Upper bound in seconds for the adaptive flush interval. | `2` |
| `flush.adaptive.low_pressure` | Chunk backpressure percentage that selects the longest flush interval. | `25` |
| `flush.adaptive.medium_pressure` | Chunk backpressure percentage that shortens the flush interval to three quarters of the `flush` value. | `50` |
| `flush.adaptive.high_pressure` | Chunk backpressure percentage that selects the shortest flush interval. | `75` |
| `flush.adaptive.up_steps` | Consecutive samples at a higher pressure level required before the flush interval is shortened. | `2` |
| `flush.adaptive.down_steps` | Consecutive samples at a lower pressure level required before the flush interval is lengthened. | `3` |
| `grace` | Set the grace time in `seconds` as an integer value. The engine loop uses a grace timeout to define wait time on exit. | `5` |
| `http_listen` | Set listening interface for HTTP Server when it's enabled. | `0.0.0.0` |
| `http_port` | Set TCP Port for the HTTP Server. | `2020` |
Expand All @@ -35,6 +43,7 @@ The `Service` section defines global properties of the service. The following ke
| `plugins_file` | Path for a `plugins` configuration file. A `plugins` configuration file defines paths for external plugins. [See an example](https://github.com/fluent/fluent-bit/blob/master/conf/plugins.conf). | _none_ |
| `scheduler.base` | Set a base of exponential backoff. Supported in v1.8.7 and greater. | `5` |
| `scheduler.cap` | Set a maximum retry time in seconds. Supported in v1.8.7 and greater. | `2000` |
| `security.fips_mode` | If enabled, the OpenSSL FIPS provider is required at startup and Fluent Bit exits if it isn't available. See [FIPS mode](../yaml/service-section.md#fips-mode). | `Off` |
| `sp.convert_from_str_to_num` | If enabled, Stream processor converts from number string to number type. | `true` |
| `streams_file` | Path for the Stream Processor configuration file. [Learn more about Stream Processing configuration](../../../stream-processing/overview.md). | _none_|
| `windows.maxstdio` | If specified, the limit of stdio is adjusted. Only provided for Windows. From 512 to 2048 is allowed. | `512` |
Expand Down
58 changes: 58 additions & 0 deletions administration/configuring-fluent-bit/yaml/service-section.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,14 @@ The `service` section of YAML configuration files defines global properties of t
| `dns.resolver` | Sets the DNS resolver implementation. Possible values: `LEGACY`, `ASYNC`. | _none_ |
| `enable_chunk_trace` | If enabled, activates chunk tracing for debugging purposes. Requires Fluent Bit to be built with the `FLB_HAVE_CHUNK_TRACE` option. Possible values: `off` or `on`. | `off` |
| `flush` | Sets the flush time in `seconds.nanoseconds`. The engine loop uses a flush timeout to define when to flush the records ingested by input plugins through the defined output plugins. | `1` |
| `flush.adaptive` | If enabled, Fluent Bit adjusts the flush interval at runtime based on chunk backpressure. See [Adaptive flush intervals](#adaptive-flush-intervals). Possible values: `off` or `on`. | `off` |
| `flush.adaptive.down_steps` | Sets how many consecutive samples at a lower pressure level are required before Fluent Bit lengthens the flush interval. | `3` |
| `flush.adaptive.high_pressure` | Sets the chunk backpressure percentage that makes Fluent Bit target its shortest flush interval. | `75` |
| `flush.adaptive.low_pressure` | Sets the chunk backpressure percentage that makes Fluent Bit target its longest flush interval. | `25` |
| `flush.adaptive.max_interval` | Sets the upper bound in seconds for the adaptive flush interval. | `2` |
| `flush.adaptive.medium_pressure` | Sets the chunk backpressure percentage that makes Fluent Bit shorten the flush interval to three quarters of the `flush` value. | `50` |
| `flush.adaptive.min_interval` | Sets the lower bound in seconds for the adaptive flush interval. | `0.5` |
| `flush.adaptive.up_steps` | Sets how many consecutive samples at a higher pressure level are required before Fluent Bit shortens the flush interval. | `2` |
| `grace` | Sets the grace time in `seconds` as an integer value. The engine loop uses a grace timeout to define the wait time on exit. | `5` |
| `hc_errors_count` | Sets the number of errors that must occur within the health check period before the health check endpoint reports an unhealthy status. | `5` |
| `hc_period` | Sets the health check evaluation period in seconds. | `60` |
Expand All @@ -32,12 +40,62 @@ The `service` section of YAML configuration files defines global properties of t
| `plugins_file` | Path for a `plugins` configuration file. This file specifies the paths to external plugins (.so files) that Fluent Bit can load at runtime. Plugins can also be declared directly in the [`plugins` section](../yaml/plugins-section.md) of YAML configuration files. | _none_ |
| `scheduler.base` | Sets the base of exponential backoff. | `5` |
| `scheduler.cap` | Sets a maximum retry time in seconds. | `2000` |
| `security.fips_mode` | If enabled, Fluent Bit requires the OpenSSL FIPS provider at startup and exits if it isn't available. See [FIPS mode](#fips-mode). Possible values: `off` or `on`. | `off` |
| `sp.convert_from_str_to_num` | If enabled, the stream processor converts strings that represent numbers to a numeric type. | `true` |
| `streams_file` | Path for the [stream processor](../../../stream-processing/overview.md) configuration file. This file defines the rules and operations for stream processing in Fluent Bit. Stream processor configurations can also be defined directly in the `streams` section of YAML configuration files. | _none_ |
| `windows.maxstdio` | If specified, adjusts the limit of `stdio`. Only provided for Windows. Values from `512` to `2048` are allowed. | `512` |

The `service` section only controls the built-in monitoring and control HTTP server. Plugin-specific HTTP listener settings such as `http_server.http2`, `http_server.buffer_max_size`, `http_server.buffer_chunk_size`, `http_server.max_connections`, `http_server.workers`, `http_server.ingress_queue_event_limit`, `http_server.ingress_queue_byte_limit`, and `http_server.idle_timeout` are configured on the relevant input plugin in the [`pipeline.inputs`](../yaml/pipeline-section.md#shared-http-listener-settings-for-inputs) section.

## Adaptive flush intervals

Adaptive flush intervals are available in Fluent Bit version 5.1 and greater.

The `flush` key sets a fixed flush interval. A short interval keeps latency low but wastes cycles when there's little data to deliver, and a long interval delays delivery when a pipeline is under load. Enabling `flush.adaptive` lets Fluent Bit move between those two behaviors on its own.

When `flush.adaptive` is enabled, Fluent Bit samples the highest chunk backpressure percentage across all input instances and maps that sample to a pressure level. Each level applies a multiplier to the configured `flush` value:

| Backpressure sample | Flush interval |
| --- | --- |
| `flush.adaptive.high_pressure` or greater | `flush` multiplied by `0.5` |
| `flush.adaptive.medium_pressure` or greater | `flush` multiplied by `0.75` |
| Greater than `flush.adaptive.low_pressure` | `flush` unchanged |
| `flush.adaptive.low_pressure` or less | `flush` multiplied by `2` |

The resulting interval is then clamped to the range set by `flush.adaptive.min_interval` and `flush.adaptive.max_interval`, so those two keys always take precedence over the multipliers.

To keep the interval from oscillating between levels, Fluent Bit requires repeated samples before it changes level: `flush.adaptive.up_steps` consecutive samples to move to a higher pressure level, and `flush.adaptive.down_steps` consecutive samples to move to a lower one. A sample that matches the current level resets the count. If Fluent Bit can't apply the new interval, it keeps the interval that's already in effect.

The following example flushes every 1.5 seconds under normal conditions, drops to 0.75 seconds when chunk backpressure reaches 80%, and backs off to a maximum of three seconds when the pipeline is idle:

{% tabs %}
{% tab title="fluent-bit.yaml" %}

```yaml
service:
flush: 1.5
flush.adaptive: on
flush.adaptive.min_interval: 0.5
flush.adaptive.max_interval: 3
flush.adaptive.high_pressure: 80
```

{% endtab %}
{% endtabs %}

## FIPS mode

FIPS mode is available in Fluent Bit version 5.1 and greater. It requires OpenSSL 3.0 or greater with the FIPS provider installed on the host.

When `security.fips_mode` is enabled, Fluent Bit activates the OpenSSL FIPS provider during startup and verifies that a FIPS-approved algorithm can be fetched from it. If the provider isn't available or can't be activated, Fluent Bit logs the OpenSSL errors and exits instead of starting with non-compliant cryptography. You can set the same behavior from the command line with the [`--enable-fips`](../../configuring-fluent-bit.md#require-fips-mode-with---enable-fips) flag.

FIPS mode can't be changed by [hot reload](../../../administration/hot-reload.md). If a reloaded configuration changes `security.fips_mode`, Fluent Bit halts the reload and keeps running with the previous configuration.

Enabling FIPS mode changes the behavior of plugins that rely on MD5:

- The [Amazon S3](../../../pipeline/outputs/s3.md) output rejects `send_content_md5` at startup, because that header requires MD5.
- The [Azure Blob](../../../pipeline/outputs/azure_blob.md) output derives block IDs using SHA-256 instead of MD5.

## Storage configuration

The following storage-related keys can be set as children to the `storage` key:
Expand Down