Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion administration/backpressure.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Backpressure

<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=63e37cfe-9ce3-4a18-933a-76b9198958c1" />
<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=63e37cfe-9ce3-4a18-933a-76b9198958c1" alt="tracking" />

It's possible for Fluent Bit to ingest or create data faster than it can flush that data to the intended destinations. This creates a condition known as _backpressure_.

Expand Down Expand Up @@ -54,6 +54,17 @@ In a few seconds, if the scheduler was able to flush the initial 700&nbsp;KB of
- If the plugin is paused, it invokes a `resume` callback.
- The input plugin can continue appending more data.

## Pause behavior for HTTP-based inputs

The HTTP-based input plugins [`http`](../pipeline/inputs/http.md), [`splunk`](../pipeline/inputs/splunk.md), [`elasticsearch`](../pipeline/inputs/elasticsearch.md), [`opentelemetry`](../pipeline/inputs/opentelemetry.md), and [`prometheus_remote_write`](../pipeline/inputs/prometheus-remote-write.md) act on the `pause` and `resume` callbacks by pausing their [shared HTTP listener](configuring-fluent-bit/yaml/pipeline-section.md#shared-http-listener-settings-for-inputs). This applies whichever backpressure mechanism paused the input.

While one of these inputs is paused:

- The listening socket stays open, but Fluent Bit accepts and immediately closes each incoming connection. Clients see a connection that closes before a response arrives rather than a refused connection.
Comment thread
eschabell marked this conversation as resolved.
- Connections that are already open are closed instead of being kept alive, and any request still in flight is dropped.

Because the client never receives a successful response for these requests, it's responsible for retrying. Configure your senders to retry on connection failure so data isn't lost while an input is paused. When the `resume` callback runs, the listener starts accepting connections again.

## Manage backpressure for filesystem buffering

If one or more active input plugins use [filesystem buffering](../pipeline/buffering.md#filesystem-buffering-hybrid), use the following settings to manage backpressure.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,8 @@ effect.

For backward compatibility, some plugins also accept the legacy aliases `http2`, `buffer_max_size`, `buffer_chunk_size`, `max_connections`, and `workers`.

These plugins also pause and resume the shared listener in response to [backpressure](../../backpressure.md#pause-behavior-for-http-based-inputs). While an input is paused, Fluent Bit accepts and immediately closes incoming connections instead of buffering the requests, so clients must retry.

### Incoming `OAuth 2.0` `JWT` validation settings

The HTTP-based input plugins that support bearer token validation share the following `oauth2.*` settings:
Expand Down