diff --git a/administration/backpressure.md b/administration/backpressure.md index ea43460ce..e93d27e46 100644 --- a/administration/backpressure.md +++ b/administration/backpressure.md @@ -1,6 +1,6 @@ # Backpressure - +tracking It's possible for Fluent Bit to ingest or create data faster than it can flush that data to the intended destinations. This creates a condition known as _backpressure_. @@ -54,6 +54,17 @@ In a few seconds, if the scheduler was able to flush the initial 700 KB of - If the plugin is paused, it invokes a `resume` callback. - The input plugin can continue appending more data. +## Pause behavior for HTTP-based inputs + +The HTTP-based input plugins [`http`](../pipeline/inputs/http.md), [`splunk`](../pipeline/inputs/splunk.md), [`elasticsearch`](../pipeline/inputs/elasticsearch.md), [`opentelemetry`](../pipeline/inputs/opentelemetry.md), and [`prometheus_remote_write`](../pipeline/inputs/prometheus-remote-write.md) act on the `pause` and `resume` callbacks by pausing their [shared HTTP listener](configuring-fluent-bit/yaml/pipeline-section.md#shared-http-listener-settings-for-inputs). This applies whichever backpressure mechanism paused the input. + +While one of these inputs is paused: + +- The listening socket stays open, but Fluent Bit accepts and immediately closes each incoming connection. Clients see a connection that closes before a response arrives rather than a refused connection. +- Connections that are already open are closed instead of being kept alive, and any request still in flight is dropped. + +Because the client never receives a successful response for these requests, it's responsible for retrying. Configure your senders to retry on connection failure so data isn't lost while an input is paused. When the `resume` callback runs, the listener starts accepting connections again. + ## Manage backpressure for filesystem buffering If one or more active input plugins use [filesystem buffering](../pipeline/buffering.md#filesystem-buffering-hybrid), use the following settings to manage backpressure. diff --git a/administration/configuring-fluent-bit/yaml/pipeline-section.md b/administration/configuring-fluent-bit/yaml/pipeline-section.md index 3fea1f1b1..1f613f45d 100644 --- a/administration/configuring-fluent-bit/yaml/pipeline-section.md +++ b/administration/configuring-fluent-bit/yaml/pipeline-section.md @@ -108,6 +108,8 @@ effect. For backward compatibility, some plugins also accept the legacy aliases `http2`, `buffer_max_size`, `buffer_chunk_size`, `max_connections`, and `workers`. +These plugins also pause and resume the shared listener in response to [backpressure](../../backpressure.md#pause-behavior-for-http-based-inputs). While an input is paused, Fluent Bit accepts and immediately closes incoming connections instead of buffering the requests, so clients must retry. + ### Incoming `OAuth 2.0` `JWT` validation settings The HTTP-based input plugins that support bearer token validation share the following `oauth2.*` settings: