From bc72ace31b938e786e099ed27f3af5e6c61e8c84 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Fri, 31 Jul 2026 18:02:04 +0200 Subject: [PATCH 1/2] docs: administration: document input rate metrics and rate gate Add the rate_window and rate_gate settings shared by all input plugins to the pipeline section, covering the pause and resume thresholds and how backpressure scales the effective limit. Add the five new fluentbit_input_rate_* metrics to the monitoring reference. Also clarify rate gate pause and resume semantics and rate gate activation requirements. Note: this update needed to cover code changes without doc PR Signed-off-by: Eric D. Schabell --- .../yaml/pipeline-section.md | 40 +++++++++++++++++++ administration/monitoring.md | 5 +++ 2 files changed, 45 insertions(+) diff --git a/administration/configuring-fluent-bit/yaml/pipeline-section.md b/administration/configuring-fluent-bit/yaml/pipeline-section.md index 74dc93afb..643395475 100644 --- a/administration/configuring-fluent-bit/yaml/pipeline-section.md +++ b/administration/configuring-fluent-bit/yaml/pipeline-section.md @@ -112,6 +112,46 @@ pipeline: enabled: false ``` +### Rate metrics and rate limiting for inputs + +Available in Fluent Bit version 5.1 and greater. + +Every input plugin measures its own ingestion rate, and can optionally pause itself when that rate gets too high. All input plugins support the following settings: + +| Key | Description | Default | +| --- | ----------- | ------- | +| `rate_window` | Time window used to measure the ingestion rate, such as `1s`, `1m`, or `1h`. Whatever window you choose, the resulting rate is always published in per-second units. | `1s` | +| `rate_gate` | Enable the rate gate, which pauses ingestion while the measured rate exceeds the configured limits. | `false` | +| `rate_gate.max_bytes` | Maximum ingestion rate in bytes per second. Set to `0` to leave the byte rate unlimited. This value must follow [unit size](../../configuring-fluent-bit.md#unit-sizes) specifications. | `0` | +| `rate_gate.max_records` | Maximum ingestion rate in records per second. Set to `0` to leave the record rate unlimited. | `0` | +| `rate_gate.backpressure` | Lower the effective limits while the input has busy chunks or pending delivery retries. | `true` | +| `rate_gate.resume_ratio` | Fraction of the effective limit that the measured rate must fall to before ingestion resumes. Accepts a value greater than `0` and less than or equal to `1`. | `0.80` | + +Rate measurement is always active, and publishes the [`fluentbit_input_rate_bytes` and `fluentbit_input_rate_records`](../../monitoring.md#v2-metrics) metrics. The rate gate is opt-in and takes effect only when you enable `rate_gate` and set at least one of `rate_gate.max_bytes` or `rate_gate.max_records` to a non-zero value. A limit of `0` is unlimited. + +When the gate is enabled and either the measured byte rate or record rate exceeds its limit, Fluent Bit pauses the input, logs a warning, and schedules a one-shot timer to reevaluate the rate. The input resumes only when both measured rates are no greater than their respective effective limits multiplied by `rate_gate.resume_ratio`. Limits set to `0` are unlimited and are excluded from both checks. This gap between the pause and resume thresholds keeps an input at the limit from repeatedly pausing and resuming. + +While `rate_gate.backpressure` is enabled, the limits you configure aren't applied directly. Fluent Bit divides each limit by the number of busy chunks plus pending retry attempts, plus one. An input with no pending work is held to the full limit, while an input whose destination is backed up is held to a progressively smaller share of it, which slows ingestion before chunks accumulate. Set `rate_gate.backpressure` to `false` to apply the configured limits as fixed values instead. + +The following example limits an HTTP input to 5 MB per second, measured over a five-second window: + +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +pipeline: + inputs: + - name: http + listen: 0.0.0.0 + port: 8888 + rate_window: 5s + rate_gate: true + rate_gate.max_bytes: 5M +``` + +{% endtab %} +{% endtabs %} + ### Shared HTTP listener settings for inputs Some HTTP-based input plugins share the same listener implementation and support the following common settings in addition to their plugin-specific parameters: diff --git a/administration/monitoring.md b/administration/monitoring.md index 027220557..a1a7f0a86 100644 --- a/administration/monitoring.md +++ b/administration/monitoring.md @@ -216,6 +216,11 @@ Some metrics are available only for specific plugins or runtime modes. For examp | `fluentbit_input_memrb_dropped_bytes` | name: the name or alias for the input instance | The number of bytes dropped by the memory ring buffer (`memrb`) storage type when the buffer is full. Only available for input plugins with `storage.type` set to `memrb`. | counter | bytes | | `fluentbit_input_memrb_dropped_chunks` | name: the name or alias for the input instance | The number of chunks dropped by the memory ring buffer (`memrb`) storage type when the buffer is full. Only available for input plugins with `storage.type` set to `memrb`. | counter | chunks | | `fluentbit_input_multiline_truncated_total` | name: the name or alias for the input instance | The total number of truncated occurrences for multiline messages. Only available for the [Tail](../pipeline/inputs/tail.md) input plugin when `multiline.parser` is configured. | counter | occurrences | +| `fluentbit_input_rate_bytes` | name: the name or alias for the input instance | The current ingestion rate in bytes per second, measured over the window set by [`rate_window`](configuring-fluent-bit/yaml/pipeline-section.md#rate-metrics-and-rate-limiting-for-inputs). | gauge | bytes | +| `fluentbit_input_rate_gate_busy_chunks` | name: the name or alias for the input instance | The number of busy chunks the [rate gate](configuring-fluent-bit/yaml/pipeline-section.md#rate-metrics-and-rate-limiting-for-inputs) counted when it last computed the effective limit. Only available when `rate_gate` is enabled. | gauge | chunks | +| `fluentbit_input_rate_gate_limited` | name: the name or alias for the input instance | Indicates whether the [rate gate](configuring-fluent-bit/yaml/pipeline-section.md#rate-metrics-and-rate-limiting-for-inputs) is currently limiting ingestion (1) or not (0). Only available when `rate_gate` is enabled. | gauge | boolean | +| `fluentbit_input_rate_gate_retry_attempts` | name: the name or alias for the input instance | The number of pending delivery retry attempts the [rate gate](configuring-fluent-bit/yaml/pipeline-section.md#rate-metrics-and-rate-limiting-for-inputs) counted when it last computed the effective limit. Only available when `rate_gate` is enabled. | gauge | retries | +| `fluentbit_input_rate_records` | name: the name or alias for the input instance | The current ingestion rate in records per second, measured over the window set by [`rate_window`](configuring-fluent-bit/yaml/pipeline-section.md#rate-metrics-and-rate-limiting-for-inputs). | gauge | records | | `fluentbit_input_records_total` | name: the name or alias for the input instance | The number of log records this input ingested successfully. | counter | records | | `fluentbit_input_ring_buffer_retries_total` | name: the name or alias for the input instance | The number of ring buffer write retries. | counter | retries | | `fluentbit_input_ring_buffer_retry_failures_total` | name: the name or alias for the input instance | The number of ring buffer write retry failures. | counter | failures | From 347afbb354c3cc41e6e5eb5b7e32670956aed6ff Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Sun, 9 Aug 2026 20:29:19 +0200 Subject: [PATCH 2/2] docs: administration: use plain space instead of   in rate gate example Signed-off-by: Eric D. Schabell --- administration/configuring-fluent-bit/yaml/pipeline-section.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/administration/configuring-fluent-bit/yaml/pipeline-section.md b/administration/configuring-fluent-bit/yaml/pipeline-section.md index 643395475..f0712db73 100644 --- a/administration/configuring-fluent-bit/yaml/pipeline-section.md +++ b/administration/configuring-fluent-bit/yaml/pipeline-section.md @@ -133,7 +133,7 @@ When the gate is enabled and either the measured byte rate or record rate exceed While `rate_gate.backpressure` is enabled, the limits you configure aren't applied directly. Fluent Bit divides each limit by the number of busy chunks plus pending retry attempts, plus one. An input with no pending work is held to the full limit, while an input whose destination is backed up is held to a progressively smaller share of it, which slows ingestion before chunks accumulate. Set `rate_gate.backpressure` to `false` to apply the configured limits as fixed values instead. -The following example limits an HTTP input to 5 MB per second, measured over a five-second window: +The following example limits an HTTP input to 5MB per second, measured over a five-second window: {% tabs %} {% tab title="fluent-bit.yaml" %}