Skip to content

docs: administration: document input rate metrics and rate gate #2637

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
eschabell wants to merge 2 commits into
base: master
Choose a base branch
from
+45 −0
Open

docs: administration: document input rate metrics and rate gate #2637

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bc72ace
docs: administration: document input rate metrics and rate gate
eschabell Jul 31, 2026
347afbb
docs: administration: use plain space instead of   in rate gate …
eschabell Aug 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions administration/configuring-fluent-bit/yaml/pipeline-section.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,46 @@
enabled: false
```

### Rate metrics and rate limiting for inputs

Available in Fluent Bit version 5.1 and greater.
Comment thread
eschabell marked this conversation as resolved.

Every input plugin measures its own ingestion rate, and can optionally pause itself when that rate gets too high. All input plugins support the following settings:

| Key | Description | Default |
| --- | ----------- | ------- |
| `rate_window` | Time window used to measure the ingestion rate, such as `1s`, `1m`, or `1h`. Whatever window you choose, the resulting rate is always published in per-second units. | `1s` |
| `rate_gate` | Enable the rate gate, which pauses ingestion while the measured rate exceeds the configured limits. | `false` |
| `rate_gate.max_bytes` | Maximum ingestion rate in bytes per second. Set to `0` to leave the byte rate unlimited. This value must follow [unit size](../../configuring-fluent-bit.md#unit-sizes) specifications. | `0` |
| `rate_gate.max_records` | Maximum ingestion rate in records per second. Set to `0` to leave the record rate unlimited. | `0` |
| `rate_gate.backpressure` | Lower the effective limits while the input has busy chunks or pending delivery retries. | `true` |
| `rate_gate.resume_ratio` | Fraction of the effective limit that the measured rate must fall to before ingestion resumes. Accepts a value greater than `0` and less than or equal to `1`. | `0.80` |

Rate measurement is always active, and publishes the [`fluentbit_input_rate_bytes` and `fluentbit_input_rate_records`](../../monitoring.md#v2-metrics) metrics. The rate gate is opt-in and takes effect only when you enable `rate_gate` and set at least one of `rate_gate.max_bytes` or `rate_gate.max_records` to a non-zero value. A limit of `0` is unlimited.

When the gate is enabled and either the measured byte rate or record rate exceeds its limit, Fluent Bit pauses the input, logs a warning, and schedules a one-shot timer to reevaluate the rate. The input resumes only when both measured rates are no greater than their respective effective limits multiplied by `rate_gate.resume_ratio`. Limits set to `0` are unlimited and are excluded from both checks. This gap between the pause and resume thresholds keeps an input at the limit from repeatedly pausing and resuming.

While `rate_gate.backpressure` is enabled, the limits you configure aren't applied directly. Fluent Bit divides each limit by the number of busy chunks plus pending retry attempts, plus one. An input with no pending work is held to the full limit, while an input whose destination is backed up is held to a progressively smaller share of it, which slows ingestion before chunks accumulate. Set `rate_gate.backpressure` to `false` to apply the configured limits as fixed values instead.

The following example limits an HTTP input to 5MB per second, measured over a five-second window:

Check warning on line 136 in administration/configuring-fluent-bit/yaml/pipeline-section.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.Units] Put a nonbreaking space between the number and the unit in '5MB'.

Raw Output:
{"message":"[FluentBit.Units] Put a nonbreaking space between the number and the unit in '5MB'.","location":{"path":"administration/configuring-fluent-bit/yaml/pipeline-section.md","range":{"start":{"line":136,"column":47},"end":{"line":136,"column":50}}},"severity":"INFO","code":{"value":"FluentBit.Units","url":"https://developers.google.com/style/units-of-measure"}}

{% tabs %}
{% tab title="fluent-bit.yaml" %}

```yaml
pipeline:
inputs:
- name: http
listen: 0.0.0.0
port: 8888
rate_window: 5s
rate_gate: true
rate_gate.max_bytes: 5M
```

{% endtab %}
{% endtabs %}

### Shared HTTP listener settings for inputs

Some HTTP-based input plugins share the same listener implementation and support the following common settings in addition to their plugin-specific parameters:
Expand Down
5 changes: 5 additions & 0 deletions administration/monitoring.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,11 @@ Some metrics are available only for specific plugins or runtime modes. For examp
| `fluentbit_input_memrb_dropped_bytes` | name: the name or alias for the input instance | The number of bytes dropped by the memory ring buffer (`memrb`) storage type when the buffer is full. Only available for input plugins with `storage.type` set to `memrb`. | counter | bytes |
| `fluentbit_input_memrb_dropped_chunks` | name: the name or alias for the input instance | The number of chunks dropped by the memory ring buffer (`memrb`) storage type when the buffer is full. Only available for input plugins with `storage.type` set to `memrb`. | counter | chunks |
| `fluentbit_input_multiline_truncated_total` | name: the name or alias for the input instance | The total number of truncated occurrences for multiline messages. Only available for the [Tail](../pipeline/inputs/tail.md) input plugin when `multiline.parser` is configured. | counter | occurrences |
| `fluentbit_input_rate_bytes` | name: the name or alias for the input instance | The current ingestion rate in bytes per second, measured over the window set by [`rate_window`](configuring-fluent-bit/yaml/pipeline-section.md#rate-metrics-and-rate-limiting-for-inputs). | gauge | bytes |
| `fluentbit_input_rate_gate_busy_chunks` | name: the name or alias for the input instance | The number of busy chunks the [rate gate](configuring-fluent-bit/yaml/pipeline-section.md#rate-metrics-and-rate-limiting-for-inputs) counted when it last computed the effective limit. Only available when `rate_gate` is enabled. | gauge | chunks |
| `fluentbit_input_rate_gate_limited` | name: the name or alias for the input instance | Indicates whether the [rate gate](configuring-fluent-bit/yaml/pipeline-section.md#rate-metrics-and-rate-limiting-for-inputs) is currently limiting ingestion (1) or not (0). Only available when `rate_gate` is enabled. | gauge | boolean |
| `fluentbit_input_rate_gate_retry_attempts` | name: the name or alias for the input instance | The number of pending delivery retry attempts the [rate gate](configuring-fluent-bit/yaml/pipeline-section.md#rate-metrics-and-rate-limiting-for-inputs) counted when it last computed the effective limit. Only available when `rate_gate` is enabled. | gauge | retries |
| `fluentbit_input_rate_records` | name: the name or alias for the input instance | The current ingestion rate in records per second, measured over the window set by [`rate_window`](configuring-fluent-bit/yaml/pipeline-section.md#rate-metrics-and-rate-limiting-for-inputs). | gauge | records |
| `fluentbit_input_records_total` | name: the name or alias for the input instance | The number of log records this input ingested successfully. | counter | records |
| `fluentbit_input_ring_buffer_retries_total` | name: the name or alias for the input instance | The number of ring buffer write retries. | counter | retries |
| `fluentbit_input_ring_buffer_retry_failures_total` | name: the name or alias for the input instance | The number of ring buffer write retry failures. | counter | failures |
Expand Down
Loading