From bb58fbb4e5b7d4e0c6ed98f48381cb6e6ebdb8c6 Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Wed, 2 Jul 2025 12:32:58 -0700 Subject: [PATCH 1/2] Pipeline: input: tcp: style Signed-off-by: Lynette Miles --- pipeline/inputs/tcp.md | 90 +++++++++++++++++++++++------------------- 1 file changed, 49 insertions(+), 41 deletions(-) diff --git a/pipeline/inputs/tcp.md b/pipeline/inputs/tcp.md index 67dba0eb5..dea11dbd9 100644 --- a/pipeline/inputs/tcp.md +++ b/pipeline/inputs/tcp.md @@ -1,6 +1,6 @@ # TCP -The **tcp** input plugin allows to retrieve structured JSON or raw messages over a TCP network interface (TCP port). +The _TCP_ input plugin lets you retrieve structured JSON or raw messages over a TCP network interface (TCP port). ## Configuration Parameters @@ -8,41 +8,59 @@ The plugin supports the following configuration parameters: | Key | Description | Default | | ------------ | ----------- | ------- | -| Listen | Listener network interface. | 0.0.0.0 | -| Port | TCP port where listening for connections | 5170 | -| Buffer\_Size | Specify the maximum buffer size in KB to receive a JSON message. If not set, the default size will be the value of _Chunk\_Size_. | | -| Chunk\_Size | By default the buffer to store the incoming JSON messages, do not allocate the maximum memory allowed, instead it allocate memory when is required. The rounds of allocations are set by _Chunk\_Size_ in KB. If not set, _Chunk\_Size_ is equal to 32 (32KB). | 32 | -| Format | Specify the expected payload format. It support the options _json_ and _none_. When using _json_, it expects JSON maps, when is set to _none_, it will split every record using the defined _Separator_ (option below). | json | -| Separator | When the expected _Format_ is set to _none_, Fluent Bit needs a separator string to split the records. By default it uses the breakline character (LF or 0x10). | | -| Source\_Address\_Key| Specify the key where the source address will be injected. | | -| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | +| `Listen` | Listener network interface. | `0.0.0.0` | +| `Port` | TCP port to listen for connections. | `5170` | +| `Buffer_Size` | Specify the maximum buffer size in KB to receive a JSON message. If not set, the default is the value of `Chunk_Size`. | `Chunk_Size` | +| `Chunk_Size` | The default buffer to store the incoming JSON messages. It doesn't allocate the maximum memory allowed; instead it allocates memory when required. The rounds of allocations are set by `Chunk_Size`. If not set, `Chunk_Size` is equal to 32 (32KB). | `32` | +| `Format` | Specify the expected payload format. Supported values: `json` and `none`. `json` it expects JSON maps. When set to `none`, every record splits using the defined `Separator`. | `json` | +| `Separator` | When `Format` is set to `none`, Fluent Bit needs a separator string to split the records. | `LF` or `0x10` (break line) | +| `Source_Address_Key`| Specify the key to inject the source address. | _none_ | +| `Threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | -## Getting Started +## Get started -In order to receive JSON messages over TCP, you can run the plugin from the command line or through the configuration file: +To receive JSON messages over TCP, you can run the plugin from the command line or through the configuration file: -### Command Line +### Command line -From the command line you can let Fluent Bit listen for _JSON_ messages with the following options: +From the command line you can let Fluent Bit listen for JSON messages with the following options: ```bash -$ fluent-bit -i tcp -o stdout +fluent-bit -i tcp -o stdout ``` -By default the service will listen an all interfaces (0.0.0.0) through TCP port 5170, optionally you can change this directly, e.g: +By default the service will listen an all interfaces (`0.0.0.0`) through TCP port `5170`. Optionally you can change this directly: ```bash -$ fluent-bit -i tcp://192.168.3.2:9090 -o stdout +fluent-bit -i tcp://192.168.3.2:9090 -o stdout ``` -In the example the JSON messages will only arrive through network interface under 192.168.3.2 address and TCP Port 9090. +In the example the JSON messages will only arrive through the network interface at `192.168.3.2` address and TCP Port `9090`. -### Configuration File +### Configuration file -In your main configuration file append the following _Input_ & _Output_ sections: +In your main configuration file append the following sections: {% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +pipeline: + inputs: + - name: tcp + listen: 0.0.0.0 + port: 5170 + chunk_size: 32 + buffer_size: 64 + format: json + outputs: + - name: stdout + match: '*' +``` + +{% endtab %} {% tab title="fluent-bit.conf" %} + ```python [INPUT] Name tcp @@ -56,37 +74,27 @@ In your main configuration file append the following _Input_ & _Output_ sections Name stdout Match * ``` -{% endtab %} -{% tab title="fluent-bit.yaml" %} -```yaml -pipeline: - inputs: - - name: tcp - listen: 0.0.0.0 - port: 5170 - chunk_size: 32 - buffer_size: 64 - format: json - outputs: - - name: stdout - match: '*' -``` {% endtab %} {% endtabs %} -## Testing +## Test the configuration -Once Fluent Bit is running, you can send some messages using the _netcat_: +When Fluent Bit is running, you can send some messages using `netcat`: ```bash -$ echo '{"key 1": 123456789, "key 2": "abcdefg"}' | nc 127.0.0.1 5170 +echo '{"key 1": 123456789, "key 2": "abcdefg"}' | nc 127.0.0.1 5170 +``` + +Run Fluent Bit: + +```shell +bin/fluent-bit -i tcp -o stdout -f 1 ``` -In [Fluent Bit](http://fluentbit.io) we should see the following output: +You should see the following output: ```bash -$ bin/fluent-bit -i tcp -o stdout -f 1 Fluent Bit v1.x.x * Copyright (C) 2019-2020 The Fluent Bit Authors * Copyright (C) 2015-2018 Treasure Data @@ -101,8 +109,8 @@ Fluent Bit v1.x.x [0] tcp.0: [1570115975.581246030, {"key 1"=>123456789, "key 2"=>"abcdefg"}] ``` -## Performance Considerations +## Performance considerations When receiving payloads in JSON format, there are high performance penalties. Parsing JSON is a very expensive task so you could expect your CPU usage increase under high load environments. -To get faster data ingestion, consider to use the option `Format none` to avoid JSON parsing if not needed. +To get faster data ingestion, consider using the option `Format none` to avoid JSON parsing if not needed. From 67960b0bf5218c7a77035904b52c7be7a3197fa7 Mon Sep 17 00:00:00 2001 From: Lynette Miles <6818907+esmerel@users.noreply.github.com> Date: Thu, 3 Jul 2025 10:04:25 -0700 Subject: [PATCH 2/2] Apply suggestions from code review Co-authored-by: Craig Norris <112565517+cnorris-cs@users.noreply.github.com> Signed-off-by: Lynette Miles <6818907+esmerel@users.noreply.github.com> --- pipeline/inputs/tcp.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/pipeline/inputs/tcp.md b/pipeline/inputs/tcp.md index dea11dbd9..92bcd1120 100644 --- a/pipeline/inputs/tcp.md +++ b/pipeline/inputs/tcp.md @@ -12,14 +12,14 @@ The plugin supports the following configuration parameters: | `Port` | TCP port to listen for connections. | `5170` | | `Buffer_Size` | Specify the maximum buffer size in KB to receive a JSON message. If not set, the default is the value of `Chunk_Size`. | `Chunk_Size` | | `Chunk_Size` | The default buffer to store the incoming JSON messages. It doesn't allocate the maximum memory allowed; instead it allocates memory when required. The rounds of allocations are set by `Chunk_Size`. If not set, `Chunk_Size` is equal to 32 (32KB). | `32` | -| `Format` | Specify the expected payload format. Supported values: `json` and `none`. `json` it expects JSON maps. When set to `none`, every record splits using the defined `Separator`. | `json` | +| `Format` | Specify the expected payload format. Supported values: `json` and `none`. When set to `json` it expects JSON maps. When set to `none`, every record splits using the defined `Separator`. | `json` | | `Separator` | When `Format` is set to `none`, Fluent Bit needs a separator string to split the records. | `LF` or `0x10` (break line) | | `Source_Address_Key`| Specify the key to inject the source address. | _none_ | | `Threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | ## Get started -To receive JSON messages over TCP, you can run the plugin from the command line or through the configuration file: +To receive JSON messages over TCP, you can run the plugin from the command line or through the configuration file. ### Command line @@ -39,7 +39,7 @@ In the example the JSON messages will only arrive through the network interface ### Configuration file -In your main configuration file append the following sections: +In your main configuration file append the following sections: {% tabs %} {% tab title="fluent-bit.yaml" %}