Skip to content

Commit 79b59cc

Browse files
eschabelleschabell
authored
Added all missing YAML examples an standardized Filters docs. Fixes #1882. (#1883)
* Adding YAML examples to Checklist filter doc, part of issue #1882. Signed-off-by: Eric D. Schabell <eric@schabell.org> * Standardizing on YAML and shell usage for grep filter doc, part of issue #1882. Signed-off-by: Eric D. Schabell <eric@schabell.org> * Adding YAML examples and standard shell usage for log to metric filter doc, part of issue #1882. Signed-off-by: Eric D. Schabell <eric@schabell.org> * Updating Fluent Bit output and small layout fixes for parser filter doc, part of issue #1882. Signed-off-by: Eric D. Schabell <eric@schabell.org> * Updated for YAML examples first for modify filter doc, part of issue #1882. Signed-off-by: Eric D. Schabell <eric@schabell.org> * Updated for YAML examples first for nest filter doc, part of issue #1882. Signed-off-by: Eric D. Schabell <eric@schabell.org> * Updated for YAML examples first for rewrite tag filter doc, part of issue #1882. Signed-off-by: Eric D. Schabell <eric@schabell.org> * Updated for YAML examples first for sysinfo filter doc, part of issue #1882. Signed-off-by: Eric D. Schabell <eric@schabell.org> * Updated for YAML examples first for type converter filter doc, part of issue #1882. Signed-off-by: Eric D. Schabell <eric@schabell.org> * Added YAML examples for WASAM filter doc, part of issue #1882. Signed-off-by: Eric D. Schabell <eric@schabell.org> --------- Signed-off-by: Eric D. Schabell <eric@schabell.org>
1 parent af34978 commit 79b59cc

10 files changed

Lines changed: 897 additions & 471 deletions

File tree

‎pipeline/filters/checklist.md‎

Lines changed: 35 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,37 @@ The plugin supports the following configuration parameters
1717

1818
## Example configuration
1919

20-
```python
20+
{% tabs %}
21+
{% tab title="fluent-bit.yaml" %}
22+
23+
```yaml
24+
pipeline:
25+
inputs:
26+
- name: tail
27+
tag: test1
28+
path: test1.log
29+
read_from_head: true
30+
parser: json
31+
32+
filters:
33+
- name: checklist
34+
match: test1
35+
file: ip_list.txt
36+
lookup_key: $remote_addr
37+
record:
38+
- ioc abc
39+
- badurl null
40+
log_level: debug
41+
42+
outputs:
43+
- name: stdout
44+
match: test1
45+
```
46+
47+
{% endtab %}
48+
{% tab title="fluent-bit.conf" %}
49+
50+
```text
2151
[INPUT]
2252
name tail
2353
tag test1
@@ -39,6 +69,9 @@ The plugin supports the following configuration parameters
3969
match test1
4070
```
4171

72+
{% endtab %}
73+
{% endtabs %}
74+
4275
The following configuration reads a file `test1.log` that includes the following values:
4376

4477
```text
@@ -64,4 +97,4 @@ The configuration uses `$remote_addr` as the lookup key, and `7.7.7.7` is malici
6497

6598
```text
6699
{"remote_addr": "7.7.7.7", "ioc":"abc", "url":"https://badurl.com/payload.htm","badurl":"null"}
67-
```
100+
```

‎pipeline/filters/grep.md‎

Lines changed: 99 additions & 80 deletions
Original file line numberDiff line numberDiff line change
@@ -42,53 +42,54 @@ When using the command line, pay close attention to quote the regular expression
4242
The following command loads the [tail](../../pipeline/inputs/tail) plugin and reads the content of `lines.txt`. Then the `grep` filter applies a regular expression rule over the `log` field created by the `tail` plugin and only passes records with a field value starting with `aa`:
4343

4444
```shell
45-
bin/fluent-bit -i tail -p 'path=lines.txt' -F grep -p 'regex=log aa' -m '*' -o stdout
45+
$ ./fluent-bit -i tail -p 'path=lines.txt' -F grep -p 'regex=log aa' -m '*' -o stdout
4646
```
4747

4848
### Configuration file
4949

5050
{% tabs %}
51-
{% tab title="fluent-bit.conf" %}
52-
53-
```python
54-
[SERVICE]
55-
parsers_file /path/to/parsers.conf
56-
57-
[INPUT]
58-
name tail
59-
path lines.txt
60-
parser json
61-
62-
[FILTER]
63-
name grep
64-
match *
65-
regex log aa
66-
67-
[OUTPUT]
68-
name stdout
69-
match *
70-
```
71-
72-
{% endtab %}
73-
7451
{% tab title="fluent-bit.yaml" %}
7552

7653
```yaml
7754
service:
7855
parsers_file: /path/to/parsers.conf
56+
7957
pipeline:
8058
inputs:
8159
- name: tail
8260
path: lines.txt
8361
parser: json
62+
8463
filters:
8564
- name: grep
8665
match: '*'
8766
regex: log aa
67+
8868
outputs:
8969
- name: stdout
9070
match: '*'
71+
```
72+
73+
{% endtab %}
74+
{% tab title="fluent-bit.conf" %}
75+
76+
```text
77+
[SERVICE]
78+
parsers_file /path/to/parsers.conf
79+
80+
[INPUT]
81+
name tail
82+
path lines.txt
83+
parser json
9184

85+
[FILTER]
86+
name grep
87+
match *
88+
regex log aa
89+
90+
[OUTPUT]
91+
name stdout
92+
match *
9293
```
9394

9495
{% endtab %}
@@ -102,7 +103,7 @@ To match or exclude records based on nested values, you can use [Record Accessor
102103

103104
Consider the following record example:
104105

105-
```javascript
106+
```text
106107
{
107108
"log": "something",
108109
"kubernetes": {
@@ -122,25 +123,27 @@ Consider the following record example:
122123
For example, to exclude records that match the nested field `kubernetes.labels.app`, use the following rule:
123124

124125
{% tabs %}
125-
{% tab title="fluent-bit.conf" %}
126-
127-
```python
128-
[FILTER]
129-
Name grep
130-
Match *
131-
Exclude $kubernetes['labels']['app'] myapp
132-
```
133-
134-
{% endtab %}
135126
{% tab title="fluent-bit.yaml" %}
136127

137128
```yaml
129+
pipeline:
130+
138131
filters:
139132
- name: grep
140133
match: '*'
141134
exclude: $kubernetes['labels']['app'] myapp
142135
```
143136
137+
{% endtab %}
138+
{% tab title="fluent-bit.conf" %}
139+
140+
```text
141+
[FILTER]
142+
Name grep
143+
Match *
144+
Exclude $kubernetes['labels']['app'] myapp
145+
```
146+
144147
{% endtab %}
145148
{% endtabs %}
146149

@@ -154,9 +157,25 @@ key fails this check.
154157
The following example checks for a specific valid value for the key:
155158

156159
{% tabs %}
160+
{% tab title="fluent-bit.yaml" %}
161+
162+
```yaml
163+
pipeline:
164+
165+
filters:
166+
# Use Grep to verify the contents of the iot_timestamp value.
167+
# If the iot_timestamp key does not exist, this will fail
168+
# and exclude the row.
169+
- name: grep
170+
alias: filter-iots-grep
171+
match: iots_thread.*
172+
regex: iot_timestamp ^\d{4}-\d{2}-\d{2}
173+
```
174+
175+
{% endtab %}
157176
{% tab title="fluent-bit.conf" %}
158177
159-
```python
178+
```text
160179
# Use Grep to verify the contents of the iot_timestamp value.
161180
# If the iot_timestamp key does not exist, this will fail
162181
# and exclude the row.
@@ -167,17 +186,6 @@ The following example checks for a specific valid value for the key:
167186
Regex iot_timestamp ^\d{4}-\d{2}-\d{2}
168187
```
169188

170-
{% endtab %}
171-
{% tab title="fluent-bit.yaml" %}
172-
173-
```yaml
174-
filters:
175-
- name: grep
176-
alias: filter-iots-grep
177-
match: iots_thread.*
178-
regex: iot_timestamp ^\d{4}-\d{2}-\d{2}
179-
```
180-
181189
{% endtab %}
182190
{% endtabs %}
183191

@@ -196,27 +204,6 @@ If you want to set multiple `Regex` or `Exclude`, you can use `Logical_Op` prope
196204
If `Logical_Op` is set, setting both `Regex` and `Exclude` results in an error.
197205

198206
{% tabs %}
199-
{% tab title="fluent-bit.conf" %}
200-
201-
```python
202-
[INPUT]
203-
Name dummy
204-
Dummy {"endpoint":"localhost", "value":"something"}
205-
Tag dummy
206-
207-
[FILTER]
208-
Name grep
209-
Match *
210-
Logical_Op or
211-
Regex value something
212-
Regex value error
213-
214-
[OUTPUT]
215-
Name stdout
216-
```
217-
218-
{% endtab %}
219-
220207
{% tab title="fluent-bit.yaml" %}
221208

222209
```yaml
@@ -225,15 +212,39 @@ pipeline:
225212
- name: dummy
226213
dummy: '{"endpoint":"localhost", "value":"something"}'
227214
tag: dummy
215+
228216
filters:
229217
- name: grep
230218
match: '*'
231219
logical_op: or
232220
regex:
233221
- value something
234222
- value error
223+
235224
outputs:
236225
- name: stdout
226+
match: '*'
227+
```
228+
229+
{% endtab %}
230+
{% tab title="fluent-bit.conf" %}
231+
232+
```text
233+
[INPUT]
234+
Name dummy
235+
Dummy {"endpoint":"localhost", "value":"something"}
236+
Tag dummy
237+
238+
[FILTER]
239+
Name grep
240+
Match *
241+
Logical_Op or
242+
Regex value something
243+
Regex value error
244+
245+
[OUTPUT]
246+
Name stdout
247+
Match *
237248
```
238249

239250
{% endtab %}
@@ -242,20 +253,28 @@ pipeline:
242253
The output looks similar to:
243254

244255
```text
245-
Fluent Bit v2.0.9
246-
* Copyright (C) 2015-2022 The Fluent Bit Authors
256+
Fluent Bit v4.0.3
257+
* Copyright (C) 2015-2025 The Fluent Bit Authors
247258
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
248259
* https://fluentbit.io
249260
250-
[2023/01/22 09:46:49] [ info] [fluent bit] version=2.0.9, commit=16eae10786, pid=33268
251-
[2023/01/22 09:46:49] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
252-
[2023/01/22 09:46:49] [ info] [cmetrics] version=0.5.8
253-
[2023/01/22 09:46:49] [ info] [ctraces ] version=0.2.7
254-
[2023/01/22 09:46:49] [ info] [input:dummy:dummy.0] initializing
255-
[2023/01/22 09:46:49] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
256-
[2023/01/22 09:46:49] [ info] [filter:grep:grep.0] OR mode
257-
[2023/01/22 09:46:49] [ info] [sp] stream processor started
258-
[2023/01/22 09:46:49] [ info] [output:stdout:stdout.0] worker #0 started
261+
______ _ _ ______ _ _ ___ _____
262+
| ___| | | | | ___ (_) | / || _ |
263+
| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' |
264+
| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| |
265+
| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ /
266+
\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/
267+
268+
269+
[2025/07/03 16:15:34] [ info] [fluent bit] version=4.0.3, commit=3a91b155d6, pid=23196
270+
[2025/07/03 16:15:34] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128
271+
[2025/07/03 16:15:34] [ info] [simd ] disabled
272+
[2025/07/03 16:15:34] [ info] [cmetrics] version=1.0.3
273+
[2025/07/03 16:15:34] [ info] [ctraces ] version=0.6.6
274+
[2025/07/03 16:15:34] [ info] [input:dummy:dummy.0] initializing
275+
[2025/07/03 16:15:34] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
276+
[2025/07/03 16:15:34] [ info] [output:stdout:stdout.0] worker #0 started
277+
[2025/07/03 16:15:34] [ info] [sp] stream processor started
259278
[0] dummy: [1674348410.558341857, {"endpoint"=>"localhost", "value"=>"something"}]
260279
[0] dummy: [1674348411.546425499, {"endpoint"=>"localhost", "value"=>"something"}]
261-
```
280+
```

0 commit comments

Comments
 (0)