Skip to content

Commit a204fa0

Browse files
tilloclaude
andcommitted
logging: add open_timeout/read_timeout to cribl ClusterOutput
fluentd out_http defaults read_timeout to unlimited. On 2026-06-11 a torn write corrupted the Cribl httpraw source persistent-queue; the source accepted connections but never processed the POST bodies, so the flush threads blocked indefinitely with no error and no retry and the WAF/modsec feed silently stalled for ~44h. open_timeout 5 / read_timeout 30 force those hung flushes to error and retry (re-resolving to a healthy worker). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 6b1b660 commit a204fa0

1 file changed

Lines changed: 10 additions & 0 deletions

File tree

monitoring-rules/13-logging-output-cribl.yml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,16 @@ spec:
1515
endpoint: http://logstream-leader-internal.cribl.svc.cluster.local:8080
1616
content_type: application/json
1717
json_array: true
18+
## Fail-fast instead of hanging a flush thread forever on a wedged Cribl
19+
## source. On 2026-06-11 a torn write corrupted the Cribl httpraw source
20+
## persistent-queue; the source accepted connections but never processed
21+
## the POST bodies. With no read_timeout (fluentd default is unlimited),
22+
## the flush threads blocked indefinitely — no error, no retry — and the
23+
## WAF/modsec feed silently stalled for ~44h. read_timeout forces those
24+
## hung flushes to error and retry (and re-resolve to a healthy worker);
25+
## open_timeout bounds the TCP connect.
26+
open_timeout: 5
27+
read_timeout: 30
1828
## Buffer locally for up to 10s then flush; on Cribl outage chunks
1929
## get retried up to retry_max_times with exponential backoff.
2030
## flush_mode: interval is required for flush_interval to take effect

0 commit comments

Comments
 (0)