parser: add csv format parser (Format=csv) - #12273
Conversation
Adds a new "csv" parser format (FLB_PARSER_CSV), picking up the idea from the never-merged fluent#5040, but built for performance: a single left-to-right scan over each record records (offset, length) pairs for every field with no intermediate copies, then a second pass writes the already-known-size msgpack map directly from those positions. Fields on the stack (up to 128) avoid heap allocation entirely for typical CSV lines and grow geometrically onto the heap only for pathological inputs; unescaping of embedded "" is only done for the (rare) fields that actually contain a quote, also via a reusable stack buffer. By default, fields are assigned numbered string keys ("0", "1", ...). An optional comma separated `csv_fields` parser config key assigns named keys instead. `time_key` designates which field carries the record timestamp - a 0-based index when no `csv_fields` is set, or a field name when it is - reusing the existing time_format/time_keep machinery shared with the other parsers. Also registers tests/internal/parser_csv.c covering basic splitting, trailing empty fields, quoted/escaped fields, named fields, time_key by index and by name, time_keep, and type casting. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (6)
📝 WalkthroughWalkthroughCSV parser support is added to the public parser API, configuration loader, build system, runtime dispatch, and internal tests. The parser supports quoted fields, named fields, timestamps, type casting, decoder processing, and MessagePack output. ChangesCSV parser support
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant ParserConfig
participant flb_parser_csv_set_fields
participant flb_parser_csv_resolve_time_field
participant flb_parser_csv_do
participant MessagePackOutput
ParserConfig->>flb_parser_csv_set_fields: configure CSV field names
ParserConfig->>flb_parser_csv_resolve_time_field: resolve timestamp index
ParserConfig->>flb_parser_csv_do: parse CSV record
flb_parser_csv_do->>MessagePackOutput: emit typed fields and timestamp
Suggested reviewers: ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f23d47d91a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (!parser->time_key) { | ||
| return; | ||
| } |
There was a problem hiding this comment.
When a CSV parser is configured with Time_Format and named fields such as CSV_Fields time,msg but omits Time_Key, this early return leaves csv_time_field_index at -1. Unlike the existing JSON/LTSV/logfmt/regex parsers, which default a missing time_key to "time", flb_parser_csv_do() will never parse the CSV timestamp, so inputs fall back to ingestion/current time and the time field is not removed when Time_Keep is false. Resolve "time" against csv_field_names when parser->time_key is NULL.
Useful? React with 👍 / 👎.
1 participant
Adds a new "csv" parser format (FLB_PARSER_CSV), picking up the idea from the never-merged #5040, but built for performance: a single left-to-right scan over each record records (offset, length) pairs for every field with no intermediate copies, then a second pass writes the already-known-size msgpack map directly from those positions. Fields on the stack (up to 128) avoid heap allocation entirely for typical CSV lines and grow geometrically onto the heap only for pathological inputs; unescaping of embedded "" is only done for the (rare) fields that actually contain a quote, also via a reusable stack buffer.
By default, fields are assigned numbered string keys ("0", "1", ...). An optional comma separated
csv_fieldsparser config key assigns named keys instead.time_keydesignates which field carries the record timestamp - a 0-based index when nocsv_fieldsis set, or a field name when it is - reusing the existing time_format/time_keep machinery shared with the other parsers.Also registers tests/internal/parser_csv.c covering basic splitting, trailing empty fields, quoted/escaped fields, named fields, time_key by index and by name, time_keep, and type casting.
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.
Summary by CodeRabbit
New Features
Tests