Feature/inline parse data fields - #285
Merged
Merged
Conversation
Eliminates per-record function call overhead and Vec allocation in the V9 field parser hot path. The inner field-parsing loop is now inlined directly, giving the optimizer a single loop nest to work with and allowing error recovery without unwinding through a function boundary. Benchmark results (V9 data parsing, 1000 flows): main: 130.04 µs (146.85 MiB/s) optimized: 120.12 µs (158.98 MiB/s) ~8% improvement Also adds hot_path_bench for targeted V9/IPFIX data parsing benchmarks.
Extract hand-rolled from_be_bytes parsers into a dedicated fast_parse module with comprehensive unit tests. nom's generic be_uint uses a Shl+Add+From<u8> trait loop that LLVM does not optimize into bswap/rev instructions at wider integer widths (u64, u128). Micro-benchmarks show 9-26x speedup for the from_be_bytes versions. Replace nom number parsers in DataNumber::parse, FieldValue IP/MAC parsing, and IPFIX variable-length field parsing. Update README with new performance optimization notes.
Inlines the parse_data_fields helper directly into the record loop, eliminating per-record function call overhead and Vec allocation. This was accidentally reverted during baseline benchmarking.
Owner
Author
|
@bmjask based off findings from their PR |
There was a problem hiding this comment.
Pull request overview
This PR focuses on improving NetFlow V9/IPFIX parsing throughput by removing avoidable per-record overhead and replacing nom’s generic big-endian number parsers with dedicated from_be_bytes-based implementations.
Changes:
- Inline V9 data-field parsing into the record loop to reduce per-record call/allocation overhead.
- Introduce
variable_versions::fast_parseand use it for hot-path big-endian parsing (DataNumber, IP/MAC parsing, and IPFIX variable-length field lengths). - Add a new Criterion benchmark (
hot_path_bench) and bump crate version to1.0.2.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| src/variable_versions/v9/parser.rs | Inlines record parsing logic and precomputes template field references/capacity for faster V9 data parsing. |
| src/variable_versions/mod.rs | Adds the new internal fast_parse module. |
| src/variable_versions/ipfix/parser.rs | Switches IPFIX variable-length field-length parsing to fast_parse helpers. |
| src/variable_versions/field_value.rs | Uses fast_parse for DataNumber parsing and IP/MAC decoding paths. |
| src/variable_versions/fast_parse.rs | New optimized big-endian primitive parsers with unit tests. |
| benches/hot_path_bench.rs | Adds a hot-path benchmark for warmed V9/IPFIX data parsing. |
| README.md | Documents the new fast parsers and V9 inlined parsing optimization. |
| Cargo.toml | Bumps version and registers the new benchmark target. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
throughput improvement)
Shl+Add+From trait loop that LLVM fails to optimize into bswap/rev instructions at wider integer widths (u64, u128), resulting in
byte-at-a-time loops. Micro-benchmarks show 9-26x speedup per parse call.