fix: bound cumulative decoded output per message

[ktsaou]

Problem

NetFlow v9 and IPFIX currently limit records per FlowSet/Set and fields per Template, but they do not limit the combined decoded output of a complete message.

A small message can therefore expand into a disproportionate number of heap-backed field values. A minimal NetFlow v9 regression uses 64 zero-width fields plus one one-byte field and 1,009 records: the current parser accepts the message and materializes 65,585 field values.

The same missing cumulative accounting affects native IPFIX data, IPFIX options data, v9-style records carried through IPFIX, and pending records replayed when a template arrives.

Fix

• Add independent, configurable per-message limits for NetFlow v9 and IPFIX.
• Default to 65,536 decoded field values and 4 MiB of decoded field content.
• Count fixed fields by declared width; count IPFIX variable-field content without its wire-length prefix; count zero-width fields as values with zero content bytes.
• Use checked, allocation-free accounting before proportional field-value materialization.
• Reject the complete message with a typed DecodedOutputLimitExceeded error when either limit is exceeded.
• Apply the remaining message budget to pending replay.
• Replay the largest complete FIFO prefix that fits, retain a temporarily blocked suffix, and drop entries proven malformed or unable to fit an otherwise empty budget.
• Add explicit finite-limit companions for the existing low-level Data and OptionsData parsers.

Compatibility

• Existing public method signatures and wire/serialization formats are unchanged.
• Configuration and error APIs are additive.
• Existing low-level parsing methods now use the same finite defaults as the stateful parser; callers needing different bounds can use DecodedOutputLimits.
• Ordinary messages below the limits retain existing behavior.

Verification

• The minimal amplification regression fails on the unmodified parser and passes with this change.
• 25 all-feature and 24 no-default-feature focused tests cover exact limits, one-over failures, both protocols, Data/Options paths, embedded v9 paths, direct parser boundaries, checked overflow, padding/completeness, and pending FIFO behavior.
• Fresh-process allocation tests keep the adversarial default-limit case below 16 MiB peak requested live bytes and a preflight-blocked replay below 1 MiB.
• A release allocation matrix across 16 warmed Direct/Auto, v9/IPFIX, parse/iterator, one/1,000-record paths matches clean main exactly and returns to zero retained live-byte delta.
• Paired, CPU-pinned release benchmarks across the same 16 paths retain at least 95% of clean-main median throughput with relative MAD at most 3%.
• scripts/check-all.sh, complete no-default-feature tests, strict Clippy, and Rust 1.88 test compilation pass on current main.

[Copilot]

Pull request overview

This PR adds cumulative, per-message decoded-output budgeting to NetFlow v9 and IPFIX parsing to prevent small inputs from expanding into disproportionate heap-backed decoded field values/payload, including during pending-flow replay when templates arrive.

Changes:

• Introduces shared decoded-output budgeting (DecodedOutputBudget) with configurable per-message limits and a typed NetflowError::DecodedOutputLimitExceeded.
• Integrates allocation-free preflight + bounded replay for pending flows, replaying the largest FIFO prefix that fits the remaining message budget and retaining a temporarily blocked suffix.
• Adds extensive regression/limit/allocation tests and updates docs/bench harnesses to reflect the new limits and hot paths.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/hot_path_allocations.rs	Adds deterministic warmed allocation profiling for common parser APIs.
tests/decoded_output_limits.rs	Adds comprehensive behavioral tests for per-message decoded output limits and pending replay behavior.
tests/decoded_output_default_regression.rs	Adds regression test for the decoded-output amplification case under default limits.
tests/decoded_output_allocation.rs	Adds fresh-process allocation gates to ensure adversarial cases stay bounded.
src/variable_versions/wire.rs	Adds shared checked wire-record sizing/padding completeness helpers for v9/IPFIX.
src/variable_versions/v9/types.rs	Wires decoded-output budgeting into v9 Data/Options parsing and adds parse_with_limits.
src/variable_versions/v9/parser.rs	Adds per-message budget reset, enforces limits, and updates pending replay to honor remaining message budget.
src/variable_versions/pending_flows.rs	Adds API to restore an unprocessed FIFO suffix back into the pending-flow cache.
src/variable_versions/output_budget.rs	Implements shared decoded-output accounting, limits types, and pending preflight/materialization logic.
src/variable_versions/mod.rs	Exposes new decoded-output limit types/constants and wires internal modules.
src/variable_versions/metrics.rs	Removes the bulk “failed_n” helper now that replay can retain suffixes instead of failing them.
src/variable_versions/ipfix/types.rs	Resets decoded-output budget per message and ensures over-limit parses fail as TooLarge for typed error reporting.
src/variable_versions/ipfix/parser.rs	Integrates budgeting into IPFIX parsing and budget-aware pending replay across IPFIX and embedded v9 paths.
src/variable_versions/config.rs	Adds decoded-output limit configuration, defaults, and validation errors.
src/tests.rs	Updates config fixtures to include new decoded-output limit defaults.
src/lib.rs	Exposes new builder knobs/constants and introduces NetflowError::DecodedOutputLimitExceeded.
SECURITY.md	Documents the new cumulative decoded-output DoS mitigation knobs and defaults.
README.md	Documents cumulative decoded-output limits and provides configuration example.
benches/hot_path_bench.rs	Updates hot-path benchmarks to cover direct/auto parse+iterator across v9/IPFIX.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.