Skip to content

Commit 5ef4048

Browse files
mikemiles-devmikemiles-dev
authored
docs: clean up README and bump version to 1.0.1 (#283)
Remove outdated 0.7.x migration guide, consolidate redundant sections, and reduce README by ~220 lines without losing any information.
1 parent 84c6d3e commit 5ef4048

3 files changed

Lines changed: 19 additions & 276 deletions

File tree

‎Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
[package]
22
name = "netflow_parser"
33
description = "Parser for Netflow Cisco V5, V7, V9, IPFIX"
4-
version = "1.0.0"
4+
version = "1.0.1"
55
edition = "2024"
66
rust-version = "1.88"
77
authors = ["Michael Mileusnich <michael.mileusnich@gmail.com>"]

‎README.md‎

Lines changed: 7 additions & 275 deletions
Original file line numberDiff line numberDiff line change
@@ -25,12 +25,9 @@ A Netflow Parser library for Cisco V5, V7, V9, and IPFIX written in Rust. Suppor
2525
- [Custom Enterprise Fields (IPFIX)](#custom-enterprise-fields-ipfix)
2626
- [Netflow Common](#netflow-common)
2727
- [Re-Exporting Flows](#re-exporting-flows)
28-
- [V9/IPFIX Notes](#v9ipfix-notes)
2928
- [Template Management Guide](#template-management-guide)
3029
- [Template Cache Metrics](#template-cache-metrics)
3130
- [Multi-Source Deployments](#multi-source-deployments)
32-
- [Template Collision Detection](#template-collision-detection)
33-
- [Handling Missing Templates](#handling-missing-templates)
3431
- [Template Lifecycle Management](#template-lifecycle-management)
3532
- [Best Practices](#best-practices)
3633
- [Performance & Thread Safety](#performance--thread-safety)
@@ -67,11 +64,6 @@ Structures fully support serialization. Below is an example using the serde_jso
6764
use serde_json::json;
6865
use netflow_parser::NetflowParser;
6966

70-
// 0000 00 05 00 01 03 00 04 00 05 00 06 07 08 09 00 01 ................
71-
// 0010 02 03 04 05 06 07 08 09 00 01 02 03 04 05 06 07 ................
72-
// 0020 08 09 00 01 02 03 04 05 06 07 08 09 00 01 02 03 ................
73-
// 0030 04 05 06 07 08 09 00 01 02 03 04 05 06 07 08 09 ................
74-
// 0040 00 01 02 03 04 05 06 07 ........
7567
let v5_packet = [0, 5, 0, 1, 3, 0, 4, 0, 5, 0, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7,];
7668
let result = NetflowParser::default().parse_bytes(&v5_packet);
7769
println!("{}", json!(result.packets).to_string());
@@ -127,11 +119,6 @@ println!("{}", json!(result.packets).to_string());
127119
```rust
128120
use netflow_parser::{NetflowParser, NetflowPacket};
129121

130-
// 0000 00 05 00 01 03 00 04 00 05 00 06 07 08 09 00 01 ................
131-
// 0010 02 03 04 05 06 07 08 09 00 01 02 03 04 05 06 07 ................
132-
// 0020 08 09 00 01 02 03 04 05 06 07 08 09 00 01 02 03 ................
133-
// 0030 04 05 06 07 08 09 00 01 02 03 04 05 06 07 08 09 ................
134-
// 0040 00 01 02 03 04 05 06 07 ........
135122
let v5_packet = [0, 5, 0, 1, 3, 0, 4, 0, 5, 0, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7,];
136123
let result = NetflowParser::default().parse_bytes(&v5_packet);
137124

@@ -171,35 +158,7 @@ for result in parser.iter_packets(&buffer) {
171158
}
172159
```
173160

174-
The iterator provides access to unconsumed bytes for advanced use cases:
175-
176-
```rust,ignore
177-
use netflow_parser::NetflowParser;
178-
179-
let buffer = /* your netflow data */;
180-
let mut parser = NetflowParser::default();
181-
let mut iter = parser.iter_packets(&buffer);
182-
183-
while let Some(packet) = iter.next() {
184-
// Process packet
185-
}
186-
187-
// Check if all bytes were consumed
188-
if !iter.is_complete() {
189-
println!("Warning: {} bytes remain unconsumed", iter.remaining().len());
190-
}
191-
```
192-
193-
### Benefits of Iterator API
194-
195-
- **Zero allocation**: Packets are yielded one-by-one without allocating a `Vec`
196-
- **Memory efficient**: Ideal for processing large batches or continuous streams
197-
- **Lazy evaluation**: Only parses packets as you consume them
198-
- **Template caching preserved**: V9/IPFIX template state is maintained across iterations
199-
- **Composable**: Works with standard Rust iterator methods (`.filter()`, `.map()`, `.take()`, etc.)
200-
- **Buffer inspection**: Access unconsumed bytes via `.remaining()` and check completion with `.is_complete()`
201-
202-
### Iterator Examples
161+
The iterator is zero-allocation, lazy, preserves V9/IPFIX template state, and composes with standard Rust iterator methods (`.filter()`, `.map()`, `.take()`, etc.). You can also inspect unconsumed bytes via `.remaining()` and check completion with `.is_complete()`.
203162

204163
```rust,ignore
205164
// Count V5 packets without collecting
@@ -213,23 +172,6 @@ for result in parser.iter_packets(&buffer).take(10) {
213172
// Handle packet
214173
}
215174
}
216-
217-
// Collect only if needed (equivalent to parse_bytes())
218-
let packets: Vec<_> = parser.iter_packets(&buffer)
219-
.filter_map(Result::ok)
220-
.collect();
221-
222-
// Check unconsumed bytes (useful for mixed protocol streams)
223-
let mut iter = parser.iter_packets(&buffer);
224-
for result in &mut iter {
225-
if let Ok(packet) = result {
226-
// Process packet
227-
}
228-
}
229-
if !iter.is_complete() {
230-
let remaining = iter.remaining();
231-
// Handle non-netflow data at end of buffer
232-
}
233175
```
234176

235177
## Parser Configuration
@@ -506,54 +448,6 @@ let parser = NetflowParser::builder()
506448

507449
This setting helps prevent memory exhaustion when processing malformed or malicious packets while still providing enough context for debugging.
508450

509-
#### Migration Guide
510-
511-
##### From 0.7.x to 0.8.0
512-
513-
**What changed:** Two major improvements to error handling:
514-
515-
1. **ParseResult** - `parse_bytes()` now returns `ParseResult` to preserve partial results on errors
516-
2. **Error Handling** - `NetflowPacket::Error` variant removed, errors now use `Result`
517-
518-
**ParseResult (prevents data loss):**
519-
520-
```rust,ignore
521-
// ❌ Old (0.7.x) - loses packets 1-4 if packet 5 errors
522-
let packets = parser.parse_bytes(&data); // Returns Vec<NetflowPacket>
523-
// Silent error: if parsing stopped at packet 5, you lost packets 1-4
524-
525-
// ✅ New (0.8.0) - keep packets 1-4 even if packet 5 errors
526-
let result = parser.parse_bytes(&data); // Returns ParseResult
527-
for packet in result.packets {
528-
// Process successfully parsed packets 1-4
529-
}
530-
if let Some(e) = result.error {
531-
eprintln!("Error at packet 5: {}", e); // But still got partial results!
532-
}
533-
```
534-
535-
**Error Handling (use Result instead of Error variant):**
536-
537-
```rust,ignore
538-
// ❌ Old (0.7.x) - errors inline with packets
539-
for packet in parser.parse_bytes(&data) {
540-
match packet {
541-
NetflowPacket::V5(v5) => { /* process */ }
542-
NetflowPacket::Error(e) => { /* error */ }
543-
_ => {}
544-
}
545-
}
546-
547-
// ✅ New (0.8.0) - use iter_packets() for Result-based errors
548-
for result in parser.iter_packets(&data) {
549-
match result {
550-
Ok(NetflowPacket::V5(v5)) => { /* process */ }
551-
Err(e) => { /* error */ }
552-
_ => {}
553-
}
554-
}
555-
```
556-
557451
### Custom Enterprise Fields (IPFIX)
558452

559453
IPFIX supports vendor-specific enterprise fields that extend the standard IANA field set. The library provides built-in support for several vendors (Cisco, VMWare, Netscaler, etc.), but you can also register your own custom enterprise fields:
@@ -643,116 +537,21 @@ When registering enterprise fields, you can use any of these built-in data types
643537

644538
See `examples/custom_enterprise_fields.rs` for a complete working example.
645539

646-
### Complete Configuration Example
647-
648-
```rust
649-
use netflow_parser::NetflowParser;
650-
use netflow_parser::variable_versions::ttl::TtlConfig;
651-
use netflow_parser::variable_versions::field_value::FieldDataType;
652-
use netflow_parser::variable_versions::enterprise_registry::EnterpriseFieldDef;
653-
use std::time::Duration;
654-
655-
let parser = NetflowParser::builder()
656-
// Cache configuration
657-
.with_v9_cache_size(1000)
658-
.with_ipfix_cache_size(2000)
659-
660-
// Security limits
661-
.with_v9_max_field_count(5000)
662-
.with_ipfix_max_field_count(10000)
663-
.with_max_error_sample_size(512)
664-
665-
// Template TTL
666-
.with_v9_ttl(TtlConfig::new(Duration::from_secs(3600)))
667-
.with_ipfix_ttl(TtlConfig::new(Duration::from_secs(7200)))
668-
669-
// Version filtering
670-
.with_allowed_versions(&[5, 9, 10])
671-
672-
// Enterprise fields
673-
.register_enterprise_fields(vec![
674-
EnterpriseFieldDef::new(12345, 1, "field1", FieldDataType::UnsignedDataNumber),
675-
EnterpriseFieldDef::new(12345, 2, "field2", FieldDataType::String),
676-
])
677-
678-
// Template lifecycle hooks
679-
.on_template_event(|event| {
680-
println!("Template event: {:?}", event);
681-
Ok(())
682-
})
683-
684-
.build()
685-
.expect("Failed to build parser");
686-
687-
// For multi-source deployments, use AutoScopedParser instead:
688-
// let scoped_parser = NetflowParser::builder()./* config */.try_multi_source().expect("valid config");
689-
```
690-
691540
## Netflow Common
692541

693-
We have included a `NetflowCommon` and `NetflowCommonFlowSet` structure.
694-
This will allow you to use common fields without unpacking values from specific versions.
695-
If the packet flow does not have the matching field it will simply be left as `None`.
696-
697-
### NetflowCommon and NetflowCommonFlowSet Struct:
698-
```rust
699-
use std::net::IpAddr;
700-
use netflow_parser::protocol::ProtocolTypes;
701-
702-
#[derive(Debug, Default)]
703-
pub struct NetflowCommon {
704-
pub version: u16,
705-
pub timestamp: u32,
706-
pub flowsets: Vec<NetflowCommonFlowSet>,
707-
}
708-
709-
#[derive(Debug, Default, Clone)]
710-
pub struct NetflowCommonFlowSet {
711-
pub src_addr: Option<IpAddr>,
712-
pub dst_addr: Option<IpAddr>,
713-
pub src_port: Option<u16>,
714-
pub dst_port: Option<u16>,
715-
pub protocol_number: Option<u8>,
716-
pub protocol_type: Option<ProtocolTypes>,
717-
pub first_seen: Option<u64>,
718-
pub last_seen: Option<u64>,
719-
pub src_mac: Option<String>,
720-
pub dst_mac: Option<String>,
721-
}
722-
```
723-
724-
### Converting NetflowPacket to NetflowCommon
542+
`NetflowCommon` and `NetflowCommonFlowSet` let you work with common fields (src/dst addr, ports, protocol, etc.) without unpacking version-specific structures. Fields not present in a given version are `None`.
725543

726544
```rust,ignore
727545
use netflow_parser::{NetflowParser, NetflowPacket};
728546
729-
// 0000 00 05 00 01 03 00 04 00 05 00 06 07 08 09 00 01 ................
730-
// 0010 02 03 04 05 06 07 08 09 00 01 02 03 04 05 06 07 ................
731-
// 0020 08 09 00 01 02 03 04 05 06 07 08 09 00 01 02 03 ................
732-
// 0030 04 05 06 07 08 09 00 01 02 03 04 05 06 07 08 09 ................
733-
// 0040 00 01 02 03 04 05 06 07 ........
734-
let v5_packet = [0, 5, 0, 1, 3, 0, 4, 0, 5, 0, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3,
735-
4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1,
736-
2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7];
737547
let result = NetflowParser::default().parse_bytes(&v5_packet);
738-
let netflow_common = result.packets
739-
.first()
740-
.unwrap()
741-
.as_netflow_common()
742-
.unwrap();
548+
let netflow_common = result.packets.first().unwrap().as_netflow_common().unwrap();
743549
744550
for common_flow in netflow_common.flowsets.iter() {
745551
println!("Src Addr: {} Dst Addr: {}", common_flow.src_addr.unwrap(), common_flow.dst_addr.unwrap());
746552
}
747-
```
748-
749-
### Flattened flowsets
750-
751-
To gather all flowsets from all packets into a flattened vector:
752-
753-
```rust,ignore
754-
use netflow_parser::NetflowParser;
755553
554+
// Or gather all flowsets from all packets into a flattened vector:
756555
let (flowsets, error) = NetflowParser::default().parse_bytes_as_netflow_common_flowsets(&v5_packet);
757556
```
758557

@@ -841,11 +640,6 @@ See `examples/manual_ipfix_creation.rs` for a complete example of creating IPFIX
841640

842641
```rust
843642
use netflow_parser::{NetflowParser, NetflowPacket};
844-
// 0000 00 05 00 01 03 00 04 00 05 00 06 07 08 09 00 01 ................
845-
// 0010 02 03 04 05 06 07 08 09 00 01 02 03 04 05 06 07 ................
846-
// 0020 08 09 00 01 02 03 04 05 06 07 08 09 00 01 02 03 ................
847-
// 0030 04 05 06 07 08 09 00 01 02 03 04 05 06 07 08 09 ................
848-
// 0040 00 01 02 03 04 05 06 07 ........
849643
let packet = [
850644
0, 5, 0, 1, 3, 0, 4, 0, 5, 0, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3,
851645
4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1,
@@ -857,21 +651,11 @@ if let Some(NetflowPacket::V5(v5)) = result.packets.first() {
857651
}
858652
```
859653

860-
## V9/IPFIX Notes
861-
862-
Parse the data (`&[u8]`) like any other version. The parser (`NetflowParser`) caches parsed templates using LRU eviction, so you can send header/data flowset combos and it will use the cached templates. Templates are automatically cached and evicted when the cache limit is reached.
863-
864-
**Template Management:** For comprehensive information about template caching, introspection, multi-source deployments, and best practices, see the [Template Management Guide](#template-management-guide) section below.
865-
866-
**IPFIX Note:** We only parse sequence number and domain id, it is up to you if you wish to validate it.
867-
868-
**FlowSet Access:** To access templates flowset of a processed V9/IPFIX flowset you can find the `flowsets` attribute on the Parsed Record. In there you can find `Templates`, `Option Templates`, and `Data` Flowsets.
869-
870654
## Template Management Guide
871655

872-
### Overview
656+
NetFlow V9 and IPFIX are template-based protocols where templates define the structure of flow records. The parser caches templates using LRU eviction, so you can send header/data flowset combos and it will use the cached templates. Access parsed templates, option templates, and data flowsets via the `flowsets` attribute on the parsed record.
873657

874-
NetFlow V9 and IPFIX are template-based protocols where templates define the structure of flow records. This library provides comprehensive template management features to handle various deployment scenarios.
658+
**IPFIX Note:** We only parse sequence number and domain id; it is up to you if you wish to validate them.
875659

876660
### Template Cache Metrics
877661

@@ -907,8 +691,7 @@ if let Some(hit_rate) = metrics.hit_rate() {
907691
- **Hits**: Successful template lookups
908692
- **Misses**: Failed template lookups (template not in cache)
909693
- **Evictions**: Templates removed due to LRU policy when cache is full
910-
- **Collisions**: Template ID reused with a **different definition** (same ID, different schema)
911-
- Note: RFC-compliant template retransmissions (same ID, same definition) are NOT counted as collisions
694+
- **Collisions**: Template ID reused with a **different definition** (same ID, different schema). RFC-compliant retransmissions (same ID, identical definition) are NOT counted as collisions. High collision rates suggest you need scoped parsing (see below).
912695
- **Expired**: Templates removed due to TTL expiration
913696

914697
### Multi-Source Deployments (RFC-Compliant)
@@ -1006,57 +789,6 @@ let router_builder = NetflowParser::builder()
1006789
let mut scoped = RouterScopedParser::<String>::try_with_builder(router_builder).expect("valid config");
1007790
```
1008791

1009-
### Template Collision Detection
1010-
1011-
Monitor when template IDs are reused with different definitions:
1012-
1013-
```rust,ignore
1014-
let v9_info = parser.v9_cache_info();
1015-
if v9_info.metrics.collisions > 0 {
1016-
println!("Warning: {} template collisions detected", v9_info.metrics.collisions);
1017-
println!("Use AutoScopedParser for RFC-compliant multi-source deployments");
1018-
}
1019-
```
1020-
1021-
**What counts as a collision:**
1022-
- Same template ID with a **different definition** (field structure changed)
1023-
- This typically indicates multiple routers using the same template ID for different schemas
1024-
1025-
**What does NOT count as a collision:**
1026-
- Retransmitting the same template (same ID, identical definition)
1027-
- RFC 7011 (IPFIX) and RFC 3954 (NetFlow v9) recommend periodic template retransmission for reliability
1028-
- Template refreshes are normal and expected behavior
1029-
1030-
### Handling Missing Templates
1031-
1032-
When a data flowset arrives before its template (IPFIX):
1033-
1034-
```rust,ignore
1035-
use netflow_parser::{NetflowParser, NetflowPacket};
1036-
use netflow_parser::variable_versions::ipfix::FlowSetBody;
1037-
1038-
let mut parser = NetflowParser::default();
1039-
let mut pending_data = Vec::new();
1040-
1041-
for packet in parser.iter_packets(&data) {
1042-
if let Ok(NetflowPacket::IPFix(ipfix)) = packet {
1043-
for flowset in &ipfix.flowsets {
1044-
if let FlowSetBody::NoTemplate(info) = &flowset.body {
1045-
println!("Missing template ID: {}", info.template_id);
1046-
1047-
// Save for retry after template arrives
1048-
pending_data.push(info.raw_data.clone());
1049-
}
1050-
}
1051-
}
1052-
}
1053-
1054-
// Retry pending data after templates arrive
1055-
for pending in &pending_data {
1056-
let _ = parser.parse_bytes(pending);
1057-
}
1058-
```
1059-
1060792
### Template Lifecycle Management
1061793

1062794
#### Template Introspection

0 commit comments

Comments
 (0)