You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- Complete benchmark documentation and methodology
28
28
- Cross-model evaluation results and scoring framework
29
29
30
-
NetOpsBench evaluates RCA agents inside a real reproducible data-center network loop: build a SONiC-VS / Containerlab fabric, inject a controlled fault, observe Pingmesh and telemetry symptoms, call an agent, and score the diagnosis against ground truth.
30
+
NetOpsBench evaluates agentic RCA inside a real reproducible data-center network loop: build a SONiC-VS / Containerlab fabric, inject a controlled fault, observe Pingmesh and telemetry symptoms, call an agent, and score the diagnosis against ground truth.
31
31
32
32
It is built for researchers and engineers who want to compare LLM-backed, symbolic, heuristic, or hybrid troubleshooting strategies on the same operational benchmark, not just on static logs or hand-written prompts.
33
33
@@ -38,7 +38,7 @@ It is built for researchers and engineers who want to compare LLM-backed, symbol
38
38
| Capability | What NetOpsBench gives you |
39
39
|---|---|
40
40
| Run the network, not just logs | Faults execute against live SONiC/Containerlab topologies with traffic and observability. |
41
-
| Bring any RCA agent| Pass a Python object with `diagnose(context)` into the public SDK. |
41
+
| Bring any agentic strategy| Pass a Python object with `diagnose(context)` into the public SDK. |
42
42
| Score localization, not only detection | Measure fault type, device, interface, runtime, tool calls, and token cost. |
43
43
| Scale with isolated worker pools | Run suites across independent labs and merge results into one `BenchmarkReport`. |
44
44
@@ -47,7 +47,7 @@ It is built for researchers and engineers who want to compare LLM-backed, symbol
47
47
NetOpsBench runtime execution requires Linux because Containerlab depends on Linux networking primitives.
Copy file name to clipboardExpand all lines: docs/content/docs/benchmark/methodology.mdx
+45-1Lines changed: 45 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,6 +17,50 @@ Each benchmark case pairs a generated scenario with ground truth:
17
17
18
18
The agent returns a `DiagnosisResult` containing a verdict, location, confidence, evidence, reasoning, and optional metadata such as tool calls or token usage.
19
19
20
+
## Benchmark test suite
21
+
22
+
### Topology scales
23
+
24
+
All topologies use a spine-leaf fabric running SONiC-VS switches. Scale increases both the number of network devices and the density of clients, making diagnosis progressively harder as the fault signal is diluted across a larger Pingmesh matrix.
"Network devices" counts switches only. Clients are Linux containers attached to leaf switches and act as Pingmesh endpoints.
34
+
35
+
### Fault coverage
36
+
37
+
The benchmark covers twelve canonical fault types across five categories. Each type corresponds to a distinct failure mode that an operations agent may encounter in a real DCN.
38
+
39
+
| Category | Fault types | Difficulty |
40
+
|---|---|---|
41
+
| Link |`link_down`, `link_flapping`| easy – hard |
42
+
| Routing |`blackhole_route`, `static_route_misconfig`, `bgp_neighbor_misconfig`, `route_policy_misconfig`| medium – hard |
43
+
| Impairment |`mtu_mismatch`, `packet_loss`, `packet_corruption`, `high_latency`| medium – hard |
44
+
| System |`device_down`| easy |
45
+
| ACL |`acl_misconfig`| hard |
46
+
47
+
All twelve fault types appear in every topology scale. Each scenario instance is placed on a randomly selected device that matches the fault template's role constraint (spine, leaf, or client-facing interface).
48
+
49
+
### Scenario breakdown
50
+
51
+
Each scale combines fault scenarios with a set of healthy negative samples. Healthy cases exercise false-positive control: the network is operating normally and the correct answer is `network_healthy`.
52
+
53
+
| Scale | Fault cases | Healthy cases | Total |
54
+
|---|---:|---:|---:|
55
+
| XS | 12 | 2 | 14 |
56
+
| Small | 12 | 3 | 15 |
57
+
| Medium | 24 | 4 | 28 |
58
+
| Large | 48 | 4 | 52 |
59
+
60
+
### Scenario generation
61
+
62
+
Scenarios are defined in `scenarios/specs/fault_campaign.yaml`, which specifies a `count_per_scale` entry for each fault template. Running `netopsbench benchmark prepare --scales <scale>` materialises that spec into individual YAML scenario files under `scenarios/generated/<scale>/`. The generator picks a qualifying device and interface at random for each instance, so every preparation pass produces a structurally identical but topologically varied set.
63
+
20
64
## Core scoring dimensions
21
65
22
66
| Dimension | Meaning |
@@ -36,7 +80,7 @@ Fault-case score is verdict-gated. If the agent misses the fault verdict, locali
36
80
When an interface is applicable, NetOpsBench uses a localization-first score:
For metric definitions, see [Benchmark Methodology](/docs/benchmark/methodology).
30
+
For topology dimensions, fault type details, and scenario counts, see [Benchmark Methodology — Test suite](/docs/benchmark/methodology#benchmark-test-suite). For metric definitions, see [Benchmark Methodology — Scoring](/docs/benchmark/methodology#core-scoring-dimensions).
30
31
31
32
## Cross-scale snapshot
32
33
33
34
Read each cell as `XS -> Large`. Large absolute values are the strongest signal for deployment readiness.
34
35
35
-
|Vendor| Detection (%) | Device Loc (%) | Interface Loc (%) | Composite (%) | Avg Time (s) | Avg Tools | Input Tokens (K) |
36
+
|Model| Detection (%) | Device Loc (%) | Interface Loc (%) | Composite (%) | Avg Time (s) | Avg Tools | Input Tokens (K) |
Most models maintain high detection. OpenAI and DeepSeek hold 100% from XS through Medium and stay near 80% on Large. MiniMax climbs from 25% on XS to 94% on Large. Detection is a starting point, not a final RCA outcome.
51
+
Most models maintain high detection. OpenAI GPT-5.4 and DeepSeek V4 Pro hold 100% from XS through Medium and stay near 80% on Large. MiniMax M2.7 climbs from 25% on XS to 94% on Large. Detection is a starting point, not a final RCA outcome.
Device localization is the first real bottleneck. Kimi reaches 91.7% on Small but drops to 33.3% on Large. DeepSeek peaks at 95.8% on Medium then falls to 37.5%. OpenAI is the strongest on Large at 45.8%, still below a production-grade RCA target.
57
+
Device localization is the first real bottleneck. Kimi K2.6 reaches 91.7% on Small but drops to 33.3% on Large. DeepSeek V4 Pro peaks at 95.8% on Medium then falls to 37.5%. OpenAI GPT-5.4 is the strongest on Large at 45.8%, still below a production-grade RCA target.
Interface precision is the weakest dimension. On Large, OpenAI and MiniMax fall to 0%; Kimi and DeepSeek reach only 14.3%. Many real fixes are interface-scoped, so this gap is the main barrier to operational automation.
63
+
Interface precision is the weakest dimension. On Large, OpenAI GPT-5.4 and MiniMax M2.7 fall to 0%; Kimi K2.6 and DeepSeek V4 Pro reach only 14.3%. Many real fixes are interface-scoped, so this gap is the main barrier to operational automation.
The composite summarizes localization quality. All four models drop sharply at Large (Kimi 34.2, DeepSeek 30.0, OpenAI 32.3, MiniMax 23.8), confirming that current agents perform well on small topologies but lose actionable precision at scale.
69
+
The composite summarizes localization quality. All four models drop sharply at Large (Kimi K2.6 34.2, DeepSeek V4 Pro 30.0, OpenAI GPT-5.4 32.3, MiniMax M2.7 23.8), confirming that current agents perform well on small topologies but lose actionable precision at scale.
69
70
70
71
## Cost: time, tools, tokens
71
72
72
73
More time, tool calls, or tokens do not reliably translate into better localization.
Kimi spends the most tokens (517K input / 8K output per case at Large); OpenAI the least (194K / 3K). Cost should always be evaluated against device and interface localization, not against detection alone.
86
+
Kimi K2.6 spends the most tokens (517K input / 8K output per case at Large); OpenAI GPT-5.4 the least (194K / 3K). Cost should always be evaluated against device and interface localization, not against detection alone.
86
87
87
88
## Model fit summary
88
89
89
90
The best choice depends on whether the priority is localization quality, fast triage, cost predictability, or scale readiness.
90
91
91
92
| Model | Strongest signal | Main risk | Best fit |
92
93
|---|---|---|---|
93
-
| Kimi | Strong Small-scale composite, early device localization | Large quality drops while time and token cost rise sharply | Deep-diagnosis runs with generous reasoning budget |
94
-
| DeepSeek | Balanced quality on XS, Small, Medium | Large localization still degrades materially | Quality-oriented RCA evaluation |
95
-
| OpenAI | Fast, low tool calls, strong detection | Interface localization collapses at Large | Fast triage, latency-sensitive flows |
96
-
| MiniMax | Lower runtime/token footprint, improving detection | Lower absolute localization at Large | Budget-sensitive baselines and broad triage |
94
+
| Kimi K2.6 | Strong Small-scale composite, early device localization | Large quality drops while time and token cost rise sharply | Deep-diagnosis runs with generous reasoning budget |
95
+
| DeepSeek V4 Pro | Balanced quality on XS, Small, Medium | Large localization still degrades materially | Quality-oriented RCA evaluation |
96
+
| OpenAI GPT-5.4 | Fast, low tool calls, strong detection | Interface localization collapses at Large | Fast triage, latency-sensitive flows |
97
+
| MiniMax M2.7 | Lower runtime/token footprint, improving detection | Lower absolute localization at Large | Budget-sensitive baselines and broad triage |
0 commit comments