You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+28-18Lines changed: 28 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -24,18 +24,11 @@ NetOpsBench is an open benchmark arena for agentic network troubleshooting — r
24
24
25
25
## Why NetOpsBench
26
26
27
-
Developing and evaluating agentic root cause analysis methods for network troubleshooting remains challenging, with three core bottlenecks hindering further advancement:
|**No fair comparison**| Varied network topologies, fault sets, observability tools, and evaluation metrics hinder the comparison of agentic troubleshooting strategies across the research community. | NetOpsBench unifies fault scenarios, observability access and scoring rules to support agent comparison on a shared benchmark. |
34
-
|**Non-reproducible faults**| Real network incidents cannot be reliably reproduced or labeled with consistent ground truth, slowing iterative improvement and evaluation of troubleshooting agents. | Containerlab + SONiC-VS inject controlled, reproducible faults with stable labels, so every run is an identical, repeatable episode. |
35
-
|**Non-Interactive Environment**| Static topology snapshots and logs cannot provide live probing and telemetry signals required by agents for diagnostic work. | NetOpsBench offers an interactive environment for agents to operate within live networks, capturing real-time Pingmesh data, gNMI telemetry and switch CLI evidence during every episode. |
36
-
37
-
27
+
Troubleshooting agents are difficult to compare when the network, incident, and evidence change from run to run. NetOpsBench turns those variables into a controlled live benchmark:
38
28
29
+
-**Reproducible incidents** — labeled faults run against repeatable SONiC-VS and Containerlab topologies.
30
+
-**Interactive evidence** — agents inspect live Pingmesh, BGP, gNMI, syslog, and switch state instead of static logs.
31
+
-**Comparable outcomes** — one evaluator measures detection, localization, efficiency, and tool use across agent strategies.
39
32
40
33
## Overview
41
34
@@ -47,6 +40,11 @@ It is built for researchers and engineers who want to compare LLM-backed, symbol
- Publish a versioned DeepSeek validation snapshot across all seven supported scales. See the [v0.2.0 release notes](docs/content/docs/releases/v0.2.0.mdx).
50
48
-**2026-05**: 🎉 **Initial Release** - NetOpsBench is now available as an open arena for agentic network troubleshooting.
51
49
- Provide public SDK with `run_scenario()` and `run_suite()` APIs to launch live network environments from Python.
52
50
- Equip native MCP tools of complete observability utilities and pre-configured SONiC-VS network covering XS, Small, Medium and Large scales.
NetOpsBench reports detection, fault type, device/interface localization, runtime, tool calls, and token usage so troubleshooting quality and operational cost can be compared together.
**Diagnosis score** is the mean end-to-end case score: healthy cases require the correct verdict, while fault cases receive localization credit only after the fault is detected. **Fault detection F1** measures the fault-versus-healthy decision independently.
100
108
101
-
Read [Benchmark Methodology](docs/content/docs/run-benchmarks/methodology.mdx) for scoring definitions and [Benchmark Results](docs/content/docs/run-benchmarks/results.mdx) for an example completed suite.
109
+
102
110
103
-
Public agent trajectory artifacts are available in the [NetOpsBench Trace Dataset](https://huggingface.co/datasets/yyyyyt/netopsbench-trace), including Harbor/ATIF traces, run reports, and summary CSVs for reproducible analysis.
111
+
The largest validated Fat-tree profile provides a compact case-level view. Each square below is one K=12 case; detailed cross-topology observability analysis remains in the full results.
112
+
113
+
114
+
115
+
Read the [v0.2.0 release notes](docs/content/docs/releases/v0.2.0.mdx), [Benchmark Methodology](docs/content/docs/run-benchmarks/methodology.mdx), and [Benchmark Results](docs/content/docs/run-benchmarks/results.mdx) for the full validation snapshot and scoring definitions.
116
+
117
+
The public [NetOpsBench Trace Dataset](https://huggingface.co/datasets/yyyyyt/netopsbench-trace) contains both the earlier cross-model snapshot and the [v0.2 seven-scale release](https://huggingface.co/datasets/yyyyyt/netopsbench-trace/tree/main/releases/netopsbench-0.2): 319 validated DeepSeek Harbor/ATIF trajectories across XS through Fat-tree K=12. Aggregate metrics and immutable publication provenance are recorded in the [v0.2 result snapshot](docs/public/assets/benchmark/deepseek_v02_release.json).
104
118
105
119
## Learn More
106
120
@@ -119,10 +133,6 @@ Public agent trajectory artifacts are available in the [NetOpsBench Trace Datase
119
133
- Global community: [NetOpsBench Slack](https://join.slack.com/t/netopsbench/shared_invite/zt-3zhhfangj-2U4dU_NSfCy1rcOM1dmuvQ)
Copy file name to clipboardExpand all lines: docs/content/docs/debug-operate/observability.mdx
+4Lines changed: 4 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -120,6 +120,8 @@ or deleted.
120
120
121
121
## Cleanup
122
122
123
+
Xlarge and Fat-tree labs can consume substantial CPU, memory, container, and telemetry capacity. Start with one worker, inspect `docker stats`, host memory and pressure, and confirm one complete baseline before increasing concurrency. Keeping a topology warm is useful for repeated experiments, but do not leave unused large workers running alongside a new campaign.
Prefer the exact runtime name. `teardown --all` is intended only when every runtime registered in the current workspace is owned by the caller; it is not a substitute for host-wide Docker cleanup. Do not use global `docker system prune`, broad container-name filters, or shared-network deletion on multi-user hosts.
Copy file name to clipboardExpand all lines: docs/content/docs/index.mdx
+6-1Lines changed: 6 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,6 +5,10 @@ description: Documentation map for implementing, running, and evaluating NetOpsB
5
5
6
6
NetOpsBench evaluates troubleshooting agents on generated data-center network fault scenarios. A run provisions a SONiC-VS / Containerlab topology, injects a controlled fault or healthy episode, exposes runtime evidence, calls an agent, and scores the returned `DiagnosisResult` against scenario ground truth.
The current release adds Xlarge CLOS, Fat-tree K=8/K=12, and the native Rust Pingmesh/traffic client agent. Read the [v0.2.0 release notes](/docs/releases/v0.2.0).
10
+
</Callout>
11
+
8
12
These docs are organized around the agent-development workflow.
9
13
10
14
## Main path
@@ -71,4 +75,5 @@ These docs are organized around the agent-development workflow.
71
75
72
76
-[System Overview](/docs/architecture/system-overview) explains the runtime loop, evidence path, worker isolation, and report aggregation.
Copy file name to clipboardExpand all lines: docs/content/docs/quickstart.mdx
+4Lines changed: 4 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,6 +5,8 @@ description: The shortest path from a fresh clone to one completed NetOpsBench r
5
5
6
6
This page verifies that the host can run one benchmark case. It stops after the first `BenchmarkReport`; agent implementation and larger benchmark runs are covered separately.
7
7
8
+
Use XS for installation verification even when the eventual target is Xlarge or Fat-tree. It exercises the same SDK, native client-agent, observability, fault, scoring, and cleanup path with a much smaller resource footprint.
The run provisions an XS topology, starts observability, injects one generated fault, calls the reference agent, scores the returned diagnosis, and writes a `BenchmarkReport`.
85
87
88
+
`netopsbench benchmark prepare` only generates topology and scenario assets. It does not deploy a lab, start telemetry, or reserve resources; deployment begins when a session or runtime command is executed.
89
+
86
90
## After success
87
91
88
92
- Open saved agent trajectories with `netopsbench trace view`; it syncs trace-enabled runs into the local Harbor viewer cache automatically.
description: Xlarge CLOS, Fat-tree, and native client-agent release notes.
4
+
---
5
+
6
+
**Released: 2026-08**
7
+
8
+
NetOpsBench v0.2.0 extends the live benchmark from the original CLOS profiles to Xlarge CLOS and two multi-tier Fat-tree profiles. It also replaces the client-side Python Pingmesh and iperf process fleet with a native Rust client agent while preserving the public observation, tool, evaluator, and SDK contracts.
Each profile contains four healthy cases and fault cases spanning physical links, devices, impairments, routing, BGP, policy, and ACL behavior. Fault placement covers access and upper-fabric tiers; endpoint Pingmesh can remain healthy when ECMP successfully preserves forwarding, while interface and control-plane telemetry still expose the degraded redundancy.
19
+
20
+
## Runtime changes
21
+
22
+
- A single Rust binary runs independent Pingmesh and background-traffic processes in every client.
- Background traffic is runtime-owned and health-checked as a complete flow matrix instead of being recreated for every case.
25
+
- Fault injection and recovery use transactional state tracking, readback, convergence checks, quarantine, and exact resource cleanup.
26
+
- Large BGP collections preserve fast transition/index data while writing stable full snapshots at a lower cadence.
27
+
28
+
## Compatibility and results
29
+
30
+
The stable `netopsbench.sdk` agent, scenario, report, and tool entrypoints remain available. The v0.2 result snapshot is a release-validation reference, not a permanent model ranking; provider behavior and agent strategies can change independently of the benchmark contract.
31
+
32
+
See [Benchmark Results](/docs/run-benchmarks/results) for accuracy, localization, observability, cost, and trace inventory, the [public v0.2 trace release](https://huggingface.co/datasets/yyyyyt/netopsbench-trace/tree/main/releases/netopsbench-0.2) for all 319 ATIF trajectories, and [Benchmark Methodology](/docs/run-benchmarks/methodology) for the exact scoring and detector semantics.
description: Earlier four-model NetOpsBench comparison retained for within-snapshot analysis.
4
+
---
5
+
6
+
This snapshot predates the NetOpsBench v0.2.0 release rerun. Its four models were evaluated under the same earlier contract, so it remains useful for within-snapshot comparison. Do not combine these values with the current [v0.2.0 Results](/docs/run-benchmarks/results) as if both snapshots used the same runtime and detector contract.
Verdict classification was generally easier than precise localization. Interface localization was the weakest quality metric on larger topologies.
30
+
31
+
## Quality metrics
32
+
33
+
34
+
35
+
36
+
37
+
38
+
39
+
40
+
41
+
## Runtime cost
42
+
43
+
44
+
45
+
46
+
47
+
48
+
49
+
50
+
51
+
Compare cost with localization quality, not verdict quality alone. Higher tool or token use does not automatically improve device or interface precision.
0 commit comments