NetOpsBench: Open Arena for NetOps in AI Infrastructure

Fair, reproducible benchmarks for agentic network troubleshooting.

Website · Quickstart · Build Your Agent · Trace Dataset

NetOpsBench is an open benchmark arena for agentic network troubleshooting — run reproducible fault scenarios on live SONiC-VS / Containerlab topologies, plug in any troubleshooting agent, and score it across quality and efficiency dimensions.

Why NetOpsBench

Troubleshooting agents are difficult to compare when the network, incident, and evidence change from run to run. NetOpsBench turns those variables into a controlled live benchmark:

• Reproducible incidents — labeled faults run against repeatable SONiC-VS and Containerlab topologies.
• Interactive evidence — agents inspect live Pingmesh, BGP, gNMI, syslog, and switch state instead of static logs.
• Comparable outcomes — one evaluator measures detection, localization, efficiency, and tool use across agent strategies.

Overview

NetOpsBench provides: (1) an interactive and realistic environment mimicking production networks, with common tracing and telemetry tooling; (2) comprehensive and reproducible benchmarks covering a wide range of faults and failures; (3) an extensible architecture with an open SDK to readily integrate with various agent paradigms and observability tools, allowing users to try out their own agentic workflows.

It is built for researchers and engineers who want to compare LLM-backed, symbolic, heuristic, or hybrid troubleshooting strategies on the same operational benchmark, not just on static logs or hand-written prompts.

News

• 2026-08: 🚀 NetOpsBench v0.2.0 - Large-topology benchmark release.
  • Add Xlarge CLOS and Fat-tree K=8/K=12 profiles, with 70 generated cases per large topology.
  • Replace per-client Python Pingmesh and iperf processes with the native Rust client agent for Pingmesh and background traffic.
  • Harden large-topology fault injection, recovery, observability, and exact runtime teardown.
  • Publish a versioned DeepSeek validation snapshot across all seven supported scales. See the v0.2.0 release notes.
• 2026-05: 🎉 Initial Release - NetOpsBench is now available as an open arena for agentic network troubleshooting.
  • Provide public SDK with run_scenario() and run_suite() APIs to launch live network environments from Python.
  • Equip native MCP tools of complete observability utilities and pre-configured SONiC-VS network covering XS, Small, Medium and Large scales.
  • Offer fault scenario generation scripts and an expanding repository of reproducible fault cases with standard ground truth labels.
  • A full-fledged benchmark evaluator that accesses detection accuracy and token utilization efficiency.

Quick Start

NetOpsBench runtime execution requires Linux because Containerlab depends on Linux networking primitives.

Install and run via CLI

git clone https://github.com/NetX-lab/NetOpsBench.git
cd NetOpsBench

python -m venv .venv
source .venv/bin/activate
pip install -e ".[agent]"

netopsbench benchmark prepare --scales xs
export OPENAI_API_KEY=...
PYTHONPATH=. python examples/01_run_scenario.py --vendor openai

The first successful run produces a BenchmarkReport with case-level scores, timing, and artifact paths. For Docker, Containerlab, and runtime setup details, read Quickstart.

Run a scenario from Python

from examples.agents import MinimalDeepAgent
from netopsbench.sdk import NetOpsBench

scenario = "scenarios/generated/xs/generated_link_down_xs_001.yaml"

with NetOpsBench(workspace=".") as bench:
    agent = bench.agents.wrap(MinimalDeepAgent(vendor="openai"))
    run = bench.sessions.run_scenario(scenario=scenario, agent=agent)
    report = run.wait()

print(report.summary)

Scenario YAML files define the benchmark case: topology scale, traffic profile, fault type, target device, and interface-level ground truth when applicable. Use the Python API Guide for run_scenario(...), run_suite(...), and workers=N; see Custom Troubleshooting Agents when you are ready to replace MinimalDeepAgent with your own strategy.

Benchmark Results

NetOpsBench reports detection, fault type, device/interface localization, runtime, tool calls, and token usage so troubleshooting quality and operational cost can be compared together.

Scale	Topology	Switches	Clients	Cases
XS	CLOS	4	2	14
Small	CLOS	6	8	15
Medium	CLOS	12	16	28
Large	CLOS	20	64	52
Xlarge	CLOS	144	128	70
Fat-tree K=8	Fat-tree	80	128	70
Fat-tree K=12	Fat-tree	180	144	70

Diagnosis score is the mean end-to-end case score: healthy cases require the correct verdict, while fault cases receive localization credit only after the fault is detected. Fault detection F1 measures the fault-versus-healthy decision independently.

The largest validated Fat-tree profile provides a compact case-level view. Each square below is one K=12 case; detailed cross-topology observability analysis remains in the full results.

Read the v0.2.0 release notes, Benchmark Methodology, and Benchmark Results for the full validation snapshot and scoring definitions.

The public v0.2 trace release contains 319 validated DeepSeek Harbor/ATIF trajectories across XS through Fat-tree K=12. Aggregate metrics and immutable publication provenance are recorded in the v0.2 result snapshot.

Learn More

Goal	Start here
Run one scenario	Quickstart
Run scenarios, suites, and batches	Running Benchmarks
Plug in your own troubleshooting agent	Custom Troubleshooting Agents
Use NetOpsBench from Python	Python API Guide
Interpret benchmark scores	Benchmark Methodology
Debug observability or runtime state	Operations
Understand the benchmark loop	System Overview

Community

• Global community: NetOpsBench Slack
• Chinese-language community: NetOpsBench Feishu group

License

NetOpsBench is released under the MIT License. See LICENSE.

Citation

If you use NetOpsBench in your research, please cite:

@software{netopsbench2026,
  author  = {Yang, Yitao and Xu, Hong},
  title   = {{NetOpsBench}: Open Arena for NetOps in AI Infrastructure},
  year    = {2026},
  url     = {https://github.com/netx-lab/NetOpsBench},
}