Initial open-source release of NetOpsBench

Author: yyyyyt123

.agents/skills/netopsbench-diagnosis-playbook/SKILL.md

@@ -0,0 +1,204 @@
+---
+name: netopsbench-diagnosis-playbook
+description: Use this skill when diagnosing NetOpsBench DCN benchmark faults with Pingmesh, topology, routing, interface, and log evidence. It helps localize a single infrastructure device or interface, avoid timeout-prone tool loops, and normalize conclusions to benchmark-friendly fault labels such as link_down, packet_loss, high_latency, packet_corruption, mtu_mismatch, blackhole_route, and static_route_misconfig.
+compatibility: Requires the NetOpsBench MCP toolset with Pingmesh, topology, BGP, route, interface, log, ping, and traceroute tools.
+allowed-tools: get_pingmesh_summary get_pingmesh_hotspots get_topology get_all_bgp_status get_bgp_neighbors get_route_table get_device_interfaces get_interface_metrics get_device_logs get_device_config get_bgp_rib ping_test traceroute
+---
+
+# NetOpsBench Diagnosis Playbook
+
+## When to use
+
+Use this skill for NetOpsBench troubleshooting tasks where you must produce a benchmark-friendly diagnosis from live MCP tool evidence.
+
+Use it especially when one of these risks appears:
+
+- Pingmesh anomalies are broad and you need to reduce the search space quickly
+- Pingmesh shows zero anomalies, but the network may still have a routing or forwarding fault
+- Pingmesh reports `mtu_or_fragmentation_suspect` but ordinary small-packet checks still look healthy
+- The evidence points to a path between multiple leaves and you must choose a single owner or stay inconclusive
+- You have enough evidence for a fault, but your current fault label is free-form instead of a benchmark label
+- You are about to return multiple devices or a wide interface set
+
+## Core goals
+
+1. Prefer one infrastructure device owner over a path-level multi-device answer.
+2. Prefer one interface only when evidence is strong; otherwise omit the interface.
+3. Use benchmark fault labels, not descriptive prose, for `fault_type`.
+4. Keep tool exploration narrow to avoid 300s/600s dead-ends.
+
+## Fast triage
+
+Always begin with exactly these three tools unless the user already gave equivalent outputs:
+
+1. `get_pingmesh_summary`
+2. `get_pingmesh_hotspots`
+3. `get_topology`
+
+Classify the case into one of these modes before doing anything else:
+
+- `latency-dominant`: latency spikes are present and packet loss is absent or minor
+- `loss-dominant`: packet loss events are present
+- `zero-anomaly`: total Pingmesh anomalies are zero
+
+After triage, choose at most two candidate devices for deeper validation.
+
+If Pingmesh explicitly reports `mtu_or_fragmentation_suspect`, treat that as a special branch even when packet loss is zero and latency is mild.
+
+## Candidate-device rules
+
+Use these tie-breakers in order:
+
+1. Pick the leaf or spine that appears most often in Pingmesh hotspots.
+2. Prefer a single leaf over a leaf pair when one side has stronger local evidence.
+3. Prefer infrastructure devices over clients unless a client-only fault is explicit.
+4. If evidence is only path-level and no owner emerges, return `inconclusive` instead of listing multiple devices.
+
+Never return comma-separated or "A and B" device locations unless the user explicitly asks for a path summary rather than a benchmark answer.
+
+## Evidence collection by mode
+
+### A. Latency-dominant
+
+Use this order:
+
+1. `get_device_interfaces` on the top candidate leaf
+2. `get_interface_metrics` on the top 2-3 interfaces most likely to carry the affected traffic
+3. `get_all_bgp_status` or targeted `get_bgp_neighbors`
+4. `get_device_logs`
+5. `get_route_table` only if routing inconsistency is still plausible
+
+Interpretation:
+
+- Healthy BGP plus localized latency spikes usually means `high_latency`, not a routing fault.
+- If one leaf clearly owns the hotspots and its network-facing ports show pressure, return that leaf.
+- If the device is clear but the exact interface is not, omit the interface instead of naming many ports.
+
+### B. Loss-dominant
+
+Use this order:
+
+1. `get_device_interfaces` on the top candidate leaf
+2. `get_interface_metrics` on likely path interfaces
+3. `get_all_bgp_status` or targeted `get_bgp_neighbors`
+4. `get_device_logs`
+5. `ping_test` or `traceroute` only for one or two decisive directed checks
+6. `get_route_table` only if routing evidence is needed to distinguish route faults from data-plane faults
+
+Interpretation:
+
+- Healthy BGP plus localized loss usually points to `packet_loss`, `link_down`, `packet_corruption`, or `mtu_mismatch`.
+- If a specific interface is down or clearly degraded, prefer `link_down`.
+- If connectivity still exists but loss is the dominant symptom, prefer `packet_loss`.
+- If the evidence suggests integrity or fragmentation behavior rather than simple drops, consider `packet_corruption` or `mtu_mismatch`, but only if route absence is not the better explanation.
+
+### D. MTU / fragmentation-special branch
+
+Use this branch whenever Pingmesh reports `mtu_or_fragmentation_suspect`, even if ordinary client-to-client ping still succeeds.
+
+Use this order:
+
+1. `ping_test` from an affected client with default small packets
+3. `ping_test` from the same client to the same destination with `payload_size=1472` and `dont_fragment=true`
+4. If needed, repeat `ping_test` with a larger DF probe such as `payload_size=2000`
+5. `get_device_interfaces` on the two most likely devices on the affected path
+6. `get_interface_metrics` on the most relevant path interfaces
+7. `get_all_bgp_status` or targeted `get_bgp_neighbors`
+8. `get_route_table` only if route absence is still plausible
+
+Interpretation:
+
+- Small-packet success does not clear `mtu_mismatch`.
+- The strongest directed symptom is: small client ping succeeds, but large DF ping fails.
+- If large DF probes fail only on paths that cross one spine or one leaf uplink, rank that device/interface higher.
+- If both small and large DF probes succeed, downgrade `mtu_mismatch` confidence and keep looking for another explanation.
+- If BGP and routing are healthy and large DF probes fail selectively, prefer `mtu_mismatch` over route faults.
+
+### C. Zero-anomaly
+
+This is the highest-priority timeout trap. Do not keep repeating Pingmesh queries.
+
+Switch immediately to this route-centric sequence:
+
+1. `get_route_table` on the suspected owner and one nearby peer device
+2. `ping_test` for one targeted failing destination only when a client-side directed check is needed
+3. `traceroute` for one targeted path if needed
+4. `get_all_bgp_status` or targeted `get_bgp_neighbors`
+5. `get_device_config` to inspect running configuration for route or BGP misconfigurations
+6. `get_device_interfaces`
+7. `get_device_logs`
+
+Interpretation:
+
+- Zero Pingmesh anomalies does not mean healthy routing.
+- If route-table evidence points to a wrong or missing static route on one device, prefer `static_route_misconfig`.
+- If traffic is dropped due to forwarding behavior but the exact static misconfiguration is not proven, prefer `blackhole_route`.
+- Do not fall back to generic congestion explanations when Pingmesh is clean.
+
+## Interface ranking
+
+Only return an interface if you can rank one interface above the others with specific evidence.
+
+Rank interfaces using this order:
+
+1. Topology relevance: is the interface actually on the affected path?
+2. Hard state: oper-down, admin-down, or clearly abnormal counters
+3. Directionality: does hotspot direction or traceroute implicate this side?
+4. Consistency: do logs and interface metrics agree?
+
+If you cannot produce one top interface, leave `location.interface` empty.
+
+Never return a long interface bundle such as `Ethernet0/Ethernet4/Ethernet8/...` in the final benchmark answer.
+
+## Fault label normalization
+
+Use only these benchmark labels in `fault_type`:
+
+- `link_down`
+- `packet_loss`
+- `high_latency`
+- `packet_corruption`
+- `mtu_mismatch`
+- `blackhole_route`
+- `static_route_misconfig`
+
+Map evidence to labels like this:
+
+- `high_latency`: latency spikes dominate, reachability mostly remains, control plane is healthy
+- `packet_loss`: drop symptoms dominate, reachability still exists, no stronger route-fault evidence
+- `link_down`: one specific link or interface is down or strongly isolated as failed
+- `packet_corruption`: integrity-like or corruption-like path degradation is better supported than simple loss
+- `mtu_mismatch`: fragmentation/selective path failure is more plausible than route absence or link-down
+- `blackhole_route`: forwarding to a destination disappears due to a route/next-hop blackhole, but static-route proof is incomplete
+- `static_route_misconfig`: route-table evidence points to a wrong or missing static route on a specific device
+
+If you are currently thinking in a free-form label like "leaf-side congestion on fabric uplinks", convert it to the closest benchmark label before returning.
+
+## Stop conditions
+
+Do not keep exploring once you have:
+
+- one clear device owner and enough evidence for a benchmark label, or
+- no owner after checking two devices and at most three interfaces per device
+
+If you hit the second condition, return `inconclusive` with concise evidence instead of opening more branches.
+
+## Final answer checklist
+
+Before returning, verify all of these:
+
+- `verdict` is not `network_healthy` when Pingmesh anomalies exist or directed checks fail
+- `fault_type` is one of the seven benchmark labels above
+- `location.device` is a single device string when you have a justified owner
+- `location.interface` is either one interface or empty
+- `evidence` cites concrete tool findings, not speculation
+
+## Failure modes to avoid
+
+- Re-running Pingmesh tools after a `zero-anomaly` classification
+- Treating ordinary small-packet ping success as proof that `mtu_mismatch` is absent
+- Using leaf/spine-originated `ping_test` or `traceroute` as decisive evidence; prefer client-source checks
+- Returning multiple devices because the path looks symmetric
+- Returning huge interface sets instead of a ranked top choice
+- Using descriptive prose as `fault_type`
+- Spending the budget on broad scans before checking routing in zero-anomaly cases

.editorconfig

@@ -0,0 +1,18 @@
+root = true
+
+[*]
+charset = utf-8
+end_of_line = lf
+insert_final_newline = true
+trim_trailing_whitespace = true
+indent_style = space
+indent_size = 4
+
+[*.{yml,yaml,json,md,toml}]
+indent_size = 2
+
+[Makefile]
+indent_style = tab
+
+[*.{sh,bash}]
+indent_size = 2

.env.example

@@ -0,0 +1,68 @@
+# =============================================================================
+# NetOpsBench Environment Configuration
+# Copy this file to .env and update the values for your environment.
+# Do not commit .env to git.
+# =============================================================================
+
+# --- LLM API Keys (for benchmark agents) ---
+MINIMAX_API_KEY=your-minimax-api-key
+OPENAI_API_KEY=your-openai-compatible-api-key
+# ZHIPU_API_KEY=your-zhipu-api-key
+# DEEPSEEK_API_KEY=your-deepseek-api-key
+# KIMI_API_KEY=your-kimi-api-key
+# OPENAI_BASE_URL=https://api.openai.com/v1
+
+# --- LangSmith Tracing (optional) ---
+LANGSMITH_API_KEY=your-langsmith-api-key
+LANGSMITH_TRACING=true
+LANGSMITH_PROJECT=NetOpsBench
+
+# --- InfluxDB ---
+NETOPSBENCH_INFLUXDB_URL=http://localhost:8086
+NETOPSBENCH_INFLUXDB_TOKEN=replace-me          # REQUIRED: set a real token before deployment
+NETOPSBENCH_INFLUXDB_ORG=netopsbench
+NETOPSBENCH_INFLUXDB_BUCKET=netopsbench
+
+# --- Grafana ---
+# NETOPSBENCH_GRAFANA_URL=http://localhost:3000
+# NETOPSBENCH_GRAFANA_PASSWORD=admin
+# NETOPSBENCH_GRAFANA_INFLUXDB_URL=http://influxdb:8086
+# NETOPSBENCH_GRAFANA_DEFAULT_BUCKET=netopsbench
+
+# --- Topology ---
+# NETOPSBENCH_TOPOLOGY_DIR=lab-topology/generated_topology_xs
+# NETOPSBENCH_TOPOLOGY_ID=dcn
+# NETOPSBENCH_LAB_NAME=dcn
+# NETOPSBENCH_MGMT_SUBNET=                     # Override management subnet (e.g. 172.20.20.0/24)
+# NETOPSBENCH_MGMT_NETWORK=                    # Override Docker management network name
+
+# --- SONiC / gNMI Telemetry ---
+# SONIC_GNMI_PORT=50051
+# SONIC_GNMI_USERNAME=admin
+# SONIC_GNMI_PASSWORD=                          # Set the SONiC gNMI password for telemetry collection
+# SONIC_GNMI_ENCODING=json_ietf
+# SONIC_GNMI_TARGET=COUNTERS_DB
+# SONIC_GNMI_SUBSCRIPTION_MODE=on_change       # on_change | sample | target_defined
+# SONIC_GNMI_SAMPLE_INTERVAL=10s               # Only used when subscription_mode=sample
+
+# --- Telemetry collectors (sFlow / syslog) ---
+# NETOPSBENCH_SFLOW_COLLECTOR=                 # sFlow collector IP (defaults to management IP)
+# NETOPSBENCH_SYSLOG_COLLECTOR=               # Syslog collector IP (defaults to management IP)
+# NETOPSBENCH_SFLOW_PORT=6343
+# NETOPSBENCH_SFLOW_POLLING_INTERVAL=20
+# NETOPSBENCH_SFLOW_SAMPLE_RATE=1000
+# NETOPSBENCH_SFLOW_SAMPLE_DIRECTION=ingress
+
+# --- Pingmesh ---
+# PINGMESH_CYCLE_INTERVAL=5                    # Probe cycle interval in seconds
+
+# --- Runtime / debug ---
+# NETOPSBENCH_LOG_LEVEL=INFO                   # DEBUG | INFO | WARNING | ERROR
+# NETOPSBENCH_NO_SUDO=0                        # Set to 1 in CI/test environments without sudo
+
+# --- Worker Pool ---
+# NETOPSBENCH_WORKER_DEPLOY_JOBS=              # Parallel worker deploy jobs (default: num_workers)
+# NETOPSBENCH_SONIC_WAIT_TRIES=180             # Max SONiC readiness retries (5s each, default: 15 min)
+# NETOPSBENCH_AGENT_TIMEOUT_SECONDS=300
+# NETOPSBENCH_WORKER_AGENT_TIMEOUT_SECONDS=600
+# NETOPSBENCH_WORKER_DISABLE_LANGSMITH=false

.github/workflows/docs-pages.yml

@@ -0,0 +1,61 @@
+name: docs-pages
+
+on:
+  push:
+    branches:
+      - main
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+
+concurrency:
+  group: docs-pages
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    env:
+      NEXT_PUBLIC_BASE_PATH: /${{ github.event.repository.name }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Node
+        uses: actions/setup-node@v4
+        with:
+          node-version: '20'
+          cache: npm
+          cache-dependency-path: docs/package-lock.json
+
+      - name: Configure Pages
+        uses: actions/configure-pages@v5
+        with:
+          enablement: true
+
+      - name: Install docs dependencies
+        working-directory: docs
+        run: npm ci
+
+      - name: Build docs site
+        working-directory: docs
+        run: npm run build
+
+      - name: Upload Pages artifact
+        uses: actions/upload-pages-artifact@v3
+        with:
+          path: docs/out
+
+  deploy:
+    needs: build
+    runs-on: ubuntu-latest
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - name: Deploy to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@v4