Merge pull request #3 from yyyyyt123/feature/tracing

Update benchmark results

Author: yyyyyt123

.gitignore

@@ -41,6 +41,7 @@ observability/telegraf.conf
 .venv*
 .coverage
 plot_*
+!scripts/plot_results.py
 
 # Fumadocs / Next.js site artifacts
 docs/node_modules/

docs/content/docs/build-your-agent/custom-agents.mdx

@@ -68,9 +68,9 @@ The reference LLM agent builds a compact prompt from topology size, symptom keys
 
 Return structured fields rather than only natural language. Free-form explanations are useful for review, but scoring depends on verdict, fault type, and location fields.
 
-For reproducibility, NetOpsBench saves a per-case runtime trace beside the raw scenario result. Agents that need private LLM message traces should use `context.trace.llm_client(...)` or a NetOpsBench-provided framework callback such as `context.trace.langchain_callback()`. The harness writes ATIF v1.7 `trajectory.atif.json` artifacts for Harbor-style inspection while keeping ground truth out of the agent trajectory; scoring details are linked separately through `traces/results.jsonl`. Use `netopsbench trace view` to sync trace-enabled runs into the local Harbor viewer cache, or `netopsbench trace view <run_id>` to ensure a specific saved run is available in the viewer.
+For reproducibility, NetOpsBench saves a per-case runtime trace beside the raw scenario result. The bundled reference agent captures private LLM and tool events by attaching `context.trace.langchain_callback()` to its LangChain-compatible runtime. Custom non-LangChain agents can use advanced manual recorder methods such as `context.trace.record_llm_request(...)` and `context.trace.record_llm_response(...)` when they need private model calls in the trace. The harness writes ATIF v1.7 `trajectory.atif.json` artifacts for Harbor-style inspection while keeping ground truth out of the agent trajectory; scoring details are linked separately through `traces/results.jsonl`. Use `netopsbench trace view` to sync trace-enabled runs into the local Harbor viewer cache, or `netopsbench trace view <run_id>` to ensure a specific saved run is available in the viewer.
 
-Trace storage preserves visible agent-environment interactions with secret redaction and per-field size limits. NetOpsBench does not monkeypatch arbitrary LLM SDKs, so fully private model prompts and responses are captured only when the agent uses the trace-aware client or callback.
+Trace storage preserves visible agent-environment interactions with secret redaction and per-field size limits. NetOpsBench does not monkeypatch arbitrary LLM SDKs, so fully private model prompts and responses are captured only when the agent uses a supported framework callback or the manual recorder methods.
 
 ## Reference agent
 

docs/content/docs/build-your-agent/python-api-guide.mdx

@@ -71,7 +71,7 @@ The `scenario_summaries[*].raw_result_path` fields point to raw JSON files for c
 
 Agent traces are saved by default and can be disabled for a run with `trace=False` or by setting `NETOPSBENCH_TRACE=0`. Disabling trace prevents private runtime trace collection and sidecar artifact creation. Ground truth and score details are written to `traces/results.jsonl`, not into the agent trajectory.
 
-NetOpsBench stores visible prompts, model messages, tool calls, and observations with secret redaction and per-field truncation. For private LLM message capture, custom agents should call models through `context.trace.llm_client(...)` or attach `context.trace.langchain_callback()` to LangChain-compatible runtimes. Set `NETOPSBENCH_TRACE_MAX_FIELD_CHARS` to tune truncation.
+NetOpsBench stores visible prompts, model messages, tool calls, and observations with secret redaction and per-field truncation. The bundled `MinimalDeepAgent` attaches `context.trace.langchain_callback()` to its LangChain-compatible runtime so private LLM messages and tool events flow into the same recorder. Non-LangChain agents can use the advanced manual recorder methods, such as `context.trace.record_llm_request(...)` and `context.trace.record_llm_response(...)`, when they need to capture private model calls. Set `NETOPSBENCH_TRACE_MAX_FIELD_CHARS` to tune truncation.
 
 Open a completed run directly in the Harbor viewer:
 

docs/content/docs/quickstart.mdx

@@ -69,7 +69,7 @@ Supported provider presets:
 
 | `--vendor` | Model | Environment variable |
 |---|---|---|
-| `openai` | gpt-5.4 | `OPENAI_API_KEY` |
+| `openai` | gpt-5.5 | `OPENAI_API_KEY` |
 | `minimax` | MiniMax-M3 | `MINIMAX_API_KEY` |
 | `deepseek` | deepseek-v4-pro | `DEEPSEEK_API_KEY` |
 | `zhipu` | glm-5.1 | `ZHIPU_API_KEY` |

docs/content/docs/run-benchmarks/methodology.mdx

@@ -53,7 +53,7 @@ Scenarios are generated from `scenarios/specs/fault_campaign.yaml`. Running `net
 
 | Dimension | Meaning |
 |---|---|
-| Detection accuracy | Whether the agent distinguishes faulted cases from healthy negative samples. |
+| Verdict quality | Whether the agent distinguishes faulted cases from healthy negative samples. The results page reports this as verdict F1-score. |
 | Device localization | Whether the predicted device matches ground truth or an accepted equivalent. |
 | Interface localization | Whether the predicted interface matches when interface-level ground truth exists. |
 | Fault type classification | Whether the predicted fault label matches the ground-truth type. |
@@ -92,7 +92,8 @@ The judge affects only fault-type classification. Verdict, device localization,
 
 | Metric | Interpretation |
 |---|---|
-| `detection_accuracy` | Fault/healthy verdict quality. |
+| `detection_accuracy` | Fault/healthy verdict accuracy. |
+| `detection_f1` | Fault/healthy verdict F1-score used in the results plots. |
 | `device_localization_rate` | Device-level troubleshooting precision. |
 | `interface_localization_rate` | Interface-level precision for applicable cases. |
 | `localization_composite_score` | Combined localization metric used for comparison. |

docs/content/docs/run-benchmarks/results.mdx

@@ -3,34 +3,34 @@ title: Benchmark Results
 description: Reference cross-model NetOpsBench results.
 ---
 
-This page records one cross-model NetOpsBench run. Treat it as a reference snapshot under the documented [methodology](/docs/run-benchmarks/methodology), not as a stable model ranking. Provider defaults, model versions, prompts, and tool policies can change absolute values.
+This page records the public trace dataset snapshot. Treat it as a reference snapshot under the documented [methodology](/docs/run-benchmarks/methodology), not as a stable model ranking. Provider defaults, model versions, prompts, and tool policies can change absolute values.
 
 ## Experiment scope
 
 | Dimension | Values |
 |---|---|
-| Models | Kimi K2.6, MiniMax M2.7, DeepSeek V4 Pro, OpenAI GPT-5.4 |
+| Models | Kimi K2.6, DeepSeek V4 Pro, OpenAI GPT-5.5, MiniMax M3 |
 | Topology scales | XS, Small, Medium, Large |
 | Fault types | 12 canonical types across link, routing, impairment, system, and ACL categories |
-| Quality metrics | Detection, device localization, interface localization, composite score |
+| Quality metrics | Verdict F1-score, device localization, interface localization, composite score |
 | Efficiency metrics | Diagnosis time, tool calls, input/output tokens |
 
 ## Cross-scale snapshot
 
 Each cell is `XS -> Large`.
 
-| Model | Detection (%) | Device Loc (%) | Interface Loc (%) | Composite (%) | Avg Time (s) | Avg Tools | Input Tokens (K) |
+| Model | Verdict F1 (%) | Device Loc (%) | Interface Loc (%) | Composite (%) | Avg Time (s) | Avg Tools | Input Tokens (K) |
 |---|---:|---:|---:|---:|---:|---:|---:|
-| Kimi K2.6 | 85.7 -> 61.5 | 83.3 -> 33.3 | 28.6 -> 14.3 | 77.1 -> 34.2 | 103.3 -> 224.3 | 22.6 -> 33.0 | 164.1 -> 517.0 |
-| DeepSeek V4 Pro | 100.0 -> 80.8 | 58.3 -> 37.5 | 85.7 -> 14.3 | 70.0 -> 30.0 | 161.5 -> 162.6 | 21.2 -> 16.7 | 153.7 -> 261.0 |
-| OpenAI GPT-5.4 | 100.0 -> 80.8 | 33.3 -> 45.8 | 42.9 -> 0.0 | 48.6 -> 32.3 | 97.0 -> 48.6 | 14.5 -> 12.4 | 105.1 -> 194.4 |
-| MiniMax M2.7 | 100.0 -> 93.8 | 66.7 -> 41.7 | 28.6 -> 14.3 | 64.3 -> 31.7 | 68.1 -> 87.1 | 12.0 -> 13.7 | 87.0 -> 209.5 |
+| Kimi K2.6 | 76.2 -> 58.0 | 66.7 -> 20.8 | 71.4 -> 21.4 | 64.3 -> 25.0 | 272.0 -> 576.8 | 36.5 -> 47.3 | 367.6 -> 1029.4 |
+| DeepSeek V4 Pro | 100.0 -> 91.7 | 83.3 -> 45.8 | 57.1 -> 39.3 | 78.6 -> 39.4 | 83.1 -> 106.2 | 24.9 -> 22.7 | 247.6 -> 564.2 |
+| OpenAI GPT-5.5 | 95.7 -> 66.7 | 83.3 -> 43.8 | 57.1 -> 10.7 | 75.0 -> 32.7 | 71.9 -> 80.5 | 17.9 -> 15.3 | 85.4 -> 147.6 |
+| MiniMax M3 | 80.0 -> 35.5 | 50.0 -> 16.7 | 71.4 -> 7.1 | 60.7 -> 16.3 | 214.3 -> 176.8 | 26.4 -> 15.0 | 133.3 -> 167.7 |
 
-The main pattern is that detection remains easier than precise localization. Interface localization is the weakest metric on larger topologies.
+The main pattern is that verdict classification remains easier than precise localization. Interface localization is the weakest metric on larger topologies.
 
 ## Quality metrics
 
-![Detection accuracy](/assets/benchmark/fig_detection.png)
+![Verdict F1-score](/assets/benchmark/fig_verdict_f1.png)
 
 ![Device localization](/assets/benchmark/fig_device_loc.png)
 
@@ -48,4 +48,4 @@ The main pattern is that detection remains easier than precise localization. Int
 
 ![Output tokens](/assets/benchmark/fig_output_tokens.png)
 
-Compare cost metrics with localization quality, not detection alone. Higher tool or token use does not automatically improve device or interface precision.
+Compare cost metrics with localization quality, not verdict quality alone. Higher tool or token use does not automatically improve device or interface precision.