The third leg of a three-repo network automation portfolio, exploring Cisco's own pyATS/Genie framework against the same 10-router MPLS L3VPN lab used in the sibling repos — and extended into a working AI-assisted operations layer using MCP and a locally-run model.
Companion repos:
mynornir-lab— Nornir + Python, hand-built TextFSM parsing, Jenkins CI/CD, full topology docs.myansible-lab— Ansible, same templates, Jenkins CI/CD.
Unlike the other two repos, this one isn't a deployment pipeline — it's a verification, exploration, and AI-assisted operations tool, built to understand what pyATS/Genie offers on top of hand-built TextFSM, and to extend that into a working local AI layer that can query real device state on request.
Six steps, in order, each one built and proven hands-on across this project's three repos:
| Step | Skill | Tool used | Where |
|---|---|---|---|
| 1 | Write the config | Jinja2 templates + YAML data | mynornir-lab / myansible-lab |
| 2 | Push the config safely | Nornir and Ansible | mynornir-lab / myansible-lab |
| 3 | Check it's actually healthy | TextFSM and Genie (learn, parse) |
mynornir-lab / this repo |
| 4 | Track every change | Git, GitHub, Pull Requests | all three repos |
| 5 | Automate the whole thing | Jenkins CI/CD | mynornir-lab / myansible-lab |
| 6 | Explain results in plain English | Ollama, running locally, no data leaves the machine | this repo |
Each step builds on the one before it. Step 6 doesn't replace step 3 — it reads and explains what step 3 already proved to be true. The AI is never the source of truth in this pipeline; the automation and health checks built in steps 1–5 are.
The complete picture, including the AI tool-calling piece proven working in this repo:
1. Write the config Jinja2 + YAML
|
v
2. Push the config Nornir + Ansible
|
v
3. Check it's healthy TextFSM + Genie
|
v
4. Track the changes Git, GitHub, PRs
|
v
5. Automate it all Jenkins CI/CD
|
v
6. Explain results in plain English Ollama, on your own machine
|
v
7. Let AI call it as a tool Tested with real data — RR1 BGP
All seven steps are done and proven with real data from a real router, end to end. This isn't a plan — it's a finished, working example.
python3 -m venv pyats-env
source pyats-env/bin/activate
pip install -r requirements.txtPython 3.12. requirements.txt pins pyats==26.6 / genie==26.6 plus ollama and mcp, needed for the AI integration scripts in 05_ai_integration/.
Credentials and device IPs live in testbed.yaml — edit them for your own lab before running anything. This lab's ssh_options include legacy KEX/HostKey/MAC algorithm overrides for older IOS SSH stacks; drop them if your devices support modern SSH defaults.
For the AI integration section (steps 6–7 below), you also need Ollama installed and running locally, with the model pulled:
ollama pull llama3.2testbed.yaml pyATS inventory — device connections, legacy SSH overrides
genie_baseline.json Captured healthy-state snapshot
requirements.txt pyats[full], genie — isolated venv, not shared with the other repos
01_basics/
learn_bgp.py First exercise: connect + learn('bgp') on a single device
diff_bgp.py Genie's Diff() on two learn() snapshots — before/after a manual change
parse_show_version.py device.parse() — command-level parsing (the direct TextFSM equivalent)
full_lab_check.py learn('bgp') across all 10 devices, sequential (see postmortem #9 — false-down entries on VRF-scoped neighbors)
02_healthcheck/
genie_healthcheck.py Parallel learn() (BGP + OSPF), baseline capture, noise-filtered diff — the trusted tool
03_aetest/
health_job.py Minimal AEtest job file used to explore genie.libs.health
health_testscript.py Minimal AEtest testscript (CommonSetup / Testcase / CommonCleanup)
04_blitz/
blitz_bgp_check.yaml Blitz exercise — YAML-only test (see postmortem #3, do not trust its result)
05_ai_integration/
ollama_explain_health.py Ollama only — feeds a saved health check report to a local model for a plain-English summary
ollama_ask_network.py Ollama only — native tool-calling, AI picks a tool, your code runs it, real data comes back
mcp_health_server.py MCP server — 12 real tools, for any MCP client (Claude Code CLI, Claude Desktop, etc); also
imported directly by ollama_ask_network.py as a plain Python module (no MCP involved there)
pyats validate testbed testbed.yamlChecks the YAML is well-formed and lists every device pyATS can see. Run this first, always. Example
(pyats-env) khau@nuc:~/pyats-env/mypyats-lab (main)$ pyats validate testbed testbed.yaml
Loading testbed file: testbed.yaml
--------------------------------------------------------------------------------
Testbed Name:
mpls_l3vpn_lab
Testbed Devices:
.
|-- CE1 [ios/router]
|-- CE2 [ios/router]
|-- CE3 [ios/router]
|-- CE4 [ios/router]
|-- P1 [ios/router]
|-- P2 [ios/router]
|-- PE1 [ios/router]
|-- PE2 [ios/router]
|-- RR1 [ios/router]
`-- RR2 [ios/router]
YAML Lint Messages
------------------
17:81 warning line too long (216 > 80 characters) (line-length)
81:1 warning too many blank lines (1 > 0) (empty-lines)
Warning Messages
----------------
- Device 'CE1' has no interface definitions
- Device 'CE2' has no interface definitions
- Device 'CE3' has no interface definitions
- Device 'CE4' has no interface definitions
- Device 'P1' has no interface definitions
- Device 'P2' has no interface definitions
- Device 'PE1' has no interface definitions
- Device 'PE2' has no interface definitions
- Device 'RR1' has no interface definitions
- Device 'RR2' has no interface definitions
(pyats-env) khau@nuc:~/pyats-env/mypyats-lab (main)$
python3 01_basics/learn_bgp.pyConnects to PE1, runs device.learn('bgp'), prints the full structured Ops object as JSON.
python3 01_basics/diff_bgp.pyLearns BGP on PE1, pauses so you can make a real change on the device, learns again, then prints exactly what changed using Genie's Diff().
python3 01_basics/parse_show_version.pyStructured fields (hostname, version, uptime, chassis_sn, ...) with zero parser code written — the direct equivalent of a custom TextFSM template.
pyats parse "show version" --testbed-file testbed.yamlSame thing, zero Python, straight from the CLI.
python3 01_basics/full_lab_check.pyLoops every device, learns BGP, prints Established/not-Established per neighbor. Sequential — see genie_healthcheck.py for the parallel, production version. Known issue: reports false [✗] ... — unknown lines for VRF-scoped neighbors (e.g. PE1's CE-facing sessions) — see postmortem #9. genie_healthcheck.py doesn't have this bug; treat this script as the quick sequential exercise it was meant to be, not a source of truth.
python3 02_healthcheck/genie_healthcheck.py --baseline # capture, only after YOU confirm the network is healthy
python3 02_healthcheck/genie_healthcheck.py # compare current state against that baselineConnects to all 10 devices in parallel via pcall(), learns BGP and OSPF, runs hand-written validation logic, and shows a noise-filtered structural diff (counters, timers, LSA refresh churn excluded — see NOISE_FIELDS).
pyats learn bgp ospf --testbed-file testbed.yaml --output genie_learn_outputSame data genie_healthcheck.py collects, zero Python written.
pyats learn bgp --testbed-file testbed.yaml --output snapshot_before
# make a change on a device
pyats learn bgp --testbed-file testbed.yaml --output snapshot_after
pyats diff snapshot_before snapshot_afterpyats run job 03_aetest/health_job.py --testbed-file testbed.yamlA real AEtest job: CommonSetup connects to PE1, one Testcase learns BGP and passes/fails on it, CommonCleanup disconnects.
pyats run genie 04_blitz/blitz_bgp_check.yaml --testbed-file testbed.yamlConfirmed unreliable as tested here — reported PASSED while a real BGP neighbor was down.
Everything in this section is a bare CLI invocation — no script, no import pyats. Same four commands as items 1, 4, 7, and 8 above, collected here with real transcripts captured against the live lab, so the whole no-Python path is in one place.
pyats validate testbed testbed.yamlSee the full transcript in item 1 above — lists every device pyATS can see and lints the YAML. Always run this first.
pyats parse "show version" --testbed-file testbed.yaml --devices PE1 0%| | 0/1 [00:00<?, ?it/s]100%|██████████| 1/1 [00:00<00:00, 1.19it/s]
{
"version": {
"chassis_sn": "2048001",
"compiled_by": "prod_rel_team",
"compiled_date": "Thu 26-Mar-15 07:36",
"hostname": "PE1",
"image_id": "I86BI_LINUX-ADVENTERPRISEK9-M",
"os": "IOS",
"platform": "Linux",
"uptime": "15 minutes",
"version": "15.5(2)T",
"version_short": "15.5"
}
}
--devices narrows to one device — drop it to parse every device in the testbed at once.
pyats learn bgp ospf --testbed-file testbed.yaml --output genie_learn_output --devices PE1Learning '['bgp', 'ospf']' on devices '['PE1']'
+==============================================================================+
| Genie Learn Summary for device PE1 |
+==============================================================================+
| Connected to PE1 |
| - Log: genie_learn_output/connection_PE1.txt |
|------------------------------------------------------------------------------|
| Learnt feature 'bgp' |
| - Ops structure: genie_learn_output/bgp_ios_PE1_ops.txt |
| - Device Console: genie_learn_output/bgp_ios_PE1_console.txt |
|------------------------------------------------------------------------------|
| Learnt feature 'ospf' |
| - Ops structure: genie_learn_output/ospf_ios_PE1_ops.txt |
| - Device Console: genie_learn_output/ospf_ios_PE1_console.txt |
|==============================================================================|
Drop --devices to learn all 10 at once — that's exactly how genie_learn_output/ in this repo was populated.
pyats learn bgp --testbed-file testbed.yaml --output snapshot_before
# make a change on a device
pyats learn bgp --testbed-file testbed.yaml --output snapshot_after
pyats diff snapshot_before snapshot_after+==============================================================================+
| Genie Diff Summary between directories snapshot_before/ and snapshot_after/ |
+==============================================================================+
| File: bgp_ios_RR2_ops.txt |
| - Diff can be found at ./diff_bgp_ios_RR2_ops.txt |
|------------------------------------------------------------------------------|
| File: bgp_ios_CE1_ops.txt |
| - Identical |
|------------------------------------------------------------------------------|
| File: bgp_ios_RR1_ops.txt |
| - Diff can be found at ./diff_bgp_ios_RR1_ops.txt |
|------------------------------------------------------------------------------|
Real content of one of the generated diff_*.txt files, from this exact run — RR1's VPNv4 path/memory counters ticking over between the two captures:
--- snapshot_before/bgp_ios_RR1_ops.txt
+++ snapshot_after/bgp_ios_RR1_ops.txt
info:
instance:
default:
vrf:
default:
neighbor:
1.1.1.1:
address_family:
vpnv4 unicast:
path:
- total_entries: 16
+ total_entries: 12
- total_memory: 3312
+ total_memory: 2992This is exactly the kind of counter-level noise genie_healthcheck.py's NOISE_FIELDS exists to filter out — pyats diff on raw captures shows everything, including changes that don't indicate a real problem.
Two separate AI paths, sharing one set of tool functions:
- Ollama scripts (
ollama_explain_health.py,ollama_ask_network.py) — localllama3.2, no MCP protocol at all. The tool-calling one imports its functions straight out ofmcp_health_server.pyas an ordinary Python module and calls them in-process. - MCP server (
mcp_health_server.py) — the same functions, run as an actual MCP server over stdio, for any MCP client: Claude Code CLI, Claude Desktop,mcp dev, etc. Ollama is never involved here.
Both paths run identical pyATS/Genie code underneath — MCP-vs-Ollama and "real tool call"-vs-"just reading a report" are two independent choices, not the same axis. The name prefix tells you which path a script is: ollama_* never speaks MCP, mcp_* never touches Ollama.
How each one actually decides to fetch real data — the part that's easy to mix up:
Ollama (ollama_ask_network.py) — one question, one disposable run:
you run the script with a question
│
▼
ollama.chat(tools=[...]) → llama3.2 names a function to call
│ (it does not run it — just names it)
▼
the script's own code looks that name up and calls the
real function itself, in the same process
│
▼
real pyATS/Genie result printed, process exits
There's no persistent connection and no memory between runs — every question is a fresh script execution, and the model can't chain a second tool call off the first result within the same run.
Claude Code CLI (.mcp.json → mcp_health_server.py) — one long-running conversation:
claude starts, loads .mcp.json, launches mcp_health_server.py
as a subprocess speaking MCP (a real client/server protocol)
│
▼
the server stays running and advertises its 12 tools;
I (Claude, in this chat) see them the same way I see any
other tool, and decide myself when to call one
│
▼
Claude Code sends the call over the MCP connection, the
server executes the real pyATS/Genie function, result comes
back to me — and I can chain another tool call right after,
based on what the result showed, without restarting anything
The short version: Ollama needs ollama_ask_network.py as glue code because the model itself can only produce text — it cannot open a network connection or run Python. The script is what actually executes the function the model named. Claude Code doesn't need that glue script because MCP is the glue — it's a standard protocol for a client (Claude Code) to discover and call a server's (mcp_health_server.py) tools directly, so I can call them myself, mid-conversation, without any script being re-run per question.
Level 1 — Ollama explains a saved report (read-only, no live device access from the AI side):
python3 02_healthcheck/genie_healthcheck.py > healthcheck_output.txt
python3 05_ai_integration/ollama_explain_health.py healthcheck_output.txtThe AI never touches the network. It only reads and summarizes a file your trusted script already produced.
Level 2 — Ollama decides which tool to call, your code fetches real data:
python3 05_ai_integration/ollama_ask_network.py "What is CE1's software version?"
python3 05_ai_integration/ollama_ask_network.py "Show me the BGP summary for RR1"
python3 05_ai_integration/ollama_ask_network.py "Show me ip of eth1/3 of RR2"The AI (llama3.2 via Ollama's native tool-calling, running locally, no data leaves the machine) reads your question, picks the correct function from a list of real tools, and your Python code — not the AI — actually connects via pyATS/Genie and returns the true answer. Note: plain ollama run llama3.2 "..." on the CLI can't do this — native tool-calling is a Python/HTTP client feature, not exposed by the bare CLI.
Level 3 — the same tools exposed as a standard MCP server:
python3 05_ai_integration/mcp_health_server.py
# or, to test interactively in a browser:
mcp dev 05_ai_integration/mcp_health_server.pySame tools, exposed the standardized way so any MCP-compatible AI client could use them, not just this project's script.
Level 4 — Claude Code CLI as the MCP client, driving the live lab directly:
# .mcp.json (repo root) registers the server for Claude Code CLI only —
# doesn't affect the Ollama scripts above
claude
# inside the session, if the server doesn't auto-connect:
/mcpClaude Code loads .mcp.json at startup and connects to network-health as a real MCP client — no Ollama, no ollama_ask_network.py glue code. Confirmed working end-to-end against the live GNS3 lab: get_bgp_summary, get_vrf_routes, get_loopback_ip, and ping_from_device all called live, including a real cross-VPN ping (CE1 → CE4's VRF_A loopback, 100% success) proving MPLS L3VPN data-plane reachability, not just control-plane state.
| Tool | What it does |
|---|---|
get_last_healthcheck() |
Returns the most recent saved BGP/OSPF health report |
get_software_version(device_name) |
IOS version + uptime for a router |
get_loopback_ip(device_name) |
Loopback0 IP for a router |
get_interface_ip(device_name, interface_name) |
IP of any named interface |
get_bgp_summary(device_name) |
BGP neighbor states for a router |
get_port_status(device_name) |
Admin status, line protocol, IP for every interface |
get_cdp_neighbors(device_name) |
CDP neighbor detail — device ID, local/remote port, platform, mgmt IP |
get_ldp_neighbors(device_name) |
MPLS LDP neighbor state and uptime |
get_ospf_neighbors(device_name) |
OSPF adjacency states per area/interface |
get_vrf_routes(device_name, vrf_name) |
VRF's RD/interfaces plus its learned VPNv4 BGP routes |
get_mpls_interfaces(device_name) |
MPLS/LDP-enabled interfaces and their operational state |
ping_from_device(device_name, target_ip, vrf_name="") |
Live reachability test from a lab router, optionally inside a VRF |
All twelve are read-only. None push config — this stays a verification/inspection layer, deliberately separate from the deploy pipelines in the sibling repos.
| Concept | What it does | Direct equivalent in mynornir-lab |
|---|---|---|
device.parse('show version') |
Structured output for one command | Custom TextFSM templates / ntc-templates |
device.learn('bgp') |
Structured feature state (multiple commands, unified model) | check_bgp() + check_vpnv4() combined |
Diff(old, new) |
Structural diff between two snapshots | The comparison logic inside healthcheck.py's check_*() functions |
pcall() |
Parallel execution across devices | Nornir's threaded runner |
| Tool (MCP) | A real Python function an AI is allowed to call, that does or fetches something real | N/A — new in this repo |
| Tool calling (Ollama) | A model deciding, from your question, which function to call and with what arguments | N/A — new in this repo |
| Skill (different concept, noted for clarity) | A written instruction guide an AI reads before a task — not a function that runs and returns data | Not built here — easy to confuse with "tool," so worth distinguishing |
The honest takeaway: Genie removes the need to write parsers, not the need to decide what "healthy" means — that logic is still hand-written. And an AI on top of all this removes the need to remember the right command, not the need to trust real data over a guess — that judgment is still yours too.
Symptom: Comparing a freshly-learn()ed snapshot against a JSON-loaded baseline failed outright — <class 'dict'> vs <class 'genie.ops.base.maker.CmdDict'>.
Root cause: learn() returns Genie's own CmdDict (a dict subclass). A baseline round-tripped through json.dump()/json.load() comes back as a plain dict. Diff() requires matching types on both sides, even when contents are structurally identical.
Fix: Normalize both sides through a JSON round-trip (json.loads(json.dumps(obj, default=str))) immediately before diffing.
Lesson: Structured "smart" data types from a library aren't always drop-in compatible with plain data loaded from disk, even when they look identical when printed.
Symptom: unicon.core.errors.EOF: Unable to read. Connection closed or not available — even after applying the same legacy-crypto ssh_options fix that resolved two earlier, unrelated SSH failures in the Nornir and Ansible repos.
What was assumed first, and was wrong: A fourth instance of the same legacy-crypto pattern. Adding KEX/MAC algorithm overrides made no difference.
What it actually was: REMOTE HOST IDENTIFICATION HAS CHANGED — the lab's base.j2 template runs crypto key generate rsa general-keys modulus 2048 on every push, regenerating the SSH host key each time, staling ~/.ssh/known_hosts.
Fix: Added -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null to testbed.yaml's ssh_options.
Lesson: Verifying with debug output before assuming "same bug as last time" avoided wasting time on a fix that wouldn't have worked.
Symptom: pyats run genie blitz_bgp_check.yaml completed with Result: PASSED, 100% success rate — while PE1's BGP session to 6.6.6.6 was confirmed Idle (Admin) on the live device at the exact same time.
Root cause: The custom test_sections written in the Blitz YAML never actually ran. The task tree in the Easypy report showed only Blitz's own default common_setup/common_cleanup steps — a generic "did the config change" check unrelated to BGP session state.
Fix: Not resolved — flagged honestly rather than papered over. genie_healthcheck.py remains the trusted tool for this lab.
Lesson: A green "PASSED" only means what you actually told the tool to check — never assume it means "everything is fine" without confirming what ran.
Symptom: Running health_job.py with --health-checks cpu memory caused common_setup to fail with Couldn't find any connected device from testbed object, even though the device had connected successfully one line earlier.
Root cause: genie.libs.health is an Easypy plugin (HealthCheckPlugin, confirmed by reading plugin.py directly) that injects its own verification steps into existing AEtest sections. Its internal get_devices API expects connections set up in a way this minimal, single-device testscript didn't satisfy.
What still worked: The plugin's config layer loaded correctly and produced a valid health_results.json — just with an empty health_data: [], since device-level collection never ran.
Lesson: pyATS degrades predictably (BLOCKED, empty result) rather than crashing outright when a plugin's assumptions aren't met.
Symptom: ollama_ask_network.py (then named ask_network_ai.py) correctly picked get_bgp_summary and correctly fetched real data (RR1 BGP summary: ... Established) — then, in a second call asking the AI to phrase that as an answer, llama3.2 returned a completely fabricated, generic BGP table and stated it "doesn't have real-time capability," directly contradicting what it had just done.
Root cause: llama3.2 (a small, locally-run model) is good at deciding to call a tool but not reliable at using the tool's result correctly afterward — a known limitation of smaller models' tool-calling support, not a bug in the surrounding Python code.
Fix: Removed the second AI call entirely. The real tool result is shown directly to the user instead of being re-explained by a second, unreliable AI step.
Lesson: Never trust an AI's second-guess over data you already know is correct — same underlying lesson as postmortem #3 (Blitz's false pass), now seen in a different part of the stack.
Symptom: Asked "Show me ip of eth1/3 of PE1", the AI called get_loopback_ip and then get_software_version — neither of which answers the question.
Root cause: No tool existed yet for "get any interface's IP," only get_loopback_ip (Loopback0 specifically). The AI didn't fail or hallucinate — it picked the closest available real tool from a genuinely incomplete toolset.
Fix: Added get_interface_ip(device_name, interface_name), a general-purpose version. Re-asking the same question correctly called the new tool, including correctly translating eth1/3 to Ethernet1/3 on its own.
Lesson: An AI agent is only as capable as the tools it's given — a wrong tool choice is often a coverage gap to fix, not a reasoning failure to fear.
Symptom: "Show me ip of eth1/3 of rr2" (lowercase) failed with Device 'rr2' not found, while the identical request in uppercase (RR2) worked.
Root cause: Device lookups used exact-match against testbed.devices, which is case-sensitive, while the AI correctly and reasonably lowercased the device name from natural language.
Fix: Added a find_device() helper doing case-insensitive matching, used consistently across all four device-lookup tools.
Lesson: When a tool's input comes from an AI's natural-language interpretation rather than a strict CLI flag, it needs to be more forgiving of reasonable variation than code written for a human typing exact commands.
Symptom: python3 05_ai_integration/mcp_health_server.py failed immediately with ModuleNotFoundError: No module named 'mcp.server.fastmcp' — even though requirements.txt correctly pins mcp==2.0.0 and the package was installed.
Root cause: The MCP Python SDK's 2.0 line renamed FastMCP to MCPServer and moved it from mcp.server.fastmcp to mcp.server.mcpserver — a breaking API change, not a missing dependency. The script was still written against the older 1.x API.
Fix: from mcp.server.mcpserver import MCPServer in place of from mcp.server.fastmcp import FastMCP, and mcp = MCPServer("network-health") in place of FastMCP(...). The @mcp.tool() decorator and .run() entry point are unchanged, so no other code needed to move.
Lesson: A pinned version in requirements.txt guarantees a reproducible install, not a stable API — worth checking a dependency's actual installed module layout (python3 -c "import mcp; print(mcp.__file__)", then ls the package) before assuming a ModuleNotFoundError means something isn't installed.
Symptom: Against the live lab, PE1 and PE2 each printed a spurious [✗] BGP 172.1.17.7 () — unknown line — no description, state unknown — right next to the correct [✓] BGP 172.1.17.7 (CE1) — Established for the exact same neighbor IP. genie_healthcheck.py, run against the same live devices at the same time, reported PE1 and PE2 fully healthy with no such entry.
Root cause: Genie's BGP Ops model lists a VRF-scoped neighbor (e.g. PE1's 172.1.17.7, CE1's session inside VRF_A) twice: once correctly under instance.default.vrf.VRF_A.neighbor, complete with session_state and description; and again under instance.default.vrf.default.neighbor as a route-table-only stub (VPNv4 prefix/path counters for that neighbor, no session_state key at all). full_lab_check.py reads neighbor_data.get('session_state', 'unknown') — the missing key on the stub entry silently defaults to the string 'unknown' and gets printed as a failure. genie_healthcheck.py's validate_bgp_state() reads neighbor_data.get('session_state') (defaulting to None) and explicitly continues when the key is absent, so the same stub entry is correctly ignored there.
Fix: Not applied — full_lab_check.py is intentionally the minimal, no-business-logic exercise script (see its own docstring: "zero custom parsing or comparison logic written by hand"). Flagging this here rather than patching it, consistent with this repo's practice of being honest about what each script actually checks.
Lesson: A dict's .get(key, default) fallback is a modeling decision, not a formality — defaulting a missing "is this healthy" field to a truthy-looking placeholder ('unknown') instead of None turns "this field doesn't apply here" into "this looks broken." Same family of lesson as postmortem #3: know exactly what a script is checking before trusting its verdict.
- Understanding the real division of labor between prebuilt parsers (Genie, ntc-templates) and hand-written validation logic — neither eliminates the need for judgment
- Recognizing noise vs. signal in diffed structured state, an empirical judgment call, not something any library provides automatically
- Correctly not assuming a new bug matches a previous one just because the surface symptom looks similar — verifying with real debug output before committing to a fix
- Working across three genuinely different SSH client implementations (Paramiko, libssh, unicon) in one project, diagnosing each on its own evidence
- Catching a tool reporting a false positive result, and being honest about it rather than trusting a green checkmark
- Building a real, working, three-level AI integration on top of an already-reliable automation base — AI reading and querying a tool layer that was trustworthy before AI was added, never the other way around
- Correctly identifying when an AI's output should and shouldn't be trusted, catching a real hallucination against known-correct data
- Iteratively closing tool-coverage gaps based on real failed requests, rather than trying to anticipate every possible question up front
- Running the entire AI stack locally, with no data leaving the lab environment