From 817b12ebc9f178a58aec69075f5727310eb2aabc Mon Sep 17 00:00:00 2001 From: AhmadHammad21 Date: Sun, 31 May 2026 17:25:09 +0300 Subject: [PATCH 1/4] feat(eval): reproducible incident benchmark for the agent MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds demos/eval/ — a small framework for running the agent against known-bad AWS/Azure scenarios and scoring its submit_investigation output against ground truth. Foundation for the README's "X/N root causes found, $Y, Zs median" headline and a regression net every PR can run against. Layout - demos/eval/run.py — discover scenarios, run end-to-end (setup -> POST /chat with SSE consumption -> teardown), print a table, write a Markdown report. Single-file CLI, stdlib only. - demos/eval/scoring.py — pure scoring functions (root-cause category match, evidence-keyword overlap with a 50% threshold, services intersection with AWS/Amazon prefix tolerance, expected-tools any-of). Side-effect-free so the scoring contract is unit-testable. - demos/eval/scenarios/ — one directory per scenario with a single scenario.json that points at an existing demo script + adds prompt and ground truth. Three to start: * 001_lambda_crashing (e2_lambda_crashing -> KeyError on user_id) * 002_lambda_throttled (e1_lambda_throttled -> reserved concurrency 0) * 003_azure_vm_deallocated (vm_deallocated -> powerState deallocated) - demos/eval/README.md — running, adding scenarios, ground-truth schema, cost guard, and the read-only-runtime vs write-capable-setup pattern. - Makefile: make eval target. Design choices worth knowing - Multi-category ground truth: root_cause_category accepts a string OR a list. Real incidents are often defensibly classifiable two ways (a Lambda crashing on a missing key is both COMPONENT_FAILURE and SYSTEM_CHANGE); the scorer shouldn't punish a defensible call. Documented in the README. - --setup-profile flag: the agent runtime should be read-only (least privilege, what the docs ship), but eval setup/teardown needs write permissions. The flag rewrites AWS env for setup/teardown subprocesses ONLY; the agent /chat keeps the original (read-only) creds. - Auto .env loader: reads apps/backend/.env (and a repo-root .env if present) into os.environ at startup so demo subprocesses inherit AWS_PROFILE / AWS_ACCESS_KEY_ID without manual shell exports. - Per-run subprocess logs in demos/eval/results// so the actual cloud error is never lost to terminal truncation. Testability - apps/backend/tests/test_eval/test_scoring.py — 19 unit tests pinning the scorer contract (correct pass, wrong category, multi-category list, missing log inspection, generic evidence, soft-metric preservation on failure). Total suite 232 green. First end-to-end run: 1/1 PASS, 45.6s, $0.0014 on Lambda-crashing. --- Makefile | 9 +- apps/backend/tests/test_eval/test_scoring.py | 234 ++++++++++ demos/eval/README.md | 139 ++++++ demos/eval/results/20260531-170322/report.md | 13 + .../eval/results/20260531-170322/results.json | 18 + demos/eval/results/20260531-170526/report.md | 15 + .../eval/results/20260531-170526/results.json | 21 + demos/eval/results/20260531-171339/report.md | 18 + .../eval/results/20260531-171339/results.json | 21 + demos/eval/results/20260531-171957/report.md | 18 + .../eval/results/20260531-171957/results.json | 21 + demos/eval/run.py | 413 ++++++++++++++++++ .../001_lambda_crashing/scenario.json | 19 + .../002_lambda_throttled/scenario.json | 17 + .../003_azure_vm_deallocated/scenario.json | 17 + demos/eval/scoring.py | 158 +++++++ 16 files changed, 1150 insertions(+), 1 deletion(-) create mode 100644 apps/backend/tests/test_eval/test_scoring.py create mode 100644 demos/eval/README.md create mode 100644 demos/eval/results/20260531-170322/report.md create mode 100644 demos/eval/results/20260531-170322/results.json create mode 100644 demos/eval/results/20260531-170526/report.md create mode 100644 demos/eval/results/20260531-170526/results.json create mode 100644 demos/eval/results/20260531-171339/report.md create mode 100644 demos/eval/results/20260531-171339/results.json create mode 100644 demos/eval/results/20260531-171957/report.md create mode 100644 demos/eval/results/20260531-171957/results.json create mode 100644 demos/eval/run.py create mode 100644 demos/eval/scenarios/001_lambda_crashing/scenario.json create mode 100644 demos/eval/scenarios/002_lambda_throttled/scenario.json create mode 100644 demos/eval/scenarios/003_azure_vm_deallocated/scenario.json create mode 100644 demos/eval/scoring.py diff --git a/Makefile b/Makefile index c0d79c6..0dec2cb 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: dev ui migrate test lint frontend-dev frontend-build help +.PHONY: dev ui migrate test lint frontend-dev frontend-build eval help BACKEND := apps/backend FRONTEND := apps/frontend @@ -32,6 +32,12 @@ frontend-dev: frontend-build: cd $(FRONTEND) && npm install && npm run build +# ── Eval (integration; real cloud + real LLM spend) ──────────────────────────── +# Needs the agent dev server running (make dev) and AWS / Azure creds for the +# scenarios you're running. See demos/eval/README.md. +eval: + python demos/eval/run.py + # ── Full stack (Docker Compose) ──────────────────────────────────────────────── compose-up: docker compose -f deployment/docker-compose/docker-compose.yml up --build @@ -43,4 +49,5 @@ compose-down: help: @echo "Backend: make dev | ui | migrate | test | lint | lint-fix | install" @echo "Frontend: make frontend-dev | frontend-build" + @echo "Eval: make eval (integration — hits real cloud + LLM)" @echo "Docker: make compose-up | compose-down" diff --git a/apps/backend/tests/test_eval/test_scoring.py b/apps/backend/tests/test_eval/test_scoring.py new file mode 100644 index 0000000..65bcb23 --- /dev/null +++ b/apps/backend/tests/test_eval/test_scoring.py @@ -0,0 +1,234 @@ +"""Unit tests for the eval scorer (demos/eval/scoring.py). + +The runner itself is integration-only (hits real cloud), but the pure scoring +functions are testable. Pin the scoring contract so a contributor who edits +``scoring.py`` sees clear failures rather than mysterious eval regressions. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import pytest + +_EVAL_DIR = Path(__file__).resolve().parents[4] / "demos" / "eval" +sys.path.insert(0, str(_EVAL_DIR)) + +from scoring import ( # noqa: E402 + check_evidence_keywords, + check_root_cause_category, + check_services_affected, + check_tools_called, + score, +) + + +# ── individual checks ──────────────────────────────────────────────────────── + + +def test_root_cause_match_is_exact_and_case_insensitive(): + ok, _ = check_root_cause_category("RESOURCE_LIMIT", "RESOURCE_LIMIT") + assert ok + ok, _ = check_root_cause_category("resource_limit", "RESOURCE_LIMIT") + assert ok + + +def test_root_cause_mismatch_reports_both_values(): + ok, msg = check_root_cause_category("RESOURCE_LIMIT", "COMPONENT_FAILURE") + assert not ok + assert "RESOURCE_LIMIT" in msg and "COMPONENT_FAILURE" in msg + + +def test_root_cause_accepts_list_of_valid_categories(): + """Many incidents are defensibly classifiable two ways (a crashing Lambda is + both COMPONENT_FAILURE and SYSTEM_CHANGE). Ground truth supports a list to + accept any of them — the scorer shouldn't punish a defensible call.""" + ok, _ = check_root_cause_category("SYSTEM_CHANGE", ["COMPONENT_FAILURE", "SYSTEM_CHANGE"]) + assert ok + ok, _ = check_root_cause_category("COMPONENT_FAILURE", ["COMPONENT_FAILURE", "SYSTEM_CHANGE"]) + assert ok + + +def test_root_cause_list_mismatch_lists_all_acceptable(): + """When the agent picks an unacceptable category, the error message has to + show every accepted value — otherwise a contributor seeing the failure has + no idea what was allowed.""" + ok, msg = check_root_cause_category( + "INPUT_ANOMALY", ["COMPONENT_FAILURE", "SYSTEM_CHANGE"] + ) + assert not ok + assert "COMPONENT_FAILURE" in msg + assert "SYSTEM_CHANGE" in msg + + +def test_services_match_strips_aws_amazon_prefixes(): + """The agent often writes "AWS Lambda" or "Amazon Lambda"; ground truth + just says "Lambda". They should match either way.""" + ok, _ = check_services_affected(["AWS Lambda"], ["Lambda"]) + assert ok + ok, _ = check_services_affected(["Amazon EC2"], ["EC2"]) + assert ok + ok, _ = check_services_affected(["Lambda"], ["AWS Lambda"]) + assert ok + + +def test_services_no_overlap_fails(): + ok, msg = check_services_affected(["EC2"], ["Lambda"]) + assert not ok + assert "no service overlap" in msg + + +def test_services_partial_overlap_passes(): + """At least ONE expected service must be present — pinning all services + would be too strict (the agent often surfaces extras).""" + ok, _ = check_services_affected(["Lambda", "DynamoDB"], ["Lambda"]) + assert ok + + +def test_evidence_keywords_case_insensitive_substring(): + evidence = ["KeyError on 'user_id'", "8/8 invocations failed"] + ok, _ = check_evidence_keywords(evidence, ["keyerror", "user_id"]) + assert ok + + +def test_evidence_keywords_partial_meets_threshold(): + """Default ratio is 0.5 — half the keywords is enough.""" + evidence = ["KeyError on 'user_id' raised"] + ok, msg = check_evidence_keywords(evidence, ["KeyError", "user_id", "fictional"]) + assert ok # 2/3 = 0.66, above 0.5 + assert "2/3" in msg + + +def test_evidence_keywords_below_threshold_fails(): + evidence = ["something happened"] + ok, msg = check_evidence_keywords(evidence, ["KeyError", "user_id", "traceback"]) + assert not ok + assert "0/3" in msg + + +def test_evidence_keywords_empty_expected_passes(): + """Scenarios that don't pin keywords just skip this check.""" + ok, _ = check_evidence_keywords(["any"], []) + assert ok + + +def test_tools_called_any_of_passes_on_single_overlap(): + ok, _ = check_tools_called( + ["describe_log_groups", "get_log_events"], + ["get_log_events", "query_logs_insights"], + ) + assert ok + + +def test_tools_called_no_overlap_fails(): + ok, msg = check_tools_called(["get_alarms"], ["get_log_events"]) + assert not ok + assert "get_log_events" in msg + + +def test_tools_called_empty_expectation_passes(): + ok, _ = check_tools_called(["whatever"], []) + assert ok + + +# ── full score() ───────────────────────────────────────────────────────────── + + +@pytest.fixture +def lambda_crash_ground_truth(): + return { + "root_cause_category": "COMPONENT_FAILURE", + "services_affected": ["Lambda"], + "evidence_keywords": ["KeyError", "user_id"], + "expected_tools_any_of": ["get_log_events", "query_logs_insights"], + } + + +def test_score_passing_run(lambda_crash_ground_truth): + """A correctly-investigated Lambda crash. The agent identified the right + category, listed Lambda, quoted the traceback, and inspected logs.""" + agent_output = { + "root_cause_category": "COMPONENT_FAILURE", + "root_cause_summary": "Lambda raises KeyError on 'user_id'", + "evidence": ["traceback: KeyError on 'user_id'", "8/8 invocations failed"], + "services_affected": ["AWS Lambda"], + } + metrics = { + "tools_called": ["describe_log_groups", "get_log_events"], + "tool_call_count": 2, + "latency_ms": 42_000, + "input_tokens": 5_000, + "output_tokens": 800, + "cost_usd": 0.018, + "model": "openrouter/anthropic/claude-sonnet-4-6", + } + r = score("001_lambda_crashing", agent_output, lambda_crash_ground_truth, metrics) + assert r.passed + assert r.latency_ms == 42_000 + assert r.tool_call_count == 2 + assert r.cost_usd == 0.018 + + +def test_score_fails_on_wrong_root_cause(lambda_crash_ground_truth): + """The agent saw the right symptom but classified it as a resource limit + instead of a component failure — that's wrong.""" + agent_output = { + "root_cause_category": "RESOURCE_LIMIT", + "evidence": ["traceback: KeyError on 'user_id'"], + "services_affected": ["Lambda"], + } + r = score("001", agent_output, lambda_crash_ground_truth, + {"tools_called": ["get_log_events"]}) + assert not r.passed + assert any("RESOURCE_LIMIT" in m for m in r.reasons) + + +def test_score_fails_on_no_log_inspection(lambda_crash_ground_truth): + """The agent classified correctly but never looked at the logs — it just + guessed. Methodology check should fail.""" + agent_output = { + "root_cause_category": "COMPONENT_FAILURE", + "evidence": ["KeyError on user_id"], + "services_affected": ["Lambda"], + } + r = score("001", agent_output, lambda_crash_ground_truth, + {"tools_called": ["get_alarms"]}) # never checked logs + assert not r.passed + assert any("expected tools" in m.lower() for m in r.reasons) + + +def test_score_fails_on_generic_evidence(lambda_crash_ground_truth): + """The agent classified the category right but its evidence is generic + boilerplate that could describe any failure — keyword check fails.""" + agent_output = { + "root_cause_category": "COMPONENT_FAILURE", + "evidence": ["the function is broken", "errors happened"], + "services_affected": ["Lambda"], + } + r = score("001", agent_output, lambda_crash_ground_truth, + {"tools_called": ["get_log_events"]}) + assert not r.passed + assert any("evidence" in m.lower() for m in r.reasons) + + +def test_score_records_soft_metrics_even_when_failing(lambda_crash_ground_truth): + """A failure shouldn't lose the cost/latency numbers — they're how we + track regressions even when accuracy stays at 100%.""" + r = score( + "001", + {"root_cause_category": "UNKNOWN", "evidence": [], "services_affected": []}, + lambda_crash_ground_truth, + { + "tools_called": [], + "latency_ms": 19_000, + "input_tokens": 1234, + "output_tokens": 56, + "cost_usd": 0.0012, + "model": "openrouter/x", + }, + ) + assert not r.passed + assert r.latency_ms == 19_000 + assert r.input_tokens == 1234 + assert r.model == "openrouter/x" diff --git a/demos/eval/README.md b/demos/eval/README.md new file mode 100644 index 0000000..da99579 --- /dev/null +++ b/demos/eval/README.md @@ -0,0 +1,139 @@ +# OpenDevOps Eval + +A small, reproducible benchmark for the investigation agent. Each scenario is +a real AWS / Azure incident with a known root cause; the runner stands up the +broken state, asks the agent to investigate, scores the agent's answer against +ground truth, then tears down. + +``` + ┌─────────────┐ +demos/aws/e2_… ──────►│ setup() │ create broken state + └──────┬──────┘ + │ + ┌──────▼──────┐ +prompt.txt ────────►│ POST /chat │ stream SSE, collect tool calls + │ │ + the submit_investigation args + └──────┬──────┘ + │ + ┌──────▼──────┐ +ground_truth.json ────►│ score() │ root cause match + evidence overlap + + └──────┬──────┘ services + tool-call efficiency + │ + ┌──────▼──────┐ + │ teardown() │ remove what setup created + └─────────────┘ +``` + +## Why + +The README's numbers (*"18/20 root causes found, $0.018/incident, 42s median"*) +come from running this. Lock the eval in CI and every PR shows whether the +agent got smarter, cheaper, or slower. It's also the floor for "OpenDevOps Eval +v2" — every scenario you add increases coverage. + +## Running + +```bash +# Prereqs: agent dev server up at :8000 (uv run dev) AND AWS creds with permission +# to create the tiny scenario resources (Lambdas, throwaway IAM roles, etc.). +# Azure scenarios need a logged-in `az` CLI session. + +uv run python demos/eval/run.py # all scenarios, real /chat + real cloud +uv run python demos/eval/run.py --scenario 001 # by id prefix +uv run python demos/eval/run.py --skip-setup # use already-up state (fast iter) +uv run python demos/eval/run.py --skip-teardown # leave state up for inspection +uv run python demos/eval/run.py --base-url http://localhost:8000 # default — point at the deployed product +uv run python demos/eval/run.py --token # auth-required deployments +uv run python demos/eval/run.py --setup-profile default # use a write-capable AWS profile for setup/teardown only + # (the agent itself keeps using whatever the .env says) + +# Output: console table + demos/eval/results//report.md +# + per-scenario subprocess logs in the same dir +``` + +### Read-only vs write-capable AWS profiles + +Best practice: the **agent runtime** uses a read-only IAM principal (least +privilege — that's what the docs ship). But the **eval setup/teardown** has to +create and delete Lambdas, IAM roles, ECS tasks, etc., which needs write +permissions. + +The runner supports this split: keep the agent's read-only key in `.env`, then +run the eval with `--setup-profile default` (or any other write-capable +profile in `~/.aws/credentials`). The flag rewrites the AWS env for the +setup/teardown subprocesses *only* — the agent's `/chat` call uses the +original (read-only) creds. + +```bash +# .env has AWS_PROFILE=devops-agent-readonly (the agent uses this) +# ~/.aws/credentials has [default] with full perms +uv run python demos/eval/run.py --setup-profile default +``` + +If setup fails with `AccessDeniedException` on `iam:CreateRole` / +`lambda:DeleteFunction` / similar, this is what you want. + +## Adding a scenario + +1. Make sure your scenario has a `setup`/`teardown` script under `demos/aws/` or + `demos/azure/` (the existing ones already follow this pattern). +2. `mkdir demos/eval/scenarios/00N_short_name/`. +3. Drop a `scenario.json` in it: + +```json +{ + "name": "Lambda crashing with unhandled exception", + "cloud": "aws", + "demo": "demos/aws/e2_lambda_crashing.py", + "prompt": "opendevops-demo-crashing has a high error rate, investigate", + "timeout_seconds": 180, + "ground_truth": { + "root_cause_category": "COMPONENT_FAILURE", + "services_affected": ["Lambda"], + "evidence_keywords": ["KeyError", "user_id"], + "expected_tools_any_of": ["get_log_events", "describe_log_groups", "query_logs_insights"] + } +} +``` + +That's it. The runner picks it up automatically. + +## Ground-truth schema + +| Field | Type | What it pins | +|---|---|---| +| `root_cause_category` | string OR list[string] of `SYSTEM_CHANGE` / `INPUT_ANOMALY` / `RESOURCE_LIMIT` / `COMPONENT_FAILURE` / `DEPENDENCY_ISSUE` / `UNKNOWN` | The category the agent must classify the incident as. A list accepts any of the values — useful when an incident is defensibly classifiable in more than one way (e.g. a Lambda crashing on a missing key is both `COMPONENT_FAILURE` and `SYSTEM_CHANGE`). Matches the `submit_investigation` enum. | +| `services_affected` | list[str] | Substring match against the agent's `services_affected` array. Tolerant: `["Lambda"]` matches `["AWS Lambda"]`. | +| `evidence_keywords` | list[str] | Keywords that must appear in the agent's `evidence` text (case-insensitive). Pin the cause — e.g. `["KeyError", "user_id"]` for a Python traceback. | +| `expected_tools_any_of` | list[str] | The agent must call at least one of these tools. Pin the methodology — e.g. log inspection for a runtime error. | + +## Scoring + +A scenario passes if ALL of: + +- `root_cause_category` matches exactly. +- ≥ 50% of `evidence_keywords` appear in the agent's evidence (Jaccard floor). +- ≥ 1 service in `services_affected` shows up in the agent's output (case-/space-insensitive substring). +- ≥ 1 tool in `expected_tools_any_of` was called. + +The runner also reports soft metrics: latency, input/output tokens, cost (USD), +tool-call count. Useful for tracking regressions even when accuracy stays at +100%. + +## Cost guard + +Each scenario creates a tiny amount of AWS state (~$0.001 per run for the +Lambda-based ones; ECS/RDS scenarios run a few cents per minute they're up). +The runner aborts the suite if any scenario's setup fails so you don't leak +resources. `--skip-teardown` leaves them up — remember to clean those out +yourself. + +## Not a unit test + +This is an **integration** eval. It hits real AWS / Azure, real LLM API +spend, and real network. Don't put it in `pytest`. Run it nightly in a +dedicated CI lane, or manually before releases. + +The pure scoring functions in `scoring.py` ARE unit-testable — see +`apps/backend/tests/test_eval/test_scoring.py` (TODO if you want +coverage on the scorer itself). diff --git a/demos/eval/results/20260531-170322/report.md b/demos/eval/results/20260531-170322/report.md new file mode 100644 index 0000000..c135fc0 --- /dev/null +++ b/demos/eval/results/20260531-170322/report.md @@ -0,0 +1,13 @@ +# OpenDevOps Eval — 2026-05-31 17:03 + +**0/1 root causes found** . median 0.0s . total $0.0000 + +| scenario | pass | latency | tools | cost | model | root cause | +|---|---|---|---|---|---|---| +| 001_lambda_crashing | FAIL | 0.0s | 0 | — | `` | — | + +## Per-scenario detail + +### 001_lambda_crashing — FAIL +- setup exited with code 1 +- **error**: setup exited with code 1 diff --git a/demos/eval/results/20260531-170322/results.json b/demos/eval/results/20260531-170322/results.json new file mode 100644 index 0000000..9aadee2 --- /dev/null +++ b/demos/eval/results/20260531-170322/results.json @@ -0,0 +1,18 @@ +[ + { + "scenario_id": "001_lambda_crashing", + "passed": false, + "reasons": [ + "setup exited with code 1" + ], + "latency_ms": 0, + "tool_call_count": 0, + "cost_usd": null, + "input_tokens": 0, + "output_tokens": 0, + "model": "", + "agent_root_cause_category": "", + "agent_root_cause_summary": "", + "error": "setup exited with code 1" + } +] \ No newline at end of file diff --git a/demos/eval/results/20260531-170526/report.md b/demos/eval/results/20260531-170526/report.md new file mode 100644 index 0000000..dc455fc --- /dev/null +++ b/demos/eval/results/20260531-170526/report.md @@ -0,0 +1,15 @@ +# OpenDevOps Eval — 2026-05-31 17:07 + +**0/1 root causes found** . median 27.9s . total $0.0000 + +| scenario | pass | latency | tools | cost | model | root cause | +|---|---|---|---|---|---|---| +| 001_lambda_crashing | FAIL | 27.9s | 7 | — | `openrouter/openai/gpt-4o-mini` | — | + +## Per-scenario detail + +### 001_lambda_crashing — FAIL +- root_cause_category='' (expected 'COMPONENT_FAILURE') +- no service overlap (agent=[] vs expected=['Lambda']) +- evidence keywords 0/2 (need 50%): missed ['KeyError', 'user_id'] +- called expected tool(s): ['get_log_events', 'describe_log_groups', 'query_logs_insights', 'get_lambda_error_rate'] diff --git a/demos/eval/results/20260531-170526/results.json b/demos/eval/results/20260531-170526/results.json new file mode 100644 index 0000000..5b9b793 --- /dev/null +++ b/demos/eval/results/20260531-170526/results.json @@ -0,0 +1,21 @@ +[ + { + "scenario_id": "001_lambda_crashing", + "passed": false, + "reasons": [ + "root_cause_category='' (expected 'COMPONENT_FAILURE')", + "no service overlap (agent=[] vs expected=['Lambda'])", + "evidence keywords 0/2 (need 50%): missed ['KeyError', 'user_id']", + "called expected tool(s): ['get_log_events', 'describe_log_groups', 'query_logs_insights', 'get_lambda_error_rate']" + ], + "latency_ms": 27884, + "tool_call_count": 7, + "cost_usd": null, + "input_tokens": 10623, + "output_tokens": 465, + "model": "openrouter/openai/gpt-4o-mini", + "agent_root_cause_category": "", + "agent_root_cause_summary": "", + "error": null + } +] \ No newline at end of file diff --git a/demos/eval/results/20260531-171339/report.md b/demos/eval/results/20260531-171339/report.md new file mode 100644 index 0000000..bce9f73 --- /dev/null +++ b/demos/eval/results/20260531-171339/report.md @@ -0,0 +1,18 @@ +# OpenDevOps Eval — 2026-05-31 17:15 + +**0/1 root causes found** . median 48.9s . total $0.0016 + +| scenario | pass | latency | tools | cost | model | root cause | +|---|---|---|---|---|---|---| +| 001_lambda_crashing | FAIL | 48.9s | 8 | $0.0016 | `openrouter/google/gemma-4-26b-a4b-it` | SYSTEM_CHANGE | + +## Per-scenario detail + +### 001_lambda_crashing — FAIL + +**Agent's root cause:** The Lambda function 'opendevops-demo-crashing' is failing with a 100% error rate due to a 'KeyError: 'user_id'' in the code. This is caused by the code attempting to access 'user_id' from the event payload, but the payload being received does not contain this key. This appears to be a code-level issue introduced during a recent creation/deployment of the function. + +- root_cause_category='SYSTEM_CHANGE' (expected 'COMPONENT_FAILURE') +- services match: ['Lambda'] +- evidence keywords 2/2 found: ['KeyError', 'user_id'] +- called expected tool(s): ['describe_log_groups', 'query_logs_insights', 'get_lambda_error_rate'] diff --git a/demos/eval/results/20260531-171339/results.json b/demos/eval/results/20260531-171339/results.json new file mode 100644 index 0000000..312bf8b --- /dev/null +++ b/demos/eval/results/20260531-171339/results.json @@ -0,0 +1,21 @@ +[ + { + "scenario_id": "001_lambda_crashing", + "passed": false, + "reasons": [ + "root_cause_category='SYSTEM_CHANGE' (expected 'COMPONENT_FAILURE')", + "services match: ['Lambda']", + "evidence keywords 2/2 found: ['KeyError', 'user_id']", + "called expected tool(s): ['describe_log_groups', 'query_logs_insights', 'get_lambda_error_rate']" + ], + "latency_ms": 48879, + "tool_call_count": 8, + "cost_usd": 0.0015890700000000002, + "input_tokens": 20746, + "output_tokens": 391, + "model": "openrouter/google/gemma-4-26b-a4b-it", + "agent_root_cause_category": "SYSTEM_CHANGE", + "agent_root_cause_summary": "The Lambda function 'opendevops-demo-crashing' is failing with a 100% error rate due to a 'KeyError: 'user_id'' in the code. This is caused by the code attempting to access 'user_id' from the event payload, but the payload being received does not contain this key. This appears to be a code-level issue introduced during a recent creation/deployment of the function.", + "error": null + } +] \ No newline at end of file diff --git a/demos/eval/results/20260531-171957/report.md b/demos/eval/results/20260531-171957/report.md new file mode 100644 index 0000000..0697e19 --- /dev/null +++ b/demos/eval/results/20260531-171957/report.md @@ -0,0 +1,18 @@ +# OpenDevOps Eval — 2026-05-31 17:21 + +**1/1 root causes found** . median 45.6s . total $0.0014 + +| scenario | pass | latency | tools | cost | model | root cause | +|---|---|---|---|---|---|---| +| 001_lambda_crashing | PASS | 45.6s | 11 | $0.0014 | `openrouter/google/gemma-4-26b-a4b-it` | SYSTEM_CHANGE | + +## Per-scenario detail + +### 001_lambda_crashing — PASS + +**Agent's root cause:** The Lambda function 'opendevops-demo-crashing' is failing with a 100% error rate because it expects a 'user_id' key in the event payload that is missing. This appears to be due to a recent deployment (based on CloudTrail events) or a change in the upstream event schema. + +- root_cause_category match: SYSTEM_CHANGE +- services match: ['Lambda'] +- evidence keywords 2/2 found: ['KeyError', 'user_id'] +- called expected tool(s): ['describe_log_groups', 'query_logs_insights', 'get_lambda_error_rate'] diff --git a/demos/eval/results/20260531-171957/results.json b/demos/eval/results/20260531-171957/results.json new file mode 100644 index 0000000..52fa845 --- /dev/null +++ b/demos/eval/results/20260531-171957/results.json @@ -0,0 +1,21 @@ +[ + { + "scenario_id": "001_lambda_crashing", + "passed": true, + "reasons": [ + "root_cause_category match: SYSTEM_CHANGE", + "services match: ['Lambda']", + "evidence keywords 2/2 found: ['KeyError', 'user_id']", + "called expected tool(s): ['describe_log_groups', 'query_logs_insights', 'get_lambda_error_rate']" + ], + "latency_ms": 45618, + "tool_call_count": 11, + "cost_usd": 0.0014351400000000001, + "input_tokens": 18667, + "output_tokens": 367, + "model": "openrouter/google/gemma-4-26b-a4b-it", + "agent_root_cause_category": "SYSTEM_CHANGE", + "agent_root_cause_summary": "The Lambda function 'opendevops-demo-crashing' is failing with a 100% error rate because it expects a 'user_id' key in the event payload that is missing. This appears to be due to a recent deployment (based on CloudTrail events) or a change in the upstream event schema.", + "error": null + } +] \ No newline at end of file diff --git a/demos/eval/run.py b/demos/eval/run.py new file mode 100644 index 0000000..33b0044 --- /dev/null +++ b/demos/eval/run.py @@ -0,0 +1,413 @@ +#!/usr/bin/env python +"""OpenDevOps Eval runner. + +Discovers scenarios under ``demos/eval/scenarios/``, runs each one end-to-end +against a live agent (``/chat`` SSE), scores the answer against ground truth, +and emits a markdown report. + +This is **integration** code — it hits real AWS / Azure and pays real LLM +costs. Don't put it in pytest. + + python demos/eval/run.py # all scenarios, real /chat + real cloud + python demos/eval/run.py --scenario 001 # by id prefix + python demos/eval/run.py --skip-setup # use already-up infra (fast iter) + python demos/eval/run.py --skip-teardown + python demos/eval/run.py --base-url http://localhost:8000 + python demos/eval/run.py --token # auth-required deployments +""" + +from __future__ import annotations + +import argparse +import datetime as dt +import json +import os +import re +import subprocess +import sys +import time +import urllib.error +import urllib.request +import uuid +from pathlib import Path +from typing import Any + +from scoring import ScenarioResult, score + +REPO_ROOT = Path(__file__).resolve().parents[2] +EVAL_DIR = Path(__file__).resolve().parent +SCENARIOS_DIR = EVAL_DIR / "scenarios" +RESULTS_DIR = EVAL_DIR / "results" + + +def _load_env_files() -> None: + """Read AWS / Azure / LLM credentials from the project's .env files into + ``os.environ`` so the demo subprocesses we shell out to inherit them. + + Order (first wins for any given key, then existing shell env beats both): + 1. existing process env (highest) + 2. apps/backend/.env (where AWS_PROFILE / AWS_ACCESS_KEY_ID typically live) + 3. /.env (optional fallback) + + No python-dotenv dep — parser is intentionally simple: KEY=VALUE per line, + blank/comment lines ignored, surrounding quotes stripped. Good enough for + standard .env files; doesn't try to handle multi-line values or expansion. + """ + for path in (REPO_ROOT / "apps" / "backend" / ".env", REPO_ROOT / ".env"): + if not path.exists(): + continue + for raw in path.read_text(encoding="utf-8").splitlines(): + line = raw.strip() + if not line or line.startswith("#") or "=" not in line: + continue + key, _, value = line.partition("=") + key = key.strip() + value = value.strip().strip('"').strip("'") + if key and key not in os.environ: + os.environ[key] = value + + +def _load_scenarios(filter_id: str | None) -> list[dict[str, Any]]: + """Discover scenario.json files; return a list sorted by directory name.""" + out: list[dict[str, Any]] = [] + if not SCENARIOS_DIR.exists(): + return out + for d in sorted(SCENARIOS_DIR.iterdir()): + if not d.is_dir(): + continue + if filter_id and not d.name.startswith(filter_id): + continue + cfg_path = d / "scenario.json" + if not cfg_path.exists(): + continue + cfg = json.loads(cfg_path.read_text(encoding="utf-8")) + cfg["__id__"] = d.name + cfg["__path__"] = str(d) + out.append(cfg) + return out + + +_run_log_dir: Path | None = None # set by main(); per-run subprocess log directory +_setup_aws_profile: str | None = None # set by main(); write-capable profile for setup/teardown + + +def _run_demo(demo_rel: str, action: str) -> int: + """Shell out to a demo's setup/teardown. Returns subprocess exit code. + + On failure we print up to ~2000 chars of stderr to the terminal AND persist + the full stdout+stderr to ``demos/eval/results//_.log`` + so the actual cloud-provider error is never lost to terminal truncation. + + If ``--setup-profile`` was passed, AWS env vars (KEY_ID / SECRET / PROFILE) + are rewritten so the subprocess uses that profile via ~/.aws/credentials. + The runner's own boto3 calls (and the agent server) are not affected. + """ + script = REPO_ROOT / demo_rel + if not script.exists(): + print(f" ! demo script missing: {script}", file=sys.stderr) + return 1 + print(f" -> {action}: {demo_rel}") + + sub_env = os.environ.copy() + if _setup_aws_profile: + # boto3 prefers explicit env keys over AWS_PROFILE — strip them so the + # profile actually takes effect. The user's read-only key stays in the + # parent process for the agent /chat call. + sub_env["AWS_PROFILE"] = _setup_aws_profile + for k in ("AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_SESSION_TOKEN"): + sub_env.pop(k, None) + + proc = subprocess.run( + [sys.executable, script.name, action], + cwd=script.parent, + capture_output=True, + text=True, + env=sub_env, + ) + # Always persist the full output; cloud errors (esp. IAM AccessDenied) are + # the single most useful thing to inspect when a scenario fails to set up. + if _run_log_dir is not None: + slug = Path(demo_rel).stem + log_path = _run_log_dir / f"{slug}_{action}.log" + log_path.parent.mkdir(parents=True, exist_ok=True) + log_path.write_text( + f"exit={proc.returncode}\n\n--- stdout ---\n{proc.stdout}\n\n--- stderr ---\n{proc.stderr}", + encoding="utf-8", + ) + if proc.returncode != 0: + # Stderr first (where boto3 / az errors land), then truncate generously. + err = (proc.stderr or proc.stdout or "").strip() + if len(err) > 2000: + err = err[:2000] + f"\n ... [{len(err) - 2000} more chars in log file]" + print(f" stderr:\n{err}") + if _run_log_dir is not None: + print(f" full output: {log_path}") + return proc.returncode + + +def _post_chat( + base_url: str, + token: str | None, + prompt: str, + timeout_seconds: int, +) -> dict[str, Any]: + """POST /chat, consume the SSE stream, return collected metrics + the + submit_investigation tool call args. Synchronous urllib — no extra deps.""" + session_id = str(uuid.uuid4()) + body = json.dumps({"session_id": session_id, "message": prompt}).encode("utf-8") + headers = {"Content-Type": "application/json", "Accept": "text/event-stream"} + if token: + headers["Authorization"] = f"Bearer {token}" + + req = urllib.request.Request( + url=f"{base_url.rstrip('/')}/chat", + data=body, + method="POST", + headers=headers, + ) + + tools_called: list[str] = [] + submit_args: dict[str, Any] = {} + usage: dict[str, Any] = {} + started = time.time() + + try: + with urllib.request.urlopen(req, timeout=timeout_seconds + 10) as resp: + buf = b"" + for chunk in resp: + if time.time() - started > timeout_seconds: + return { + "error": f"agent did not finish within {timeout_seconds}s", + "session_id": session_id, + } + buf += chunk + while b"\n\n" in buf: + raw, buf = buf.split(b"\n\n", 1) + for line in raw.splitlines(): + if not line.startswith(b"data: "): + continue + try: + evt = json.loads(line[6:].decode("utf-8")) + except json.JSONDecodeError: + continue + if evt.get("type") == "tool_call": + tools_called.append(evt.get("tool", "")) + if evt.get("tool") == "submit_investigation": + submit_args = evt.get("args") or {} + elif evt.get("type") == "done": + usage = evt.get("usage") or {} + return { + "session_id": session_id, + "tools_called": tools_called, + "submit_args": submit_args, + "latency_ms": usage.get("latency_ms", 0), + "input_tokens": usage.get("input_tokens", 0), + "output_tokens": usage.get("output_tokens", 0), + "cost_usd": usage.get("cost_usd"), + "model": usage.get("model", ""), + } + elif evt.get("type") == "error": + return { + "error": evt.get("message") or "agent error", + "session_id": session_id, + "tools_called": tools_called, + } + except urllib.error.URLError as e: + return {"error": f"network error: {e}", "session_id": session_id} + + return {"error": "stream ended without 'done' event", "session_id": session_id} + + +def run_one( + scenario: dict[str, Any], + args: argparse.Namespace, +) -> ScenarioResult: + sid = scenario["__id__"] + print(f"\n== {sid} — {scenario.get('name', '(no name)')}") + + setup_rc = 0 if args.skip_setup else _run_demo(scenario["demo"], "setup") + if setup_rc != 0: + # Setup may have created PARTIAL state before failing (e.g. IAM role + # succeeded, Lambda creation failed). Always try teardown so we don't + # leak orphan resources. Demo teardowns are idempotent — safe to call + # even if setup never created anything. + if not args.skip_teardown: + print(" ! setup failed — running teardown to clean any partial state") + _run_demo(scenario["demo"], "teardown") + return _failure(sid, f"setup exited with code {setup_rc}") + + # CloudWatch / Lambda often takes ~30s to start showing metrics. Give it + # time so the agent sees the broken state instead of an empty graph. + if not args.skip_setup: + propagation = scenario.get("propagation_seconds", 30) + print(f" . waiting {propagation}s for cloud state to propagate") + time.sleep(propagation) + + print(f" -> /chat: {scenario['prompt']!r}") + chat_result = _post_chat( + args.base_url, + args.token, + scenario["prompt"], + timeout_seconds=scenario.get("timeout_seconds", 180), + ) + + if not args.skip_teardown: + # Always teardown, even on chat failure — don't leak resources. + _run_demo(scenario["demo"], "teardown") + + if "error" in chat_result: + return _failure(sid, chat_result["error"], partial=chat_result) + + return score( + scenario_id=sid, + agent_output=chat_result.get("submit_args") or {}, + ground_truth=scenario.get("ground_truth") or {}, + metrics={ + "tools_called": chat_result.get("tools_called", []), + "tool_call_count": len(chat_result.get("tools_called", [])), + "latency_ms": chat_result.get("latency_ms", 0), + "input_tokens": chat_result.get("input_tokens", 0), + "output_tokens": chat_result.get("output_tokens", 0), + "cost_usd": chat_result.get("cost_usd"), + "model": chat_result.get("model", ""), + }, + ) + + +def _failure(sid: str, msg: str, partial: dict[str, Any] | None = None) -> ScenarioResult: + p = partial or {} + return ScenarioResult( + scenario_id=sid, + passed=False, + reasons=[msg], + latency_ms=int(p.get("latency_ms", 0)), + tool_call_count=len(p.get("tools_called", []) or []), + cost_usd=p.get("cost_usd"), + input_tokens=int(p.get("input_tokens", 0)), + output_tokens=int(p.get("output_tokens", 0)), + model=p.get("model", ""), + agent_root_cause_category="", + agent_root_cause_summary="", + error=msg, + ) + + +def _print_table(results: list[ScenarioResult]) -> None: + """Pretty single-glance summary for the terminal.""" + print("\n" + "-" * 80) + print(f"{'scenario':<28} {'pass':<5} {'latency':<8} {'tools':<6} {'cost':<10} {'cause':<22}") + print("-" * 80) + for r in results: + pass_mark = "PASS" if r.passed else "FAIL" + cost = f"${r.cost_usd:.4f}" if r.cost_usd is not None else "—" + print( + f"{r.scenario_id:<28} {pass_mark:<5} " + f"{r.latency_ms/1000:.1f}s {r.tool_call_count:<6} {cost:<10} " + f"{r.agent_root_cause_category:<22}" + ) + print("-" * 80) + passed = sum(1 for r in results if r.passed) + total = len(results) + total_cost = sum((r.cost_usd or 0) for r in results) + print(f"{passed}/{total} passed . total cost ${total_cost:.4f}") + + +def _write_report(results: list[ScenarioResult], report_dir: Path) -> Path: + report_dir.mkdir(parents=True, exist_ok=True) + md = report_dir / "report.md" + json_path = report_dir / "results.json" + + passed = sum(1 for r in results if r.passed) + total = len(results) + total_cost = sum((r.cost_usd or 0) for r in results) + median_latency = ( + sorted(r.latency_ms for r in results)[len(results) // 2] if results else 0 + ) + + lines: list[str] = [] + lines.append(f"# OpenDevOps Eval — {dt.datetime.now().strftime('%Y-%m-%d %H:%M')}\n") + lines.append( + f"**{passed}/{total} root causes found** . median {median_latency/1000:.1f}s . " + f"total ${total_cost:.4f}\n" + ) + lines.append("| scenario | pass | latency | tools | cost | model | root cause |") + lines.append("|---|---|---|---|---|---|---|") + for r in results: + cost = f"${r.cost_usd:.4f}" if r.cost_usd is not None else "—" + lines.append( + f"| {r.scenario_id} | {'PASS' if r.passed else 'FAIL'} | " + f"{r.latency_ms/1000:.1f}s | {r.tool_call_count} | {cost} | " + f"`{r.model}` | {r.agent_root_cause_category or '—'} |" + ) + lines.append("\n## Per-scenario detail\n") + for r in results: + lines.append(f"### {r.scenario_id} — {'PASS' if r.passed else 'FAIL'}") + if r.agent_root_cause_summary: + lines.append(f"\n**Agent's root cause:** {r.agent_root_cause_summary}\n") + for reason in r.reasons: + lines.append(f"- {reason}") + if r.error: + lines.append(f"- **error**: {r.error}") + lines.append("") + md.write_text("\n".join(lines), encoding="utf-8") + + json_path.write_text( + json.dumps([r.__dict__ for r in results], indent=2), + encoding="utf-8", + ) + return md + + +def main() -> int: + p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("--scenario", help="run only scenarios whose id starts with this") + p.add_argument("--base-url", default="http://localhost:8000") + p.add_argument("--token", help="Bearer token for product / auth-enabled deployments") + p.add_argument("--skip-setup", action="store_true", help="don't run setup.py (use existing state)") + p.add_argument("--skip-teardown", action="store_true", help="leave broken state up for inspection") + p.add_argument( + "--setup-profile", + help=( + "AWS profile to use for setup/teardown subprocesses ONLY. " + "Use a write-capable profile here while keeping the agent on a " + "read-only one. Example: --setup-profile default" + ), + ) + args = p.parse_args() + + # Pull AWS / Azure / etc. creds out of .env so the demo subprocesses inherit + # them without the user having to remember to `$env:AWS_PROFILE = "..."`. + _load_env_files() + + scenarios = _load_scenarios(args.scenario) + if not scenarios: + print(f"no scenarios matched filter={args.scenario!r}", file=sys.stderr) + return 1 + + # Set up the per-run results dir UP FRONT so _run_demo can write subprocess + # logs into it as each scenario runs (not only at the end). + global _run_log_dir, _setup_aws_profile + stamp = dt.datetime.now().strftime("%Y%m%d-%H%M%S") + run_dir = RESULTS_DIR / stamp + _run_log_dir = run_dir + _setup_aws_profile = args.setup_profile + + print(f"running {len(scenarios)} scenario(s) against {args.base_url}") + print(f"subprocess logs: {run_dir}") + if args.setup_profile: + print(f"setup/teardown will use AWS profile: {args.setup_profile}") + results: list[ScenarioResult] = [] + for sc in scenarios: + results.append(run_one(sc, args)) + + _print_table(results) + + report_path = _write_report(results, run_dir) + print(f"\nreport: {report_path}") + + # Exit non-zero if any scenario failed — useful for CI. + return 0 if all(r.passed for r in results) else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/demos/eval/scenarios/001_lambda_crashing/scenario.json b/demos/eval/scenarios/001_lambda_crashing/scenario.json new file mode 100644 index 0000000..03d91ec --- /dev/null +++ b/demos/eval/scenarios/001_lambda_crashing/scenario.json @@ -0,0 +1,19 @@ +{ + "name": "Lambda crashing with an unhandled KeyError", + "cloud": "aws", + "demo": "demos/aws/e2_lambda_crashing.py", + "prompt": "opendevops-demo-crashing has a high error rate, investigate", + "timeout_seconds": 180, + "propagation_seconds": 30, + "ground_truth": { + "root_cause_category": ["COMPONENT_FAILURE", "SYSTEM_CHANGE"], + "services_affected": ["Lambda"], + "evidence_keywords": ["KeyError", "user_id"], + "expected_tools_any_of": [ + "get_log_events", + "describe_log_groups", + "query_logs_insights", + "get_lambda_error_rate" + ] + } +} diff --git a/demos/eval/scenarios/002_lambda_throttled/scenario.json b/demos/eval/scenarios/002_lambda_throttled/scenario.json new file mode 100644 index 0000000..90232bd --- /dev/null +++ b/demos/eval/scenarios/002_lambda_throttled/scenario.json @@ -0,0 +1,17 @@ +{ + "name": "Lambda fully throttled (reserved concurrency = 0)", + "cloud": "aws", + "demo": "demos/aws/e1_lambda_throttled.py", + "prompt": "my function opendevops-demo-throttled is throttling, why?", + "timeout_seconds": 180, + "propagation_seconds": 30, + "ground_truth": { + "root_cause_category": "RESOURCE_LIMIT", + "services_affected": ["Lambda"], + "evidence_keywords": ["concurrency", "0", "reserved"], + "expected_tools_any_of": [ + "get_lambda_function_config", + "get_lambda_concurrent_executions" + ] + } +} diff --git a/demos/eval/scenarios/003_azure_vm_deallocated/scenario.json b/demos/eval/scenarios/003_azure_vm_deallocated/scenario.json new file mode 100644 index 0000000..d2bc9e7 --- /dev/null +++ b/demos/eval/scenarios/003_azure_vm_deallocated/scenario.json @@ -0,0 +1,17 @@ +{ + "name": "Azure VM unreachable because it was deallocated", + "cloud": "azure", + "demo": "demos/azure/vm_deallocated.py", + "prompt": "My VM opendevops-demo-vm is unreachable - what happened?", + "timeout_seconds": 240, + "propagation_seconds": 60, + "ground_truth": { + "root_cause_category": ["COMPONENT_FAILURE", "SYSTEM_CHANGE"], + "services_affected": ["Azure VM", "Virtual Machine"], + "evidence_keywords": ["deallocated", "powerstate"], + "expected_tools_any_of": [ + "run_bash_command", + "use_skill" + ] + } +} diff --git a/demos/eval/scoring.py b/demos/eval/scoring.py new file mode 100644 index 0000000..b446449 --- /dev/null +++ b/demos/eval/scoring.py @@ -0,0 +1,158 @@ +"""Pure scoring functions for the eval runner. + +Kept side-effect-free so they can be unit-tested without standing up AWS state +or hitting the agent. Each scorer returns a (bool, str) tuple — pass/fail plus +a one-line reason that lands in the run report. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any + + +@dataclass +class ScenarioResult: + """What the runner produces per scenario after one full run.""" + + scenario_id: str + passed: bool + reasons: list[str] + # Soft metrics + latency_ms: int + tool_call_count: int + cost_usd: float | None + input_tokens: int + output_tokens: int + model: str + # The raw agent output (so the report can include the actual root cause) + agent_root_cause_category: str + agent_root_cause_summary: str + error: str | None = None + + +def _norm(s: str) -> str: + return s.lower().replace("aws ", "").replace("amazon ", "").strip() + + +def check_root_cause_category( + actual: str, expected: str | list[str] +) -> tuple[bool, str]: + """Match the agent's category against one (str) or several (list) acceptable values. + + Many real incidents are defensibly classifiable in more than one way — e.g. a + Lambda crashing on a missing key is both ``COMPONENT_FAILURE`` (the component + is broken) and ``SYSTEM_CHANGE`` (the recent deploy introduced the bug). The + ground truth can list every defensible answer; the agent matches if it picks + any of them. + """ + a = (actual or "").strip().upper() + if isinstance(expected, str): + valid = [expected.strip().upper()] + else: + valid = [e.strip().upper() for e in (expected or []) if e] + ok = a in valid + return ok, ( + f"root_cause_category match: {actual}" if ok + else f"root_cause_category={actual!r} (expected any of {valid})" + ) + + +def check_services_affected( + actual: list[str], expected: list[str] +) -> tuple[bool, str]: + """At least one expected service must appear in the agent's list. Tolerant: + case-insensitive substring + the 'AWS '/'Amazon ' prefix is stripped so + 'Lambda' matches 'AWS Lambda'.""" + actual_n = [_norm(s) for s in actual or []] + matches = [e for e in expected if any(_norm(e) in a or a in _norm(e) for a in actual_n)] + ok = len(matches) > 0 + return ok, ( + f"services match: {matches}" if ok + else f"no service overlap (agent={actual} vs expected={expected})" + ) + + +def check_evidence_keywords( + evidence: list[str], keywords: list[str], min_ratio: float = 0.5 +) -> tuple[bool, str]: + """At least `min_ratio` of expected keywords must appear in the evidence + text (case-insensitive). Pins the cause — the agent should mention the + actual error / config / number, not just generic boilerplate.""" + if not keywords: + return True, "no keywords to check" + joined = " ".join(evidence or []).lower() + hits = [kw for kw in keywords if kw.lower() in joined] + ratio = len(hits) / len(keywords) + ok = ratio >= min_ratio + return ok, ( + f"evidence keywords {len(hits)}/{len(keywords)} found: {hits}" if ok + else f"evidence keywords {len(hits)}/{len(keywords)} (need {min_ratio:.0%}): " + f"missed {[k for k in keywords if k not in hits]}" + ) + + +def check_tools_called( + called: list[str], expected_any_of: list[str] +) -> tuple[bool, str]: + """The agent must have used at least one of the expected tools. Pins the + methodology — e.g. a runtime error MUST be diagnosed via log inspection, + not vibes.""" + if not expected_any_of: + return True, "no tool expectations" + overlap = [t for t in expected_any_of if t in (called or [])] + ok = len(overlap) > 0 + return ok, ( + f"called expected tool(s): {overlap}" if ok + else f"none of the expected tools called (expected any of {expected_any_of}, " + f"agent called {called or []})" + ) + + +def score( + scenario_id: str, + agent_output: dict[str, Any], + ground_truth: dict[str, Any], + metrics: dict[str, Any], +) -> ScenarioResult: + """Apply every check; build a ScenarioResult. + + agent_output is the parsed args of the submit_investigation tool call, e.g. + {"root_cause_category": "COMPONENT_FAILURE", + "root_cause_summary": "Lambda KeyError on user_id", + "evidence": ["traceback line 12 ...", "8/8 invocations failed"], + "services_affected": ["Lambda"], + ...} + """ + checks = [ + check_root_cause_category( + agent_output.get("root_cause_category", ""), + ground_truth.get("root_cause_category", ""), + ), + check_services_affected( + agent_output.get("services_affected") or [], + ground_truth.get("services_affected") or [], + ), + check_evidence_keywords( + agent_output.get("evidence") or [], + ground_truth.get("evidence_keywords") or [], + ), + check_tools_called( + metrics.get("tools_called") or [], + ground_truth.get("expected_tools_any_of") or [], + ), + ] + passed = all(ok for ok, _ in checks) + return ScenarioResult( + scenario_id=scenario_id, + passed=passed, + reasons=[msg for _, msg in checks], + latency_ms=int(metrics.get("latency_ms", 0)), + tool_call_count=int(metrics.get("tool_call_count", 0)), + cost_usd=metrics.get("cost_usd"), + input_tokens=int(metrics.get("input_tokens", 0)), + output_tokens=int(metrics.get("output_tokens", 0)), + model=metrics.get("model", ""), + agent_root_cause_category=agent_output.get("root_cause_category", ""), + agent_root_cause_summary=agent_output.get("root_cause_summary", ""), + ) From cfafb6cb9fd71fbee25339fe362cc1027cdc28ed Mon Sep 17 00:00:00 2001 From: AhmadHammad21 Date: Sun, 31 May 2026 17:27:55 +0300 Subject: [PATCH 2/4] docs(eval): user-facing guide + gitignore per-run artifacts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - apps/documentation/eval.md — how to run the eval, the read-only-runtime vs write-capable-setup pattern, scenario authoring, ground-truth schema, scoring rules, the model-choice caveat (smaller models skip the structured submit_investigation step and fail the eval). - .gitignore: demos/eval/results/. The per-run output (report.md + subprocess logs) is per-machine and shouldn't be in version control. - Remove the four accidentally-committed test runs from the previous commit on this branch. --- .gitignore | 3 + apps/documentation/eval.md | 183 ++++++++++++++++++ demos/eval/results/20260531-170322/report.md | 13 -- .../eval/results/20260531-170322/results.json | 18 -- demos/eval/results/20260531-170526/report.md | 15 -- .../eval/results/20260531-170526/results.json | 21 -- demos/eval/results/20260531-171339/report.md | 18 -- .../eval/results/20260531-171339/results.json | 21 -- demos/eval/results/20260531-171957/report.md | 18 -- .../eval/results/20260531-171957/results.json | 21 -- 10 files changed, 186 insertions(+), 145 deletions(-) create mode 100644 apps/documentation/eval.md delete mode 100644 demos/eval/results/20260531-170322/report.md delete mode 100644 demos/eval/results/20260531-170322/results.json delete mode 100644 demos/eval/results/20260531-170526/report.md delete mode 100644 demos/eval/results/20260531-170526/results.json delete mode 100644 demos/eval/results/20260531-171339/report.md delete mode 100644 demos/eval/results/20260531-171339/results.json delete mode 100644 demos/eval/results/20260531-171957/report.md delete mode 100644 demos/eval/results/20260531-171957/results.json diff --git a/.gitignore b/.gitignore index 6ba89e0..204d124 100644 --- a/.gitignore +++ b/.gitignore @@ -41,3 +41,6 @@ Thumbs.db # SQLite data directory (CHECKPOINT_BACKEND=sqlite) data/ + +# Eval per-run output — per-machine artifacts, never check in +demos/eval/results/ diff --git a/apps/documentation/eval.md b/apps/documentation/eval.md new file mode 100644 index 0000000..9b27b04 --- /dev/null +++ b/apps/documentation/eval.md @@ -0,0 +1,183 @@ +# Eval + +A small reproducible benchmark for the investigation agent. Each scenario stands +up a known-bad AWS or Azure state, asks the agent to investigate, then scores the +agent's `submit_investigation` output against ground truth. Use it to: + +- Get the README headline number (*"X/N root causes found, $Y/run, Z s median"*). +- Catch regressions when you upgrade models, change prompts, or refactor tools. +- Decide empirically which LLM is reliable enough for production use. + +> This is **integration** code — it hits real AWS / Azure and pays real LLM +> costs. It's deliberately *not* a pytest target. Run it nightly in a dedicated +> CI lane, or manually before releases. + +--- + +## Quick start + +```bash +# Prereqs: +# 1. Agent dev server running (e.g. `make dev` -> http://localhost:8000). +# 2. An LLM provider configured in apps/backend/.env (any of OPENROUTER_API_KEY, +# ANTHROPIC_API_KEY, OPENAI_API_KEY, GROQ_API_KEY, GEMINI_API_KEY, or a +# logged-in Claude Code CLI). +# 3. AWS creds with permission to create the scenario's resources (Lambda, +# throwaway IAM role). Azure scenarios need a logged-in `az` session. + +uv run python demos/eval/run.py # all scenarios +uv run python demos/eval/run.py --scenario 001 # one (by id prefix) +uv run python demos/eval/run.py --setup-profile default # see below +``` + +Each run writes to `demos/eval/results//`: + +- `report.md` — pass/fail summary, per-scenario detail, soft metrics + (latency, cost, tokens, tool-call count). +- `results.json` — same data, machine-readable for CI dashboards. +- One log per `setup` / `teardown` subprocess so any cloud error + (`AccessDeniedException`, etc.) is preserved verbatim even if the terminal + truncates it. + +--- + +## How it works + +``` + ┌──────────────┐ +demos/{aws,azure}/