feat(eval): reproducible incident benchmark for the agent

[AhmadHammad21]

Summary

Adds demos/eval/ — a small framework that runs the agent against known-bad AWS / Azure scenarios and scores its submit_investigation output against ground truth. Foundation for the README's "X/N root causes found, $Y/run, Zs median" line and a regression net every PR can run against.

Verified: 1/1 PASS on a real Lambda-crashing investigation (45.6s, $0.0014, 11 tool calls).

What's in the box

• demos/eval/run.py — discover scenarios, run end-to-end (setup → POST /chat with SSE consumption → score → teardown), print a table, write a Markdown + JSON report. Single-file CLI, stdlib only.
• demos/eval/scoring.py — pure scoring functions (category match, evidence-keyword overlap with 50% threshold, services intersection with AWS/Amazon prefix tolerance, expected-tools any-of). Side-effect-free so the contract is unit-testable.
• demos/eval/scenarios/ — one directory per scenario with a single scenario.json that points at an existing demo script + adds prompt and ground truth. Three to start:
  • 001_lambda_crashing (e2_lambda_crashing → KeyError on user_id)
  • 002_lambda_throttled (e1_lambda_throttled → reserved concurrency 0)
  • 003_azure_vm_deallocated (vm_deallocated → powerState deallocated)
• demos/eval/README.md — implementation reference (how scenarios compose, scoring internals).
• apps/documentation/eval.md — user-facing guide (running, authoring scenarios, model-choice caveats).
• apps/backend/tests/test_eval/test_scoring.py — 19 unit tests pinning the scorer contract. Total suite: 232 green.
• Makefile: make eval target.

Design choices worth knowing

• Multi-category ground truth: root_cause_category accepts a string OR a list. Real incidents are often defensibly classifiable two ways (a Lambda crashing on a missing key is both COMPONENT_FAILURE and SYSTEM_CHANGE); the scorer shouldn't punish a defensible call. First run uncovered exactly this — see lesson baked into the docs.
• --setup-profile flag: the agent runtime should be read-only (least privilege, what iam_setup.md ships), but eval setup/teardown needs write perms. The flag rewrites AWS env for setup/teardown subprocesses ONLY; the agent /chat keeps the original (read-only) creds.
• Auto .env loader: reads apps/backend/.env (and a repo-root .env if present) into os.environ at startup so demo subprocesses inherit AWS_PROFILE / keys without manual shell exports.
• Per-run subprocess logs in demos/eval/results/<timestamp>/ so a cloud error (AccessDeniedException, etc.) is never lost to terminal truncation. results/ is gitignored.
• Not a pytest target: this is integration code that hits real cloud + LLM. Documented as such. Run nightly in a dedicated CI lane or manually before releases.

Test plan

• cd apps/backend && uv run pytest tests/test_eval/ — 19 passing
• cd apps/backend && uv run pytest — 232 passing
• End-to-end against openrouter/openai/gpt-4o-mini on 001_lambda_crashing — PASS in 45.6s, $0.0014
• Run remaining scenarios (002, 003) — follow-up, low risk
• Add more scenarios from the existing demo set — follow-up (7 more demos available)

What this unlocks

• A real README headline once we sweep all scenarios.
• An empirical model-comparison story (Sonnet vs GPT-4o vs Gemma — different cost/accuracy trade-offs visible per run).
• A clear failure-mode log: "this model is too small to close the loop on tool use" becomes visible immediately, not three weeks into a customer test.