Version: 1.1 · Owner: Solo builder · Last updated: 2026-06-02 Status: Built & verified offline (19 tests) — remaining: Vertex run, Cloud Run deploy, demo video · Deadline: 2026-06-12 02:30 IST (= 2026-06-11 14:00 PDT, verified on official Devpost page 2026-05-17) · Target ship: 2026-06-11 (buffer)
v1.1 note: v1 shipped and then went deeper. The loop now also does root-cause analysis, live replay of the failing input, and adversarial red-team; evaluation runs live (Phoenix MCP has no run-experiment tool); Cassandra traces and grades itself and publishes its own MCP server. Reasoning core is Gemini 3 / OpenAI / OpenRouter.
Cassandra is a meta-agent: an autonomous agent whose sole responsibility is the continuous quality supervision of other LLM agents in production. It observes a target agent through its Arize Phoenix telemetry, detects failure modes, converts real failures into reproducible evaluation assets, validates fixes experimentally, and proposes deployable prompt patches — closing the observability → evaluation → improvement loop with no human in the path until the final approval gate.
Elevator pitch: "It is an agent that babysits agents. When your production agent hallucinates, Cassandra catches it, proves it with an eval, and hands you a tested fix before your on-call engineer has finished their coffee."
LLM agents fail in ways traditional monitoring cannot see:
- Hallucination — confident, fluent, factually wrong output (e.g. inventing a refund policy that does not exist).
- Prompt drift — behavior degrades after a model/version change while the prompt is unchanged; or instruction-following decays over long contexts.
- Tool-call failure — a tool errors or returns malformed data and the agent papers over the gap with a fabrication instead of surfacing the error.
Current mitigation is manual and reactive: humans sample traces, eyeball outputs, hand-author eval datasets, and tweak prompts by intuition. This does not scale with agent count or traffic, has high latency to detection, and misses the long tail entirely.
- G1 — Detect the three failure classes above from live Phoenix traces with explainable, per-span verdicts.
- G2 — Write verdicts back into Phoenix as span annotations (failures are visible in the customer's existing tool, not a separate silo).
- G3 — Auto-synthesize a reproducible adversarial eval dataset from any confirmed failure.
- G4 — Quantify any proposed fix with a Phoenix experiment (LLM-as-judge, current vs. candidate prompt).
- G5 — Propose a versioned, A/B-ready prompt patch via Phoenix prompt management.
- G6 — Present the entire loop on a live dashboard suitable for a 3-minute demo.
- G7 — Close the loop on itself: trace Cassandra's own reasoning into Phoenix and grade its own diagnostic accuracy (the Arize track's explicit self-improvement bonus criterion).
- G8 — Publish Cassandra's supervision as an MCP server callable by any external agent/IDE.
- NG1 — Auto-deploying patches without human approval (we queue an A/B; we do not flip it live).
- NG2 — Supervising non-LLM systems or infra metrics (that is the Dynatrace bucket's job).
- NG3 — Multi-tenant SaaS, auth, billing, RBAC.
- NG4 — Fine-tuning or model training. Cassandra only touches prompts and evals.
- NG5 — Supporting observability backends other than Phoenix.
| Persona | Need | How Cassandra serves it |
|---|---|---|
| LLM Platform Engineer ("the on-call for agents") | Know within minutes when an agent regresses, with a reproducible case | Real-time alerts + annotated failing span + auto eval dataset |
| Applied AI / Prompt Engineer | A tested prompt fix, not a hunch | Candidate patch with measured pass-rate delta from a Phoenix experiment |
| Eng Manager / Hackathon Judge (Arize SE) | See Phoenix used deeply and correctly | Near-full Phoenix MCP surface exercised in a coherent autonomous loop |
Primary persona for the demo narrative: the Platform Engineer woken at 2am because a support agent is inventing policies.
- US-1 As a platform engineer, I want Cassandra to watch my agent's Phoenix project so that failures are caught without me sampling traces.
- US-2 As a platform engineer, I want each detected failure annotated on the exact Phoenix span so that I can see why it was flagged in the tool I already use.
- US-3 As a prompt engineer, I want a failure turned into a 12-row adversarial dataset so that the bug is reproducible and regression-tested forever.
- US-4 As a prompt engineer, I want a Phoenix experiment comparing the current and proposed prompt so that I trust the fix with numbers, not vibes.
- US-5 As a platform engineer, I want the proposed patch versioned in Phoenix prompt management with an A/B queued so that rollout is one approval click.
- US-6 As a judge, I want to watch this entire loop happen live in under 3 minutes.
| Capability | Status |
|---|---|
| Scheduled trace ingestion from Phoenix via MCP | ✅ |
| LLM-as-judge classification of spans (Gemini / OpenAI / OpenRouter) | ✅ |
| Incident severity (class × confidence) | ✅ |
| Write-back annotations to Phoenix | ✅ |
| Root-cause analysis (culprit + causal chain + fix strategy) | ✅ |
| Adversarial dataset synthesis + upload to Phoenix | ✅ |
| Live baseline-vs-candidate evaluation (real numbers, not stubbed) | ✅ |
| Cost/latency efficiency delta | ✅ |
| Prompt patch proposal + Phoenix prompt versioning + unified diff | ✅ |
| Live replay of the original failing input (before/after FIXED) | ✅ |
| Adversarial red-team against the live agent | ✅ |
| A/B queued (not auto-promoted) | ✅ NG1 |
| Live single-file dashboard (full cascade + Phoenix deep links) | ✅ |
| "The Patient" fragile victim + deterministic seeder | ✅ |
| Self-evaluation scorecard (grades its own diagnoses) | ✅ |
Self-tracing into cassandra-meta |
✅ |
Published cassandra-mcp server (supervision-as-tools) |
✅ |
| On-product Phoenix experiments via Phoenix client | ✅ optional (flag) |
| Vertex Agent Engine run · Cloud Run hosted URL · demo video | ⛔ pending (env-gated) |
| BigQuery analytics · Slack/email alerting · auto-promotion | ❌ out of scope |
- Primary: Top-3 placement in the Arize bucket ($2k–$5k).
- Submission accepted: public repo + top-level LICENSE + hosted URL + ≤3-min video + Devpost form + Arize track selected.
- Detection latency: seeded failure → annotated Phoenix span in < 10 seconds.
- Diagnostic precision: ≥ 90% correct classification on the seeded trap set (hand-labeled, ~20 cases).
- Eval lift demonstrated: candidate prompt beats current prompt by a visible margin (target: current ≤ 4/12 pass, candidate ≥ 10/12 pass on the synthesized dataset).
- MCP surface coverage: ≥ 5 distinct Phoenix MCP tool families exercised end-to-end (spans, annotations, datasets, experiments, prompts).
- Solo builder, ~25 working days.
- Mandatory stack: Gemini 3, Agent Builder/ADK, ≥1 partner MCP (Phoenix). Non-negotiable.
- Phoenix Cloud free tier is sufficient for demo scale; self-host on Cloud Run is the fallback if cloud limits bite.
- Judges will not run the code — the video and hosted URL carry the submission.
- Assumption: Phoenix MCP exposes tools for spans, annotations, datasets, experiments, and prompt management. Validated by a Day-1 spike before any feature work.
- R1 — Phoenix MCP tool surface differs from assumptions → Day-1 enumeration spike.
- R2 — Gemini 3 / Agent Engine quota or availability → confirm Day 1, Cloud Run fallback.
- R3 — Failure not reproducible on camera → deterministic seeder, never live randomness.
- R4 — Scope creep → NG list is binding; stretch items only if ahead at Day 20.
The product is "done" when, from a clean deploy, a single seeded customer message causes — with no human action — an annotated Phoenix span, a synthesized Phoenix dataset, a completed Phoenix experiment showing a pass-rate lift, and a versioned candidate prompt with an A/B queued, all reflected on the live dashboard within the demo window; and the repository is public with a detectable top-level Apache-2.0 LICENSE.