Agent Eval Platform

AI agent observability & evaluation platform — LLM-as-a-judge scoring across four dimensions, root cause pipelines tracing regressions to the data layer, and longitudinal quality dashboards.

---

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                     Daily Eval Pipeline                          │
│                                                                  │
│  seeds/datasets/         agent/mock_agent.py                    │
│  (50 Q&A pairs)  ──────► (FastAPI + @observe) ──────────────►  │
│                                │                Langfuse v3      │
│                                │               (traces + scores) │
│                                ▼                    │            │
│             eval/evaluators/   │          ingest/   │            │
│           ┌─────────────────┐  │        langfuse_   │            │
│           │ answer_quality  │◄─┘        exporter.py │            │
│           │ retrieval_checks│                ▼       │            │
│           │ latency_checks  │          DuckDB        ◄───────────┘
│           └─────────────────┘       (bronze table)
│                                          │
│                                    dbt transform
│                                  (silver → gold)
│                                          │
│                              fct_agent_quality
│                              fct_retrieval_trends
│                                          │
│                                   Evidence.dev
│                               (localhost:3001)
│                                    dashboard
└─────────────────────────────────────────────────────────────────┘

Tools: Mock Agent (FastAPI) · Langfuse v3 (self-hosted) · Portkey (LLM gateway) · DuckDB · dbt · Prefect · Evidence.dev

---

Portfolio Context

This project is the quality layer of a three-part AI data platform:

1. beacon-lakehouse — Structured operational data layer (client metrics, lead counts, ad spend). Dagster + DuckDB + dbt.
2. client-fact-library — Unstructured knowledge retrieval layer (facts, confidence scores, source types). Qdrant + FastAPI.
3. agent-eval-platform (this project) — Quality observability layer. Evaluates whether agents using the above two layers are actually performing well.

Together: trusted structured data + trusted knowledge retrieval + quality observability = a complete AI data platform story.

---

What This Demonstrates

• LLM-as-a-judge pipelines — Portkey-managed judge prompts scoring accuracy, completeness, and groundedness independently
• Langfuse instrumentation — @observe() decorators on every agent span, metadata on every observation, dataset-based experiments
• Root cause data pipelines — traces a quality regression back to stale facts, low KB coverage, or retrieval degradation
• Longitudinal eval dashboards — Evidence.dev querying DuckDB gold models, regression alert flags
• CI-gated regression detection — make ci-eval fails the build when score thresholds breach

---

Quick Start

# 1. Generate Langfuse secrets
make setup

# 2. Start all services (Langfuse + mock agent)
make up
# Wait ~60 seconds, then open http://localhost:3000

# 3. Configure Portkey (see SETUP.md Step 3) then:
make seed-datasets

# 4. Run the full pipeline
make pipeline

# 5. Open the quality dashboard
make dashboard
# → http://localhost:3001

See SETUP.md for detailed first-run instructions including Portkey configuration.

---

Evaluation Dimensions

Dimension	Type	Target Span	Threshold (CI)
accuracy	LLM-as-a-judge	generation span	≥ 0.80
completeness	LLM-as-a-judge	generation span	—
groundedness	LLM-as-a-judge	generation span	≥ 0.75
retrieval_hit	Code evaluator	retrieval span	—
kb_present	Code evaluator	retrieval span	≥ 0.90 rate
source_diversity	Code evaluator	retrieval span	—
staleness	Code evaluator	retrieval span	—
total_latency_ms	Code evaluator	root trace	—

Example scores from a demo run:

Client	Accuracy	Groundedness	Completeness	KB Present Rate
demo-dental-001	0.89	0.83	0.86	0.92
demo-legal-001	0.85	0.79	0.84	0.88

---

Root Cause Pipeline

When a client's quality score drops, the root cause pipeline determines which data layer caused the regression:

result = run_root_cause_analysis("client-001", date(2026, 6, 26))

# RootCause(
#     client_id="client-001",
#     primary_cause="stale_facts",
#     evidence={
#         "avg_fact_age_days": 47.3,
#         "hit_rate_delta": -0.08,
#         "model_versions": ["gemini-2.5-flash"],
#         "kb_groundedness": 0.61,
#     }
# )

Cause → diagnosis mapping:

• stale_facts → avg_fact_age_days > 30 → refresh facts in client-fact-library
• low_retrieval_quality → hit_rate_delta < -0.20 → check Qdrant index or embeddings
• none → all metrics healthy

---

Quality Dashboard

The Evidence.dev dashboard (http://localhost:3001 after make dashboard) provides:

• Overview page — sparkline accuracy trend, clients sorted by worst groundedness, regression alerts
• Per-client page (/client/[client_id]) — all four dimensions over time, root cause panel
• Experiments page — all eval runs compared, best groundedness highlighted

The regression_flag column in fct_agent_quality is set when any quality dimension drops more than 0.10 points week-over-week. The overview dashboard shows a warning banner listing all flagged clients.

---

Langfuse Datasets

Two ground-truth datasets live in Langfuse (seeded via make seed-datasets):

Dataset	Items	Industry	Categories
industry_qa_dental	25	Dental practice	pricing, insurance, procedure, hours, services
industry_qa_legal	25	Law firm	pricing, legal, procedure, timeline, services

Each item has expected_fact_types, expected_source_types, and expected_answer_contains metadata — used by code evaluators to check KB presence and answer content.

Seed files: seeds/datasets/industry_qa_dental.json · seeds/datasets/industry_qa_legal.json

---

CI Eval Gate

make ci-eval (or the ci-eval GitHub Actions job on main) fails if:

• Mean accuracy < 0.80
• Mean groundedness < 0.75
• kb_present rate < 0.90 for KB-triggering questions

Example failing run:

CI EVAL FAILED: accuracy 0.71 < 0.80; groundedness 0.68 < 0.75

To test a failing gate locally: edit the judge system prompt to return low scores, then run make ci-eval.

---

Production Swap Path

Component	Local	Production
Langfuse	Self-hosted Docker	Langfuse Cloud — change LANGFUSE_BASE_URL only
DuckDB	Local file	Snowflake — change dbt profile + Evidence.dev source
Evidence.dev	Local dev server	Static build deployed to Netlify/Vercel
Prefect	Local server	Prefect Cloud — change PREFECT_API_URL
Mock agent	Local FastAPI	Replace with real agent URL in runner

---

Project Structure

agent-eval-platform/
├── agent/                  # Mock agent server (FastAPI + Langfuse @observe)
├── eval/
│   ├── evaluators/         # answer_quality, retrieval_checks, latency_checks
│   ├── runner.py           # Orchestrates eval loop + Langfuse score submission
│   └── root_cause.py       # Root cause analysis pipeline
├── ingest/
│   ├── langfuse_exporter.py  # Pulls scores from Langfuse API → DuckDB
│   └── score_schema.py       # Pydantic models for score records
├── transform/
│   └── quality_metrics/    # dbt project: bronze → silver → gold
├── pipeline/
│   ├── flows.py            # Prefect flows
│   └── tasks.py            # Prefect tasks
├── dashboard/              # Evidence.dev quality dashboard
├── seeds/datasets/         # 50 ground-truth Q&A pairs (JSON)
├── tests/                  # pytest test suite
├── docs/adr/               # 4 Architecture Decision Records
├── docker-compose.yml      # Langfuse v3 stack + project services
├── Makefile
└── SETUP.md

---

License

MIT — see LICENSE