Autonomous PRD-to-Prototype Agent

Turns a raw product idea into a structured PRD, user stories, and an interactive prototype spec – then grades its own output against a product-quality rubric and sends it back for repair if it falls short.

The generation is the easy part. The evaluation harness is the point.

Demo mode replays a recorded run with no API key and no backend running. The trace shows the eval agent scoring 3.20, the router sending a critique back, and revision 1 scoring 3.60 – then the prototype spec rendered as a wireframe whose buttons actually navigate.

           raw idea
              │
          interview
              │
         requirements
              │
              ▼
   ┌───────► PRD
   │          │
   │     ┌────┴────┐
   │     ▼         ▼
   │  stories  prototype
   │     │         │
   │     └────┬────┘
   │          ▼
   │       evaluate
   │          │
   │          ▼
   │        router ───► finalize
   │          │
   └─ repair ─┘

---

Why this exists

Anything can generate a PRD. Almost nothing can tell you whether the PRD is any good, and that is the actual product problem. This repo is built around three claims that are stated so they can be checked:

1. The multi-agent loop earns its complexity. Measured as repair_lift – the delivered spec's score minus its first draft's. If it is zero, the architecture is decorative.
2. The judge is trustworthy. Measured as agreement with human labels (quadratic weighted kappa), and withheld until a human has actually labelled.
3. Claims are grounded. Every requirement carries the interview turns it came from, so "did the model make this up?" is a question about evidence rather than plausibility.

The finding that justifies the whole harness

The first measurement of repair_lift was exactly 0.00. The loop ran twice and changed nothing.

The harness traced it to a real defect. The PRD mandated exact UI copy ("No handover notes yet.") while the prototype spec independently invented a different string for the same element. The cause was in generate_prototype: it received requirement titles and descriptions but not the acceptance criteria, which is where that copy lives. It could not see what it was supposed to match – and since it regenerates in parallel with the PRD from the same critique, the two re-diverged on every revision. The loop could not converge.

Passing the full criteria and declaring the PRD authoritative for literal copy:

	before	after
repair_lift	0.00	+0.40
blocking ambiguities	5	2

A second finding fell out of the same data: scores went 3.20 → 3.60 → 3.40. Revision 1 helped, revision 2 made it worse – on both runs measured. So the graph now returns its best revision rather than its last, and max_revisions dropped from 2 to 1.

That is the argument for building the evals first. Without them this ships looking fine.

Architecture

graph TD
    START([idea]) --> PM[pm_agent<br/>interview: human or simulated]
    PM --> EX[requirements_extractor<br/>attaches source_turn_ids]
    EX --> PRD[spec_prd]
    PRD --> ST[spec_stories]
    PRD --> PT[spec_prototype]
    ST --> EV[eval_agent<br/>19 deterministic checks + judge k=3]
    PT --> EV
    EV -->|below threshold<br/>or blocking defect| REV[prepare_revision<br/>builds targeted critique]
    REV --> PRD
    EV -->|passes| FIN[finalize<br/>keeps best revision]
    FIN --> END([PRD · stories · prototype · scorecard])

Loading

Stories and prototype genuinely run in parallel; both depend on the PRD for requirement ids, so the fan-out starts after it. State is a Pydantic model threaded through LangGraph, and the trace it accumulates drives the live UI, the demo replay, and the eval harness from one representation.

The judge runs on a different model family from the spec writer. A model grading its own output scores it generously.

Evaluation

Two layers, kept structurally separate because they fail differently.

Deterministic (api/checks.py) – sixteen checks, no model involved. Is every requirement testable? Does every story map to a real requirement? Is every prototype screen reachable from the entry screen (a graph traversal)? These cannot hallucinate, so they are the floor.

LLM judge (api/agents/eval_agent.py) – five rubric dimensions, k=3 samples with the median taken and the spread reported. Every score must cite a verbatim quote, and those quotes are then checked mechanically against the source. A score whose evidence cannot be found in the artifact is a fabricated score.

Golden dataset – 14 cases, four deliberately adversarial:

Case	What it tests
vague-one-liner-03	"An app for pets." A confident PRD here is fabrication.
contradictory-reqs-04	Offline + real-time + always-consistent + serverless. Cannot all hold.
scope-creep-11	Eleven features, no priority. Does it force sequencing?
hallucination-trap-07	Bait to invent integrations nobody mentioned.

Adversarial cases are supposed to score low. See docs/EVAL_METHODOLOGY.md for rubric definitions, metric formulas, and a frank list of limitations – including why the judge calibration number is currently withheld.

Running it

brew install python@3.13
python3.13 -m venv .venv && .venv/bin/pip install -r requirements-dev.txt
echo 'AI_GATEWAY_API_KEY=...' > .env      # vercel.com → AI Gateway → API Keys

python scripts/resolve_models.py           # resolve real model slugs, then edit config.yaml
python -m api.run --case oncall-handover-09   # one run, headless
python -m evals.run_benchmark --workers 4  # full benchmark → docs/EVAL_REPORT.md
python -m evals.label                      # assign human labels for judge calibration
pytest                                     # 33 tests, no API key needed

Frontend and API together:

uvicorn api.main:app --port 8000     # Python service
cd web && npm run dev                # Next.js, proxies /api in development

Stack

LangGraph for orchestration · Pydantic for structured output enforced end-to-end · Vercel AI Gateway for model access (one key, every provider, and live pricing so cost is reported in dollars rather than tokens) · FastAPI streaming SSE · Next.js frontend with a wireframe renderer driven straight off the prototype spec.

Model slugs are resolved from the gateway at setup time rather than hardcoded – they change often enough that a memorised slug is a 404 waiting to happen.

Repo map

Path
api/graph.py	the state machine, fan-out, and repair loop
api/schemas/	Pydantic contracts; provenance lives here
api/checks.py	deterministic checks – start here
api/agents/eval_agent.py	judge, evidence verification, critique builder
evals/dataset/	14 golden cases
evals/metrics.py	kappa, repair lift, coverage, hallucination
web/	Next.js app: runner, wireframe, eval dashboard
docs/EVAL_METHODOLOGY.md	how this is measured, and where it is weak
docs/how-it-works.html	full step-by-step walkthrough – open it in a browser