Attune

Behavioral evaluation infrastructure for AI companions and social agents. Point Attune at any OpenAI-compatible chatbot endpoint, pick a predefined compliance package, and get a structured report covering per-dimension scores, flagged conversations, and judge rationales.

v1 MVP — local, single-user. Targets California SB 243 disengagement behavior: 10 scenarios × 5 personas × 3 research-grounded evaluation dimensions.

Run it locally

Prereqs: uv, Node 20+, an OPENAI_API_KEY with active billing.

Set your key — anywhere in the repo, create a .env:

OPENAI_API_KEY=sk-...

Start the backend (FastAPI + LangGraph, port 8000):

cd backend
uv sync
uv run uvicorn attune.main:app --port 8000

Start the frontend (Next.js, port 3000), in a second terminal:

cd frontend
npm install
npm run dev

Open http://localhost:3000.

Demo flow

1. Home → New run.
2. Target agent: Reference agent → Manipulative reference.
3. Package: California SB 243 — Disengagement Audit.
4. Start run → watch scenarios execute in the live view (SSE-driven).
5. View report → per-dimension averages, flagged conversations, transcript drill-down with highlighted turns and judge rationales.

Expected calibration: running the package against manipulative_reference scores ~4.0 on disengagement_manipulation; against healthy_baseline scores ~1.0. A working install separates them by at least 2 points.

What's in each folder

• content/ — authored environments, personas, scenarios, dimensions, packages, reference agents. This is the library the product sells.
• backend/ — FastAPI app. src/attune/: Pydantic models, YAML content loader, SQLite DB, LangGraph conversation + judge graphs, orchestrator, routes.
• frontend/ — Next.js 15 App Router + Tailwind v4 + shadcn/ui. Warm Anthropic-style palette with infra-tool density.
• docs/superpowers/specs/ — design spec that defines scope and acceptance criteria.
• docs/superpowers/plans/ — task-by-task implementation plan.

Architecture

• One Python process serves HTTP, SSE, and runs LangGraph orchestration. No queue, no background worker.
• Scenarios execute concurrently with asyncio.Semaphore(5) against the target endpoint.
• Per-conversation judging runs one LangGraph StateGraph per applicable dimension.
• All content is YAML on disk, validated against Pydantic models on startup.
• Operational state (runs, scenario_executions, conversations, evaluations, run_events) lives in SQLite at backend/runs.db.

Tests

cd backend && uv run pytest -v

24 backend tests cover models, content loader, DB, both LangGraph graphs, OpenAI adapter stubs, orchestrator, routes, and SSE replay.

Out of scope for v1

Auth, multi-user, Postgres, billing, PDF export, additional environments (Crisis Disclosure, Identity Probing), voice/multi-modal, customer-authored scenarios, production observability.