Session report callback server for agent-transport with a built-in dashboard UI and a shadcn component registry for embedding observability views in your own app.
When a voice agent call ends, the agent-transport SDK uploads a session report containing:
- Chat transcript — full conversation with per-turn metrics (e2e latency, TTS TTFB, LLM TTFT, STT delay)
- Audio recording — OGG/Opus call recording (optional)
- Session metadata — session ID, start time, duration
This server receives that report, extracts session metrics, optionally uploads the audio to S3, and saves everything to Postgres. The dashboard UI lets you browse sessions and view detailed performance metrics.
The easiest way to run the server is Docker Compose. It builds the frontend, starts Postgres, runs migrations, and serves the API plus dashboard on http://localhost:9090.
Prerequisite: Docker with Compose.
git clone https://github.com/plivo-labs/agent-observability
cd agent-observability
cp .env.example .env
docker compose up --buildThe compose file points DATABASE_URL at the bundled Postgres container and
sets AUTO_MIGRATE=true. Edit .env only when you want optional basic auth or
S3 recording upload settings.
Use the Bun flow when you want to work on the backend, frontend, or component registry directly.
Prerequisites:
- Bun runtime
- Postgres database
Install dependencies:
bun install
cd frontend && bun install && cd ..
# Optional, only if working on the component library
cd packages/ui && bun install && cd ..
cp .env.example .env # set DATABASE_URL (required); AGENT_OBSERVABILITY_USER/PASS enable basic authSet DATABASE_URL in .env to your local Postgres database.
Run the backend and frontend dev servers:
# Terminal 1: Backend (Hono server on :9090)
bun run dev
# Terminal 2: Frontend (Vite dev server on :5173, proxies /api to :9090)
bun run dev:frontendOpen http://localhost:5173 for the dashboard.
bun run build:frontend
bun run start # serves API + static frontend on :9090# Server tests
bun test
# UI component tests
cd packages/ui && bun testThe dashboard components are also available as a shadcn registry at packages/ui/. Consumers install components into their own project via npx shadcn add — code is copied in, fully customizable, and wired up with a provider + hooks pattern.
# Install a component into your project
npx agent-observability-ui@latest add metric-summary-cards
# Install the full dashboard (pulls in everything)
npx agent-observability-ui@latest add session-detail-pageSee the full documentation for usage, available components, and hooks. Releases are automated — see Releasing for the publish flow.
Live playground: https://plivo-labs.github.io/agent-observability/ — browse every component with mock data, no install required.
To run locally:
cd docs
bun install
bun run dev # http://localhost:3000/agent-observability/Language-native test-framework plugins stream eval runs into the same
dashboard. Each pytest or vitest invocation lands as one eval_run
with every test surfacing as an eval_case — function-call assertions,
LLM-judge verdicts, agent handoffs, and failure detail are captured
automatically.
| Package | Framework | Docs |
|---|---|---|
agent-observability-sdk |
pytest (Python) | Judges + pytest plugin: install, configure, env vars, and how to invoke pytest from a FastAPI server |
agent-observability-sdk |
Vitest (Node/TS) | Vitest reporter: install, configure, env vars, and how to invoke Vitest from a Bun/Node HTTP server via startVitest |
Runnable reference suites for both frameworks — including simple agents,
a multi-agent banking example, LLM-generated scenarios, and the HTTP
runners — live under plugins/examples/.
| Variable | Required | Description |
|---|---|---|
DATABASE_URL |
Yes | Postgres connection string |
AGENT_OBSERVABILITY_USER |
No | Basic auth username — when set with AGENT_OBSERVABILITY_PASS, native ingest routes accept Basic credentials |
AGENT_OBSERVABILITY_PASS |
No | Basic auth password (see above) |
LIVEKIT_API_KEY |
No | Issuer identifier for LiveKit Bearer JWTs. The LiveKit SDK requires this pair to initialize and signs every observability payload (recordings, OTLP) with it — the observability server must verify against the same pair, since that's the credential the SDK signs with. You generate the pair yourself; see Generating a LiveKit API key/secret. |
LIVEKIT_API_SECRET |
No | HS256 signing secret paired with LIVEKIT_API_KEY. Both env vars are required to enable LiveKit Bearer auth. |
AUTO_MIGRATE |
No | Run SQL migrations on startup (true/false, default: false) |
PORT |
No | Server port (default: 9090) |
S3_BUCKET |
No | Enable S3 upload for audio recordings |
S3_REGION |
No | AWS region (default: us-east-1) |
S3_ACCESS_KEY_ID |
No | Required if S3_BUCKET is set |
S3_SECRET_ACCESS_KEY |
No | Required if S3_BUCKET is set |
S3_ENDPOINT |
No | Custom S3 endpoint (for S3-compatible services) |
S3_PREFIX |
No | Key prefix for uploads (default: recordings) |
| Method | Path | Description |
|---|---|---|
GET |
/health |
Health check (always unauthenticated) |
POST |
/observability/recordings/v0 |
Session report (multipart with JSON or protobuf MetricsRecordingHeader + JSON chat_history + optional OGG audio). Accepts Basic auth or LiveKit Bearer JWT. |
POST |
/observability/logs/otlp/v0 |
OTLP log records emitted by the LiveKit SDK Tagger or hand-built equivalents. Accepts JSON / protobuf, gzip-encoded or not. Persists tags, judge evaluations, outcomes, and session-report patches. |
POST |
/observability/traces/otlp/v0 |
OTLP traces — accepted but not persisted yet (200 no-op). |
POST |
/observability/metrics/otlp/v0 |
OTLP metrics — accepted but not persisted yet (200 no-op). Per-turn agent metrics ride on chat_history items in the recording payload, not here. |
POST |
/observability/evals/v0 |
Eval run payload from the pytest / vitest plugins |
| Method | Path | Description |
|---|---|---|
GET |
/api/sessions |
List sessions (paginated: ?limit=20&offset=0) |
GET |
/api/sessions/:id |
Session detail |
GET |
/api/evals |
List eval runs |
GET |
/api/evals/:run_id |
Single eval run with its cases |
GET |
/api/evals/:run_id/cases/:case_id |
One case with transcript, judgments, failure |
In production, the Vite-built frontend is served as static files from the same server. In development, the Vite dev server proxies API requests to the backend.
| Column | Type | Description |
|---|---|---|
session_id |
TEXT | Call session identifier |
account_id |
TEXT | Account identifier (multi-tenant) |
started_at |
TIMESTAMPTZ | Call start time |
ended_at |
TIMESTAMPTZ | Call end time |
duration_ms |
BIGINT | Call duration in milliseconds |
turn_count |
INTEGER | Number of conversation turns |
has_stt |
BOOLEAN | Speech-to-text was used |
has_llm |
BOOLEAN | LLM was used |
has_tts |
BOOLEAN | Text-to-speech was used |
chat_history |
JSONB | Full transcript with per-turn metrics |
session_metrics |
JSONB | Aggregated latency metrics |
record_url |
TEXT | S3 URL for audio recording |
Two tag names are a wire contract read by the eval judges: amd:voicemail / amd:screening (the sender's machine-detection verdict) and transfer:human (the sender confirms a transfer to a human executed; metadata {"intent": "<handoff intent>", "next_node": "<optional target>"}, which drives the code-derived human_transfer judge). A session without transfer:human gets no human_transfer row — absence is never treated as "not transferred". Send tags in the same OTLP batch as the agent config.
Populated by the OTLP logs ingest path; joined to a session via session_id.
| Table | Purpose |
|---|---|
ao_session_tags |
Tagger annotations (e.g. agent.session, account_id:…, transport:sip). Unique on (session_id, name, source). |
ao_session_external_evals |
LiveKit JudgeGroup outcomes — one row per (session, judge): judge_name, verdict, tag, reasoning, instructions, raw. |
ao_session_outcomes |
High-level pass/fail outcome summaries. Unique on (session_id, source). |
All AO tables carry an ao_ prefix (they share the core DB); the prefix is applied by migration 023_ao_table_prefix.sql. Migrations run automatically when AUTO_MIGRATE=true.
Set these in the agent process to enable session report upload:
AGENT_OBSERVABILITY_URL=https://your-server:9090
# Option A — legacy basic auth (older agent-transport clients)
AGENT_OBSERVABILITY_USER=your_user
AGENT_OBSERVABILITY_PASS=your_pass
# Option B — LiveKit-native auth (agent-transport >= 0.1.10)
# The LiveKit SDK requires this pair to initialize and signs every payload
# it emits (recordings, OTLP logs/traces) with it. The observability server
# verifies against the same pair because that is the only credential the SDK
# signs with. See "Generating a LiveKit API key/secret" below for how to
# create the values.
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secretThe server accepts whichever auth header the client sends. Either option on its own is enough; configure both during a migration window if you have mixed clients.
LIVEKIT_API_KEY and LIVEKIT_API_SECRET are not issued by a LiveKit
cloud service — they are an HS256 keypair you generate locally and
configure on both sides:
- The agent process passes them to the LiveKit SDK, which signs Bearer
JWTs (and the OTLP payloads) with the secret using the key as the
issclaim. - The observability server reads the same pair from its env and verifies
incoming JWT signatures against the secret, requiring
issto equal the key.
Generate them once with openssl (or any source of cryptographic
randomness) and store them in your secrets manager:
LIVEKIT_API_KEY="API$(openssl rand -hex 6)" # short identifier, e.g. APIa1b2c3d4e5f6
LIVEKIT_API_SECRET="$(openssl rand -base64 48)" # high-entropy HS256 signing secretDistribute the same values to every agent process and to the observability server. Rotating the pair is a coordinated change: update the secret store, redeploy the agents (so the SDK picks up the new signing key), and redeploy the observability server (so it verifies against the new key) within the same window.
agent-observability/
├── src/ # Backend (Bun/Hono)
├── frontend/ # Dashboard app (Vite + React)
├── packages/ui/ # shadcn component registry
│ ├── registry/ # Component source
│ └── tests/ # Unit tests
├── docs/ # Docs site (preview app, deployed to GH Pages)
├── plugins/ # Language SDKs + runnable examples
│ ├── agent-observability-sdk/ # Python SDK: judges + pytest plugin
│ ├── agent-observability-sdk-node/ # Node SDK: Vitest reporter + helpers
│ └── examples/ # Runnable eval suites (python/ + node/)
├── migrations/ # SQL migrations
└── tests/ # Server tests