Observability: transcript + offer-event persistence with compliance audit trail

[teetangh]

Context

Transcripts and outcomes are only durably recorded for the offline eval loop (data/eval_runs/*.json). Live calls leave nothing queryable: the streaming path's conversation lives in the in-process Pipecat context, the TwiML path's history dies with the in-memory ChatSession, and offers proposed/validated via propose_offer (apps/voice/streaming.py:558-572) are only in logs. For a debt-collection product this is both a conversions-measurement gap and an FDCPA/CFPB auditability gap — you must be able to prove what was said, what was offered, and what was agreed.

Implementation plan

1. Schema (extends the voice_calls metrics table work): call_transcripts (session_id, turn_index, role, text, ts) and call_events (session_id, ts, kind [propose_offer|offer_validated|offer_rejected|end_call|disclosure], payload JSONB).
2. Streaming path capture: hook the context aggregator / TranscriptProcessor to append turns as they finalize; emit call_events rows from handle_propose_offer (both the LLM's attempted offer and the validator verdict + canonical suggested block) and handle_end_call.
3. TwiML/chat path capture: persist ChatSession.history turns on each /voice/gather cycle (piggyback on the Redis session-store refactor or write-through to Postgres).
4. Disclosure audit: record a disclosure event when the opener (AI identity + recording notice) is spoken — direct evidence for the compliance rules in data/policy/v1.yaml / learning/compliance.py.
5. Retention/PII: document retention policy; store borrower identifiers by internal id only; last-4 masking in transcripts mirrors the existing \d{9,} leak rule.
6. Read path: GET /calls/{session_id}/transcript operator endpoint; conversions become queryable (SELECT outcome, count(*) ... GROUP BY outcome).
7. Tests: simulator call → transcript rows + propose_offer events present; unit tests for event emission from the tool handlers.

Acceptance criteria

• After a simulator streaming call: full turn-by-turn transcript retrievable via the endpoint; propose_offer events queryable with validator verdicts; end_call outcome recorded.
• TwiML-path sessions persist history the same way.
• Disclosure event present for every call that got past the opener.

Dependencies

• Observability: per-call metrics (TTFB per stage, cost, outcome) → Postgres + report endpoints #8 per-call metrics → Postgres (shares the DB plumbing).
• Reliability: Redis-backed session store (replace in-memory _SESSIONS/_PENDING) #4 Redis-backed session store (TwiML path write-through).