Reliability: Redis-backed session store (replace in-memory _SESSIONS/_PENDING)

[teetangh]

Context

All conversation/session state is in process-local module globals:

• _SESSIONS: dict in apps/api/chat_session.py:394 (chat + TwiML voice sessions)
• _PENDING: dict of in-flight LLM asyncio tasks in apps/api/voice_twiml.py:95
• Streaming session registry used by dial-init → TwiML → WS handoff in apps/api/voice_twiml.py

Consequences: any multi-worker/multi-replica deployment breaks (Twilio's follow-up webhooks land on a different process than the one holding the session), restarts drop mid-call state, and cumulative_cost_usd is process-global rather than per-session. Redis is already provisioned in docker-compose.yml and REDIS_URL exists in agents/settings.py, but nothing uses it. Code comments already say "production would use Redis keyed by borrower_id" (chat_session.py:9).

Implementation plan

1. Add a SessionStore abstraction (apps/api/session_store.py): get/put/delete(session_id), in-memory impl (default, keeps tests hermetic) + Redis impl selected by REDIS_URL presence or a settings flag. Use redis.asyncio.
2. Make ChatSession serializable (it's mostly message history + stage + counters; pydantic model or to_dict/from_dict). Store per-session cumulative_cost_usd in the session, not a global.
3. Replace _SESSIONS reads/writes in chat_session.py and voice_twiml.py with the store; set a TTL (e.g. 2 h) so abandoned sessions expire.
4. _PENDING (asyncio tasks) cannot live in Redis — instead store the result in Redis when the task completes, and make /voice/wait/{sid}/{turn}/{cycle} poll the store, so any worker can serve the wait webhook. Task handles stay process-local only as an optimization.
5. Streaming path: persist the dial-init session record (borrower context, ws_token material) in the store so /voice/streaming/twiml/{session_id} and the WS handler work cross-worker.
6. Tests: store round-trip unit tests (fakeredis or the in-memory impl); integration check running uvicorn with --workers 2 against scripts/simulate_twiml_call.py.

Acceptance criteria

• TwiML simulator passes against a 2-worker uvicorn with Redis enabled.
• Restarting the API mid-session (between webhooks) resumes the conversation.
• Unit suite stays green with no Redis available (in-memory fallback).

Dependencies

• None (unblocks: WS lifecycle hardening, contact-frequency compliance counter, container-per-call deployment).