Resilience: circuit breaker for Anthropic calls + fake-upstream load-test harness

[mahsa7haft]

Goal

Make PaidUp resilient to a slow or failing external dependency (primarily the Anthropic API, secondarily the Parliament APIs), and build a cost-free, no-real-traffic way to load-test it.

Two outcomes:

1. A circuit breaker around the Claude calls so a slow/dead dependency fails fast instead of exhausting worker threads.
2. A fake-upstream + Locust harness so we can generate traffic and inject faults without ever calling real Claude (no cost) or hammering the public Parliament API.

Why

• A single /lookup fans out to 3 external APIs in parallel; /analyze calls Anthropic. If any one goes slow (not down — slow), worker threads block and the whole app degrades, even for cached requests.
• The Anthropic client currently has no explicit timeout (SDK default ~10 min) — a hang can pin a thread for minutes. This is the prerequisite to fix: a breaker is useless without timeouts.

Design

Stability patterns compose, innermost → outermost: timeout → (retry) → circuit breaker → fallback. We implement timeout + breaker + fallback now; retry/bulkhead and the chatbot fallback-chain are future work.

The breaker is a small, dependency-free, thread-safe module (State machine inside, Proxy/Decorator outside). Faking is done at the network boundary (a local fake server the app talks to), not in-process — so the real timeout/exception code path is exercised and the breaker is genuinely validated.

Locust (fake users) ──> PaidUp ──> fake_upstream (localhost)
                                     knobs: latency_ms, fail_rate
                          │
                   CircuitBreaker round the Claude call
                          │
                   open? -> graceful 503 fallback

Steps

• 1. Circuit breaker module — src/app/circuit_breaker.py (CLOSED/OPEN/HALF_OPEN, thread-safe, configurable failure_threshold / recovery_timeout / expected_exception)
• 2. Timeout + wire into ai.py — add explicit timeout to the Anthropic client; route the analyze() Claude call through a shared _anthropic_breaker
• 3. Graceful fallback in main.py — catch CircuitOpenError in /analyze, return a clean 503 ("high demand, try again shortly")
• 4. Fault-injection unit tests — tests/test_circuit_breaker.py: closed→open after N failures, fail-fast while open, half-open trial, success resets. (This is the cheap, deterministic "crash test".)
• 5. Env-configurable base URLs — parliament.py, theyworkforyou.py, ai.py read base URLs from env with current values as defaults (production unchanged)
• 6. Fake upstream server — loadtest/fake_upstream.py: mimics Members/Interests/TWFY/Anthropic endpoints with canned responses + latency_ms and fail_rate knobs
• 7. Locust file — loadtest/locustfile.py: drives /lookup (heavy) and /analyze (light), all against the fake
• 8. Run + tune — load test against fakes, watch latency/error rate, tune breaker thresholds; dial up fail_rate to confirm the breaker trips and recovers

Out of scope (future)

• Retry with exponential backoff + jitter
• Bulkhead (separate thread pools for /analyze vs /lookup)
• Chatbot fallback chain for the future /ask section: Sonnet → Haiku → cached → message, with error-type-aware handling (429 backoff vs 500/timeout trip) and streaming-aware failure detection

Branch

feat/circuit-breaker

Testing

• Correctness: fault-injection unit tests (step 4) — no network, deterministic
• Effectiveness: Locust against fake_upstream (step 8) — no cost, no real API load

[mahsa7haft]

Progress — steps 1–4 done (the circuit breaker itself + validation)

Branch feat/circuit-breaker:

• ✅ 1. Breaker module — src/app/circuit_breaker.py (38e3d60)
• ✅ 2. Timeout + wired into ai.py — 60s Anthropic timeout, analyze() routed through shared _anthropic_breaker; trips on connection/timeout + 5xx only (57f97e3)
• ✅ 3. Graceful 503 — /analyze catches CircuitOpenError (27d27c2)
• ✅ 4. Fault-injection tests — 11 tests, fake clock, prove open/fail-fast/recover (cef17d4)

Full suite: 76 passed (was 65). The breaker is now correct and proven.

Remaining (the fake-upstream load-test harness, no real cost/traffic):

• 5. Env-configurable base URLs
• 6. Fake upstream server (latency/fail knobs)
• 7. Locust file
• 8. Run + tune