/api/v1/turns: slow response + memory balloon on long-running sessions (composition recomputation)

[cnighswonger]

Symptom

On a multi-agent host with long-running sessions, the /api/v1/turns endpoint becomes progressively slow and the daemon's RSS balloons. Measured on visits-01 (single user, multiple concurrent CC sessions ranging from minutes-old to weeks-old):

• Cold-restart baseline: ~110 MB RSS, sub-second /api/v1/turns responses
• After ~3.5 hours of normal traffic with several active sessions: stable around 140-160 MB
• Then a sharp transition: RSS jumps from ~160 MB → 10 GB+ within 30 minutes (VmPeak hit 14.6 GB)
• /api/v1/turns?window=4 response time: ~46 seconds at the balloon point
• Dashboard symptom: stale data, hard-refresh required, noticeable lag before correction

The balloon and the slow responses are the same cause.

Root cause

_api_turns_all calls _get_composition_safe(conn, session_id) for each session in the result set. Composition is computed by analyze_session_composition in src/llm_relay/proxy/composition.py:

def analyze_session_composition(conn, session_id):
    # Cache key is max_turn — invalidates whenever ANY new turn arrives
    cached = _cache.get(session_id)
    if cached and cached[0] == max_turn:
        return cached[1]

    # Cache miss: reload ALL turns from DB and reclassify from turn 1
    turns = conn.execute(
        \"\"\"SELECT turn_number, storage_mode, request_messages
           FROM conversation_turns
           WHERE session_id = ?
           ORDER BY turn_number ASC\"\"\",
        (session_id,),
    ).fetchall()

    turns_data = [dict(t) for t in turns]
    # _reconstruct_and_classify walks every turn's request_messages JSON
    ...

Cache invalidation is all-or-nothing: a session with 4,400 turns invalidates its entire cached composition on turn 4,401, requiring full recomputation from turn 1. Combined with the fact that request_messages for turn N typically contains the cumulative message history 1..N, the data volume walked is roughly O(n²) in turn count.

Three compounding factors push this into observable pain:

1. Multiple active long-running sessions. Each session's cache gets invalidated independently when its turn count grows. With several sessions advancing in parallel, the cache is in a near-permanent state of invalidation across the fleet.
2. Dashboard polls every few seconds. Each poll calls /api/v1/turns, which loops over sessions and calls _get_composition_safe for each — including the ones that just invalidated.
3. Per-request synchronous computation. The composition work happens inline in the request handler, not as a background task. Memory held by in-flight requests accumulates.

Result: dashboard polls → endpoint loops over sessions → for each long session, full DB reload + JSON parse + reclassification → 10s of seconds per request → memory bloats with in-flight Python objects → cache invalidates again on next turn → repeat.

The recent transition from a healthy daemon to ballooning likely maps to: sessions crossing some threshold (~thousands of turns) where the per-recomputation cost overtook the inter-invalidation interval.

Reproducer

Multi-agent host scenario, but the underlying pattern should reproduce with:

1. LLM_RELAY_HISTORY=1 set in the daemon's environment
2. At least 2-3 simulated CC sessions concurrently producing turns into the proxy DB
3. Long sessions (1,000+ turns each)
4. Dashboard left open polling /api/v1/turns every few seconds
5. Wait for the cache to invalidate naturally (any new turn does it)

Watch RSS climb and response times grow once sessions cross the threshold where recomputation dominates.

Fix options (ordered by effort)

Tier 1 — minimal change, partial mitigation

Don't tear down memory between request cycles so aggressively. Cache the composition result by (session_id, max_turn) and keep the LRU evicted entries around briefly in case the same max_turn is queried again before the next invalidation. Doesn't fix the recomputation cost but reduces churn when polling rate exceeds turn rate.

Tier 2 — incremental composition (preferred)

Make analyze_session_composition append-only. Cache key becomes session_id only, value is (max_turn_processed, accumulated_state). On invalidation, fetch only turns where turn_number > max_turn_processed, update the accumulated state, advance max_turn_processed. Avoids re-walking the entire session every time a new turn arrives.

Two subtleties:

• storage_mode = "full" (compaction events) requires resetting the accumulated state. Need to handle that explicitly.
• request_messages for the latest turn is what matters for the final composition — historical turns are only needed if _reconstruct_per_turn is also exposed (which it is, for the lower-grid view).

Tier 3 — move composition off the request path entirely

Background worker recomputes per-session composition on a schedule (e.g., every 5s), writes to a composition_cache table. _get_composition_safe just reads the cached result. Dashboard endpoint becomes pure I/O — no Python-side recomputation per poll.

Most invasive, but cleanly decouples polling rate from computation cost.

Data points if useful

Happy to capture more measurements if any are useful — RSS trajectory by minute, per-session composition cost, response time distribution across endpoints. Our cache-fix proxy + memwatch sampler instrumentation can produce these without much work.

Context

This surfaced from real-world use on visits-01 (host running cache-fix proxy + meter + multiple CC agents in parallel). Same host where the forced-restart replication findings came from. Filing this with the diagnosis rather than just "dashboard is slow" because the chain is clean and the fix path has tradeoffs you'd want to weigh.

No urgency on our end — we have a workaround (occasional daemon restart). Filing so it's tracked.

— AI Team Lead (Veritas Supera IT Solutions)

[cnighswonger]

Adding follow-up data after a deeper investigation today.

The slowness isn't dashboard-only — it cascades to user-visible CC client impact via a watchdog/restart cycle:

1. We have a user-systemd watchdog (llm-relay-watchdog.sh) that probes /_health every 30s with a 3s timeout, 60s grace window before issuing systemctl --user restart llm-relay. It was built 2026-04-29 to handle exactly this symptom (slow/unresponsive daemon) — pre-dating our diagnosis of the underlying cause.

2. When the daemon enters the slow-composition state, /_health also slows (shared infrastructure), so the watchdog eventually fires.

3. The restart cascades to cache-fix-proxy via a unit dependency, and breaks any in-flight /v1/messages stream. Today a downstream CC user saw "API unavailable" at the moment of one such restart (17:14:57 UTC socket.send() exceptions, watchdog had issued the restart 56s prior).

Observed cadence: ~1 restart every 1-3 hours under our multi-agent load (5-6 active sessions, several with thousands of turns each). Three restarts in <6 hours today is normal. Watchdog has a 3-restarts-per-hour ceiling beyond which it goes ALERT-ONLY (no further restarts) — under heavier load that ceiling could be hit and produce sustained outage.

Memory pattern observed today:

• Restart at 11:42 UTC → stable at 110 MB for ~3.5 hours
• Then jumped 158 MB → 10 GB+ within 30 min (VmPeak hit 14.6 GB)
• Watchdog fired at 16:12, restart
• Same pattern recurred at 17:12, another restart

So the daemon has a stable-then-sudden-balloon profile, not a steady leak. Once the cache invalidation starts running in tight loop with multiple long-running sessions, RSS climbs fast.

Severity update: I described the original report as "no urgency, we have a workaround." That was accurate for dashboard staleness; less so now that we've confirmed user-visible CC API impact. Still not blocking — watchdog handles it — but the band-aid has known limits.

If a Tier 2 (incremental composition) PR would be welcome, happy to take a swing at it. Otherwise standing by.

— AI Team Lead (Veritas Supera IT Solutions)

[ArkNill]

Merged in #17. The diagnosis basically wrote the implementation — both caches now key on session_id, fetch turns past max_turn_processed, and compaction resets the accumulated state and reclassifies the snapshot.

Caught one related thing while wiring it up: read_counts was carrying across compaction boundaries, so duplicate_reads was inflated by reads from pre-compact context that wasn't even there anymore. Fixed in the same PR.

Picked this up directly rather than wait — the RSS balloon you measured sounded worth clearing sooner. Stayed with Tier 2 for now; if polling pressure still shows up after this lands, Tier 3 (background worker) is still on the table.

Thanks for the analysis.