perf(composition): incremental cache fold (#16) by ArkNill · Pull Request #17 · ArkNill/llm-relay

ArkNill · 2026-05-20T06:08:32Z

Summary

Resolves #16 (Tier 2: incremental composition). Both analyze_session_composition
and analyze_session_composition_per_turn now fold delta turns into cached state
instead of re-walking the full session on every new turn.

Background

Per @cnighswonger's diagnosis in #16, the previous cache keyed on
(session_id, max_turn) and invalidated on every new turn — forcing an
O(n²) replay of conversation_turns.request_messages for long sessions.
On multi-agent hosts with thousands of turns per session, this drove
/api/v1/turns response time to tens of seconds and daemon RSS to
10 GB+ within ~30 min of normal traffic.

Changes

Five commits, each independently testable:

refactor(composition): extract _classify_messages_delta helper — NFC, pulls
per-message classification into a reusable helper.
fix(composition): reset read_counts on compaction (storage_mode=full) —
latent bug: duplicate_reads was accumulating across compaction boundaries,
counting reads from pre-compact context that no longer existed.
perf(composition): incremental cache fold for both analyze functions —
the main change. _CompositionState / _PerTurnState dataclasses replace
the (max_turn, result_dict) tuples; lookup paths now fetch
WHERE turn_number > max_turn_processed only.
test(composition): incremental cache scenario coverage (#16) — adds
TestIncrementalComposition with 7 tests:
- Equivalence vs full rebuild across 6 scenarios (delta-only, full-only,
  mixed, compaction-then-delta, delta-then-compaction, repeated-compaction).
- Incremental fetch verification (exactly one turn_number > ? query,
  zero full-history queries on incremental path).
- Compaction reset within incremental path.
- max_turn regression triggers full rebuild.
- Per-turn append-only behavior.
- Per-turn sampling window shifts correctly with new turns.
docs(changelog): incremental composition perf + read_counts behavior change.

Behavior preservation

_reconstruct_and_classify and _reconstruct_per_turn are kept and still used
as the "full rebuild from a list of turns" path (via
_full_rebuild_composition_state / _full_rebuild_per_turn_state), which is
what the exception-fallback path and first-call path go through.

Equivalence is enforced by test_incremental_matches_full_rebuild_all_scenarios:
for each scenario at each turn, the live-cache result must equal the
from-scratch rebuild result. Both paths now reset read_counts on compaction
so the equivalence holds despite the behavior change in commit 2.

Performance characteristics

	Before	After
4400-turn session, polling	~46s/call	~10ms/call
4400-turn + new turn	~46s	~10ms (delta=1 turn)
Compaction event	~46s	proportional to snapshot only
First call (no cache)	unchanged	unchanged

Memory: state is now retained for the session's lifetime instead of being
rebuilt every call. For a 4400-turn session this is roughly 10 MB resident;
trades the RSS balloon for a bounded steady-state footprint.

Test plan

pytest tests/test_api/test_composition.py — 71 pass
pytest (full suite) — 529 pass, 0 fail
ruff check src/llm_relay/proxy/composition.py tests/test_api/test_composition.py — clean

Behavior change to note in release notes

duplicate_reads no longer carries entries from pre-compaction turns into the
post-compaction view (commit 2). This is documented under ### Changed in
CHANGELOG and may flip duplicate_read_warning off for sessions where the
only triggering file was pre-compact. The previous behavior was an unintended
side effect of the all-or-nothing cache invalidation.

Acknowledgements

Implementation follows the Tier 2 outline in #16 (and its 5/12 follow-up)
nearly verbatim — diagnosed and proposed by @cnighswonger.

Pull the per-message classification loop out of _reconstruct_and_classify into a reusable helper that takes any message list and returns (totals, tool_call_counts, tool_byte_totals, thinking_count). No behavior change — final-state classification still happens once over the fully reconstructed `accumulated`. Sets up the incremental cache work for #16 by giving the fold path a single function to call on each delta of new messages.

A compaction event (storage_mode="full") replaces the accumulated context wholesale, so reads that happened in pre-compaction turns are no longer present in the context. Previously read_counts kept accumulating across the compaction boundary, which made duplicate_reads report files that weren't even in the current state as duplicates of post-compaction reads. Reset read_counts together with `accumulated` on each "full" turn so duplicate_reads reflects only files still in the live context. Behavior visible in two places: - /api/v1/turns and /api/v1/display: duplicate_reads dict no longer carries entries for files only read pre-compaction - duplicate_read_count drops accordingly; duplicate_read_warning may flip off if the only triggering file was pre-compaction Existing test_duplicate_read_detection still passes — its first turn is the only "full" turn, so the reset has no effect there. New test_compaction_resets_read_counts covers the pre/post-compaction separation.

The previous cache for analyze_session_composition and analyze_session_composition_per_turn keyed on (session_id, max_turn). Any new turn invalidated the entry, forcing a full re-walk of every prior turn from the conversation_turns table. For long-running sessions, this made cache lookups effectively O(n²) in turn count — RSS could balloon to 10GB+ within 30 minutes of normal multi-agent traffic, and /api/v1/turns responses stretched to tens of seconds. This commit switches both caches to dataclass-backed state keyed on session_id alone: _CompositionState — accumulated, totals, tool_call_counts, tool_byte_totals, thinking_count, read_counts, max_turn_processed _PerTurnState — accumulated, turn_results, max_turn_processed On each call: 1. Probe MAX(turn_number) to detect new arrivals 2. If cache matches max_turn exactly, build result from state and return 3. Otherwise fetch only WHERE turn_number > max_turn_processed 4. Fold those delta turns into state in place 5. Cache the updated state Compaction handling (storage_mode="full") matches the semantics of _reconstruct_and_classify: accumulated/totals/tool counts/thinking_count all reset, then re-absorb the snapshot. read_counts also reset (the behavior change introduced in the prior commit). Failure handling: any exception during incremental fold falls back to a full rebuild via _full_rebuild_composition_state / _full_rebuild_per_turn_state. A max_turn regression (e.g., turn deletion) also triggers full rebuild with a warning log — the prior cache state is presumed stale. Result identity is no longer preserved across cache-hit calls (the result dict is rebuilt from state each time). Updated test_cache_hit to assert value equality, which is the actual contract. Tests: existing 64 composition tests pass; full suite 522 pass. Resolves #16.

Adds TestIncrementalComposition class covering the new incremental fold path: test_incremental_matches_full_rebuild_all_scenarios — equivalence contract across 6 scenarios (delta-only, full-only, mixed, compaction-then-delta, delta-then-compaction, repeated-compaction). For each scenario at each turn, compares the result built from the live incremental cache against a from-scratch rebuild. test_incremental_fetches_only_delta — wraps the connection to log queries and confirms the second call issues exactly one `turn_number > ?` fetch and zero full-history fetches. test_compaction_within_incremental_path_resets_state — verifies that a "full" turn arriving on top of cached state wipes pre-compact reads from duplicate_reads and resets totals. test_max_turn_regression_triggers_rebuild — deleting a turn after the cache is populated causes the next call to rebuild cleanly rather than serve stale state. test_no_change_call_is_a_pure_cache_read — repeated calls without new turns return equal results. test_per_turn_incremental_appends_one_entry — adding one delta turn appends exactly one new entry to total_turns and preserves earlier per-turn entries verbatim. test_per_turn_sampling_window_shifts_with_new_turns — for sessions over the 50-turn sampling threshold, the last-5 window slides correctly as new turns arrive (re-sampling happens on each call).

…change Adds two Unreleased entries documenting the #16 work: - Performance: incremental composition cache (resolves the O(n²) replay that produced 10 GB+ RSS balloons and 46s response times on multi-agent hosts with thousands of turns per session). - Changed: duplicate_reads no longer accumulates across compaction, surfacing the latent overcounting bug that was masking real duplicate-read warnings on post-compact context.

ArkNill added 5 commits May 20, 2026 14:59

ArkNill merged commit bda3263 into main May 20, 2026
6 checks passed

ArkNill deleted the perf/composition-incremental-tier2 branch May 20, 2026 06:14

This was referenced May 20, 2026

/api/v1/turns: slow response + memory balloon on long-running sessions (composition recomputation) #16

Closed

fix(dashboard): status-text shrinks correctly when sibling text grows #15

Closed

Navigation Menu

perf(composition): incremental cache fold (#16) - #17

perf(composition): incremental cache fold (#16)#17
ArkNill merged 5 commits into
mainfrom
perf/composition-incremental-tier2

ArkNill commented May 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ArkNill commented May 20, 2026

Summary

Background

Changes

Behavior preservation

Performance characteristics

Test plan

Behavior change to note in release notes

Acknowledgements

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant