Context economy: prompt caching, reachable compaction, bounded outputs (0.6.0) - #88
Merged
Conversation
Coordinator runs re-send the whole growing conversation every turn; in a real 26-iteration loopy-loop program that produced ~28M prompt tokens against 185k completion tokens. Four levers, all knob-gated: - prompt_cache="auto": cache_control breakpoints (system + latest boundary) for Anthropic-family coordinator models, deterministic tool schema ordering for automatic OpenAI-side caching, and cached-token counts surfaced in usage records. - compact_above_tokens: an absolute-token compaction trigger that fires at user OR tool-result boundaries, so SDK runs can compact mid-run (the near-limit rule alone was unreachable: ~967k threshold vs ~96k peaks). The initial task is pinned verbatim across compactions and a trailing tool result is folded into the summary so the rebuilt history stays valid. - read_agent_output tail_bytes clamped to read_output_max_tail_bytes (16 KB) with a banner naming the full logs; read_new_agent_output cap now configurable; spawn footer asks workers for a <=15-line result card with paths instead of report dumps. - run.json persists tool results bounded to run_log_tool_result_max_bytes (8 KB) with a truncation note; the live message list is never touched. compact_above_tokens and prompt_cache are also TeamHarness ctor params so SDK callers (loopy-loop's signature-introspecting passthrough) can set them per run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EPYwF6cRYs2mfsfHF1hibc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Coordinator context growth is the dominant cost in long harness runs: the whole conversation is re-sent every turn, uncached, and compaction could never fire in SDK runs (threshold ≈ model limit − 33k vs ~96k peaks, user-boundary-only gate). Measured in a real loopy-loop program: ~28M prompt tokens vs 185k completion tokens over 26 iterations.
Changes (all knob-gated, defaults preserve current behavior except the read ceiling):
prompt_cache="auto": content-partcache_controlbreakpoints (system + latest message boundary) for Anthropic-family coordinator models; deterministic tool-schema ordering so OpenAI-family automatic caching gets a stable prefix; cached-token counts pass through intoTurnRecord.usage(additivecached_prompt_tokens).compact_above_tokens: absolute-token compaction trigger, firing at user or tool-result boundaries. Initial task pinned verbatim across (repeated) compactions; a trailing tool result is folded into the summary so the rebuilt message sequence stays valid.read_agent_outputtail clamped toread_output_max_tail_bytes(16 KB) with a banner naming the full logs;read_new_agent_outputcap configurable; the spawn footer now asks workers to end with a ≤15-line result card + file paths.run_log_tool_result_max_bytes(8 KB) with a note; live in-memory messages untouched; usage keys unchanged for downstream metering.compact_above_tokens/prompt_cacheareTeamHarnessctor params so SDK callers (loopy-loop's signature-introspecting passthrough) can set them per run.Design doc: loopy-loop
design/designs/context-and-eval-economy.md§A1–A4.Test plan
🤖 Generated with Claude Code
https://claude.ai/code/session_01EPYwF6cRYs2mfsfHF1hibc