SessionStart injects ~2.8k tokens once, but every API call re-reads them — the real cost is a per-turn slope, not a one-time write (measured A/B)

[RoxsLee]

Summary

The cost of the injected ruleset is usually discussed as a one-time write (#219 was closed on a ~340-token one-shot estimate; #121 observed "ON → more cost" without attribution). We ran a controlled A/B plus a multi-turn mechanism study and found the dominant term is different: the injection is resident in context, and every API call within every turn re-reads it. An agentic turn averages ~6 API calls, so a 5-turn session re-reads the ruleset ~30 times. That turns a one-time ~$0.016 write into a per-turn drag that never amortizes — the gap vs. a one-sentence prompt grows with session length instead of shrinking.

Measured headline: +6.9% cost [95% CI +3.9%, +10.0%], p < 1e-4 vs. a fair one-sentence opponent, and a per-turn marginal cost that stays +8–13% higher across 5-turn sessions with no break-even point.

Setup

• Claude Code 2.1.220, Sonnet 5, headless; plugin pinned at 14a0d79 (main as of 2026-07-10 — code-identical to current main, whose only later commit, 16f2980, touches README docs only, so all line references below hold at HEAD).
• Arm A (ponytail): plugin installed, default full mode.
• Arm B (fair opponent): no plugin; the base prompt plus one sentence asking to leave a runnable check behind — i.e. what a user would type to get ponytail's most valuable behavior (tests delivered) without a plugin.
• Single-shot: 3 arms (ponytail / one-sentence opponent / bare baseline) × 5 tasks × n=8, same batch, cluster bootstrap (10,000×) + stratified permutation for CI/p.
• Multi-turn: 5-turn sessions, two carriers (single-process stream-json multi-turn, and --resume chains), n=4 per cell.
• All costs are nominal at public list pricing computed from per-call usage (four categories split), not billing exports.

What we measured

1. Injection semantics: exactly once per session; resume does not re-inject

The SessionStart matcher includes resume (hooks/claude-codex-hooks.json:5), but in practice we observed exactly one hook_response event per session — --resume continuation turns do not re-inject (per-turn cache-write deltas vs. no-plugin were ≈ 0 for turns 1–4). So this is not a repeated-injection bug report.

The one injection is large, though: the observed SessionStart hook_response event was 11,158 characters ≈ 2.8k tokens (ruleset via getPonytailInstructions at hooks/ponytail-activate.js:42, which reads the full SKILL.md, hooks/ponytail-instructions.js:86-88, plus hook framing). Three independent estimates agree: first-turn cache-write delta vs. no-plugin was +2,752 tokens (carrier A, median after removing cross-session prefix-cache reuse), +2,942 (carrier B), and +2,465 in the single-shot batch.

2. Single-shot A/B: +6.9% vs. the one-sentence opponent

Comparison	Δ cost	95% CI	Verdict
ponytail vs. bare baseline	+0.7%	[−6.5%, +9.8%]	cost-neutral
one-sentence opponent vs. bare baseline	−5.8%	[−11.7%, +1.1%]	slightly cheaper (n.s.)
ponytail vs. one-sentence opponent	+6.9%	[+3.9%, +10.0%]	significantly more expensive (p < 1e-4)

To be fair to the plugin: it also delivered −20% total LOC and −12% output tokens, and it was the only arm whose safety-oracle pass rate never dropped. The point is not "ponytail is bad" — it's that its cost story is currently mispriced, and the price is mostly avoidable (see fix below).

Per-cell token breakdown shows where the +6.9% lives: cache writes +23% (≈ one injection; write price is 2× base input under 1h caching, $6/M vs. $3/M) — and the resident re-reads below.

3. Multi-turn: the intercept amortizes, the slope does not

Per-turn accounting over 5-turn sessions:

• A turn averages ~6 API calls (6.2 with plugin, 5.8 opponent), and each call re-reads the entire context, resident ruleset included. The plugin arm's average per-call context was ~3.1k tokens thicker — the ruleset's size — which works out to ~17k extra cache-read tokens per turn paid for the same instructions, every turn, forever.
• Marginal per-turn cost (turns 1–4, injection already sunk): +13% (carrier A: $0.0843 vs. $0.0747) and +8% (carrier B: $0.0864 vs. $0.0798) vs. the one-sentence opponent.
• Cumulative gap over k turns: gap(k) ≈ $0.016 (intercept) + k × $0.010 (slope). There is no break-even point: amortizing the intercept would require a lower marginal cost per turn, and the measured marginal cost is higher. Over 5 turns the gap was +14.2% (carrier A, p=0.114) / +7.0% (carrier B, p=0.343) — n=4 has no power by design, but the direction matches the significant single-shot +6.9%.
• Why cache reads matter despite being the cheapest category ($0.30/M): they are the largest line item — 730k–880k tokens per 5-turn session, roughly half the bill. "Reads are cheap" means cached content beats resending it 10×; it does not make resident bytes free. The true cost of injected text is size × calls-per-turn × turns × read price, not the write.

This reframes #219's closing estimate: ~340 tokens written once is indeed not worth optimizing — but the write is only the intercept. The term that estimate leaves out is the × per-call re-read slope, and that term dominates for anything resident.

Relation to existing issues

• SDK benchmark: Ponytail ON → more tool calls & cost, fewer output lines; skillCount ≠ skills read #121 (closed): third-party SDK benchmark saw the same surface pattern (ON → more calls, more cost) across models but didn't attribute it. This measurement supplies the mechanism and a controlled opponent.
• Reduce context overhead: route through the ponytail conductor instead of N always-loaded skills #219 (closed): same cost category (resident context), estimated as a one-time write; see above.
• SubagentStart always sends the full SKILL.md — reuse the shipped condensed fallback for subagent workers #597 (open): SubagentStart sends the full SKILL.md and proposes reusing getFallbackInstructions. This issue is the SessionStart-side complement — same fix, different hook.
• lite / full / ultra are 96 of 99 lines identical — the level changes a stance sentence, not a rule #664 (open): lite/full/ultra are 96/99 lines identical, so switching levels can't slim the injection today.

Proposed fixes

1. Reuse the shipped condensed instructions for SessionStart too. hooks/ponytail-activate.js:42 calls getPonytailInstructions(mode) (full SKILL.md); the plugin already exports getFallbackInstructions(mode) (hooks/ponytail-instructions.js:43-75, ~670 tokens per SubagentStart always sends the full SKILL.md — reuse the shipped condensed fallback for subagent workers #597's measurement), currently reachable only when the SKILL.md read throws (hooks/ponytail-instructions.js:89-90). Making the condensed set the default (or a config option) is the same one-line shape as SubagentStart always sends the full SKILL.md — reuse the shipped condensed fallback for subagent workers #597's option 1, and it shrinks the slope, not just the intercept: resident size drops from ~2.8k to a few hundred tokens, and the ~17k/turn re-read tax scales down proportionally. The condensed text already carries the ladder, the rules, the output contract, and the "when NOT to be lazy" rails — the parts a model needs every call.
2. Make lite genuinely lighter (lite / full / ultra are 96 of 99 lines identical — the level changes a stance sentence, not a rule #664): if the level gated whole sections instead of one stance sentence, lite would double as the low-overhead choice for cost-sensitive setups.

Scope and honest caveats

• All figures are nominal cost at API list prices (Sonnet 5, 1h caching: write $6/M, read $0.30/M). Subscription metering formulas are not public; the monotonic argument (plugin arm consumes strictly more in the largest categories) transfers, exact percentages don't.
• Workload was headless micro-task chains (~$0.08–0.10 per turn). On tasks 10× the per-turn cost the relative intercept shrinks (~2% of a turn), but the slope argument — resident size × per-call re-reads — is workload-independent.
• Single model (Sonnet 5), single client version (Claude Code 2.1.220), plugin at 14a0d79. Interactive sessions, subagent-heavy sessions (SubagentStart always sends the full SKILL.md — reuse the shipped condensed fallback for subagent workers #597's territory), 5-minute-TTL accounts, and auto-compact re-injection were out of scope.
• The numbers above are self-contained: every figure derives from per-call usage four-way splits (input / cache write / cache read / output) priced at public list rates, with the estimator described where it is used (medians for the injection delta after excluding two cross-session prefix-cache-reuse outliers; cluster bootstrap and stratified permutation for the single-shot CI/p).