perf: TTFT roadmap after control-plane work — placement, prompt caching, container wake

[Yevanchen]

Context

The 2026-07-20/21 TTFT effort merged seven control-plane PRs (#369, #379–#384). All of them are provider-agnostic: they act before the model call or inside the Sandbox layer (accept path, hydration, dispatch CAS, filesystem RPC concurrency, telemetry deferral, read dedup). After that work, a 32-pair crossover on the two dedicated perf stacks (start 77eb6a84 vs final ba47018b, warm-ping instrument, 64/64 runs correct) shows:

• End-to-end TTFT paired median +2.28s with bootstrap 95% CI [−1.75s, +6.95s] — crosses zero; no causal end-to-end claim. The two crossover phases disagree in sign (−3.4s vs +8.8s), consistent with every earlier cohort: provider first-token drift (±10s) dominates.
• Per-cell TTFT medians (cattle cold bring-up): 11.9s (final @ stack-b) / 23.0s (final @ stack-a); start build: 12.4s / 24.5s. The same build differs by ~11s across two identically configured stacks.
• Control plane is no longer the bottleneck: hydration ~0.2s, real dispatch ~0.1s. Remaining waits: container wake (prepareFilesystem 2.4–4.8s), driver/CLI readiness, provider first token (one ping = 12,413 input tokens with cached_read=0).

Lab artifacts (perf workspace): warm-ping-ab-v26…v29-* JSON series; nightly ledger autoresearch-ledger-20260721.md.

This issue records the three next optimization directions with mechanics and cost estimates.

1. Placement engineering ("stop renting a bad stack")

Same config ≠ same physics. Measured decomposition of the ~11s stack gap:

• D1 round trips: stack-a consistently ~70ms/read slower (hydration stage delta ~0.69s, driver_turn delta ~40ms over 2 writes) → the two D1 instances were created in different locations.
• Container node quality: stack-a min 11.3s ≈ stack-b median → a scheduling-luck distribution problem, not a constant.
• Residual ~7–9s sits in the driver/provider window per stack (suspected slow-node knock-on; not yet isolated).

Plan:

Step	Action	Effort
Instrument	/api/health?deep=1: timed SELECT 1×3 (D1), DO echo, container_api_ready marker	~0.5 d
Attribute	Run probes on both stacks; split the 11s into D1 / node / provider buckets	~0.5 d
Fix	Recreate D1 with location hints (staging data disposable); DO locationHint; Worker Smart Placement; optionally container instance_type upgrade (unit cost ×2–4, gate on measurement)	config-level
Loop	Probe readings into observability; alert + recreate on bad placement	0.5–1 d

Total ~2–3 engineer-days plus optional instance-type run cost. Ceiling: bring every stack to today's stack-b level.

2. Provider-layer prompt caching (the only provider-specific work)

Evidence: a ping run pays full prefill for 12,413 input tokens every message (cached_read=0) while the prompt's bulk (system prompt, tool schemas, skills) is byte-identical within a session.

Runtime	Mechanism	Work	Estimate
openai	Automatic prefix caching (≥1024 tokens, byte-stable prefix)	cached=0 ⇒ the prefix is being broken. Audit prompt assembly in the driver/app-server for volatile content near the top (timestamps, run/session ids, unstable tool ordering); stabilize serialization, move volatile parts to the end	0.5–1 d audit + 0.5 d fix + 0.5 d A/B (driver repo)
anthropic (claude-agent-sdk)	Explicit cache_control (reads −90%, writes +25%)	Agent SDK likely already enables it — verify cached_read in benchmark usage rows first (unverified)	0.5 d verify, probably zero change
opencode / OpenAI-compatible	Same as openai shape	Rides along with the openai fix	—

Expected: cache hits typically remove 1–3s of provider TTFT at this prompt size (to be A/B'd), and cut input cost by ~50% (OpenAI) / ~90% (Anthropic reads) on the cached share. The only lever that reduces latency and the bill at the same time. Total ~2–3 engineer-days, concentrated in the driver repo.

3. Container wake cost (the cost-model front)

prepareFilesystem 2.4–4.8s is the first RPC waking the container (image boot + entrypoint). Cattle policy tears the world down after every terminal run by design (closeOnRunTerminal, idleReleaseDelayMs=0), so every cattle run pays it. Ladder, cheapest first:

Rung	Action	Cost	Expected
a. Pay less often	Pet already has a 5-min idle grace; lengthening it is a one-line policy value traded against container-hours. Cattle keeps teardown semantics where isolation demands it	config + residency cost	2nd+ message in a session pays zero wake
b. Make each payment cheaper	Image diet: 413MB compressed vs <150MB reference — multi-stage build, strip toolchains/caches; measure entrypoint first via container_api_ready	2–4 d (driver repo), no run cost	est. −30–50% wake time (to be measured)
c. Pay in advance	Warm pool: N pre-booted unassigned containers, bind on demand; moves the 2.4–4.8s to when nobody is waiting	~1–2 wk (pool mgmt, reclaim, leak guards) + N × instance residency	wake → sub-second; the decisive step for a stable sub-10s p95

Suggested order: b-measure → a → c (delete and simplify before automating spend).

Sequencing note

Directions 1 and 2 are ~3 engineer-days each with near-zero run cost; direction 3 climbs from free config to real standby spend. All three are additive to the merged provider-agnostic control-plane work.

[Yevanchen]

Progress update — all three directions ran one Auto Research iteration each (small-batch screens only, per plan; the 32-pair empirical run is reserved for after all three directions conclude).

Direction 1 — placement engineering

• Instrument merged: GET /api/health?deep=1 (feat(api): add deep health probe for worker-to-d1 rtt #389) returns three timed sequential D1 round trips.
• Attribution complete (direct measurement): via the SEA ingress colo, stage-a D1 RTT ≈ 52ms/read vs stage-b ≈ 13ms/read — the two identically configured D1 instances sit on opposite ends of the US. ~38ms/read × 15–20 round trips per run ≈ the 0.6–1s per-run stack gap seen in the crossover.
• Candidate: Smart Placement ([env.perf_a.placement] mode="smart", config-only): deployed, but after ~30 min and ~200 requests cf-placement stayed local-SEA — Smart Placement does not trigger at staging traffic levels, so staging cannot produce evidence for it. It remains a reasonable production candidate (production has real traffic), gated on the deep probe for before/after.
• Deterministic alternative (parked): recreate stage-a's D1 with a location hint (~1h ops including fixture re-setup). The known asymmetry is currently useful as a crossover covariate.

Direction 2 — provider prompt caching

• Anthropic (claude-agent-sdk): zero work needed. usage_event rows show cache_read 12,598 / input 12,600 (99.98% hit) and ~3.7M of 3.7M input tokens cached across 306 runs — the Agent SDK's automatic caching is already active. Provider first-token variance (±10s) is therefore not prefill-driven on this runtime.
• OpenAI runtime: root cause narrowed. The driver does not assemble the OpenAI prompt at all — the embedded app-server binary does, and our code only passes through its reported cachedInputTokens (app-server-event-mapping.ts); there are no cache knobs in app-server-env.ts. The cached_read=0 on a 12,413-token production ping is app-server-internal behavior. Next step requires an openai-runtime fixture (+key via the existing secret mechanism) or a production re-capture; parked as a conditioned item.

Direction 3 — container wake

• Decomposition from existing phase data (no new code): on an already-warm container the same RPC classes take 43–345ms, so ~95% of the 2.4–4.8s prepareFilesystem is container boot, not filesystem work. Image diet mainly helps pull/disk, not the boot floor.
• Mechanism test: instance_type basic → standard-1 (config-only, stage-b): cold boots measured 3,996/4,044ms — inside the basic range. Instance upgrade does not buy boot time; discarded and reverted. (Side finding: standard-1 × max_instances=1000 exceeds account memory quota; test ran with mi=10.)
• Accidental strong evidence for the warm-pool endgame: one probe ping hit a not-yet-recycled warm container and measured prepare_run 1,335ms (fs 345 + conversation 78 + driver ready 779) versus 4–6s cold. That is the measured prize of warm reuse: prepare 4–6s → ~1.3s.
• Conclusion for this direction: warm reuse (pool / lifecycle) is the only lever that moves the boot floor; node class does not, image size mostly does not.

Ledger state

• Kept & merged this round: feat(api): add deep health probe for worker-to-d1 rtt #389 (deep probe).
• Discarded with evidence: standard-1 upgrade.
• Parked with findings: Smart Placement (staging can't test it), stage-a D1 rebuild, openai app-server caching (needs fixture/key).
• The final 32-pair empirical comparison will run once all three directions conclude.

[Yevanchen]

Round 2 on direction 3 (container wake) — cattle idle grace merged (#390, main cabb7cfb).

• Mechanism: cattle idleReleaseDelayMs 0 → 90s; terminal release no longer recycles the per-session sandbox immediately, the existing kind-agnostic deadline sweep reclaims it after the session goes quiet. Session isolation unchanged (cattle subject scope is per-session by contract; instant recycle was resetting state mid-session).
• Evidence (4-pair same-stack screen, artifact warm-ping-ab-v30-cattle-grace-4pairs.json): TTFT 7,606/7,846/8,277/8,571ms — median 8.06s, inside the 5–10s target band, variance collapsed from ±3s to ±0.5s; prepare_run 3,511–3,809ms vs 4.3–9.3s historical; 4/4 pairs faster, 10/10 correct.
• Remaining gap to the measured warm floor (~1.3s): follow-up turns still pay ~2.2s of container re-wake — the sandbox naps between turns because the conversation session closes per terminal run. Naively flipping closeOnRunTerminal would leak containers: the inactive deadline is only armed on lease release when no active conversation remains (runtime-run-lease-store.ts). Next candidate (needs design, not a one-liner): decouple "conversation stays open" from "deadline armed" — arm the deadline unconditionally on lease release, or add a conversation-idle sweep — then keep the conversation open through the grace window. Expected prize: prepare 3.7s → ~1.3s, TTFT median ~8s → ~6s.

[Yevanchen]

Final certification (crossover, openai instrument, 29 valid pairs) — before a840973c (2026-07-21 starting point) vs after 6963ffba (current main, deployed to production today):

Metric	Before	After
TTFT median	11,415ms	6,395ms (−44.0%)
TTFT p95	15,728ms	9,868ms
Paired median	—	−6,271ms, bootstrap 95% CI [−7,554, −1,062] — entirely below zero
Pair wins	—	22/29 (phase1 16/16; after placement swap 6/13)
context_hydration / prepare_run / driver_turn medians	1,654 / 7,767 / 3,691ms	236 / 3,946 / 96ms

Against the lab's retention gate: paired improvement median 51.4% (≥20% required), after median 6.4s ≤10s (p95 also <10s), CI below zero — all three met. The gpt-5.5 first-token window (~0.4–1.5s) removes the ±10s provider noise that blocked every earlier end-to-end claim on the anthropic instrument.

Caveats: warm-ping public-API instrument (not the frozen golden harness — no per-run container identity forensics); 29/30 pairs (one pair lost to a client-side interruption); openai-runtime scope. Artifacts: warm-ping-openai-v32-cert-phase1-16pairs.json, warm-ping-openai-v32-cert-phase2-14pairs.log (7bea1c18…).

Also from today: production now carries #389/#390/#339; the production deep probe's first reading shows D1 RTT 47–62ms on try.mosoo.ai — production shares the far-D1 placement issue (direction 1 applies there); openai prompt-cache routing affinity filed as langgenius/mosoo-agent-driver#82.

[Yevanchen]

Direction 3 / V-C2 warm-floor follow-up is now submitted in #400.

• Keeps cattle conversations and their resident driver alive through a 5-minute idle grace, with reclamation handled by the conversation idle sweep.
• Paired staging gate: 32/32 pairs faster; TTFT batch medians moved from 7.9–11.8s on main to 3.1–3.5s, with warm prepare_run at ~1.0s.
• Prompt-cache affinity recovered, and the full conversation → subject deadline → sandbox reclamation path was verified end to end.
• Cost ceiling: up to 5 minutes of container residency after the last run.

This links the implementation and evidence to the roadmap; the issue remains open for roadmap-level closure.

[vr-varad]

hey @Yevanchen would love to pick this issue if its open for contribution

[Yevanchen]

Hi @vr-varad — yes, I think this issue is open for contributions. The key consideration is that validating performance improvements depends heavily on results from a real deployed environment: improvements measured locally do not necessarily translate into improvements in the cloud.

Therefore, any proposed change would need:

1. Validation in a staging environment.
2. At minimum, a before/after A/B test demonstrating a clear performance improvement.

I’d be very happy to discuss this with you. Do you already have a specific improvement or direction in mind?

[Yevanchen]

Production follow-up found why the #400 staging win did not hold for this user report.

The two reported no-tool runs (01KYNP1C4XG9KC16SX8RP2843K, 01KYNP5RKZW1NSYJ9G3WDBVWD9) both completed, but their Drivers failed 3–5 seconds later with terminal status could not be delivered. The API had already persisted the terminal command, then synchronously performed linked run/lease cleanup before returning the ACK. At production D1 latency (~41–62 ms/round trip), that path exceeds the Driver's 250 ms per-attempt deadline. The Driver retries and self-fails; the next turn therefore creates/resumes another runtime and loses #400's resident-session benefit.

A Chrome cold/warm reproduction on production measured 11.96 s / 10.90 s, but the two runs used different Driver IDs and both Drivers failed with the same terminal-delivery error. This also explains the staging gap: the performance gate measured TTFT/prepare improvements, but did not assert post-run Driver readiness or same-Driver reuse, and staging D1 was fast enough not to expose the ACK deadline coupling.

Fix submitted in #431: persist the command transition, return the ACK, and keep lease cleanup alive via Durable Object waitUntil. A regression test blocks cleanup and proves ACK completion is independent. API typecheck, 33 related runtime tests, and all 1,063 API tests pass. Production deployment/post-deploy API recertification remains after merge.

[vr-varad]

@Yevanchen I haven't started investigating this issue, so no approach as of now, but would love to discuss and share my insights.
Do u guys have any community for discussing?

[Yevanchen]

One more note on communication: our team primarily collaborates in an internal Lark group, but I’m not sure whether Lark is part of your usual workflow. Please feel free to continue the discussion here and tag @Yevanchen. Our team members, myself included, will keep an eye on this thread and respond promptly.