Problem
The system prompt + tool definitions are large and static, but they are re-processed on every one of the several sequential round-trips per question. When the provider is configured as "OpenAI Compatible" against https://api.anthropic.com/v1/, Anthropic prompt caching is typically not passed through. This drives both latency (time-to-first-token) and cost.
Proposal
Send the static prefix (system prompt + tool definitions, and any always-loaded context) with cache_control: {type: "ephemeral"} breakpoints — either via the native Anthropic Messages API, or by extending the OpenAI-compatible path to set Anthropic cache headers.
Benefit
- Markedly lower time-to-first-token on follow-up round-trips within the 5-minute cache window.
- ~90% cheaper cached input tokens.
Context
Feedback from production use with ERPNext v15 + Aakvatech PropMS, where a single question can trigger 3–5 sequential model round-trips, each re-processing the full static prompt.
Problem
The system prompt + tool definitions are large and static, but they are re-processed on every one of the several sequential round-trips per question. When the provider is configured as "OpenAI Compatible" against
https://api.anthropic.com/v1/, Anthropic prompt caching is typically not passed through. This drives both latency (time-to-first-token) and cost.Proposal
Send the static prefix (system prompt + tool definitions, and any always-loaded context) with
cache_control: {type: "ephemeral"}breakpoints — either via the native Anthropic Messages API, or by extending the OpenAI-compatible path to set Anthropic cache headers.Benefit
Context
Feedback from production use with ERPNext v15 + Aakvatech PropMS, where a single question can trigger 3–5 sequential model round-trips, each re-processing the full static prompt.