Skip to content

Support cache_control breakpoints on system prompt and message content blocks #116

Description

@flechilla

Problem

Currently, LanguageModelRequest::builder().system() accepts a plain String, which gets converted to a single system message. There's no way to attach cache_control metadata to content blocks in the system prompt or messages.

This means providers that require explicit cache breakpoints (notably Anthropic via cache_control: {"type": "ephemeral"}) can't leverage prompt caching through aisdk, even though the SDK already reads cached token counts from responses (AnthropicCacheCreation).

Impact

For multi-turn tool-use agents, the system prompt + tool declarations are large (30-60K chars) and identical across turns within a conversation. Without cache_control breakpoints:

  • Anthropic models: 0% cache hit rate (confirmed via telemetry)
  • Google/Z-AI models: ~35-46% cache hit rate (automatic caching, no explicit markers needed)

Enabling Anthropic caching would reduce input token costs by ~90% on cache reads for the static prefix.

Proposed API

Allow passing structured content blocks with optional cache_control to .system() and within messages:

// Option A: Builder method for cache breakpoint on system prompt
let request = LanguageModelRequest::builder()
    .model(model)
    .system_with_cache("system prompt text", CacheControl::Ephemeral)
    .messages(messages)
    .build();

// Option B: Content block array with cache markers
let request = LanguageModelRequest::builder()
    .model(model)
    .system(vec![
        ContentBlock::text("static system prompt").with_cache(CacheControl::Ephemeral),
    ])
    .messages(messages)
    .build();

Provider Mapping

The cache_control should be mapped to each provider's native format:

Provider Format Min Tokens Notes
Anthropic cache_control: {"type": "ephemeral"} on content blocks 1,024 (Sonnet/Opus) Max 4 breakpoints
OpenAI Automatic (no explicit control needed) 1,024 Could be a no-op
Google cachedContent resource 32,768 Different mechanism
DeepSeek Supports cache_control (Anthropic-compatible) 1,024 Via OpenAI-compatible API

For providers without explicit caching support, the cache_control metadata can be silently ignored.

Current Workaround

None — the system prompt path (LanguageModelOptions.system: Option<String>) doesn't support structured content blocks, and the conversion in openai_chat_completions/conversions.rs always produces a single ChatMessage with a plain string content.

Context

We're using aisdk 0.5.2 via OpenRouter to power an AI assistant with 65 tools. After optimizing tool declarations (41% size reduction), caching the static prefix is the next biggest cost-saving opportunity.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions