Skip to content

feat: model OpenAI prompt cache fields for transparent forwarding - #2468

Open
Aias00 wants to merge 1 commit into
envoyproxy:mainfrom
Aias00:feat/openai-prompt-cache-breakpoint
Open

feat: model OpenAI prompt cache fields for transparent forwarding#2468
Aias00 wants to merge 1 commit into
envoyproxy:mainfrom
Aias00:feat/openai-prompt-cache-breakpoint

Conversation

@Aias00

@Aias00 Aias00 commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Description

OpenAI GPT-5.6 and later support explicit prompt cache breakpoints and related request-level fields for marking stable prompt prefixes for reuse. The OpenAI to OpenAI translator already forwards request and response bodies opaquely (raw bytes, only sjson-patching the model), so these fields already reach the upstream and client unchanged. This change promotes them to first-class fields in the OpenAI type model so the gateway recognizes them when parsing request/response bodies (and preserves them through the redaction/debug-log re-marshal path) instead of relying on opaque passthrough alone.

Modeled fields:

  • request-level prompt_cache_key and prompt_cache_options (with mode) on ChatCompletionRequest
  • prompt_cache_breakpoint (with mode) on ChatCompletionContentPartTextParam; the content union marshaler delegates to the concrete struct marshal, so the field round-trips
  • prompt_tokens_details.cache_write_tokens on PromptTokensDetails (cached_tokens was already modeled)

The response body remains opaque passthrough, so cache_write_tokens reaches the client unchanged regardless; modeling it makes it available when the response is parsed. New metrics for cache_write_tokens are intentionally out of scope here (cache-write vs cache-creation semantics warrant a separate maintainer decision); the field is modeled but not yet emitted as a metric.

Tests cover a Chat Completions request with an explicit breakpoint on a text content block (field capture plus re-marshal preservation), byte-for-byte opaque request forwarding, and cache_write_tokens preservation/parsing in the response.

Related Issues/PRs (if applicable)

Refs #2454

Special notes for reviewers (if applicable)

  • No behavior change on the data path; opaque passthrough already forwarded these fields. The change is additive type modeling plus regression tests.
  • Local golangci-lint reports two pre-existing SA5008 json:"type," warnings on ThinkingDisabled/ThinkingAdaptive (present on main, not introduced here).
  • Metrics wiring for cache_write_tokens is deliberately deferred (see Description).

OpenAI GPT-5.6 and later support explicit prompt cache breakpoints and
related request-level fields. The OpenAI to OpenAI translator already
forwards the request and response bodies opaquely (raw bytes, only sjson
patching the model), so these fields already reach upstream and client
unchanged. This change promotes them to first-class fields in the OpenAI
type model and adds regression coverage so they are recognized when the
request/response is parsed (and preserved through the redaction/debug-log
re-marshal path) rather than relying on opaque passthrough alone.

Modeled fields:
- request-level prompt_cache_key and prompt_cache_options (mode) on
  ChatCompletionRequest
- prompt_cache_breakpoint (mode) on ChatCompletionContentPartTextParam
- prompt_tokens_details.cache_write_tokens on PromptTokensDetails
  (cached_tokens was already modeled)

Tests cover a Chat Completions request with an explicit breakpoint on a
text content block (field capture + re-marshal preservation), opaque
request forwarding byte-for-byte, and cache_write_tokens preservation in
the response.

Refs envoyproxy#2454

Signed-off-by: liuhy <liuhongyu@apache.org>
@Aias00
Aias00 requested a review from a team as a code owner August 1, 2026 08:10
Copilot AI review requested due to automatic review settings August 1, 2026 08:10
@dosubot dosubot Bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Aug 1, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.04%. Comparing base (b80bc3e) to head (23258f5).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2468      +/-   ##
==========================================
+ Coverage   85.03%   85.04%   +0.01%     
==========================================
  Files         159      159              
  Lines       22634    22634              
==========================================
+ Hits        19247    19250       +3     
+ Misses       2219     2217       -2     
+ Partials     1168     1167       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR promotes OpenAI prompt-caching request/response fields to first-class schema fields so they survive decode/re-marshal paths (e.g., redaction/debug logging) instead of relying solely on opaque byte passthrough in the OpenAI→OpenAI translator.

Changes:

  • Adds request modeling for prompt_cache_key, prompt_cache_options.mode, and prompt_cache_breakpoint.mode (on text content parts).
  • Adds response modeling for usage.prompt_tokens_details.cache_write_tokens.
  • Adds regression tests validating round-trip preservation and opaque forwarding behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
internal/translator/openai_openai_test.go Adds translator-level tests ensuring prompt-cache fields are forwarded byte-for-byte and cache_write_tokens is parsed/preserved.
internal/apischema/openai/vendor_fields_test.go Adds schema-level test verifying prompt-cache fields survive unmarshal and re-marshal.
internal/apischema/openai/openai.go Extends OpenAI schema types with prompt-cache request fields and cache_write_tokens in token usage details.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants