feat: model OpenAI prompt cache fields for transparent forwarding - #2468
Open
Aias00 wants to merge 1 commit into
Open
feat: model OpenAI prompt cache fields for transparent forwarding#2468Aias00 wants to merge 1 commit into
Aias00 wants to merge 1 commit into
Conversation
OpenAI GPT-5.6 and later support explicit prompt cache breakpoints and related request-level fields. The OpenAI to OpenAI translator already forwards the request and response bodies opaquely (raw bytes, only sjson patching the model), so these fields already reach upstream and client unchanged. This change promotes them to first-class fields in the OpenAI type model and adds regression coverage so they are recognized when the request/response is parsed (and preserved through the redaction/debug-log re-marshal path) rather than relying on opaque passthrough alone. Modeled fields: - request-level prompt_cache_key and prompt_cache_options (mode) on ChatCompletionRequest - prompt_cache_breakpoint (mode) on ChatCompletionContentPartTextParam - prompt_tokens_details.cache_write_tokens on PromptTokensDetails (cached_tokens was already modeled) Tests cover a Chat Completions request with an explicit breakpoint on a text content block (field capture + re-marshal preservation), opaque request forwarding byte-for-byte, and cache_write_tokens preservation in the response. Refs envoyproxy#2454 Signed-off-by: liuhy <liuhongyu@apache.org>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2468 +/- ##
==========================================
+ Coverage 85.03% 85.04% +0.01%
==========================================
Files 159 159
Lines 22634 22634
==========================================
+ Hits 19247 19250 +3
+ Misses 2219 2217 -2
+ Partials 1168 1167 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR promotes OpenAI prompt-caching request/response fields to first-class schema fields so they survive decode/re-marshal paths (e.g., redaction/debug logging) instead of relying solely on opaque byte passthrough in the OpenAI→OpenAI translator.
Changes:
- Adds request modeling for
prompt_cache_key,prompt_cache_options.mode, andprompt_cache_breakpoint.mode(on text content parts). - Adds response modeling for
usage.prompt_tokens_details.cache_write_tokens. - Adds regression tests validating round-trip preservation and opaque forwarding behavior.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| internal/translator/openai_openai_test.go | Adds translator-level tests ensuring prompt-cache fields are forwarded byte-for-byte and cache_write_tokens is parsed/preserved. |
| internal/apischema/openai/vendor_fields_test.go | Adds schema-level test verifying prompt-cache fields survive unmarshal and re-marshal. |
| internal/apischema/openai/openai.go | Extends OpenAI schema types with prompt-cache request fields and cache_write_tokens in token usage details. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
OpenAI GPT-5.6 and later support explicit prompt cache breakpoints and related request-level fields for marking stable prompt prefixes for reuse. The OpenAI to OpenAI translator already forwards request and response bodies opaquely (raw bytes, only sjson-patching the model), so these fields already reach the upstream and client unchanged. This change promotes them to first-class fields in the OpenAI type model so the gateway recognizes them when parsing request/response bodies (and preserves them through the redaction/debug-log re-marshal path) instead of relying on opaque passthrough alone.
Modeled fields:
prompt_cache_keyandprompt_cache_options(withmode) onChatCompletionRequestprompt_cache_breakpoint(withmode) onChatCompletionContentPartTextParam; the content union marshaler delegates to the concrete struct marshal, so the field round-tripsprompt_tokens_details.cache_write_tokensonPromptTokensDetails(cached_tokenswas already modeled)The response body remains opaque passthrough, so
cache_write_tokensreaches the client unchanged regardless; modeling it makes it available when the response is parsed. New metrics forcache_write_tokensare intentionally out of scope here (cache-write vs cache-creation semantics warrant a separate maintainer decision); the field is modeled but not yet emitted as a metric.Tests cover a Chat Completions request with an explicit breakpoint on a text content block (field capture plus re-marshal preservation), byte-for-byte opaque request forwarding, and
cache_write_tokenspreservation/parsing in the response.Related Issues/PRs (if applicable)
Refs #2454
Special notes for reviewers (if applicable)
SA5008 json:"type,"warnings onThinkingDisabled/ThinkingAdaptive(present onmain, not introduced here).cache_write_tokensis deliberately deferred (see Description).