U1: Response cache aggressive mode#9
Conversation
There was a problem hiding this comment.
Pull request overview
Enables “aggressive mode” response caching for non-stream chat completions by default, increasing cache hit rate and reducing repeat LLM work at the gateway layer.
Changes:
- Enable response caching by default via
RESPONSE_CACHE_ENABLED !== "0". - Increase Redis TTL from 300s to 3600s and simplify skip logic to only exclude streaming.
- Expand cache key inputs to include
tool_choicein addition to model/messages/temperature/tools.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const messages = body.messages; | ||
| const model = body.model ?? "auto"; | ||
| const temperature = body.temperature ?? 0; | ||
| const tools = body.tools ?? null; | ||
| const payload = JSON.stringify({ model, messages, temperature, tools }); | ||
| const tool_choice = body.tool_choice ?? null; | ||
| const payload = JSON.stringify({ model, messages, temperature, tools, tool_choice }); | ||
| const hash = createHash("sha256").update(payload).digest("hex").slice(0, 32); |
There was a problem hiding this comment.
The cache key only hashes {model, messages, temperature, tools, tool_choice}. However the gateway accepts and forwards other request parameters that materially change the output (e.g., response_format/json_schema, max_tokens/max_completion_tokens, top_p, stop, presence_penalty, frequency_penalty, logit_bias, n, seed, etc.). With aggressive caching enabled by default, two requests that differ in any of those fields can incorrectly collide and return an incompatible cached response. Consider hashing a normalized copy of the full request body (at least all output-affecting fields), excluding only non-semantic fields like stream (and possibly stream_options/store after confirming they don't affect provider output).
Summary
RESPONSE_CACHE_ENABLED=0to disable)temp > 0.3andtools.length > 0skip conditions; keepstream === trueskiptool_choicein cache key hash alongside existing tools/messages/model/temperatureCross-response bleeding risk is mitigated because the sha256 key includes the full messages array and tool definitions — two requests only collide if every field is byte-identical.
Test plan
rtk npx next build-> Errors: 0 | Warnings: 0