Where
What's wrong
There is last line in concept section which means caching is strictly prefix only if even one prefix token different then after the violate token everything not gone cache
The invariant. All three cache prefixes only. If any token differs between requests, everything after the first differing token is a miss. Put the stable parts at the top, the variable parts at the bottom.
So section: The cache friendly layout
why we have retrieved documents above from conversation history, if we get different documents from retrieval then conversation history also not gone cache! which must be cache
[retrieved documents] <-- cache if reused, else don't
[conversation history] <-- cache up to last turn
Screenshot or logs
Where
What's wrong
There is last line in concept section which means caching is strictly prefix only if even one prefix token different then after the violate token everything not gone cache
So section: The cache friendly layout
why we have retrieved documents above from conversation history, if we get different documents from retrieval then conversation history also not gone cache! which must be cache
Screenshot or logs