fix: preserve per-member caches with memory saving

[jmkuebler]

Jonas: We had a bug when using memory saving mode together with cached predictions since the batching PR #1312. We are now not using memory saving mode anymore for cached predictions. Those memory peaks can anyways be bound by setting the maximum number of test query rows when prediction with cache. so another saving is not needed imo.

---

Generated by Codex

Memory-saving chunking split cached v3/v3.5 queries without slicing their K/V caches, so later ensemble members used the first member's cache and repeatedly dequantized the full batch. Keep cached attention on the intact member batch while retaining MLP memory saving and inference-engine test-row chunking. Add a tiny CPU regression test for both architectures with plain and int8 caches.

Issue

Fixes #1364.

Public API Changes

• No public API changes.

How Has This Been Tested?

All four new regression cases fail before the fix and pass afterward. The 18 small cache unit tests pass; Ruff and pre-commit hooks pass. No checkpoint download or GPU is needed for the new test.

Reproduction and runtime evidence

The regression test uses two members, five cached rows, one query, and eight features. Distinct cache values make the wrong-member lookup observable. The attention output projection is nonzero so initialization cannot hide the defect.

The prior targeted RTX PRO 6000 experiment tested the equivalent cached-attention change on 200,000 training rows × 58 columns, eight estimators, and a single cached query:

Setting	Median predict	Class-1 probability
Original batching with memory saving	115.511 ms	0.7800719142
Corrected cached attention	49.141 ms	0.9951955080
Unbatched reference	77.517 ms	0.9951962829

The corrected and original batched settings reused the same fitted cache. The correction replaces 64 single-member ICL attention calls with eight correctly aligned batch calls. Other memory-saving operations remain enabled. No full accuracy benchmark was run.

Checklist

• Tested locally.
• Public API unchanged; no usage documentation change needed.
• Changelog entry added.
• Style checks pass.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want fixes drafted automatically? Bugbot Autofix can create code changes for findings. A team admin can enable Autofix in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit f780182. Configure here.}

[bejaeger]

Thanks!