perf: DBCache robustness knobs + bn-block experiment (edge-dit borrow) - #13
Conversation
…uous) + bn-block experiment Borrowed from cache-dit / edge-dit.cpp, all defaulting to the existing per-step-threshold behavior (verified byte-identical to before): - reuse window: --cache-window-start/end cache only mid-schedule steps; first/last steps stay full-compute (safer at the extremes) - UCache-style accumulated-error gate: --cache-error-decay + -limit enables forced recompute once the accumulated skipped delta exceeds a limit - --cache-max-continuous caps consecutive cached steps - --cache-bn-blocks: recompute the last N tail blocks on a hit (middle+back host split). Implemented + ablated: it regresses a boundary (33->32 notes) because the extra host round-trip re-introduces fused-vs-split fp drift — kept at default 0 and documented as not recommended. Verified (CPU Q8 n8, 10s, seed 42): default 2.81s/33 notes byte-identical; decay 2.78s/33; window[0.4,0.6] 5.0s/33 (restrictive); maxcont1 3.9s/33.
|
@coderabbitai full review |
|
|
Important Review available on request
Reviews should be triggered manually for repositories with fewer than 10 stars. Select Trigger review above or comment ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…rk annex + cache knob CLI docs
1 participant
1/2/3 全试（edge-dit.cpp 借鉴落地）
① DBCache 增强（落地，默认零回归）
window[0.4,0.6]=5.0s/33（更保守）；decay=2.78s/33；maxcont1=3.9s/33。
③ bn-blocks（尾部末尾块命中时强制重算）——实现了但不推荐
实测 33→32 音符：middle↔back 的额外 host 往返再次引入 fused-vs-split 的 fp 漂移（与 fused 路径的教训一致），弊大于利；默认 0，文档注明。
② 注意力 cont/permute 削减——评估后无需改
我们的 \ggml_cont\（对连续 view）是 no-op；唯一实拷贝是 rope(ne2=T 布局) → permute → cont 以适配 \ggml_flash_attn_ext\ 的 (D,T,H,B) 布局；mainline ggml 无此布局直出算子（edge 的“layout-aware RoPE”是其专有算子）。主路径已是最少拷贝。
附带