pxq_llama: PXA-native low-bit MoE quants (PXQ2/3/6 + PXQU) + fused Pascal/Volta kernels #1
poisonxa16
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
This is a fork of ik_llama.cpp that adds PXQ - a PXA-native low-bit quant family for MoE models, with fused CUDA kernels written for Pascal (sm_60) and Volta (sm_70). The goal was to put a real 35B MoE on the cheap 16 GB datacenter cards that modern quant kernels skip.
Results (all measured; repro harnesses in bench/):
Format sketch: expert tensors use a learned codebook + per-row E16 scales (a per-row fp16 anchor amortized over a 64-row panel + a 4-bit sub-scale per 16-elem block); backbone stays MXFP4. Every fast kernel is memcmp bit-exact against its baseline (bench/determinism-gates.md).
We also publish a negative result: protecting the attention backbone at q6 (like ik IQ_K) is a KLD wash at fixed size on Pascal and costs 3-5% decode, so we shipped the faster version - the override code is in the repo for anyone who wants the trade.
Built on ikawrakow's ik_llama.cpp - thank you for the base engine and the IQ_K line of work. Weights: https://huggingface.co/poisonxa/PXA-Fusion2-35B-GGUF
Feedback on the format or the kernels very welcome.
All reactions