Architecture Question: Desktop CPU Inference (SIMD & BitNet)

[shifulegend]

Hey team,

First off, LLM-Hub looks fantastic. I saw in the README that native Windows and macOS desktop apps are on the roadmap. I've been doing a lot of work on local CPU inference engines and had a couple of architectural questions about how you plan to handle the desktop backend, specifically since mobile (QNN/Metal) is very different from desktop x86/ARM CPUs:

1. SIMD Auto-Calibration: Will the desktop engine rely purely on Llama.cpp for CPU execution? On x86 (e.g., Intel/AMD), having a runtime dispatcher that auto-calibrates to choose the best kernel (AVX-512 VNNI vs AVX2 vs scalar) is critical for hitting high tokens/s without OOMs. I implemented this runtime auto-calibration in Project Zero (a pure C99 engine) and it made a massive difference across different host machines.
2. Ternary Model Support: Are there plans to support Microsoft's BitNet b1.58 (I2_S ternary format)? Llama.cpp currently struggles with or doesn't support the raw ternary unpacking efficiently. In Project Zero, I wrote a custom AVX-512 VBMI kernel that skips dequantization entirely for ternary weights, hitting 36 tok/s on a Xeon.

Would love to know if LLM-Hub plans to stick to standard GGUF dense models via llama.cpp for desktop, or if there's an appetite to integrate highly optimized pure-CPU backends for these edge cases! Happy to share notes if you end up going down the custom SIMD route.

[timmyy123]

Hey team,

First off, LLM-Hub looks fantastic. I saw in the README that native Windows and macOS desktop apps are on the roadmap. I've been doing a lot of work on local CPU inference engines and had a couple of architectural questions about how you plan to handle the desktop backend, specifically since mobile (QNN/Metal) is very different from desktop x86/ARM CPUs:

1. SIMD Auto-Calibration: Will the desktop engine rely purely on Llama.cpp for CPU execution? On x86 (e.g., Intel/AMD), having a runtime dispatcher that auto-calibrates to choose the best kernel (AVX-512 VNNI vs AVX2 vs scalar) is critical for hitting high tokens/s without OOMs. I implemented this runtime auto-calibration in Project Zero (a pure C99 engine) and it made a massive difference across different host machines.
2. Ternary Model Support: Are there plans to support Microsoft's BitNet b1.58 (I2_S ternary format)? Llama.cpp currently struggles with or doesn't support the raw ternary unpacking efficiently. In Project Zero, I wrote a custom AVX-512 VBMI kernel that skips dequantization entirely for ternary weights, hitting 36 tok/s on a Xeon.

Would love to know if LLM-Hub plans to stick to standard GGUF dense models via llama.cpp for desktop, or if there's an appetite to integrate highly optimized pure-CPU backends for these edge cases! Happy to share notes if you end up going down the custom SIMD route.

Hello
Desktop version is in early idealization step, still exploring model runtime options
For macos there's already runaywhere sdk available, on windows I need to find a similar framework that supports cuda

[shifulegend]

A no-GPU fallback path is worth keeping in the design even once you pick a primary backend. Most people asking about desktop CPU inference are asking specifically because they don't have a CUDA/Metal-capable machine on hand, not because they're waiting for one. That's the whole premise Project Zero got built around from the start, no GPU dependency anywhere in the stack, and the ternary/BitNet path in particular still gets real throughput without one to fall back on. Worth keeping around, even as a fallback.