Windows-native local LLM inference — a portable, tiny, single-exe multi-engine orchestrator for llama.cpp (official master + Tom TurboQuant fork). Built for GGUF models on Windows with easy Foundry builds from source , State of the art MEMORY forecast, and live fusion telemetry. -support MTP, Dflash and TurboQuant out of the box.
No WSL. No Electron. No Docker required. Native desktop app for running and managing local llama servers on Windows.
This project is my testament to open source and a local inference first mission.
You do not need Linux to run serious LLM workloads on your own hardware. Blackwell Ops exists to challenge it — directly, on Windows, with top performance and minimal memory footprint.
Blackwell Ops is a native Windows desktop app for local LLM / llama.cpp users. It orchestrates multiple llama-server instances (GGUF models), provides a GGUF model library with VRAM estimation, one-click Foundry CMake builds, and real-time telemetry — all without Docker, WSL, or cloud dependencies. Fully portable and open source.
Drop the installer or portable folder anywhere. The app recreates its ecosystem around itself: configs, runtime engines, foundry artifacts, and user preferences — all relative to the install directory. Great for portable local LLM setups or moving between machines.
At v1.0.x we strongly focus on GGML / llama.cpp (official master + Tom TurboQuant fork bundled). Any llama-compatible fork can be easily wired in by users in 2 clicks. You will get most of funcionality right away, including 1click builds form source anytime. The architecture is semi–backend-agnostic by design — support grows over time.
| Core binary | ~14 MB Rust executable |
| Typical RAM (app shell) | ~40 MB running — about half what Windows Notepad needs |
| Engine slots (factory) | Up to 64 concurrent instances (GGML master) |
| Stress-tested | 64 engine instances orchestrated with ~400 MB total app RAM overhead |
| Build investment | ~2,500+ hours across ~5 months |
| GPU targets | BLACKWELL (heavily optimized), also AMPERE · ADA |
Engine VRAM is separate — these figures are the ops layer, not model weights.
People running serious local LLMs on Windows often hit friction with existing tools:
| Tool | Windows Native | Multiple Engines | Easy Source Builds | Portable | Low Overhead | Full Config Freedom |
|---|---|---|---|---|---|---|
| Blackwell Ops | ✅ | ✅ (up to 64 engines) | ✅ (Foundry) | ✅ | ✅ | ✅ |
| Ollama | Partial (WSL) | Limited | No | No | Medium | Limited |
| LM Studio | ✅ | Limited | No | No | Medium | Medium |
| llama-server (bare) | ✅ | Manual | Manual | Partial | ✅ | ✅ (CLI only) |
Blackwell Ops is the Windows-native alternative to Ollama and LM Studio for power users who want full control over llama.cpp (and Tom TurboQuant builds) — multiple engines, source builds, and proper Windows integration.
Most users run models in a single session. But llama-server supports multiple parallel slots on one instance.
In Blackwell Ops you simply:
- Set 4× or 8× parallel slots on a single model / single port
- Tell your agent harness (OpenCode, etc.) something like:
"Use up to 8 parallel agents for any suitable work or sub-task"
Real results on the same hardware:
- Qwen3.6-27B single slot + MTP → ~185 TPS per session, but prefill is halved (painful for coding)
- Same model, 8× parallel, MTP off → 330+ total TPS on one GPU
- 2× RTX Pro with tensor split → 850 TPS combined, with full prefill speed
You get dramatically higher aggregate throughput and much snappier prefill behavior for agentic workflows — with almost no extra power draw.
(You can also run completely separate engine instances if you prefer full isolation.)
Blackwell Ops treats faster generation as a first-class feature, not a hidden CLI flag. Multi-Token Prediction (MTP) and DFlash speculative decoding are wired through the whole stack — catalog, pairing, launch, and telemetry — so you can rapidly test a ton of settings just by clicking.
| MTP | DFlash | |
|---|---|---|
| What it is | Draft tokens baked into the main GGUF (nextn layers) |
Separate lightweight draft model loaded alongside the main |
| Best for | Single-session speed on MTP-capable weights (Qwen 3.x, etc.) | Multi-slot / agent workloads — full prefill speed with parallel agents |
| Setup | Turn spec on — no second file | Family-matched draft picker finds the right .gguf in your library |
| Parallel slots | Use 1× parallel (MTP + multi-slot conflict) | 4× / 8× parallel — aggregate TPS without halving prefill |
What you get in the app
- Smart catalog — MAIN / DRAFT / ALL filter; DFLASH badges on external draft models; Gemma, Qwen, and family-aware pairing (no random cross-family matches).
- Engine config — spec group with ON/OFF toggle, live MTP / DFLASH mode badge, and a ★ best-match draft picker when DFlash is selected.
- Both on the same main — a model can ship with baked-in MTP and support an external DFlash draft; pick the mode that fits the workload (MTP for one chat, DFlash when you crank parallel agents).
- FULL-AUTO + Essentials — Regular Joe path: spec turns on when your model supports it, only MTP and DFLASH shown, sensible N-max / N-min presets applied automatically; Full config view stays untouched for power users.
- Launch-safe — draft-only GGUFs cannot be launched as mains; DFlash launches with
--fit offand a resolved--spec-draft-modelpath.
DFlash / external drafts are on GGML master today; Tom TurboQuant remains MTP-focused for now.
- Native Rust + Tauri — tiny footprint, no Electron bloat or Linux subsystem tax
- Foundry — one-click build
llama-serverfrom source with your preferred CUDA / VS version - Portable — works from USB, relative paths everywhere
- Multi-engine stack — orchestrate many
llama-serverinstances (or parallel slots) with a shared config system - Fusion telemetry — real-time metrics from stderr + /slots without extra overhead
Architecture — click to expand
Process model
- Each
llama-serverinstance runs in a private Job Object (engine_job,KILL_ON_JOB_CLOSE) - Teardown is PID-only via
stop_child_fast— never port-based carpet-bombing - Launch verifies orphan status via
engine_port_lock::reclaim_our_ghost_or_fail(checked against verified orphans only) - App exit:
teardown_all_for_app_exit→std::process::exit(0)(skips Tauri Drop to avoid heap corruption0xC0000374)
Fusion telemetry
- Single constant
TELEMETRY_TICK_MSdrives: stderr batch flush,/slotspoll, andRENDER_INTERVAL_MS - Currently 25ms (~80 HTTP polls/s per active engine)
- Merges
NewPrompt(prefill) +/slots(decode) metrics —n_decodedcompared per-request, not absolute - Spec flags respect per-param
hiddenfrom ConfigPage — spec group fully omitted from CLI when OFF
Foundry
- CMake cache retained in
work/when fingerprint matches (.blackwell-foundry-cache-key+CMakeCache.txt) - Durable runtime binary:
foundry/artifacts/.../Release/ - Supports VS2022 (CUDA 12.8 /
stable) and VS2026 (CUDA 13.3 /frontier)
Speculative decoding
- MTP: draft tokens baked into main GGUF (
nextnlayers) — single-slot best - DFlash: separate lightweight draft, family-aware pairing (Gemma/Qwen) via
llama-fit-params - Toggle state reads all group params; rows use
specVisibleParams(respectshidden)
Performance
| Metric | Value |
|---|---|
| Core binary | ~14 MB |
| App RAM (idle) | ~40 MB |
| Max concurrent engines | 64 |
| Stress test RAM overhead | ~400 MB (64 instances) |
| Telemetry poll rate | 25ms per active engine |
| Parallel slot throughput (8× Qwen3.6 27B) | 330+ TPS |
| 2× RTX PRO tensor split | 850 TPS combined |
- Native Rust, Win32/Tauri shell — no Electron bloat, no Linux subsystem tax
- Foundry — build
llama-serverfrom source on your machine (VS2022 / VS2026 + CUDA 12.8–13.3) - Portable path model — clone/move the folder, it still works, even from a flash drive.
- Full config freedom — factory templates merge with your overrides; nothing hidden behind a SaaS panel
- Low idle cost — run many engine slots without many heavy processes until you launch
Pre-built binaries ship for multiple toolchain generations — pick the profile that matches your GPU and driver stack:
| Profile | CUDA | Toolchain |
|---|---|---|
| FRONTIER | 13.3 | VS Build Tools 2026 |
| STABLE | 12.8 | VS Build Tools 2022 |
- Includes GGML llama (master) and Tom-llama runtimes.
- Users can Foundry-build their own engines anytime (5minutes 1 click), or download asset packages later — the app does not lock you to shipped binaries.
- Rapid onboarding — two-click setup for local LLM on Windows from zero to first inference
- GGUF Model Library — catalog, metadata scan, VRAM fit estimation and benchmarks for your models
- Multi-engine stack — run, monitor, bench or stop many llama.cpp / llama-server instances side-by-side
- Advanced Provider Config — full params editor with 250+ parameters and factory templates
- Foundry Builds — update, configure, compile and publish llama-server binaries directly from the UI (custom CMake flags supported)
- Fusion Telemetry — real-time metrics (generation, prefill, progress) from multiple sources
- MTP & DFlash speculative decoding — catalog pairing, mode badges, draft picker, FULL-AUTO presets, full launch validation
- Unified Console & Logs — dockable logs for Engines, Foundry, Errors with search and syntax highlight (supports 64+ concurrent streams)
- Portable & Lightweight — relative paths, tiny RAM usage, works from USB stick, no registry
- Hardware Monitoring — GPU/CPU stats and vital signs
- Zero Telemetry — completely offline, no calling home, no data collection
Perfect for users looking for a Windows native llama.cpp GUI, multiple llama engines, or portable local LLM server.
A short auto-playing demo GIF lives in the Quick Start section below.
| Main dashboard | Engine stack | Foundry build |
|---|---|---|
 |
 |
 |
- Download the latest
Blackwell Ops_*_x64-setup.exefrom Releases. - Install (or extract portable layout if you ship a zip).
- Point Setup Guide → Step 1 at your GGUF model folder (or LM Studio / Ollama path).
- Pick a provider profile (FRONTIER or STABLE).
- Launch an engine.
First-run onboarding walks the rest.
Quick demo — click the thumbnail to play (30s MP4):
- Windows 10/11 x64
- NVIDIA GPU recommended (CUDA builds bundled), AMD, Intel will follow
- GGUF models — not included; you bring your own weights
Optional and trongly recommended BUILD TOOLS package -for Foundry cmake builds BUILD TOOLS package.
I vibe-coded this app on local models — mostly Qwen3.5 236B + Qwen3.6 27B + Step3.7-flash — with hardening passes on Composer 2.5, on hand build custom workstation (2× RTX PRO 6000 · 256 GB VRAM).
I baked in 30+ years of love for PC hardware and Windows — how machines should feel, how software should respect RAM, how inference should stay on your desk.
Roughly 3,000 hours went into this across ~six months. Time not spent with my family and my five-year-old daughter, who I love more than anything. This repo is what that time became.
If Blackwell Ops helps one person run serious local inference on Windows without apologizing for their OS choice, it was worth it. PS: This is my first coding endeavor. I openly state this, as inspiration to anyone hesitating to use AI besides chating. AI scene, including the local only is very powerfull already - DO NOT WAIT - JUST DO IT NOW, anyone can. -This was very difficult project as a "starter", now it is progressively more easy. I had been fully determined to achieve this, since i had a strong skill in HW suite and clear idea how i want this to work. I would NEVER achieve that without AI - NEVER!
-I'am etternally gratefull to my family, to be supportive and respecting the need to pursue my purpose in this. Love you Karla.
- In-app binary updates (GitHub release assets)
- Broader backend adapters beyond GGML llama and TOM llama
- Deeper fusion metrics for third-party forks
- and one surprise very soon ;-)
- Releases: https://github.com/Seen-Tomorrow/blackwell-ops/releases
- Issues: https://github.com/Seen-Tomorrow/blackwell-ops/issues
- Third-party notices: THIRD_PARTY_NOTICES.md (includes Nvidia Inspector / Orbmu2k attribution)
Local inference first. Windows is not the compromise.
Built with open source engines, open source tools, and closed-door family time I'll try to win back.