Every model tested on this setup, summarized. Flag-by-flag configuration lives in SETUP.md; the reasoning behind each choice is in JOURNAL.md. Speed figures in benchmarks.md; the hardware these numbers came from is in hardware.md.
| Model | Purpose | Quant | RAM/GPU Usage | Speed | Quality | Strengths | Weaknesses | Recommendation |
|---|---|---|---|---|---|---|---|---|
| Chat / coding / agent-mode | Unsloth Dynamic Q4_K_XL | ~530MB RSS, ~6GB Vulkan-visible | ~9–9.8 tok/s | Reliable structured tool-calling, no hallucinated function calls in 9/9 real agent-mode tests | Dense architecture (no recurrent-memory caching bug), correct multi-turn tool_calls extraction, generic autoparser handles Open WebUI's real request shape cleanly once --reasoning off is set |
Shares a chat template with a "thinking" variant it doesn't use, causing a reasoning-detection bug that needed a specific flag/testing to fix; needs -c 24576 for real headroom |
Deleted 2026-08-05 — this censored model file and its launch script were removed; only uncensored models are kept on this box now | |
| Qwen3-4B-Instruct-2507-heretic-av2 (abliterated, arnomatic/"Heretic") | Chat / coding / agent-mode, blunt/direct on sensitive-but-legal topics | mradermacher Q4_K_M | Same base as above | ~9–11.6 tok/s | Refusals dropped from 100/100 to 19/100 per its own model card; scores higher NatInt (17.77 vs. 13.76) than the untouched base on the UGI Leaderboard | Same base model/template as the (now-deleted) censored Qwen3, so it inherits the same stability and tool-calling path rather than a bespoke one; needed explicit repeat-penalty/DRY/length-cap tuning after a real long-form generation looped | Confirmed to confabulate confidently on niche real-world facts — not a factual reference; untested against Open WebUI's real tool-calling payload shape, not verified for agent-mode duty | Production — port 8081, the only Qwen server on this box as of 2026-08-05; run mutually exclusive with Gemma-Uncensored (combined memory doesn't fit) |
| Vision chat | Unsloth Dynamic Q4_K_XL | TODO RSS, TODO Vulkan-visible | ~9–9.8 tok/s | Only vision-capable model this repo has run; correctly separates reasoning_content from content when a tools block is present |
Genuine image understanding via OpenAI-style image_url blocks |
More crash/hang-prone than Qwen3 beyond the shared iGPU fence-timeout bug; --reasoning off breaks at scale with many tool declarations (produces garbage token output); audio does not work (two separate unfixable upstream blockers) |
Deleted 2026-08-05 — this censored model file, mmproj-F16.gguf, and the launch script were removed; Gemma-4-E4B-Uncensored-HauhauCS-Aggressive below is its uncensored replacement but is text-only, not a vision substitute |
|
| Gemma-4-E4B-Uncensored-HauhauCS-Aggressive (biprojected abliteration, HauhauCS) | Chat, blunt/direct — text-only, no vision | Q4_K_P (custom quant) | ~5.3GB Vulkan-visible | ~7.4 tok/s gen, ~41 tok/s prompt | Coherent output verified via real generation test; base checkpoint predates Gemma-4's 2026-07-15 family-wide refresh, so llama.cpp applies chat-template compatibility workarounds on load | Same "blunt, unfiltered" role as the Qwen uncensored variant, on its own port so either can run standalone | No --mmproj loaded — despite the Gemma-4 family's vision capability, this script is text-only; heavier than the old censored Gemma at the same quant tier (5.34GB vs 4.8GB) |
Production — port 8082, run mutually exclusive with Qwen-Uncensored (combined memory doesn't fit) |
| Qwen2.5-Coder-1.5B (Windows only) | Autocomplete | Q4_K_M | Full offload, -ngl 99 |
~25 tok/s (per author's notes) | TODO | Fast enough for inline autocomplete without blocking on a larger model | Not ported to Fedora; autocomplete-only, not a chat/agent model | Production (Windows only) — paired alongside a larger chat model for the coding-assistant setup |
| Qwen3.5-4B | Chat (retired) | Unsloth Dynamic Q4_K_XL | — | — | — | Hybrid attention + Mamba2/SSM architecture, more parameter-efficient recurrent memory in theory | Upstream multi-turn caching bug (ggml-org/llama.cpp#21831, still open as of 2026-07-30): cached_tokens sticks at a constant regardless of turn count — a checkpoint-restore-selection bug, not fixable by cache-frequency tuning |
Retired — switched away from on 2026-07-26; revisit if the upstream bug is ever fixed |
| Qwen3-8B (OpenVINO backend) | Backend evaluation | INT4 (NNCF, calibrated OpenVINO IR) | GPU-resident (OVMS container) | ~11.12–11.51 tok/s (GPU), ~7.56–7.64 tok/s (CPU) | Not evaluated for output quality — benchmarking/backend-comparison only | Reasoning + tool-calling parsers (qwen3/hermes3) confirmed working through OVMS |
Separate inference stack (OpenVINO, not llama.cpp) — GGUF-incompatible with this repo's other tooling; confirmed no genuine CPU+iGPU split possible for a single request (JOURNAL.md 2026-08-03) | Evaluated, not in daily use — start-ovms-qwen3-8b.sh, port 8084, see SETUP.md |
| Qwen3.5-9B (text) (OpenVINO backend) | Second chat option, registered in Open WebUI | INT4 (pre-converted int4-ov, ASYM) |
GPU-resident (OVMS container) | ~10.2–10.3 tok/s (GPU) | Correct answers verified via curl and real Open WebUI chat turns; found a real behavioral bug via use — open-ended/persona system prompts (e.g. "speak caveman") can send its <think> trace into a repeated self-doubt loop for minutes on trivial input, vs. seconds on deterministic questions |
Registered as an actual Open WebUI connection (port 8085), not just curl-reachable; --cache_dir on a named volume gets warm restarts to ~18s |
Hybrid thinking model, worse than Qwen3-8B's known slowness — needs enable_thinking:false as a default, not confirmed set yet; concurrency ceiling stuck at ~15 concurrent requests for a reason not yet identified (cache size ruled out) |
Registered, not the daily driver — start-ovms-qwen3.5-9b-text.sh, port 8085, see SETUP.md |
| Qwen3.5-9B (vision, CPU) (OpenVINO backend) | Vision requests, routed off the GPU | Same IR as above | CPU-resident (OVMS container, P-cores pinned) | Not measured | Not tested — script written but never launched | Exists specifically to route around a GPU-only hallucination bug in the vision-merger path (openvino#37223, filed and root-caused 2026-08-04) | Untested — no automatic routing between this and the GPU text instance; a caller has to pick the right port | Written, unverified — start-ovms-qwen3.5-9b-vision.sh, port 8086 |
| TinyLlama-1.1B-Chat (OpenVINO backend) | Backend evaluation | INT4 (NNCF, OpenVINO IR) | GPU-resident (OVMS container) | ~61.34 ± 5.55 tok/s (GPU) | Weak — small model, observed echoing/repeating the prompt back rather than answering it | Fast enough to sanity-check the OVMS pipeline end-to-end with low latency | Same separate-stack caveat as Qwen3-8B above; too small to be a real chat candidate | Evaluated, one-off — no dedicated launch script yet, see JOURNAL.md 2026-08-03 |
The abliterated model selection (see JOURNAL.md, 2026-07-30 entry) is the most fully-documented example of the evaluation approach used here: cross-checking a live leaderboard (UGI Leaderboard) rather than trusting SEO-blog claims, verifying the quantizer's reputation, and explicitly preferring a same-base-model option over a higher-parameter alternative from a different family because it de-risks integration. Turning this into a repeatable checklist is an open roadmap item — see SETUP.md.
- Qualitative "Quality" notes for Gemma 4 E4B and Qwen2.5-Coder-1.5B
- RAM/GPU usage figures for Gemma 4 E4B (RSS, Vulkan-visible)
- benchmarks.md — full performance numbers behind the Speed column above
- hardware.md — the machine every model here was run on
- troubleshooting.md — issues specific to individual models
- SETUP.md — flag-by-flag configuration for every model