[testing] standalone qwen3vl encoder + CPU repack (do not merge)#178
Draft
olyasir wants to merge 2 commits into
Draft
[testing] standalone qwen3vl encoder + CPU repack (do not merge)#178olyasir wants to merge 2 commits into
olyasir wants to merge 2 commits into
Conversation
github-actions
Bot
added
documentation
Vision-encoder weights (q8_0/q4_0 2D matmul matrices) were allocated in the default CPU buffer, so their mul_mat used the generic path. Allocate the repackable ones in ggml's CPU "repack" extra buffer type instead, so they get repacked into the interleaved layout at load and hit the fast i8mm/AVX2 GEMM. Selection is per-tensor via a ggml_backend_dev_supports_op probe (mirrors llama's buft_supported): only tensors the backend can actually repack+matmul go there, so conv weights / biases / norms stay in the default buffer, and on arches with no repack kernel it is a no-op. Weight loading is made per-tensor host-aware since weights now span the host default buffer and the non-host repack buffer. Output is bit-identical. Opt out with MTMD_CLIP_NO_REPACK=1. On Pixel 9 Pro (Qwen3.5-0.8B mmproj, q8_0, arm64 i8mm build) this cuts the CPU vision-encode from ~1807 ms to ~1114 ms.
tools/mtmd/qwen3vl-encoder: a small executable that runs only the Qwen3-VL vision encoder (ViT + patch-merge projection) through ggml, with no libllama dependency (it compiles the clip sources directly). It loads an mmproj gguf, preprocesses an image, and times clip_image_batch_encode across CPU / Vulkan / OpenCL, with a cosine-similarity check (--dump / --ref) to keep changes lossless. This gives a fast edit -> build -> run loop for optimizing the shared Vulkan/OpenCL/CPU kernels of the vision encoder without the heavy llama build. See tools/mtmd/qwen3vl-encoder/standalone_loop.md for the local + Android + Firebase workflow.
olyasir
force-pushed
the
feat/qwen3vl-standalone-encoder
branch
from
3791bdf to
a3bc3d8
Compare
This was referenced Jul 2, 2026
Closed
olyasir
changed the title
Author
|
Split: the production change (q8_0 CPU repack, clip.cpp only) now lives in #179 for review/merge. This branch keeps the repack plus the standalone |
Review StatusCurrent Status: ❌ PENDING Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Speeds up the CPU vision encoder (mtmd/clip) and adds a standalone
encoder benchmark tool used to find and validate the win.
On a Pixel 9 Pro, the Qwen3.5-0.8B vision encode (q8_0 mmproj, elephant.jpg,
4 threads) drops from ~1807 ms → ~1114 ms (1.6×), with bit-identical
output.
The production arm64-android build already runs with
GGML_CPU_ALL_VARIANTS=ON,so it ships an
armv8.6_1(i8mm/dotprod) CPU variant that is dlopen-dispatchedat runtime — i.e. the fast i8mm GEMM path is already active in production. This
change is the incremental win on top of that; no build-config change is
required. (A CPU build with no i8mm/dotprod variant runs ~2689 ms and this is
a safe no-op there.)
1. Repack quantized CLIP weights into the CPU i8mm buffer
clip.cppallocated all weights in the default CPU buffer, so the q8_0/q4_0matmul matrices used the generic GEMM instead of the interleaved i8mm/AVX2
kernels. This change allocates the repackable weights in ggml's CPU "repack"
extra buffer type, so they are repacked into the interleaved layout at load.
ggml_backend_dev_supports_opprobe (mirrorsllama's
buft_supported): only tensors the backend can actually repack+matmulgo to the repack buffer. Conv weights / biases / norms stay in the default
buffer (otherwise their non-matmul ops become unsupported and the scheduler
aborts). On arches/variants with no repack kernel the probe rejects everything
and this is a no-op.
default buffer and the non-host repack buffer.
golden). Opt out with
MTMD_CLIP_NO_REPACK=1.2. Standalone
qwen3vl-encoderbenchmark tooltools/mtmd/qwen3vl-encoder: runs only the vision encoder (ViT + patch-mergeprojection) through ggml with no libllama dependency (it compiles the clip
sources directly). Loads an mmproj gguf, preprocesses an image, and times
clip_image_batch_encodeacrosscpu | vulkan | opencl, with acosine-similarity check (
--dump/--ref) so kernel changes stay lossless.This gives a seconds-scale edit → build → run loop for optimizing the shared
vision-encoder kernels without the heavy llama build.
standalone_loop.mddocuments the local + Android + Firebase workflow.
Testing
verified bit-identical to the non-repack path.
Test Lab game-loop.