test[notask]: overlay qvac-fabric onto qwen3vl CPU-repack branch (fabric #178)#3012
Closed
olyasir wants to merge 2 commits into
Closed
test[notask]: overlay qvac-fabric onto qwen3vl CPU-repack branch (fabric #178)#3012olyasir wants to merge 2 commits into
olyasir wants to merge 2 commits into
Conversation
Phase A of the fabric-change rollout (docs/fabric-change-rollout.md): add a shared overlay port at packages/ports/qvac-fabric pinning the fabric branch commit for PR tetherto/qvac-fabric-llm.cpp#178 (q8_0 CPU weight repack in mtmd/clip), and wire "overlay-ports": ["../ports"] into every fabric consumer so CI builds the change end-to-end. The registry is untouched and no baseline is bumped — the overlay shadows only qvac-fabric (version unchanged at 9341.1.0). Once #178 merges and a fabric version is published, revert this overlay and bump each consumer to the new version.
Contributor
Review StatusCurrent Status: ❌ PENDING Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member. |
Contributor
VLM Matrix BenchmarkRun #216 — full report
|
Contributor
VLM Matrix BenchmarkRun #215 — full report
|
Contributor
VLM Matrix BenchmarkRun #217 — full report VLM Matrix — several-sources / cognitive (run #217)Mode: several sources (engine varies; model fixed) · Engine: addon Preset: one fixed model across inference engines · quality = lmms-eval (VQA / ANLS / relaxed / MC), equal-weight mean across tasks. 1 · HighlightsInference engines on the same model: addon@baseline, addon@candidate. Quality — overall % per source
Speed — mmproj-encode ms per source (lower = faster)
2 · DetailsSources — resolved versions
Models & origins (Source = Registry / HF / S3 / URL · pinned commits)
Provenance — hardware & softwarelinux · cpu (runner
macmini · cpu (runner
macos · cpu (runner
windows · cpu (runner
iphone17pro — Apple iPhone 17 Pro (AWS Device Farm)
Quality (%)
Quality by task (% — higher better, mean across platforms; one column per source)
Speed
Peak memory (RSS)
3 · Test Results (per platform)
4 · Image samples
|
Contributor
VLM Matrix BenchmarkRun #218 — full report VLM Matrix — several-sources / cognitive (run #218)Mode: several sources (engine varies; model fixed) · Engine: addon Preset: one fixed model across inference engines · quality = lmms-eval (VQA / ANLS / relaxed / MC), equal-weight mean across tasks. 1 · HighlightsInference engines on the same model: addon@baseline, addon@candidate. Quality — overall % per source
Speed — mmproj-encode ms per source (lower = faster)
2 · DetailsSources — resolved versions
Models & origins (Source = Registry / HF / S3 / URL · pinned commits)
Provenance — hardware & softwarelinux · cpu (runner
macmini · cpu (runner
macos · cpu (runner
windows · cpu (runner
s25 — Samsung Galaxy S25 Ultra (AWS Device Farm)
iphone17pro — Apple iPhone 17 Pro (AWS Device Farm)
Quality (%)
Quality by task (% — higher better, mean across platforms; one column per source)
Speed
Peak memory (RSS)
3 · Test Results (per platform)
4 · Image samples
|
Contributor
VLM Matrix BenchmarkRun #217 — full report VLM Matrix — several-sources / cognitive (run #217)Mode: several sources (engine varies; model fixed) · Engine: addon Preset: one fixed model across inference engines · quality = lmms-eval (VQA / ANLS / relaxed / MC), equal-weight mean across tasks. 1 · HighlightsInference engines on the same model: addon@baseline, addon@candidate. Quality — overall % per source
Speed — mmproj-encode ms per source (lower = faster)
2 · DetailsSources — resolved versions
Models & origins (Source = Registry / HF / S3 / URL · pinned commits)
Provenance — hardware & softwarelinux · cpu (runner
macmini · cpu (runner
macos · cpu (runner
windows · cpu (runner
pixel9 — Google Pixel 9a (AWS Device Farm)
s25 — Samsung Galaxy S25 Ultra (AWS Device Farm)
iphone17pro — Apple iPhone 17 Pro (AWS Device Farm)
Quality (%)
Quality by task (% — higher better, mean across platforms; one column per source)
Speed
Peak memory (RSS)
3 · Test Results (per platform)
4 · Image samples
|
Contributor
VLM Matrix BenchmarkRun #219 — full report VLM Matrix — several-sources / cognitive (run #219)Mode: several sources (engine varies; model fixed) · Engine: addon Preset: one fixed model across inference engines · quality = lmms-eval (VQA / ANLS / relaxed / MC), equal-weight mean across tasks. 1 · HighlightsInference engines on the same model: addon@baseline, addon@candidate. Quality — overall % per source
Speed — mmproj-encode ms per source (lower = faster)
2 · DetailsSources — resolved versions
Models & origins (Source = Registry / HF / S3 / URL · pinned commits)
Provenance — hardware & softwarelinux · cpu (runner
macmini · cpu (runner
macos · cpu (runner
windows · cpu (runner
pixel9 — Google Pixel 9a (AWS Device Farm)
s25 — Samsung Galaxy S25 Ultra (AWS Device Farm)
iphone17pro — Apple iPhone 17 Pro (AWS Device Farm)
Quality (%)
Quality by task (% — higher better, mean across platforms; one column per source)
Speed
Peak memory (RSS)
3 · Test Results (per platform)
4 · Image samples
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase A of the fabric-change rollout for the
Qwen3-VL CPU vision-encoder speedup (q8_0 weight repack). This wires the
fabric branch into the monorepo via a shared overlay port so CI validates it
end-to-end before anything is published to the registry.
Companion fabric PR: tetherto/qvac-fabric-llm.cpp#178 (q8_0 CPU weight repack
in mtmd/clip → ~1807 ms → ~1114 ms on Pixel 9 Pro, bit-identical).
Changes
packages/ports/qvac-fabric/— copied from theregistry port, with
portfile.cmakere-pinned to the fabric branch commit(
a3bc3d8d…, PR Adding compiler check to the prebuild step on macOS #178) viavcpkg_from_githubREF+SHA512. Version unchanged(
9341.1.0); the overlay shadows onlyqvac-fabric."overlay-ports": ["../ports"]added to every fabric consumer:llm-llamacpp,embed-llamacpp,translation-nmtcpp,ocr-ggml,classification-ggml,vla-ggml. No baseline bump; registry untouched.Not for merge to
mainas-isThis is a test overlay (Phase A). Once #178 merges and a fabric version is
cut in the registry (Phase B), revert the overlay port +
../portsentries andbump each consumer to the published version. Until then this PR is for CI
validation of the fabric change across all consumers.