QVAC-21841 test: add mobile smoke test for VLM continuous batching#3279
Open
RamazTs wants to merge 2 commits into
Open
QVAC-21841 test: add mobile smoke test for VLM continuous batching#3279RamazTs wants to merge 2 commits into
RamazTs wants to merge 2 commits into
Conversation
The continuousBatching group is already registered for both ios and android in test-groups.json, but every test in continuous-batching.test.js gates on skipHeavyPlatform (isMobile || isDarwin), so both Device Farm runs import the module and execute zero tests. VLM continuous batching has no mobile coverage. Add a mobile smoke test at parallel: 2 with one image slot beside one text slot, skipped only on darwin so it runs on the Device Farm pools and on Linux. Vision encode is serialized across slots, so pairing an image with a text prompt is what catches an encode barrier starving the text slot. It asserts both outputs are correct and that the slots decoded together. Add a desktop test for rolling admission: the existing MTMD tests submit 4 image prompts into 4 slots, so every sequence is admitted in the initial wave and no slot is ever recycled. Halving the slots forces the second half of IMAGE_CASES to wait in pending_ and be admitted into freed slots, exercising a per-slot MTMD driver resetting its media and vision-encoding a new image while the other slot keeps decoding. Stage the SmolVLM2 model and mmproj for runContinuousBatchingTest in model-manifest.json. The entry listed only Llama-3.2-1B, so the Android pre-stage would have missed the vision models and downloaded them on-device from HuggingFace. Entries are added by hand rather than regenerating, since the generator scrapes through the _image-common.js require and would also stage Qwen3VL-2B and its mmproj, which this file never loads.
Contributor
Review StatusCurrent Status: ❌ PENDING Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member. |
RamazTs
added
run-mobile-addon-tests
Contributor
Mobile integration tests — @qvac/llm-llamacpp (iOS)Result: passed
|
Contributor
Mobile integration tests — @qvac/llm-llamacpp (Android)Result: passed
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 What problem does this PR solve?
continuousBatchinggroup is already registered for bothiosandandroidintest/mobile/test-groups.json, but every test incontinuous-batching.test.jsgates onskipHeavyPlatform(isMobile || isDarwin). Both Device Farm runs currently import the module and execute zero tests.parallel: '4'slots, so every sequence is admitted in the initial wave and no slot is ever recycled onto a new image.runContinuousBatchingTestentry intest/mobile/model-manifest.jsonlisted onlyLlama-3.2-1B, so SmolVLM2 + mmproj would be downloaded on-device from HuggingFace — the flaky path pre-staging exists to avoid.Follows #2327 (text) and #2543 (VLM/images). Scope per @Olya's call: mobile coverage only, no extra VLM-Benchmark throughput leg (vision encode is serialized, so single-step encode optimisations carry over to batched inference anyway).
📝 How does it solve it?
continuous batching MTMD: mobile smoke — image and text slots decode together):parallel: '2', oneelephant.jpgimage slot beside one text slot, skipped only on darwin so it runs on the Device Farm pools and on Linux desktop.avgConcurrentSeq > 1.0(a mean pinned at exactly 1.0 means batching collapsed to serial).parallel: 2against the defaultctx_size: 4096leaves each slot a 2048-token window — comfortably above SmolVLM2-500M's ~256 vision tokens plus prompt and output, which matters because oversized prompts are rejected rather than truncated.continuous batching MTMD: image prompts outnumber slots and roll through freed slots): reuses the existing 4IMAGE_CASESbut halves the slots toparallel: '2', forcing the second half to wait inpending_and be admitted into freed slots. This exercises a per-slot MTMD driver resetting its media and vision-encoding a new image while the other slot keeps decoding.IMAGE_CASESasks two questions each about two different images, so a slot that fails to reset media on recycle answers from the wrong image and fails the assertion. Also bracketsavgConcurrentSeqat the slot cap to check admission respectsn_seq_max.runContinuousBatchingTest. Added by hand rather than regenerating withscripts/generate-model-manifest.js, because the generator scrapes through therequire('./_image-common.js')and would also stageQwen3VL-2B+ mmproj (~1.5 GB per Android run) that this file never loads.No new fixtures or wiring needed:
elephant.jpgis already used on-device byimage-mmproj-gpu.test.jsandmrope-sliding-context.test.jsviaglobal.assetPaths, andrunContinuousBatchingTestis already assigned to a group on both platforms, sotest-groups.jsonandintegration.auto.cjsare untouched.🧪 How was it tested?
node --checkpass locally.verified+run-mobile-addon-tests+run-desktop-addon-teststo exercise the Device Farm pools and the Linux leg.avgConcurrentSeqthresholds (> 1.0mobile,> 1.2desktop rolling) are reasoned estimates, not measured. They are the flake risk in this PR and want calibrating against the first green Device Farm run — happy to adjust or drop them tot.commentif they prove noisy on device.📋 Notes for reviewers
Llama-3.2-1B(~770 MB) is still pre-staged for this group even though its test is skipped on mobile. The manifest is keyed per test-function and can't express "Android only". Worth a follow-up if Device Farm time is tight.