[Model] Add VisionPsy-Nano-460M / -Flash support - #1613
Open
cdpark92 wants to merge 1 commit into
Open
Conversation
Author
|
Please refer this blog post for more information about this model. Thanks! |
cdpark92
force-pushed
the
visionpsy-support
branch
from
August 27, 2026 07:49
8e7ed12 to
02c8d8c
Compare
New adapter in vlmeval/vlm/visionpsy.py. The checkpoints are Hub-packaged and bundle their own processor and modeling code, so the default backend is transformers with trust_remote_code and needs no extra install; use_vllm=True serves the same checkpoint through an in-process vLLM engine via the VisionPsy vLLM plugin. The base/Flash preprocessing variants are resolved from the checkpoint config. Existing adapters are untouched.
cdpark92
force-pushed
the
visionpsy-support
branch
from
September 3, 2026 08:40
02c8d8c to
273ff22
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds support for VisionPsy-Nano-460M and VisionPsy-Nano-460M-Flash
(qvac/VisionPsy-Nano-460M,
qvac/VisionPsy-Nano-460M-Flash),
two ~460M-parameter vision-language models built for on-device use
(SigLIP2-base-512 vision tower + SmolLM2-360M decoder, pixel-shuffle
projector, dynamic 512px tiling). The Flash variant keeps the native
resolution with a min-side clamp instead of resizing to the max side, trading
a little accuracy for much lower time-to-first-token.
Changes
All additive — no existing file or adapter is modified.
vlmeval/vlm/visionpsy.py(new) — self-contained adapter with twobackends:
their own processor and modeling code, so they load with
trust_remote_code=Trueand need no extra install (INSTALL_REQ = False).--use-vllm) — serves the same checkpoint through theVisionPsy vLLM plugin, which registers the out-of-tree architecture. The
engine is built before any tokenizer/torch work: HF tokenizers and torch
spawn worker threads, and vLLM forks its engine-core subprocess, so the
reverse order deadlocks.
The base/Flash preprocessing variants are resolved from the checkpoint's own
config, so one class serves both models.
vlmeval/vlm/__init__.py(+1) export,vlmeval/config.py(+9)registry entries,
vlmeval/inference.py(+1) forwardsuse_vllmforVisionPsy model names.
Usage
Verification
--use-vllm), greedy decoding, loaded through the registered names.pre-commit runpasses on the added/changed files.Notes
nanoVLM-460M-8k (MIT), but
ship as standalone Hub packages with their own processor, so they get their
own adapter rather than reusing
nanovlm.py.transformers>=5.14.1(fix: include transitive relative imports when loading from local directory huggingface/transformers#46022 fixed transitive relative imports for local
trust_remote_codeloads); the registered Hub ids work on older versionsas well.
proposed in [Fix] MM-IFEval: import rule-based verify functions into module globals #1601.