Skip to content

[Model] Add VisionPsy-Nano-460M / -Flash support - #1613

Open
cdpark92 wants to merge 1 commit into
open-compass:mainfrom
cdpark92:visionpsy-support
Open

[Model] Add VisionPsy-Nano-460M / -Flash support#1613
cdpark92 wants to merge 1 commit into
open-compass:mainfrom
cdpark92:visionpsy-support

Conversation

@cdpark92

@cdpark92 cdpark92 commented Jul 29, 2026

Copy link
Copy Markdown

What

Adds support for VisionPsy-Nano-460M and VisionPsy-Nano-460M-Flash
(qvac/VisionPsy-Nano-460M,
qvac/VisionPsy-Nano-460M-Flash),
two ~460M-parameter vision-language models built for on-device use
(SigLIP2-base-512 vision tower + SmolLM2-360M decoder, pixel-shuffle
projector, dynamic 512px tiling). The Flash variant keeps the native
resolution with a min-side clamp instead of resizing to the max side, trading
a little accuracy for much lower time-to-first-token.

Changes

All additive — no existing file or adapter is modified.

vlmeval/vlm/visionpsy.py (new) — self-contained adapter with two
backends:

  1. transformers (default) — the checkpoints are Hub-packaged and bundle
    their own processor and modeling code, so they load with
    trust_remote_code=True and need no extra install (INSTALL_REQ = False).
  2. in-process vLLM (--use-vllm) — serves the same checkpoint through the
    VisionPsy vLLM plugin, which registers the out-of-tree architecture. The
    engine is built before any tokenizer/torch work: HF tokenizers and torch
    spawn worker threads, and vLLM forks its engine-core subprocess, so the
    reverse order deadlocks.

The base/Flash preprocessing variants are resolved from the checkpoint's own
config, so one class serves both models.

vlmeval/vlm/__init__.py (+1) export, vlmeval/config.py (+9)
registry entries, vlmeval/inference.py (+1) forwards use_vllm for
VisionPsy model names.

Usage

# transformers (eager) — no extra install
python run.py --data MMStar --model VisionPsy-Nano-460M

# in-process vLLM
pip install "git+https://github.com/tether-ai-research/qvac-visionpsy-nano#subdirectory=vllm-inference"
python run.py --data MMStar --model VisionPsy-Nano-460M-Flash --use-vllm

Verification

  • Both models generate correctly on both backends (transformers and
    --use-vllm), greedy decoding, loaded through the registered names.
  • pre-commit run passes on the added/changed files.

Notes

@cdpark92

Copy link
Copy Markdown
Author

Please refer this blog post for more information about this model. Thanks!
https://huggingface.co/blog/qvac/visionpsy

@cdpark92 cdpark92 changed the title [Model] Support VisionPsy-Nano-460M / -Flash (hub-packaged nanoVLM) with an optional in-process vLLM path [Model] Add VisionPsy-Nano-460M / -Flash support Aug 27, 2026
New adapter in vlmeval/vlm/visionpsy.py. The checkpoints are Hub-packaged
and bundle their own processor and modeling code, so the default backend is
transformers with trust_remote_code and needs no extra install; use_vllm=True
serves the same checkpoint through an in-process vLLM engine via the VisionPsy
vLLM plugin. The base/Flash preprocessing variants are resolved from the
checkpoint config. Existing adapters are untouched.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant