In vision-language models (VLMs), visual tokens usually consume a significant amount of computational overhead, despite their sparser information density compared to text tokens. To address this, existing methods extract more compact image representations by modifying the image encoder or projector. While some recent works further sparsify vision tokens during the decoding, they still ignore the guidance from the language tokens, which contradicts the multimodality paradigm. We argue that visual tokens should be sparsified adaptively based on the question prompt, as the model might focus on different parts (e.g., foreground or background) when dealing with various questions, as shown in Figure below. Unlike previous methods with text-agnostic visual sparsification (c) e.g., recent FastV, our SparseVLM (b) is guided by question prompts to select relevant visual patches.
- Clone this repository and navigate to SparseVLMs folder
git clone https://github.com/hyiii0204/FlashVLMs.git
cd FlashVLMs- Install necessary package
conda create -n FlashVLMs python=3.10 -y
conda activate FlashVLMs
pip install -e .- Download Multimodal Benchmark
Please follow the detailed instruction in LLaVA-Evaluation.
Specifically, --sparse in script indicates whether to perform sparseness, while --scale and --bias control the degree of token sparsity.
- Example for evaluating MME results (192 tokens, scale = 13.5, bias = 0.0):
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mme.sh- Example for evaluating POPE results (128 tokens, scale = 9, bias = 6):
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/pope.sh- Example for evaluating TextVQA results (64 tokens, scale = 0.8, bias = 0.0):
CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/textvqa.shThis project is released under the Apache 2.0 license.
We extend our gratitude to the open-source efforts of TCFormer, LLaVA, MiniGemini and VideoLLaVA.