The Evaluation Toolkit for AndesVL Series Models

 

AndesVL Technical Report: An Efficient Mobile-side Multimodal Large Language Model
AndesVL Team, OPPO AI Center

We are very excited to introduce AndesVL, a state-of-the-art model designed for mobile-side applications with 0.6B to 4B parameters.

🔥 News

• 2025/10/13: AndesVL Technical Report is now available at arxiv and AndesVL Models is available at huggingface.

🎬 Quick Start

Inference using the non-thinking version of AndesVL

# require transformers>=4.52.4

import torch
from transformers import AutoModel, AutoTokenizer, CLIPImageProcessor

model_dir = "OPPOer/AndesVL-4B-Instruct"

model = AutoModel.from_pretrained(model_dir, trust_remote_code=True,torch_dtype=torch.bfloat16).cuda()
tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
image_processor = CLIPImageProcessor.from_pretrained(model_dir, trust_remote_code=True)

messages = [
                {
                    "role": "user",
                    "content": [
                        {"type": "text", "text": "描述这张图片。"},
                        {
                            "type": "image_url",
                            "image_url": {
                                "url": "https://i-blog.csdnimg.cn/blog_migrate/2f4c88e71f7eabe46d062d2f1ec77d10.jpeg" # image/to/path
                            },
                        }
                    ],
                },
        ]
response = model.chat(messages, tokenizer, image_processor, max_new_tokens=1024, do_sample=True, temperature=0.6)
print(response)

Inference using the thinking version of AndesVL

# require transformers>=4.52.4
import torch
from transformers import AutoModel, AutoTokenizer, CLIPImageProcessor

model_dir = "OPPOer/AndesVL-4B-Thinking"
model = AutoModel.from_pretrained(model_dir, trust_remote_code=True,torch_dtype=torch.bfloat16).cuda()
tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
image_processor = CLIPImageProcessor.from_pretrained(model_dir, trust_remote_code=True)
messages = [
                {
                    "role": "user",
                    "content": [
                        {"type": "text", "text": "描述这张图片。"},
                        {
                            "type": "image_url",
                            "image_url": {
                                "url": "https://i-blog.csdnimg.cn/blog_migrate/2f4c88e71f7eabe46d062d2f1ec77d10.jpeg" # image/to/path
                            },
                        }
                    ],
                },
        ]
response = model.chat(messages, tokenizer, image_processor, max_new_tokens=1024, do_sample=True, temperature=0.6, Thinking=True)
print(response)

📚Evaluation

Setup essential keys

To infer with API models (GPT-4v, Gemini-Pro-V, etc.) or use LLM APIs as the judge or choice extractor, you need to first setup API keys.You can place the required keys in $VLMEvalKit/.env or directly set them as the environment variable.

Environment

Prepare the environment, install the required libraries:

$ cd AndesVL_Evaluation
$ conda create --name AndesVL_Evaluation python==3.11
$ conda activate AndesVL_Evaluation
$ pip install -r requirements.txt

Run evaluation

Evaluation script for thinking models using vLLM

bash scripts/run_andesvl_thinking.sh

Evaluation script for instruct models using vLLM

bash scripts/run_andesvl_instruct_vllm.sh

Evaluation script for instruct models using deepspeed

bash scripts/run_andesvl_general.sh

Performance Discrepancies

Model performance may vary across different environments. These differences could be attributed to variations in versions of libraries such as transformers, cuda, and torch.

If you encounter unexpected performance, we recommend first reviewing the local generation records ({model}_{dataset}.xlsx) or the evaluation records ({model}_{dataset}_{judge_model}.xlsx). This may help you better understand the evaluation outcomes and identify potential issues.

Acknowledgement

The project build upon VLMEvalKit, which you can visit for relevant details.

Citation

If you find our work helpful, feel free to give us a cite.

@misc{jin2025andesvltechnicalreportefficient,
      title={AndesVL Technical Report: An Efficient Mobile-side Multimodal Large Language Model}, 
      author={AndesVL Team, OPPO AI Center},
      year={2025},
      eprint={2510.11496},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2510.11496}, 
}