v1: Learning to Point Visual Tokens
for Multimodal Grounded Reasoning

Jiwan Chung^*  Junhyeok Kim^*  Siyeol Kim  Jaeyoung Lee  Minsoo Kim  Youngjae Yu

🎉 v1 is accepted at COLM 2026!

Installation

conda create -n v1 python=3.10 -y
conda activate v1
pip install -r requirements.txt
pip install flash-attn --no-build-isolation

Demo

Gradio Web UI

Highly Recommended as the copy tokens are displayed on image.

python run_gradio.py

Inference

python inference.py

The script uses a default image URL and text prompt. To use your own inputs, you can modify the image variable within the messages list and the text field for the user prompt.

Training dataset: v1g

v1g is the dataset used to train the released v1-7B model. It contains 312,373 long-form multimodal reasoning traces for mostly mathematical visual problems. The responses interleave natural-language reasoning with detect(...) calls that connect referenced objects to bounding boxes in the image.

Property	Value
Examples	312,373
Storage	74 Parquet shards (~33GB download)
Image data	PNG bytes embedded in each row
Dialogue	ShareGPT-style human question and gpt grounded reasoning trace
Grounding	JSON mapping from <|objK|> pointer tokens to [x1, y1, x2, y2] boxes

Each row has the following fields:

Field	Description
id	Unique example identifier
image	Decoded problem image
conversations	Question and grounded reasoning response
regions	JSON-encoded pointer-token-to-box mapping
image_size	[width, height] coordinate space used by regions
num_tokens	Approximate trace length

Load the full training split with Hugging Face Datasets:

from datasets import load_dataset

dataset = load_dataset("kjunh/v1g", split="train")
example = dataset[0]
print(example["id"], example["image"].size, example["image_size"])

The images stay inside the Parquet data, so no JSON or PNG materialization step is required. Hugging Face caches the downloaded shards locally; set HF_HOME to place the ~33GB cache on a scratch disk. A 100-item sample is available for inspecting the schema or running a quick data-path check.

The v1g reasoning traces and grounding annotations are released under Apache-2.0. The embedded images retain the licenses and terms of their source datasets; see the dataset card for provenance details.

Training

Install the training extras on top of the base requirements:

pip install -r requirements-train.txt

By default, train.py loads kjunh/v1g directly from the Hub and adapts its ShareGPT conversations and grounding regions lazily during training.

Launch training (8×GPU with DeepSpeed ZeRO-3, the configuration used for the released v1-7B):

bash train.sh

Small-data debug run using the public 100-item sample:

python train.py --data kjunh/v1g-sample --debug --max-steps 3 --batch-size 1

Reproducibility notes

• The released v1-7B checkpoint was trained with transformers==4.50.0 (the pin in requirements-train.txt); newer versions may not be compatible with the bundled v1/ modeling code.
• Effective hyperparameters of the released run: lr 3e-5 (linear schedule, warmup ratio 0.03), per-device batch 2 × grad-accum 4 × 8 GPUs, bf16, gradient checkpointing, max grad norm 0.5, z_loss_weight 1e-5, max sequence length 8192, images capped at 672px (longer side).
• Data ordering of the released run corresponds to the HF Trainer default shuffling seed (42), independent of the --seed flag value used at launch time.

Release status

• Inference code
• Training data sample
• Training data
• Training code

License

This project is released under the Apache License 2.0.

Citation

If you find our work valuable, please cite:

@misc{chung2025v1learningpointvisual,
      title={v1: Learning to Point Visual Tokens for Multimodal Grounded Reasoning}, 
      author={Jiwan Chung and Junhyeok Kim and Siyeol Kim and Jaeyoung Lee and Min Soo Kim and Youngjae Yu},
      year={2025},
      eprint={2505.18842},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.18842}, 
}

(The COLM 2026 proceedings BibTeX will replace this arXiv entry once available.)