AnyGroundBench

AnyGroundBench: A Specialized-Domain Benchmark for Video Grounding in Vision-Language Models

Rintaro Otsubo*^1,2 · Ryo Fujii*^1,2 · Reina Ishikawa^1,2 · Taiki Kanaya^1,2 · Kanta Sawafuji^1,2
Hiroki Kajita^1,3 · Shigeki Sakai^1,3 · Hideo Saito^1,2 · Ryo Hachiuma⁴

_{¹ Keio University   ² Keio AI Research Center   ³ Keio University School of Medicine   ⁴ NVIDIA}

_{* Equal contribution.}

AnyGroundBench is a domain-adaptation benchmark for evaluating video grounding in vision-language models across specialized domains with dense temporal, spatial, and spatio-temporal annotations.

Overview

AnyGroundBench evaluates how vision-language models adapt to specialized-domain video grounding across animal, industry, sports, surgery, and public-security scenarios using unified temporal, spatial, and spatio-temporal annotations.

Datasets

AnyGroundBench is organized by domain at runtime. Each domain is built from one or more source datasets:

Dataset	Source type	Domain	Videos / source
mouse	Newly captured	Animal	Hugging Face
animal_kingdom	Existing dataset	Animal	Original source
enigma	Existing dataset	Industry	Original source
meccano	Existing dataset	Industry	Original source
dota	Existing dataset	Public security	Original source
uca	Existing dataset	Public security	Original source
american_football	Newly captured	Sports	Hugging Face
multisports	Existing dataset	Sports	Original source
cholectrack20	Existing dataset	Surgery	Original source
egosurgery	Existing dataset	Surgery	Original source

The source dataset names describe where the videos come from. The benchmark runtime reads the merged domain-level data under data/<domain_name>/.

Data Preparation

The runtime layout is domain-level:

data/<domain_name>/
├── meta-data
│   ├── t_train.json
│   ├── t_test.json
│   ├── s_train.json
│   ├── s_test.json
│   ├── st_train.json
│   └── st_test.json
├── simrank
│   └── video_top100_alpha<alpha>.json
└── videos
    ├── clips
    │   └── <video_name>.mp4
    └── clips4spatial
        └── <clip_name>.mp4

The meta-data/ and simrank/ files are included in the AnyGroundBench Hugging Face release. Source videos must be reconstructed into data/<domain_name>/videos/ using the domain-specific preparation scripts.

Follow the corresponding preparation guide:

Domain	Preparation guide
animal	create_anygroundbench/animal/README.md
industry	create_anygroundbench/industry/README.md
safety	create_anygroundbench/safety/README.md
sports	create_anygroundbench/sports/README.md
surgery	create_anygroundbench/surgery/README.md

General workflow:

1. Download the AnyGroundBench release files under data/<domain_name>/.
2. Download the required original source videos for the target domain.
3. Follow the corresponding create_anygroundbench/<domain_name>/README.md guide above to reconstruct videos/clips.
4. Run the domain spatial clip script to create videos/clips4spatial.

Do not use data/<source_dataset>/ as the runtime input. Source dataset directories such as data/animal_kingdom/, data/MECCANO/, or data/DoTA/ are raw inputs for the reconstruction scripts. The benchmark runtime reads data/<domain_name>/....

Inference

Run inference with test.py by specifying a task, domain, and model.

For API-based models, copy .env.example to .env and set credentials:

GEMINI_API_KEY=<your_gemini_api_key>

Gemini 2.5 Flash zero-shot:

uv run --project envs/gemini python test.py --task temporal --domain_name animal --model_name gemini --model_id gemini-2.5-flash --n_shot 0

Gemini 2.5 Flash 2-shot:

uv run --project envs/gemini python test.py --task temporal --domain_name animal --model_name gemini --model_id gemini-2.5-flash --n_shot 2 --alpha 0.5

Qwen-3VL-8B zero-shot:

uv run --project envs/qwen python test.py --task temporal --domain_name animal --model_name qwen3 --model_id Qwen/Qwen3-VL-8B-Instruct --n_shot 0

Qwen-3VL-8B 2-shot:

uv run --project envs/qwen python test.py --task temporal --domain_name animal --model_name qwen3 --model_id Qwen/Qwen3-VL-8B-Instruct --n_shot 2 --alpha 0.5

See scripts/inference_0shot.sh and scripts/inference_2shot.sh for full command lists across tasks, domains, and models.

Evaluation

Evaluation examples are provided in scripts/eval/.

The evaluation scripts infer the domain from each prediction filename. Use domain suffixes such as animal, industry, sports, surgery, or safety in the prediction JSON filename. For example, evaluate an animal-domain inference output as results/<task>/<timestamp>_animal.json.

Temporal Grounding evaluation examples

# Gemini 2.5 Flash temporal grounding
uv run --project envs/download scripts/eval/run_temporal_grounding_bundle.py \
  --model gemini-2.5-flash \
  --pred-jsons results/temporal/<timestamp>_animal.json

# Qwen-3VL-8B temporal grounding
uv run --project envs/download scripts/eval/run_temporal_grounding_bundle.py \
  --model qwen3-vl-8b \
  --pred-jsons results/temporal/<timestamp>_animal.json

Spatial grounding evaluation examples

# Gemini 2.5 Flash spatial grounding
uv run --project envs/download scripts/eval/run_spatio_temporal_grounding_bundle.py \
  --task spatial \
  --model gemini-2.5-flash \
  --pred-jsons results/spatial/<timestamp>_animal.json

# Qwen-3VL-8B spatial grounding
uv run --project envs/download scripts/eval/run_spatio_temporal_grounding_bundle.py \
  --task spatial \
  --model qwen3-vl-8b \
  --pred-jsons results/spatial/<timestamp>_animal.json

Spatio-Temporal Grounding evaluation examples

# Gemini 2.5 Flash spatio-temporal grounding
uv run --project envs/download scripts/eval/run_spatio_temporal_grounding_bundle.py \
  --task spatio_temporal \
  --model gemini-2.5-flash \
  --pred-jsons results/spatio_temporal/<timestamp>_animal.json

# Qwen-3VL-8B spatio-temporal grounding
uv run --project envs/download scripts/eval/run_spatio_temporal_grounding_bundle.py \
  --task spatio_temporal \
  --model qwen3-vl-8b \
  --pred-jsons results/spatio_temporal/<timestamp>_animal.json

License

The newly captured mouse and american_football data are under CC BY-NC-SA 4.0.

Third-party datasets remain governed by their original licenses and terms. Users are responsible for complying with the original dataset providers' licenses, data use agreements, and redistribution restrictions.

Citation

@misc{otsubo2026anygroundbenchspecializeddomainbenchmarkvideo,
      title={AnyGroundBench: A Specialized-Domain Benchmark for Video Grounding in Vision-Language Models},
      author={Rintaro Otsubo and Ryo Fujii and Reina Ishikawa and Taiki Kanaya and Kanta Sawafuji and Hiroki Kajita and Shigeki Sakai and Hideo Saito and Ryo Hachiuma},
      year={2026},
      eprint={2607.02269},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.02269},
}

Contact

For questions contact at rintarootsubo@keio.jp.