FingerCap: Fine-grained Finger-level Hand Motion Captioning

This repository contains the code for the paper "FingerCap: Fine-grained Finger-level Hand Motion Captioning" (ECCV 2026). FingerCap is a benchmark for fine-grained hand and finger motion captioning from videos.

Dataset

The FingerCap dataset is available on Baidu Cloud Drive:

• Download Link: https://pan.baidu.com/s/1v6W6T05unTkVLB4J96l5dg
• Extraction Code: ywc7

Dataset Structure

After downloading, extract the dataset and organize it as follows:

data/
├── test/
│   ├── annotations.json
│   └── videos/
│       ├── video_001.mp4
│       ├── video_002.mp4
│       └── ...

annotations.json format:

[
  {
    "video_name": "video_001.mp4",
    "caption": "Right hand grips a stack of cards while the left hand rubs the cards, fanning them out.",
    "data_source": "FingerCap",
    "view": "TPV"
  }
]

Installation

Prerequisites

• Python 3.9+
• Conda (recommended)

Step 1: Create Environment

conda create -n fingercap python=3.11
conda activate fingercap

Step 2: Install LLaMA-Factory

For open-source model inference (Qwen3-VL), we use LLaMA-Factory:

git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e ".[torch,metrics]" --no-build-isolation
cd ..

Step 3: Install FingerCap

git clone <this-repo-url>
cd FingerCap
pip install -e .

Step 4: Install BLEURT (for evaluation)

git clone https://github.com/google-research/bleurt.git
cd bleurt
pip install .
wget https://storage.googleapis.com/bleurt-oss-21/BLEURT-20.zip
unzip BLEURT-20.zip
cd ..

Step 5: Set API Keys

For Gemini (closed-source model) and HandJudge evaluation, set the following environment variables:

export GEMINI_API_KEY="your_gemini_api_key"
export OPENAI_API_KEY="your_openai_api_key"

Step 6: Verify Installation

fingercap --help

Supported Models

Model	Type	Description
qwen3-vl-8b-instruct	Open-source	Qwen3-VL 8B (runs locally via LLaMA-Factory)
gemini-2.5-pro	Closed-source	Google Gemini 2.5 Pro (API-based)

Quick Start

Inference

Run inference using YAML configuration files:

# Open-source model (Qwen3-VL)
fingercap inference configs/inference/qwen3.yaml

# Closed-source model (Gemini)
fingercap inference configs/inference/gemini.yaml

Resume interrupted inference:

fingercap inference configs/inference/qwen3.yaml --resume

Skip rephrasing step (use raw predictions):

fingercap inference configs/inference/qwen3.yaml --no-rephrase

Evaluation

Run evaluation on inference results:

# Evaluate with BLEU, BLEURT, and HandJudge
fingercap evaluation configs/evaluation/qwen3.yaml --use-bleu --use-bleurt --use-handjudge

# Evaluate Gemini results
fingercap evaluation configs/evaluation/gemini.yaml --use-bleu --use-bleurt --use-handjudge

Resume interrupted evaluation:

fingercap evaluation configs/evaluation/qwen3.yaml --use-bleu --use-bleurt --use-handjudge --resume

Evaluate raw predictions (without rephrasing):

fingercap evaluation configs/evaluation/qwen3.yaml --use-bleu --no-rephrase

Configuration

Inference Configuration

Example: configs/inference/qwen3.yaml

model: qwen3-vl-8b-instruct

model_config:
  chat_args:
    temperature: 0.2
    top_p: 0.9

data:
  input_file: ./data/test/annotations.json
  input_videos_dir: ./data/test/videos
  output_dir: ./outputs

resume: false
rephrase_model: gpt-4o-mini

Evaluation Configuration

Example: configs/evaluation/qwen3.yaml

model: qwen3-vl-8b-instruct

data:
  input_file: ./data/test/annotations.json
  input_videos_dir: ./data/test/videos
  output_dir: ./outputs

resume: false
use_bleu: false
use_bleurt: false
use_handjudge: false
handjudge_model: gpt-4o-mini
bleurt_checkpoint: ./bleurt/BLEURT-20

Output Files

Inference Output

• Predictions file: outputs/{model}_predictions.json

Format:

[
  {
    "video_name": "video_001.mp4",
    "caption": "Reference caption...",
    "original_prediction": "Raw model output...",
    "prediction": "Rephrased prediction..."
  }
]

Evaluation Output

• Results file: outputs/{model}_evaluation_results.json

Format:

[
  {
    "video_name": "video_001.mp4",
    "caption": "Reference caption...",
    "prediction": "Model prediction...",
    "bleu": 0.4521,
    "bleurt": 0.7821,
    "HandJudge_details": {
      "finger": 4.0,
      "motion": 3.5,
      "contact": 4.5,
      "completeness": 4.0,
      "explanation": "..."
    }
  }
]

Evaluation Metrics

Metric	Description
BLEU	N-gram overlap between prediction and reference
BLEURT	Learned metric for semantic similarity
HandJudge	Fine-grained evaluation across 4 dimensions: finger identification, motion trajectory, contact/interaction, and motion completeness (0-5 scale each)

Project Structure

FingerCap/
├── handsmotion/           # Core package
│   ├── __init__.py
│   ├── cli.py             # CLI entry point
│   ├── config.py          # Configuration parser
│   ├── inference.py       # Inference module
│   ├── evaluation.py      # Evaluation module
│   └── prompt.py          # Prompt templates
├── configs/               # Configuration files
│   ├── inference/         # Inference configs
│   │   ├── qwen3.yaml
│   │   └── gemini.yaml
│   └── evaluation/        # Evaluation configs
│       ├── qwen3.yaml
│       └── gemini.yaml
├── data/                  # Dataset directory
├── outputs/               # Output directory
├── setup.py
├── requirements.txt
└── README.md

Citation

If you find this work useful, please cite our paper:

@article{shen2025fingercap,
  title={FingerCap: Fine-grained Finger-level Hand Motion Captioning},
  author={Shen, Xin and Zhu, Rui and Shen, Lei and Wang, Xinyu and Zhang, Kaihao and Zhu, Tianqing and Wu, Shuchen and Miao, Chenxi and Li, Weikang and Li, Yang and others},
  journal={arXiv preprint arXiv:2511.16951},
  year={2025}
}

License

This project is licensed under the MIT License.