GazeVLA: Learning Human Intention for Robotic Manipulation

This is the official data processing repository for "GazeVLA: Learning Human Intention for Robotic Manipulation," accepted at the 2026 Conference on Robot Learning (CoRL).

This is the human data processing script used in the GazeVLA paper. It supports hand motion alignment and 2D & 3D visualization for egocentric human datasets, and additionally integrates latest datasets including xperience-10m, VITRA, and EgoScale. All data has been standardized and unified into a consistent format, aiming to contribute to the research community.

📋 Table of Contents

• 🛠 Environment Setup
• 💡 Example
• 🙏 Acknowledgements
• ✍️ Citation

🛠 Environment Setup

Step 1: Clone the Repository

git clone git@github.com:lichy2004/GazeVLA-Data.git GazeVLA-Data
cd GazeVLA-Data

Step 2: Set Up Python Environment

# Create a conda environment
conda create -n gazevla_data python=3.11 -y
conda activate gazevla_data

# Install requirements
pip install -r requirements.txt

# You can try installing PyTorch3D using the following method
wget https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch3d/linux-64/pytorch3d-0.7.5-py38_cu117_pyt201.tar.bz2
conda install -y --use-local ./pytorch3d-0.7.5-py38_cu117_pyt201.tar.bz2

Step3: Download MANO model

Hand annotations in the MANO format can be downloaded after accepting their license agreement.

We requires the MANO_RIGHT.pkl and MANO_LEFT.pkl files for loading and rendering of hand poses. These files can be obtained from the mano_v1_2.zip file located in the "Models & Code" section of the MANO website. After downloading, extract the zip file to your local disk, and the *.pkl files can be found at the following path: assets/mano_v1_2/models.

💡 Example

Each dataset must first be downloaded from the official website with the correct file structure. Refer to DATA.md for instructions on downloading datasets and organizing the data. An example of data processing and visualization is shown below.

# processing
python src/dataset/EgoDex.py

# visualize
python examples/vis.py --file_path data/EgoDex/processed/00000000.hdf5

TODO

The following features are planned for future implementation:

• EgoDex
• HOT3D
• HoloAssist
• OAKINK2
• TACO
• HOI4D
• H2O
• xperience-10m
• VITRA
• EgoScale

🙏 Acknowledgements

Our visualization code is built upon Viser and our data processing scripts refer to the official code of each dataset.. These code serve as an essential foundation for our implementation, and we deeply appreciate the time, effort, and expertise they shared with the community.

✍️ Citation

If you find our work useful, please cite us:

@article{li2026gazevla,
  title={GazeVLA: Learning Human Intention for Robotic Manipulation}, 
  author={Li, Chengyang and Xiong, Kaiyi and Xu, Yuan and Qian, Lei and Wang, Yizhou and Zhu, Wentao},
  journal={arXiv preprint arXiv:2604.22615},
  year={2026}
}

License

This work and the dataset are licensed under CC BY-NC 4.0.