GazeVLA: Learning Human Intention for Robotic Manipulation

This is the official code repository for "GazeVLA: Learning Human Intention for Robotic Manipulation," accepted at the 2026 Conference on Robot Learning (CoRL).

We propose a novel learning-from-human framework that explicitly models intention to capture the causal structure of manipulation behavior.

📋 Table of Contents

• 🛠 Environment Setup
• 🧩 Model Architecture
• 💡 Experiments
• 🙏 Acknowledgements
• ✍️ Citation

🛠 Environment Setup

Step 1: Clone the Repository

git clone https://github.com/lichy2004/GazeVLA-Code.git GazeVLA
cd GazeVLA

Step 2: Set Up Python Environment

# Create a conda environment
conda create -n gazevla python=3.11 -y
conda activate gazevla

# Install requirements
pip install -r requirements.txt

# Install openpi-client
cd packages/openpi-client
pip install -e .
cd ../..

# Install GazeVLA
pip install -e .

🧩 Model Architecture

Our model receives a task description, an egocentric observation, and the human or robot state as inputs. It first predicts discrete intention tokens, followed by continuous action generation via an intention–action reasoning chain. By explicitly modeling intention as an intermediate representation, the framework bridges high-level task understanding and low-level control. We instantiate intention as gaze, parameterized as 2D image coordinates.

💡 Experiments

📦 Data Preparation

Download the AV-ALOHA dataset and place it in ./data/av-aloha/dataset. The directory structure should look like this:

./data/av-aloha
├── av_aloha_sim_cube_transfer
├── av_aloha_sim_peg_insertion
├── av_aloha_sim_slot_insertion
├── av_aloha_sim_hook_package
├── av_aloha_sim_pour_test_tube
└── av_aloha_sim_thread_needle

Then run examples/process_data_1.py and examples/process_data_2.py to process the data: the first step converts the raw data into HDF5 files, and the second step saves the results in RLDS format.

🔥 Training

Please run the following command for training, and you can modify the configuration in src/openpi/training/config.py.

python scripts/compute_norm_stats.py --config-name lfa-cam1-chunk25
CUDA_VISIBLE_DEVICES=0 \
    torchrun --nnodes=1 --nproc_per_node=1 --master_port=29500 \
    scripts/train/train.py lfa-cam1-chunk25 \
    --exp_name=lfa-cam1-chunk25

🧪 Evaluation

You need to first download the official LFA repository, make some modifications. We will open-source this part of the code in the future.

TODO

The following features are planned for future implementation:

• Training code.
• Pre-trained model checkpoints.
• Evaluation scripts.

🙏 Acknowledgements

Our code is built upon Openpi and GraspVLA. These code serve as an essential foundation for our implementation, and we deeply appreciate the time, effort, and expertise they shared with the community.

✍️ Citation

If you find our work useful, please cite us:

@article{li2026gazevla,
  title={GazeVLA: Learning Human Intention for Robotic Manipulation}, 
  author={Li, Chengyang and Xiong, Kaiyi and Xu, Yuan and Qian, Lei and Wang, Yizhou and Zhu, Wentao},
  journal={arXiv preprint arXiv:2604.22615},
  year={2026}
}

License

This work and the dataset are licensed under CC BY-NC 4.0.