[ETRA 2026] Learning Alignments of Human Gaze and Fine-grained Task Descriptions

Paper: Learning Alignments of Human Gaze and Fine-grained Task Descriptions

This repository provides the training and evaluation scripts of GTANet.

Prerequisites

• NVIDIA-SMI 550.54.15
• Driver Version: 550.54.15
• CUDA Version: 12.4
• python3.10+
• uv 0.9+

1. Environment Setup

Build the environment from the repository root:

cd /home/nishiyasu/git/GTANet
bash setup_uv.sh
source .venv/bin/activate
export PYTHONNOUSERSITE=1

2. Dataset Preparation

GTANet uses two datasets for compatibility prediction: MHUG and AiR.

• MHUG should be available under ../work/nishiyasu/VQA_MHUG_dataset/
• AiR should be available under ../work/nishiyasu/AiR_dataset/

3. Generate Dummy Gaze Data

Run the dummy gaze generation script:

cd compatibility_prediction/gen_dummy_gaze
bash run_gen_dummy_gaze.sh

This script generates gaze dictionaries for both datasets and all splits:

• generated_dummy_gaze/dummy_gaze_air_train.json
• generated_dummy_gaze/dummy_gaze_air_validation.json
• generated_dummy_gaze/dummy_gaze_air_test.json
• generated_dummy_gaze/dummy_gaze_mhug_train.json
• generated_dummy_gaze/dummy_gaze_mhug_validation.json
• generated_dummy_gaze/dummy_gaze_mhug_test.json

4. Generate Dummy Questions

4.1 Configure API key

Set your Gemini API key in:

• compatibility_prediction/gen_dummy_questions/api_key.py

You can use api_key_template.py as a template.

4.2 Run Gemini generation

cd compatibility_prediction/gen_dummy_questions
bash run_gemini_for_dummy_question_air.sh
bash run_gemini_for_dummy_question_mhug.sh

This creates CSV files in generated_dummy_questions_tmp/.

4.3 Convert and post-process generated questions

bash gen_dummy_question.sh

gen_dummy_question.sh runs the following pipeline:

1. python3 extract_dummy_questions.py
2. Move extracted_dummy_questions/tmp_air_*.json to air_questions/
3. Move extracted_dummy_questions/tmp_mhug_*.json to mhug_questions/
4. python3 add_random_q.py --add_random_num 19 --dataset_types train validation test --difficulty middle

Final files used by training include:

• air_questions/air_train_middle.json
• air_questions/air_validation_middle.json
• air_questions/air_test_middle.json
• mhug_questions/mhug_train_middle.json
• mhug_questions/mhug_validation_middle.json
• mhug_questions/mhug_test_middle.json

5. Train and Evaluation

Run from compatibility_prediction:

cd compatibility_prediction
bash run_train_test.sh

The script runs training and test for both datasets (air, mhug) with --train_data_mode middle, and evaluates with --test_data_mode easy and --test_data_mode middle.

Citation

Takumi Nishiyasu, Zhiming Hu, Andreas Bulling, and Yoichi Sato, "Learning Alignments of Human Gaze and Fine-grained Task Descriptions," to appear in ACM Symposium on Eye Tracking Research and Applications (ETRA 2026), June 2026.