Gaze-to-Task Inference in Chart Reading

This repository is the official project page accompanying the paper "Gaze-to-Task Inference in Chart Reading: Best Practices for Integrating Human Attention with Multimodal LLMs" by Takumi Nishiyasu and Yoichi Sato.

Publication: Proceedings of the ACM on Computer Graphics and Interactive Techniques, Volume 9, Issue 2 (ETRA 2026), Article No.: 17, Pages 1–16.

DOI: https://doi.org/10.1145/3803536

Program page: https://programs.sigchi.org/etra/2026/program/content/253162

Table of Contents

• Overview
• Resources
  • Dataset
  • Prepared few-shot samples
  • Prompting and templates
  • Gaze representations
  • Data Formats
• Reproducing the Experiments
  • Environment Setup
  • Quick Start
  • Notes
• Citation

Overview

This repository provides the code, data preprocessing pipeline, prompt templates, and evaluation scripts to reproduce MLLM-based gaze-to-task inference experiments on chart images (ETRA 2026). See the sections below for resources and reproduction instructions.

Resources

Dataset

This project uses the Taskvis eye-tracking dataset:

• Taskvis dataset page: http://vgg.fiit.stuba.sk/2018-02/taskvis/
• Direct ZIP download: taskvis.zip

Expected dataset structure:

• taskvis/images/ - chart images
• taskvis/fixations/ - fixation logs
• taskvis/tasks.txt - task descriptions
• taskvis/tasktypes.txt - task type labels

Prompting and templates

• The prompt templates and Chain-of-Thought strategies used in our experiments are provided in automatic_prompt_templates/. We support two high-level CoT prompting strategies:
  • Bottom-up: Visualization Overview → Gaze Pattern Analysis → Question Selection
  • Top-down: Hypothesis Generation → Hypothesis Verification → Conclusion
• Templates are style-specific (e.g., default, colormap, saliency, bubbleview) and paired with corresponding few-shot samples for in-context learning.

Gaze representations

• We provide multiple visual encodings of gaze for use as image inputs to MLLMs, including heatmaps, bubbleviews, and saliency-style overlays. These representations are generated by the preprocessing pipeline (see preprocessing/ and output_gaze/ directories).
• Our experiments found the heatmap representation to be the most effective encoding for gaze-to-task inference; details are in the paper.

See preprocessing/ and automatic_prompt_templates/ for generation scripts and example templates.

Prepared few-shot samples

• A set of prepared few-shot samples (images, gaze visualizations, and metadata) used in the experiments will be made available from this project. The few-shot sample bundles are generated under preprocessing/fewshot_samples_same_images/. A downloadable archive or a direct link will be posted here soon.

Data Formats

Few-Shot Sample JSON

Actual few-shot files are generated under preprocessing/fewshot_json/ and include full metadata. The key fields used by prompt/prediction scripts are shown below.

[
  {
    "image_id": 20,
    "subject_id": "p05",
    "image_path": "path/to/sample1.png",
    "task_type": "A",
    "task_content": "What is the value of X?",
    "question_candidates": {
      "A": "What is the value of X?",
      "B": "Which items satisfy condition X?",
      "C": "Which item has the highest value?"
    }
  },
  {
    "image_id": 20,
    "subject_id": "p11",
    "image_path": "path/to/sample2.png",
    "task_type": "B",
    "task_content": "Which items satisfy condition Y?",
    "question_candidates": {
      "A": "What is the value of Y?",
      "B": "Which items satisfy condition Y?",
      "C": "What is the maximum value?"
    }
  }
]

Prediction Output JSON

[
  {
    "image_id": 1,
    "subject_id": "subject1",
    "task_type": "A",
    "predicted_task_type": "A",
    "generated_text": "[Step 0: ...]\n[Step 1: ...]\n[ANSWER]: A",
    "image_path": "path/to/image.png"
  }
]

Reproducing the Experiments

Environment Setup

Requirements

• NVIDIA GPU with a CUDA-capable driver (compatible with CUDA 12.6)
• Singularity or Apptainer (required for the container workflow)
• Sufficient disk space to build and store the .sif image

The container defined in env.def provisions the following runtime automatically:

• Ubuntu 22.04 base image (nvidia/cuda:12.6.2-base-ubuntu22.04)
• Miniconda environment: scanpath_analysis (Python 3.10)
• PyTorch stack: torch==2.7.0, torchvision==0.22.0, torchaudio==2.7.0
• Additional Python dependencies from requirements.txt

Singularity (Ubuntu 22.04, CUDA 12.6)

# Build the container image
sudo singularity build mllm_gaze.sif env.def

# Start a GPU-enabled shell
singularity shell --nv mllm_gaze.sif

# Optional sanity check inside the container
# source /opt/conda/etc/profile.d/conda.sh
# conda activate scanpath_analysis
# python -c "import torch; print(torch.cuda.is_available())"

Quick Start (Reproduce the Pipeline)

Run all commands from the project root unless stated otherwise.

1) Preprocessing and Prompt Preparation

cd preprocessing
bash run_preprocess.sh

This stage runs the end-to-end preprocessing pipeline:

• gaze visualization generation
• few-shot sample image preparation
• metadata generation (gaze_metadata_new.json)
• CoT prompt generation (bottom-up and top-down templates)

Key output directories include:

• output_gaze/, output_gaze_official/
• fewshot_samples_same_images/{style}/
• fewshot_prompts/

2) Run MLLM Predictions

# GPT-4
bash scripts/run_pred_qtype_gpt4.sh

# Claude
bash scripts/run_pred_qtype_claude.sh

# Gemini
bash scripts/run_pred_qtype_gemini.sh

# Top-down CoT variants
bash scripts/run_for_topdowncot_gpt4.sh
bash scripts/run_for_topdowncot_gemini25flash.sh

To inspect available prediction runners:

ls scripts/run_for_*.sh
ls scripts/run_pred_*.sh

3) Calculate Accuracy Scores

bash scripts/run_calculate_scores.sh

This script auto-detects available prediction directories (for example, predictions_topdown_rebuttal).

4) Analyze Prediction Distributions (Optional)

python3 utils/analyse_predictions.py

5) Generate Figures

python3 utils/make_charts_for_results.py

Notes

• Verify few-shot sample image paths before running prediction scripts.
• Configure API keys in api_key.py before running OpenAI-dependent steps.
• If running utils/calculate_accuracy.py directly, add --fewshot_dir preprocessing/fewshot_samples_same_images when needed.

Citation

@article{nishiyasu2026gaze,
  title={Gaze-to-Task Inference in Chart Reading: Best Practices for Integrating Human Attention with Multimodal LLMs},
  author={Nishiyasu, Takumi and Sato, Yoichi},
  journal={Proceedings of the ACM on Computer Graphics and Interactive Techniques},
  volume={9},
  number={2},
  pages={1--16},
  year={2026},
  publisher={ACM New York, NY, USA}
}