AlEYEgnment

This repository contains the code for the paper

“AlEYEgnment: Leveraging Eye-Tracking-while-Reading to Align Language Models with Human Preferences”

All the eye-tracking data used for models training is available on OSF: https://osf.io/367qz/

---

Repository structure

training the models

Folder: dpo_training

Dataset construction scripts

• aggregated pairs dataset creation.py
  Creates pairwise dataset where eye-tracking features are aggregated across readers.

• random pairs dataset creation.py
  Samples grammatical–ungrammatical pairs to randomly pair the eye-tracking features across participants.

Sentence-level data

• dataset_with_labels_v3.csv
  sentence-level eye-tracking features

• sentences_labels.csv
  sentence id - grammaticality label mapping.

Training scripts

training different variants of the model:

• mistral train abl cos.py
  Ablation experiment with a cosine-similarity-based additional loss term.

• mistral train ET all feat.py
  Gaze-augmented DPO with all eye-tracking features.

• mistral train ET count feat.py
  Gaze-augmented DPO using features based on fixation counts.

• mistral train text.py
  Text-only DPO training script (no gaze loss).

• sft_train_text.py
  Supervised fine-tuning baseline on text.

synthetic data generaration, processing and training the model augmented with the synthetic data

Folder: synthetic_data

This folder contains the code and data needed for generating synthetic scanpaths for CoLA sentences and training the model using reading measures computed on these scanpaths.

Checkpoints & configs

• CELoss_COLA_tuned_text_eyettention_*.pth
  Trained Eyettention checkpoint used to generate synthetic scanpaths for the CoLA sentences.

• cola_features_norm.json
  Normalisation statistics for CoLA features used to train the Eyettention model.

Synthetic scanpaths → features → DPO datasets

• synthetic_data.csv
  Raw synthetic scanpaths generated for CoLA sentences.

• synth_scanpaths_generate.py
  Script that loads the Eyettention checkpoint and generates the synthetic scanpaths, saving them to synthetic_data.csv.

• synth_read_meas.py
  Script that converts raw synthetic scanpaths into eye-tracking measures and writes them to computed_et_fix_features.csv.

• computed_et_fix_features.csv
  Sentence-level synthetic eye-tracking features derived from the synthetic scanpaths.

• sentences_labels.csv
  CoLA sentences and their grammaticality labels for the subset used in the synthetic experiments.

• pairs_synth_dataset_create.py
  Script that builds grammatical–ungrammatical sentence pairs with the corresponding eye-tracking measures.

• dpo_synthetic_data.csv
  Pairwise DPO training data with synthetic gaze features and label.

• dpo_dataset_count_feat_train_synth.json
  Data in the format used to train the model (training part).

• dpo_dataset_count_feat_validation_synth.json
  Data in the format used to train the model (validation part).

• model.py
  Eyettention model.

• mistral_train_ET_fix_synth.py
  Main training script for gaze-augmented DPO with synthetic features.

evaluating the models

Folder: dpo_evaluation

• eval_datatset_construct.py
  Construct datasets for model evaluation.
• eval_mistral.py
  Evaluate the models