measurement-reliability-naturalistic-reading

Code for analyzing measurement reliability of naturalistic reading data

Data

Simulation data

The simulation data is stored in the simulation folder.

Empirical data

The empirical data is stored in the data/indico folder. The InDiCo data is downloaded from here: https://drive.switch.ch/index.php/s/faxvHpFurfjKSfo/download.

Code

Surprisal computation

We recomputed the surprisal values for the InDiCo data using the script src/surprisal.ipynb (need to fetch it from the server which is currectly down), with the Hugging Face model benjamin/gerpt2.

Data analysis scripts

The data analysis scripts are stored in the r-script folder, including the scripts for running models: correlation_estimate.Rmd, simulation scripts: simulation.Rmd, and visualization scripts: visualization.Rmd.

Results

Models

We store the models in the models folder and for simulations, we store the models in the simulation/seed123|100|1000_sd0025tau05/models folder. The folder name is a combination of the simulation settings.

It can take weeks to run all the models. You can directly download the ready-for-analysis models from : our OSF repository

Note: there are some heavy formulas in Section 2 and in the Appendix in the corresponding paper. We also store the intermediate steps in our OSF repository in the proof folder for readers who are interested in the proofs.

plots

We store the plots in the visualization folder.

Intermediate data

We store the intermediate data in the stats folder.

License

This project is licensed under the MIT License - see the LICENSE file for details.