HORIZON: A Benchmark for In-the-wild User Behaviour Modeling

Official repository for HORIZON, a large-scale benchmark for evaluating user behavior modeling in realistic, long-horizon, cross-domain settings.

📄 ACL 2026 Paper: https://aclanthology.org/2026.findings-acl.1503/
📄 arXiv: https://arxiv.org/abs/2604.17259

---

Overview

Understanding user behavior requires models that generalize across domains, time, and diverse interaction histories. Existing benchmarks primarily focus on single-domain recommendation or next-item prediction, making them poor proxies for real-world personalization systems.

HORIZON addresses this gap by reformulating the public Amazon Reviews 2023 dataset into a standardized benchmark for evaluating modern user modeling systems under realistic deployment settings.

The benchmark enables evaluation across:

• 54M users
• 35M products
• Cross-domain interaction histories
• Long-horizon sequential user modeling
• Temporal generalization across future interactions
• In-distribution and out-of-distribution evaluation
• Multiple user modeling tasks beyond conventional next-item prediction

Rather than releasing any new data, this repository provides reproducible scripts and guidelines for reconstructing the benchmark from publicly available resources, enabling transparent and reproducible research.

---

Highlights

• Large-scale benchmark for realistic user behavior modeling
• Built from the public Amazon Reviews 2023 dataset
• Cross-domain user histories spanning 33 product categories
• Long-horizon sequential recommendation setting
• Temporal train/validation/test splits
• Evaluation on both in-distribution and out-of-distribution users
• Reproducible benchmark construction pipeline
• Baseline implementations and evaluation protocols from our ACL 2026 paper

---

Citation

If you use HORIZON in your research, please cite:

@misc{goel2026horizonbenchmarkinthewilduser,
      title={HORIZON: A Benchmark for In-the-wild User Behaviour Modeling}, 
      author={Arnav Goel and Pranjal A Chitale and Bhawna Paliwal and Bishal Santra and Amit Sharma},
      year={2026},
      eprint={2604.17259},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2604.17259}, 
}

---

Objective

Our objective is to provide a standardized benchmark for evaluating user modeling algorithms under realistic temporal and cross-domain settings.

---

Audience

HORIZON is intended for researchers, AI practitioners, and industry professionals working on:

• User modeling
• Sequential recommendation
• Personalized retrieval
• Large language models for recommendation
• Representation learning for user behavior

---

Intended Uses

HORIZON is designed as a standardized evaluation platform for benchmarking both existing and newly proposed user modeling algorithms.

The benchmark is most relevant for applications involving products similar to those contained in the Amazon Reviews 2023 dataset. The benchmark spans 33 product categories, available here: Link

---

Out-of-Scope Uses

HORIZON is not intended:

• to circumvent policies adopted by LLM providers;
• to evaluate the safety or alignment of language models;
• as a direct proxy for online production performance.

The benchmark is specifically designed for e-commerce user modeling and may not directly transfer to other recommendation domains or user populations.

---

Evaluation

We evaluate a diverse collection of state-of-the-art user modeling algorithms on HORIZON. Experimental details, benchmark tasks, and baseline results are available in our ACL 2026 paper.

---

Limitations

• HORIZON provides an offline evaluation benchmark. Offline performance may differ from online deployment performance.
• The benchmark contains English-language products from Amazon Reviews 2023.
• Performance depends on the diversity and representativeness of the underlying source data.
• The benchmark focuses on e-commerce user behavior and is not intended to represent every recommendation setting.

---

Usage

This repository is intended primarily for research and experimental purposes.

Although HORIZON is built from large-scale real-world data, we strongly recommend additional validation before deploying algorithms evaluated on this benchmark in production environments.

---

Feedback and Collaboration

We welcome feedback, bug reports, and research collaborations.

If you have suggestions, questions, or would like to contribute, please open an Issue or submit a Pull Request.

---

HORIZON Benchmark Construction

a. Curating the Full Dataset:

The scripts for constructing the HORIZON benchmark are provided in the data folder. Follow the following steps to reproduce the benchmark:

1. Clone the repository:

gh repo clone microsoft/horizon-benchmark
cd horizon-benchmark

2. Create and activate the environment from the YAML file:

conda env create -f environment.yaml
conda activate <your-env-name>

3. Give necessary permissions to the bash scripts:

chmod +x running_phase_one.sh
chmod +x running_phase_two.sh
chmod +x running_phase_three_metadata.sh

4. Run the bash scripts to curate the dataset files:

./running_phase_one.sh
./running_phase_two.sh
./running_phase_three_metadata.sh

Summary of the Bash Scripts:

• The process is extremely RAM-intensive due to the massive size of the corpus being created and the multiprocessing/batching optimizations performed to make it efficient. If your system doesnt support it, consider tweaking the hyperparameters.
• Phase One is the process of retrieving the category-wise data from the Amazon Reviews 2023 open-source repository and storing it in category-wise parquet files
• Phase Two is the process of merging these category-wise data to get a final merged_user_all.json file which contains the merged / category-agnostic user history of all the users in the Benchmark.
• Phase Three is the process of curating the metadata from Amazon Reviews and storing it in a JSON, Parquet and DB file. Necessary post-proc like filtering missing users/events and removing review texts is done to get lighter versions of the benchmark.
• At the end of Phase Three, the following files shall be created:
  • amazon_parquet_data/metadata_titles.db: SQLite database containing 3 columns i.e. (1) Product ASIN, (2) Product title and (3) Product category for all items in the catalog
  • amazon_parquet_data/merged_users_all_final_filtered.json: Cleaned final full data JSON file with all users (removing those users with 0 lengths or missing titles in the metadata). The structure of this final JSON is as follows:

    {
      "{user_id}": {
            "{history}" : [I_1, I_2, .... , I_T],
            "{timestamps}" : [t_1, t_2, .... , t_T],
            "{ratings}" : [r_1, r_2, .... , r_T],
            "{reviews}" : [review_1, review_2, .... , review_T],      
    }

  • amazon_parquet_data/merged_users_all_final_filtered_no_reviews.json: Cleaned final full data JSON file with all users (removing those users with 0 lengths or missing titles in the metadata) and no reviews. We provide a lighter version of the full data for those who do not plan to use the reviews field in their study.

b. Preparing the Splits

The script for generating the splits as described in the paper are shared in the splits folder. Follow the steps below to generate the splits:

1. Generate user IDs for the 4 splits (as described in the paper) and save them in a txt file. You would need the amazon_parquet_data/merged_users_all_final_filtered_no_reviews.json file from the previous steps for this. A random seed of 42 is set for the sampling and the temporal thresholds for validation is set at 2019 and for test at 2020:

python3 prepare_split_ids_full.py

This generates 4 txt files corresponding to in-distribution and out-of-distribution validation and test set users.

2. Populate the JSON files with the complete data of each corresponding split in the same format as the JSON shared before:

python3 write_splits_to_jsons_full.py

This generates 4 JSON files corresponding to the in-distribution and out-of-distribution validation and test set users.

---

Privacy Statement:

Microsoft Privacy Statement