Empirical Rigor Through Benchmarking in Operations Research

About

This repo accompanies the paper Empirical Rigor Through Benchmarking in Operations Research (Dong, Mittal, Namkoong, 2026). The paper makes the case for a rigorous benchmarking ecosystem in stochastic operations research, with reusable simulation environments, structured instance families, theory-grounded baselines, and standardized evaluation protocols. It demonstrates the utility of such benchmarks through a brief illustrative case study on two canonical queueing models.

The codebase here implements that case study. It covers two queueing-network settings: a two-class single-server queue, and an N-Model. The first two studies probe the brittleness of analytical baselines once classical assumptions are perturbed; the next three turn to learning-based methods and probe the sample complexity of competing algorithms, the hyperparameter sensitivity of any one of them across an instance family, and the unexpected effects of imposing structural restrictions on the policy class. The companion website walks through the findings.

• src/or_benchmark/: installable Python package exposing PPO, REINFORCE, DP, and pathwise (QueueTorch) trainers, plus the env and baseline implementations for both case studies (CS1: a two-class single-server queue with state-dependent slowdown; CS2: an N-Model with service disruptions during non-stationary periods). Use it to run any (network config, training config) pair.
• paper_experiments/: the recipe that turns the library into the website's specific results: sweep configs, per-figure final configs, the analysis pipeline, small precomputed artifacts, and orchestration shell scripts.

Quick start

1. Set up the environment

git clone https://github.com/dakshmittal30/OR-benchmark-experiments
cd OR-benchmark-experiments
conda env create -f environment.yml   # installs or_benchmark + QueueTorch
conda activate or-benchmark

2. Run a quick experiment

Small PPO training run on each case study (CPU is fine, ~minutes):

bash src/or_benchmark/cs1/examples/ppo_run.sh    # CS1: two-class single-server queue
bash src/or_benchmark/cs2/examples/ppo_run.sh    # CS2: N-Model

Other entry points (REINFORCE, DP, pathwise) live alongside under src/or_benchmark/cs1/examples/ and src/or_benchmark/cs2/examples/.

3. Replicate the paper experiments

For retraining individual experiments in the paper or the full reproduction sweep, see paper_experiments/README.md.

Citation

@article{dong2026empirical,
  title   = {Empirical Rigor Through Benchmarking in Operations Research},
  author  = {Dong, Jing and Mittal, Daksh and Namkoong, Hongseok},
  journal = {Operations Research (under review)},
  year    = {2026},
  url     = {https://arxiv.org/abs/XXXX.XXXXX},
}

Acknowledgements

The pathwise-gradient pipeline in this codebase builds on QueueTorch, a PyTorch-based differentiable discrete-event simulator for queueing networks. A modified copy is vendored at external/QueueTorch/ and the modifications we made are documented in external/README.md.