HiBayES

A Python package for analysing data from Inspect logs using statistical modeling techniques presented in HiBayES: A Hierarchical Bayesian Modeling Framework for AI Evaluation Statistics and LLM-as-a-Judge Evaluations.

Please see the docs for installation instructions, examples and a breakdown of the main features.

Citation

HiBayES

@article{luettgau2025hibayeshierarchicalbayesianmodeling,
      title={HiBayES: A Hierarchical Bayesian Modeling Framework for AI Evaluation Statistics},
      author={Lennart Luettgau and Harry Coppock and Magda Dubois and Christopher Summerfield and Cozmin Ududec},
      year={2025},
      archivePrefix={arXiv},
      url={https://arxiv.org/abs/2505.05602},
}

LLM-as-a-Judge Evaluations

@inproceedings{dubois2026quantifying,
      title={Quantifying Biases in {LLM}-as-Judge Evals},
      author={Dubois, Magda and Coppock, Harry and Giulianelli, Mario and Jorgensen, Ole Kristian and Flesch, Timo and Luettgau, Lennart and Ududec, Cozmin},
      year={2026},
      booktitle={43rd International Conference on Machine Learning ({ICML})},
      url={https://icml.cc/virtual/2026/poster/61443}
}