BnMMLU is a Bengali massive multitask language understanding benchmark with 134,375 multiple-choice question-option pairs across 41 domains. It includes MathML-preserved math content, a harder stress-test subset called BnMMLU-HARD, and scripts for dataset construction, model evaluation, batch processing, and result analysis. The dataset is hosted on Hugging Face and the benchmark paper is published in Findings of ACL 2026.
- June 2026: BnMMLU is officially published in Findings of the Association for Computational Linguistics: ACL 2026. Read it on the ACL Anthology or download the PDF.
- The preprint remains available on arXiv: arXiv:2505.18951.
We present BnMMLU, a massive multitask language understanding benchmark for Bengali spanning 41 domains across STEM, humanities, social sciences, and general knowledge, with 134,375 multiple-choice question-option pairs. The dataset preserves mathematical content via MathML, introduces BnMMLU-HARD for stress testing, and benchmarks 24 model variants across 11 LLM families under standardized prompting and context regimes. The benchmark is published in Findings of the Association for Computational Linguistics: ACL 2026.
The BnMMLU dataset consists of 134,375 multiple-choice question-option pairs across 41 domains. It preserves mathematical content via MathML and includes BnMMLU-HARD, a compact subset designed for more difficult stress testing. The dataset is available on Hugging Face:
- Comprehensive evaluation across 41 domains
- 134,375 multiple-choice question-option pairs
- Mathematical content preserved via MathML
BnMMLU-HARDsubset for difficult-case stress testing- Benchmarks covering 24 model variants across 11 LLM families
- Standardized evaluation under direct vs. chain-of-thought prompting and 0-shot vs. 5-shot settings
Evaluation, batch-processing, dataset-construction, and analysis scripts are available in scripts/. Install the common dependencies with:
pip install -r requirements.txtThe vLLM-based open-weight model evaluators require a Linux/CUDA environment:
pip install -r requirements-vllm.txtUse .env.example as the local environment template for API keys. Do not commit .env files.
If you use BnMMLU in your research, please cite:
@inproceedings{joy-shatabda-2026-bnmmlu,
title = "{B}n{MMLU}: Measuring Massive Multitask Language Understanding in {B}engali",
author = "Joy, Saman Sarker and
Shatabda, Swakkhar",
editor = "Liakata, Maria and
Moreira, Viviane P. and
Zhang, Jiajun and
Jurgens, David",
booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
month = jul,
year = "2026",
address = "San Diego, California, United States",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.findings-acl.593/",
doi = "10.18653/v1/2026.findings-acl.593",
pages = "12211--12230",
ISBN = "979-8-89176-395-1",
abstract = "Large-scale multitask benchmarks have driven rapid progress in language modeling, yet most emphasize high-resource languages such as English, leaving Bengali underrepresented. We present BnMMLU, a comprehensive benchmark for measuring massive multitask language understanding in Bengali. BnMMLU spans 41 domains across STEM, humanities, social sciences, and general knowledge, and contains 134,375 multiple-choice question{--}option pairs-the most extensive Bengali evaluation suite to date. The dataset preserves mathematical content via MathML, and includes BnMMLU-HARD, a compact subset constructed from questions most frequently missed by top systems to stress difficult cases. We benchmark 24 model variants across 11 LLM families, spanning open-weights general/multilingual, Bengali-centric open-weights, and proprietary models, covering multiple parameter scales and instruction-tuned settings. We evaluate models under standardized protocols covering two prompting styles (Direct vs. Chain-of-Thought) and two context regimes (0-shot vs. 5-shot), reporting accuracy consistently across families. Our analysis highlights persistent gaps in reasoning and application skills and indicates sublinear returns to scale across model sizes. We release the dataset and evaluation templates to support rigorous, reproducible assessment of Bengali language understanding and to catalyze progress in multilingual NLP."
}This project is licensed under the CC BY-SA 4.0 license. See the LICENSE file for details.
Saman Sarker Joy - saman.sarker.joy@gmail.com