The official scorer for OIWER (Orthographically Informed Word Error Rate) as used in the Voice of India (VoI) ASR benchmark (arXiv:2604.19151) — a large-scale evaluation of ASR across 15 Indian languages. This is the same implementation used to produce the benchmark's official results.
OIWER was introduced in prior work (see below) as an orthography-aware alternative to plain WER; VoI adapts it for the benchmark. Instead of penalizing legitimate spelling and transliteration variants, it scores a hypothesis against a lattice of accepted reference variations per word/phrase, so valid orthographic variants aren't counted as errors.
Normalization runs in Python (indicnlp-based); the edit-distance alignment over the lattice is implemented in Rust for speed.
pip install voi_oiwerWorks on CPython 3.8+ (Linux, macOS, Windows). Prebuilt wheels — no Rust toolchain needed.
from voi_oiwer import oiwer
score, final_h, final_r, final_ops, (ins, dele, sub), \
total_ref_words, metadata, std_refs = oiwer(
hypothesis="नमस्ते दुनिया",
reference_lists=[["नमस्ते"], ["दुनिया", "विश्व"]], # lattice: variations per slot
input_language="hindi",
)
print(score) # OIWER, as a percentage| arg | type | description |
|---|---|---|
hypothesis |
str |
the ASR output text |
reference_lists |
list[list[str]] |
the reference lattice — one list of accepted variations per slot (a variation may span multiple words) |
input_language |
str |
a language name, e.g. "hindi", "tamil", "urdu" (see list below) |
A tuple:
(oiwer_score, # float — error percentage
final_h, # list[str] — aligned hypothesis words
final_r, # list[str] — aligned reference words (" " = insertion)
final_ops, # list[str] — per slot: 'c' correct, 's' sub, 'd' del, 'i' ins
(insertions, deletions, substitutions),
total_reference_words, # int
final_metadata, # list — [lat_idx, var_idx, word_idx] per reference word (None for insertions)
standardized_reference_lists) # list[list[str]] — the normalized lattice used internally
assamese, bengali, bodo, dogri, gujarati, hindi, kannada,
kashmiri, konkani, maithili, malayalam, manipuri, marathi,
nepali, odia, punjabi, sanskrit, santali, sindhi, tamil,
telugu, urdu, chattisgarhi, bhojpuri.
Scoring runs in two stages. First, text normalization (indicnlp-based) runs in
Python — script-specific character handling, punctuation stripping, and Unicode
normalization. Then the edit-distance alignment over the reference lattice runs
in a compiled Rust core, which is where the speedup comes from. The normalized
lattice used internally is returned as standardized_reference_lists so results
are inspectable.
The metric, VoI's adaptation of OIWER, and the lattice-based reference design are described in full in the benchmark paper (arXiv:2604.19151).
The Rust core is built with maturin:
pip install maturin
maturin develop --release # compile the Rust core into your active venv
pytest -q # run the test suiteIf you use this metric or the Voice of India benchmark, please cite:
@misc{bhogale2026voiceindialargescalebenchmark,
title={Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India},
author={Kaushal Bhogale and Manas Dhir and Amritansh Walecha and Manmeet Kaur and Vanshika Chhabra and Aaditya Pareek and Hanuman Sidh and Mahima Manik and Sagar Jain and Bhaskar Singh and Utkarsh Singh and Tahir Javed and Shobhit Banga and Mitesh M. Khapra},
year={2026},
eprint={2604.19151},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2604.19151},
}This metric builds on the original OIWER formulation:
@misc{bhogale2026orthographicallyinformedevaluationspeechrecognition,
title={Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages},
author={Kaushal Santosh Bhogale and Tahir Javed and Greeshma Susan John and Dhruv Rathi and Akshayasree Padmanaban and Niharika Parasa and Mitesh M. Khapra},
year={2026},
eprint={2603.00941},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2603.00941},
}MIT © JoshTalks