This is the official code repository for OmniOCR, a generalist OCR framework for low-resource ethnic-minority scripts.
OmniOCR: Generalist OCR for Ethnic Minority Languages
Bonan Liu, Zeyu Zhang†, Bingbing Meng, Han Wang, Hanshuo Zhang, Chengping Wang, Daji Ergu, Ying Cai*
*Corresponding author. †Project lead.
ICIC 2026
Most OCR systems focus on well-resourced scripts such as Latin and Chinese. Ethnic-minority languages are substantially harder because of complex writing systems, scarce annotations, and diverse historical and modern forms. OmniOCR addresses this setting with Dynamic LoRA, which allocates adaptation capacity across layers and scripts while preserving the base model. A sparsity regularizer removes redundant updates, producing compact adapters without an additional inference stage. The paper evaluates TibetanMNIST, Shui, Ancient Yi, and Dongba, and reports substantial gains over zero-shot foundation models and standard post-training baselines.
data/loader/Tibetan/
tibetan_zero_shot.py zero-shot VLM baseline
tibetan_fully_fine_tuning.py full-parameter fine-tuning
tibetan_lora.py static LoRA fine-tuning
tibetan_dynamic_lora.py Dynamic LoRA / CoDyRA fine-tuning
tibetan_other-models.py optional API baseline (credentials required)
models/layers/
dynamic_lora.py reusable Dynamic LoRA implementation
Evaluate.py Shui test-set evaluation
Select_datasets.py class-balanced dataset selection helper
data/loader/*/Documentation.txt notes for Shui, Yi, and Dongba preparation
The repository contains code and dataset documentation only. The base VLM checkpoint and the image datasets must be obtained separately according to their respective licenses.
The training scripts use PyTorch, Hugging Face Transformers, and CUDA. Python 3.10 or newer is recommended.
conda create -n omniocr python=3.10 -y
conda activate omniocr
# Select the PyTorch wheel matching your CUDA driver.
python -m pip install torch torchvision
python -m pip install transformers accelerate bitsandbytes
python -m pip install pillow scikit-learn tqdm backoff openaibitsandbytes is used by the 4-bit zero-shot baseline. If that baseline is
not needed, it can be omitted. Install a CUDA-enabled PyTorch build for
training; the scripts use cuda:0, bfloat16, and mixed-precision training.
The scripts expect a local vision-language model directory named RolmOCR.
It must contain the model configuration, tokenizer, processor, and weights
needed by AutoModelForVision2Seq.from_pretrained. The base checkpoint is not
stored in this repository. You can use another compatible checkpoint by
changing the model_path constant at the top of the script you run.
Before running a script, set model_path, dataset_path, and output_dir to
absolute paths. The current scripts were written with several historical
relative-path defaults, so explicit paths avoid working-directory mistakes.
The Tibetan experiments expect one directory per digit:
datasets/TibetanMNIST/
0/*.png 1/*.png 2/*.png ... 9/*.png
The zero-shot, static LoRA, and Dynamic LoRA scripts create a deterministic 60% train / 30% test / 10% validation split with seed 42. The full fine-tuning script uses 250 training, 150 test, and 40 validation images per class, so each class must contain at least 440 images.
models/layers/Evaluate.py expects twelve class directories (0 through
11) under datasets/shui_datasets and uses 250 train, 150 test, and 40
validation images per class. Images may be PNG, JPG, or JPEG.
The documentation files under data/loader/Shui_sctipt/,
data/loader/Yi_script/, and data/loader/Dongba/ describe the required
script-specific segmentation, LoRA calibration, and prompt preparation. The
provided training entry points are the Tibetan and Shui implementations; adapt
the dataset path and class prompts for additional scripts.
The Tibetan scripts are self-contained programs rather than configurable CLI tools. Edit their path constants first, then run them from the repository root or from the directory containing the script after making those constants absolute.
python data/loader/Tibetan/tibetan_dynamic_lora.pyThe default configuration trains for one epoch with learning rate 5e-6 and
gradient accumulation of two steps. It replaces the attention and MLP linear
layers (q_proj, k_proj, v_proj, out_proj, mlp.fc1, mlp.fc2) with a
sparsity-aware Dynamic LoRA module. The best model and processor are saved to
the configured output_dir using save_pretrained.
The same implementation is also available as
models/layers/dynamic_lora.py for reuse in other training drivers.
python data/loader/Tibetan/tibetan_lora.pyThis baseline uses the same 60/30/10 split and trains for one epoch with
learning rate 5e-6 and gradient accumulation of eight steps. Its default
output directory is RolmOCR_lora_tuned_shrunk_fixed.
python data/loader/Tibetan/tibetan_fully_fine_tuning.pyThe full fine-tuning baseline uses five epochs, learning rate 2e-6, and
gradient accumulation of four steps. Its default output directory is
TibetanDigit_fully_tuned.
python data/loader/Tibetan/tibetan_zero_shot.pyThis baseline loads the base model with optional 4-bit quantization and does not update model parameters. It evaluates the generated digit predictions on the test split.
The Tibetan training scripts automatically evaluate the validation split after
each epoch and, when training succeeds, reload the best checkpoint and report
test accuracy plus a classification report. Logs are written beside the
working directory with names such as codyra_fine_tune_rolmocr_log.txt and
lora_model_test_log.txt.
To evaluate a trained Shui adapter, edit model_path and dataset_path near
the top of models/layers/Evaluate.py, then run:
python models/layers/Evaluate.pyThe script reports test accuracy, macro recall, macro F1, and a per-class
classification report. It loads the model with device_map={"": "cuda:0"}
and therefore requires a CUDA device.
For an API-based baseline, data/loader/Tibetan/tibetan_other-models.py
contains a DeepSeek-compatible OpenAI client and reports test accuracy. Its
DEEPSEEK_API_KEY, DEEPSEEK_BASE_URL, and MODEL_NAME values are placeholders;
configure them locally or refactor the script to read environment variables.
Never commit an API key.
models/layers/Select_datasets.py selects up to the 30 classes with the most
images, optionally caps samples per class, copies the selected images, and
writes top30_most_samples.json plus omni_selection_metadata.json.
Set DATASET_DIR and OUTPUT_DIR to absolute paths before running:
python models/layers/Select_datasets.pyThe helper accepts PNG, JPG, JPEG, BMP, and GIF files and preserves file
metadata with shutil.copy2.
- The Tibetan split seed is fixed at
42; file ordering from the filesystem can still affect results if the dataset is modified. - The scripts assume a CUDA GPU and may require reducing image size or batch settings for smaller cards.
- Several paths and hyperparameters are module constants instead of command line arguments. Change them explicitly for a new dataset or model.
- Model checkpoints, datasets, and benchmark results are not included because of size and licensing constraints.
@misc{liu2026omniocrgeneralistocrethnic,
title={OmniOCR: Generalist OCR for Ethnic Minority Languages},
author={Liu, Bonan and Zhang, Zeyu and Meng, Bingbing and Wang, Han and Zhang, Hanshuo and Wang, Chengping and Ergu, Daji and Cai, Ying},
year={2026},
eprint={2602.21042},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2602.21042}
}No license file is currently included. Add the license required by your project, base model, and datasets before redistribution.