📊 Towards Reliable Multilingual Judge Models: An Empirical Study

This repository contains the code accompanying our paper, "Towards Reliable Multilingual Judge Models: An Empirical Study".

In this work, we study how to build reliable multilingual LLM-as-a-Judge systems when evaluation data is only available in English, with experiments across English, Spanish, and Basque.

We specifically analyze two practical settings:

• When in-domain training data is available, where judge models can be fine-tuned directly for the target evaluation task.
• When in-domain training data is not available, requiring either zero-shot transfer or fine-tuning on related but out-of-domain data.

For the first scenario, we investigate the effect of instruction translation, monolingual versus multilingual supervision, and model scale. Our experiments show that multilingual training consistently improves performance in Spanish and Basque compared to strictly monolingual training, suggesting beneficial cross-lingual transfer during judge model learning. We further observe that keeping evaluation instructions and rubrics in English often produces more stable multilingual judgments than translating prompts into the target language. Interestingly, smaller open 8B models fine-tuned on in-domain data achieve performance comparable to substantially larger 70B models and rival proprietary judge models, highlighting the effectiveness of efficient open models for multilingual evaluation.

For the latter scenario, we find that larger models in zero-shot settings are generally more robust across domains and languages, while fine-tuning 70B models on mismatched or out-of-domain supervision can significantly degrade evaluation quality. Smaller models benefit from additional supervision on related data, but their overall performance in this setting remains substantially below that of larger zero-shot models.

Overall, our findings provide practical guidance for building efficient and reliable multilingual evaluation pipelines, particularly in scenarios where task-specific multilingual supervision is limited.

---

Key Components

translate/

Code used to translate train and test datasets.

evaluation/

Code for running inference on benchmark datasets, computing evaluation metrics, and generating the analysis plots used in the paper.

llama-recipes/

Fork and adaptation of Meta’s Llama cookbook used for model fine-tuning on feedback data. This directory retains the original licensing and attribution as provided by Meta. See the corresponding license files for details.

---

Datasets

English data:

• Train: Feedback-Collection
• Test: RECON and FLASK

Basque and Spanish Data:

• Train: es_eu-Feedback-Collection
• Test: es_eu-RECON and es_eu-FLASK

Please refer to the original dataset licenses before redistribution or commercial use.

Citation

@misc{zubiaga2026reliablemultilingualllmsasajudgeempirical,
      title={Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study}, 
      author={Irune Zubiaga and Aitor Soroa and Rodrigo Agerri},
      year={2026},
      eprint={2605.28710},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.28710}, 
}