The Meaning of Beatus: Disambiguating Latin with Contemporary AI Models

The objective of this work is to assess the performance of Large Language Models (LLMs) on the task of Word Sense Disambiguation (WSD) for Latin. We evaluate state-of-the-art LLMs—including GPT-4o-mini and LLaMA variants—in both zero-shot and fine-tuned settings, using a dataset derived from the SemEval-2020 Latin Lexical Semantic Change task. Our study aims to determine whether instruction tuning and task-specific fine-tuning can significantly improve the models’ ability to disambiguate Latin word senses.

Results show that while LLMs demonstrate a non-trivial baseline ability in zero-shot settings, fine-tuning – particularly instruction-based – provides improvements in accuracy and F1 scores. These findings highlight the potential of LLMs when applied to under-resourced historical languages.

Resources

• Latin WSD LLM based on Llama-3.1-8B-Instruct HuggingFace
• Latin WSD LLM for the binary task based on Llama-3.1-8B-Instruct HuggingFace
• Training and testing data: data folder
• Training and testing code: code folder
• Models' output: out folder

Publication

Please cite

@inproceedings{ghizzota2025meaning,
  title={The Meaning of Beatus: Disambiguating Latin with Contemporary AI Models},
  author={Ghizzota, Eleonora and Basile, Pierpaolo and Siciliani, Lucia and Semeraro, Giovanni},
  booktitle={Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025)},
  pages={469--479},
  year={2025}
}