LID-For-Romance-Languages

Author: Alessio Taruffi

Year: 2025

---

Overview

This project addresses the Language Identification (LID) task for Romance languages — Italian, French, Spanish, Portuguese, and Romanian.
It compares a classical TF–IDF + SVM baseline with a fine-tuned XLM-RoBERTa model to evaluate performance across both long, structured texts and short, informal sentences.

Inspired by the work of Truica et al. (2015), this project explores how transformer-based multilingual models perform against traditional machine learning approaches in distinguishing between closely related languages.

---

Models Compared

1️ TF–IDF + SVM Baseline

• Classical approach using character n-grams (range 3–5).
• Lightweight and efficient, providing a strong and interpretable baseline.

2️ XLM-RoBERTa (Fine-Tuned)

• Transformer-based multilingual model trained on 100+ languages.
• Fine-tuned on subsets of Romance languages for better cross-lingual generalization.
• Implemented using HuggingFace Transformers and PyTorch.

---

Datasets

Two evaluation datasets were created using samples from open multilingual corpora:

Dataset	Source	Text Type	Samples per Language
WiLI-2018	Wikipedia	Long, structured sentences	1000
Tatoeba	Open-source translations	Short, informal sentences	1000

Each dataset was split 50/50 into training and testing subsets to ensure balanced evaluation.

---

Experimental Scenarios

1. Train/Test on WiLI-2018 → clean, well-structured texts.
2. Train/Test on Tatoeba → short, informal sentences.
3. Train on WiLI-2018, Test on Tatoeba → cross-domain generalization test.

---

Results Summary

Scenario	Model	Accuracy	F1 Score
WiLI-2018	TF–IDF + SVM	0.9916	0.9916
WiLI-2018	XLM-RoBERTa	0.9892	0.9892
Tatoeba	TF–IDF + SVM	0.9780	0.9780
Tatoeba	XLM-RoBERTa	0.9844	0.9844
Cross-domain	TF–IDF + SVM	0.9040	0.9036
Cross-domain	XLM-RoBERTa	0.9364	0.9370

Key Insights

• The baseline performs slightly better on clean, in-domain data.
• XLM-RoBERTa shows superior robustness and generalization to unseen, noisy data.
• Both models struggle slightly with Spanish–Portuguese and Italian–Romanian confusion due to lexical similarity.

---

Evaluation Metrics

• Accuracy – overall correctness of predictions.
• F1 Score – harmonic mean of precision and recall, computed per language.
• Confusion Matrix – visual inspection of common misclassifications.

---