Multilingual Translator with mBART

A multilingual translation system based on mBART for German, Dutch, and Hindi, using Spanish as a pivot language.

Objective

To train a many-to-many automatic translation model to analyze how grammatical distance between languages influences the training process. The core of this work focused on building a robust data engineering pipeline to process, normalize, and manage diverse linguistic corpora.

Data Pipeline and Processing

For this project, the following steps were implemented:

• Pivot Language Strategy: Spanish was used as an intermediary language for training, which allowed for managing resource disparities between the target languages.
• Processing and Cleaning: Original data were sourced from three distinct corpora (OpenSubtitles, Tatoeba, and TED Talks) in plain text files. Custom scripts were developed to normalize formats, remove noise, and unify entries into a standard <2lang> src ||| tgt structure.
• Augmentation and Balancing: A bidirectional data augmentation process was implemented, duplicating pairs to cover both translation directions. Subsequently, a sub-sampling filter was applied to balance the dataset and prevent languages with higher data volumes from dominating the training.
• Partitioning and Structuring: After unification and cleaning, the data were partitioned into training, validation, and test sets, stored in JSONL format to streamline the training process.
• Training Management: A custom ParallelTextDataset implementation was developed in PyTorch, handling tokenization, causal mask generation, and dynamic padding.

Model Specifications

• Architecture: mBART (Multilingual BART) using the Hugging Face transformers library.
• Tokenization: Custom tokenizer trained with SentencePiece to ensure optimal multilingual vocabulary coverage.
• Inference: Translation engine based on beam search, featuring an interactive interface developed with Gradio.

Project Structure

/
├── scripts/
│   ├── processing/           # Normalization, balancing, and dataset partitioning
│   │   ├── prepare_pair.py   # Normalization and bidirectional augmentation
│   │   ├── merge_multilingual.py # Unification of data sources
│   │   └── split_multilingual.py # Management of train/test/valid splits
│   ├── training/             # Model and tokenizer training
│   │   ├── train_mbart.py
│   │   └── train_tokenizer.py
│   └── utils/                # Model logic and utilities
│       ├── dataset.py        # PyTorch dataset class
│       ├── model.py          # Architecture and configuration
│       └── translate_interactive.py # Local translation engine
├── app/
│   └── app_gradio.py         # Web inference interface

Installation and Execution

1. Clone the repository

git clone [https://github.com/AlejandroHZ-sys/traductor-multilingue.git](https://github.com/AlejandroHZ-sys/traductor-multilingue.git)
cd traductor-multilingue

2. Execution

Ensure you have torch, transformers, and sentencepiece installed in your Python environment. The inference engine can be executed directly via the web interface:

python app/app_gradio.py

Note: You must have the trained model weights located in the path specified within the scripts to perform inference.