This is the submission for the Deep Past Challenge posted on Kaggle.
To build a Translation system for Old Assyrian, in order to help Decode everyday business records of ancient Assyrian merchants, using data from 8000 cuneiform texts.
Since Old Assyrian is a sumerogram-heavy and ighly-inflicted language, the way we handle data or small tweaks can lead to massive jumps in accuracy. -> Using Unsloth AI Library: (i) Unsloth's QLoRA kernels are highly efficient in terms of memory and speed. This helps in training the model quickly and leave behind a smaller footprint. (ii) Since it is mostly optimized for LLMs such as Llama and Mistral, most strategies involve ByT5.
-> Synthethic Data Generator(Data Fuel): (i) Using a synthetic data generator, to "translate-back" English trade records into Old Assyrian transliteration is an optimal strategy.
(ii) To focus only on the "Merchant talks" such as textiles, debt, tin etc:- since Old Assyrian is more of a dialect of Akkadian.
-> Reinforcement Learning: (i) Although there is not enough data, GRPO(Group Relative Policy Optimization) technique is being tested for low-resource translation
(ii) Reward Function: We can use Geometric Mean and BLEU(Bilingual Evaluation Understudy i.e 0 to 1 or 0-100 metric measuing MT text against references focusing on n-gram, which includes a brevity penalty) and chrF++(autmated MT and MLG evaluation metric that calculates F-Score based on character n-grams and word n-gram).
(iii) Risks involved in RL, includes "reward hacking", where the model generates grammatically weird sentences that happen to have the right characters to boost chrF++.
(i) The "Scribal Mess" : The competition data is full of modern scientific notations like !, ? and . Need to make sure test data isn't messy.
(ii) Evaluation Matrix: Score is caluclated using √(BLEU + chrF++ ). Need to make sure it is properly optimized for both BLEU and chrF++
Dealing with Noise(Scribal Mess): -> Problem: If you train on clean data, the model will hallucinate when the test set contains an '['.
-> Solution: Integrate "Noise-Aware" Augmentation into synthetic generator. Instead of just using generating clean pairs, randomy injecting scholarly "mess" onto the source text.
-> Akkadian-Specific Rules to keep in mind:
In Old Assyrian, [KÙ.BABBAR] (Silver) often has brackets because the tablet is broken. The generator should be able to learn that [KÙ.BABBAR] and KÙ.BABBAR both map to "silver" in English