Beyond Chinchilla: Scaling Laws Break Down for Truly Low-Resource Languages

BABEL studies from-scratch language model scaling for Amharic, Yiddish, and Tibetan.

---

Results at a glance

We fit $L(N, D) = E + A/N^\alpha + B/D^\beta$ to held-out test loss on a 5×5×3 grid (5 model sizes × 5 data splits × 3 seeds = 75 runs per language/source).

Language / Source	α	β	R²	Best 1m model
Amharic CC-100	1.6972	0.0958	0.978	s
Amharic HF	1.7073	0.0967	0.977	s
Yiddish CC-100	1.3157	0.0955	0.978	xs
Yiddish HF	1.4577	0.0900	0.980	xs
Tibetan HF	1.5818	0.2121	0.982	s

Chinchilla baselines: α = 0.3392, β = 0.2849. β is suppressed and α is inflated in the sub-million-token regime; the best 1M-token model is always xs or s.

---

1. Setup

You need: Python 3.10+, a CUDA GPU for the full training grid (≥24 GB VRAM), and a HuggingFace token for HPLT corpora (--source hf).

git clone <repository-url>
cd BABEL
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -r requirements.txt
pip install -r requirements-dev.txt   # optional: pytest
export HF_TOKEN=<your_token>          # only required for --source hf

Resource	Full 75-run grid	Smoke test
GPU	1× H200/A100-class	not required
RAM	~140 GB (cluster)	~4 GB
Walltime	~24 h / language job	~2 min
Disk	~10 GB / language-source	~100 MB

Verify the install:

python scripts/smoke_test.py
python -m pytest tests/test_pipeline.py -q

---

2. Reproduce one language/source

Pick a language and corpus. Phase 4 (04_train.py) runs inside Phase 5 — call phases 1, 2, 3, 5, 6, 7 in order. Phase 8 is optional (needs a GPU to run Qwen2.5-7B) but recommended before Phase 7 — if its output exists, Figures 1–2 overlay a pretrained-baseline reference line and switch to bits-per-character (BPC) so the from-scratch model and the pretrained model are on a common, vocab-size-independent scale.

LANG=amharic
SOURCE=cc100    # or: hf  (HPLT — needs HF_TOKEN)

python scripts/01_prepare_data.py        --lang $LANG --source $SOURCE
python scripts/02_train_tokenizer.py     --lang $LANG --source $SOURCE
python scripts/03_build_model_configs.py --lang $LANG --source $SOURCE
python scripts/05_run_grid.py            --lang $LANG --source $SOURCE
python scripts/06_collect_results.py     --lang $LANG --source $SOURCE
python scripts/08_eval_pretrained_baseline.py --lang $LANG --source $SOURCE   # optional, needs GPU
python scripts/07_generate_figures.py    --lang $LANG --source $SOURCE

Valid combinations:

--lang	--source cc100	--source hf
amharic	✓	✓
yiddish	✓	✓
tibetan	—	✓

Tibetan has no CC-100 URL in languages.yaml. CC-100 sources download from the public web and do not need HF_TOKEN.

What each phase produces:

Phase	Command	Output
1	01_prepare_data.py	data/<lang>/<source>/ — cleaned text + word splits
2	02_train_tokenizer.py	tokenizer/<lang>/<source>/ — BPE model + token splits
3	03_build_model_configs.py	configs/<lang>/<source>/ — xs–xl YAML configs
5	05_run_grid.py	runs/<lang>/<source>/ — 75 runs, summary.json per cell
6	06_collect_results.py	analysis/<lang>/<source>/results.json — fitted α, β, R²
8	08_eval_pretrained_baseline.py	analysis/<lang>/<source>/pretrained_baseline.json — zero-shot Qwen2.5-7B loss/BPC per split (optional)
7	07_generate_figures.py	figures/<lang>/<source>/ — fig1–fig4 (PDF + PNG); overlays Phase 8 baseline if present

Phase 6 uses fit_law() in scripts/06_collect_results.py (log-parameterized Chinchilla form; Wald t-tests vs. Chinchilla α, β).

---

3. Reproduce all paper results

The repo ships code and Qwen eval grids — not figures/ or analysis/. Generate them by running the pipeline.

Option A — SLURM (recommended): three jobs cover all five language/source cells.

export HF_TOKEN=<your_token>
for lang in amharic yiddish tibetan; do
  sbatch --export=ALL,HF_TOKEN=$HF_TOKEN cluster_scripts/train_${lang}.sh
done

Each Amharic/Yiddish job runs CC-100 and HPLT. Tibetan runs HPLT only. Logs: logs/.

Option B — manual: repeat Section 2 for every cell:

for combo in amharic:cc100 amharic:hf yiddish:cc100 yiddish:hf tibetan:hf; do
  LANG=${combo%:*}
  SOURCE=${combo#*:}
  python scripts/01_prepare_data.py        --lang $LANG --source $SOURCE
  python scripts/02_train_tokenizer.py     --lang $LANG --source $SOURCE
  python scripts/03_build_model_configs.py --lang $LANG --source $SOURCE
  python scripts/05_run_grid.py            --lang $LANG --source $SOURCE
  python scripts/06_collect_results.py     --lang $LANG --source $SOURCE
  python scripts/07_generate_figures.py    --lang $LANG --source $SOURCE
done

Check your numbers against the table above:

python -c "
import json, pathlib
for p in sorted(pathlib.Path('analysis').glob('*/*/results.json')):
    r = json.loads(p.read_text())['fit_result']
    print(f'{p.parent.parent.name}/{p.parent.name}: '
          f\"alpha={r['alpha']:.4f} beta={r['beta']:.4f} R2={r['r2']:.3f}\")
"

---

4. Supplementary Qwen2.5 experiments

Fine-tuning Qwen2.5 (0.5B–14B) on the same corpora. Separate from the from-scratch grid.

Pre-computed eval grids (in repo): qwen2.5-finetuned/*/eval_results.json

Directory	Corpus
cc100_amharic/, cc100_yiddish/	CC-100
hplt_amharic/, hplt_yiddish/, hplt_tibetan/	HPLT

Re-run (GPU + pip install -r requirements-qwen.txt):

export HF_TOKEN=<your_token>

# HPLT example
python qwen2.5-finetuned/run_finetune.py --lang yiddish --source hf

# CC-100 example (run Phase 1 first)
python scripts/01_prepare_data.py --lang amharic --source cc100
python qwen2.5-finetuned/run_finetune.py --lang amharic --source cc100

# Plot
python qwen2.5-finetuned/plot_results.py --results-dir qwen2.5-finetuned/hplt_yiddish

Settings: 1 epoch, lr 5e-5, block size 128, scales 100k–10M tokens. See run_finetune.py for batch sizes and vocab expansion.

---

5. Model architecture

Decoder-only transformer (GQA, pre-norm RMSNorm, SwiGLU, RoPE), trained from scratch.

Size	Params (non-emb)	d_model	Layers	Dropout
xs	~197K	64	4	0.00
s	~1.1M	128	6	0.00
m	~5.5M	256	8	0.05
l	~25M	512	12	0.10
xl	~103M	768	16	0.20

Per-split epoch budgets (20–100), validation holdout (15–20% on small splits), early stopping (patience 10).

---

Repository layout

BABEL/
├── scripts/01–07_*.py       # Main pipeline
├── scripts/lang_utils.py    # Shared config helpers
├── scripts/smoke_test.py    # Fast sanity check
├── cluster_scripts/         # SLURM: one job per language
├── qwen2.5-finetuned/       # Qwen runner + eval_results.json
├── languages.yaml           # Language/corpus registry
├── tests/                   # pytest suite
├── requirements.txt         # Main pipeline
├── requirements-dev.txt     # pytest
└── requirements-qwen.txt    # Qwen supplementary

Local only (gitignored, kept on disk): data/, runs/, analysis/, figures/, docs/, logs/, training_stats.json, and optional scripts (plot_convergence.py, plot_model_architecture.py, 08_eval_pretrained_baseline.py).

---

Adding a language

Add a block to languages.yaml, then run the pipeline with --lang <name> --source cc100 or hf:

my_language:
  name: My Language
  code: xx
  cc100_url: http://data.statmt.org/cc-100/xx.txt.xz   # optional
  hf_dataset:
    repo: HPLT/HPLT2.0_cleaned
    config: xxx_Script
  fasttext_tag: xx
  char_ratio_threshold: 0.5
  tokenizer:
    vocab_size: 8000
    model_type: bpe

---

Data & license

Corpus	Languages	Access
CC-100	Amharic, Yiddish	Public
HPLT 2.0 Cleaned	Amharic, Yiddish, Tibetan	Gated (HF token)

Code: MIT License.