AoA: tokenizer loading fallbacks for transformers-5.x-saved checkpoints - #5

Open

bylinina wants to merge 2 commits into

babylm-org:mainfrom

bylinina:aoa-tokenizer-fallbacks

Commits on Jul 15, 2026

AoA: 3-level tokenizer fallback (AutoProcessor -> AutoTokenizer -> PreTrainedTokenizerFast), matching other pipeline call sites
bylinina
committed
f3abaed
AoA: tokenizer fallback in scoring step (run.py)
bylinina
committed
05ad2fe