fingerprint.py

BPE tokenizer merge order fingerprint. Recover a tokenizer's training-corpus signature from its own vocabulary.

Built with Claude Code. Inspired by Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data? (Hayase et al., NeurIPS 2024).

Also loosely related: glitch-token research (e.g. "SolidGoldMagikarp", Rumbelow & Watkins)

Why this works

A BPE tokenizer's vocabulary is built by repeatedly merging the most frequent symbol pair in its training corpus, in order. That means each token's native vocab id is a frozen record of how early BPE needed it. Compare that "merge order" against a general-purpose English frequency ranking (wordfreq), and the gap between the two reveals words the training corpus leaned on more (or less) than general English does.

How it works

1. Filter the vocab down to real words. Raw BPE vocab is full of subword fragments and punctuation-glued junk (differe, ##ing, Ġthe) — comparing frequency signal only makes sense on things that are actually words, so tokens are kept only if they're alphabetic, at least --min-len characters, and clear a wordfreq plausibility check.

2. Rank that word list two ways. Merge order is just the tokens' native vocab ids — BPE assigns ids in the order it merged pairs during training, so a low id means the training corpus needed that word early. Freq order is the same words re-sorted by general-English wordfreq score, i.e. the ranking you'd expect if the tokenizer had trained on "average" English instead.

3. Compute drift = freq_rank − merge_rank for each word. This is the actual comparison: how far out of step is a word's "training-corpus urgency" rank from its "general English commonness" rank? A big positive drift means BPE learned the word suspiciously early for how rare it is in general English — it's overrepresented in the tokenizer's own training corpus. A big negative drift is the opposite: a common English word BPE didn't bother with until late, i.e. underrepresented.

4. Sort by drift and look at the extremes. The middle of the distribution is just "words learned in roughly the order you'd expect," which isn't interesting — the training-corpus signal lives in the tails.

5. --outliers reuses the same per-word drift map on a text file instead of the whole vocab. Drift is a property of the tokenizer, not the input text, so once it's computed it can be looked up for any text to see which of that text's words are least (or most) "native" to the tokenizer's training corpus.

6. Ladders are cached to ladders/*.json. Building one requires transformers/wordfreq and, the first time, a model download — caching means results stay reproducible across machines without re-fetching every run.

Setup

python3 -m venv .venv
source .venv/bin/activate
pip install transformers wordfreq torch

# only needed for the blind-test pipeline (see BLIND_TEST.md) --
# training your own tokenizer from scratch on a chosen corpus
pip install tokenizers

Usage

# fingerprint one tokenizer
python fingerprint.py bert --min-zipf 3 --min-len 5 --top 20

# fingerprint two, side by side
python fingerprint.py gpt2 --other whisper --min-zipf 3 --min-len 5

# check which words in an actual text sample are least "native" to a
# tokenizer's own training corpus
python fingerprint.py bert --outliers draft.md --top 10

--min-zipf 3 --min-len 5 (the defaults) matter: lower thresholds let short BPE substrings through (differe, somet) since wordfreq gives near-zero but nonzero scores to almost any string that appears anywhere in its corpus, including typos and word fragments — not what you want when looking for real vocabulary signal.

Results

bert (trained on Wikipedia + BookCorpus):

=== bert: overrepresented in training corpus ===
  glanced   sighed   nodded   leaned   frowned   whispered   shrugged   chuckled

That cluster of dialogue-tag verbs is BookCorpus (unpublished novels) showing through — one of BERT's two documented training sources, recovered here with no access to the actual training data.

whisper (trained on multilingual internet audio + transcripts):

=== whisper: overrepresented in training corpus ===
  faire   mundo   monde   ...alongside auf, sie (German) at looser filters

faire/monde/temps (French), mundo (Spanish) — genuine multilingual leakage into an English-vocab tokenizer, consistent with Whisper's documented multilingual training. Same pattern shows up in t5 (faire, seine, monde, temps — French, despite C4 being marketed as an English corpus).

Notably, this finding is English-only: faire clears the default --min-zipf 3 cutoff on a score of 3.14 in wordfreq's "en" set — real signal (French leaking into "general English" web text), but a thin margin. Raise the threshold to 3.2 and the finding breaks: faire drops below the cutoff and disappears from the list entirely, along with the rest of the French/Spanish words above.

gpt2 (trained on WebText, curated external articles linked from Reddit):

=== gpt2: overrepresented in training corpus ===
  harass   spokeswoman   infect   bytes   subsection   marketable

News/tech-article register — plausible for WebText's curated-links character, though noisier and less immediately obvious than the BERT and multilingual findings above. Note this list, not the "underrepresented" one: the underrepresented list (didnt, fuckin, congratulations, grandma) looks tempting as a Reddit-slang finding at first glance, but it means the opposite — those words are common in wordfreq's general baseline (which itself blends in Reddit comments, Twitter, and movie subtitles) but comparatively rare in GPT-2's own curated corpus. Getting this direction backwards was a real mistake made while building this tool — worth flagging since it's an easy one to make.

Run --outliers <file> on any text sample and you get the same signal applied to a passage instead of the whole vocabulary — e.g. a piece of narrative fiction checked against bert will surface exactly the glanced/sighed-style verbs above as its most "native" words, while an encyclopedic paragraph won't.

$ python fingerprint.py bert --outliers narrative.txt --top 5
=== bert: text words most OVERrepresented in its training corpus ===
  glanced   sighed   nodded   leaned   frowned

$ python fingerprint.py bert --outliers encyclopedic.txt --top 5
=== bert: text words most OVERrepresented in its training corpus ===
  membrane   occurs   mitochondrial   organism   signaling

Same tokenizer, same fingerprint — different text, different overlap. The narrative sample (dialogue-tag verbs: glanced, sighed, nodded, leaned, frowned...) lines up with BookCorpus; the encyclopedic one (membrane, mitochondrial, organism...) barely registers against it — no dialogue-tag words appear at all, because that register isn't what BERT's training corpus over-learned.

Adding tokenizers

PRESETS needs a HuggingFace model id and which word-boundary marker style the tokenizer uses — four styles cover most tokenizers in use:

style	marker	tokenizers
prefix_g	Ġword	GPT-2, RoBERTa, Whisper
prefix_u	▁word (SentencePiece)	T5, Llama, Mistral, ALBERT — most modern LLMs
suffix_w	word</w>	CLIP
bare	##continuation	BERT, DistilBERT (WordPiece)

Included out of the box: gpt2, clip, whisper, bert, roberta, distilbert, t5, albert, plus llama and mistral (gated on Hugging Face — run huggingface-cli login with granted access first). Any other HF tokenizer just needs its style identified by inspecting a few sample tokens from tokenizer.get_vocab().

Validating the technique blind

Every example above is checked against a tokenizer whose training corpus is already public knowledge — that's confirmation bias, not proof the technique works. BLIND_TEST.md covers a DIY protocol to test it properly: train a from-scratch tokenizer on a corpus mix that's randomly chosen and sealed from you, predict its composition from fingerprint.py's output alone, then reveal the recipe and score yourself.

Running that protocol several times over: detecting what registers are present in a corpus holds up well; estimating how much of each is present does not. Word-count density in the drift list tracks actual mixing ratio only when the competing sources differ a lot in how much distinctive vocabulary they naturally contain per page — it breaks down between a lexically sparse source and a lexically dense one, regardless of their actual proportions. See BLIND_TEST.md's results section for the trial-by- trial detail. Worth noting this actually matches the tool's own framing above ("which words... over/under-learned") — the technique was always about what, not how much; the "how much" question belongs to the calibrated quantitative method this tool is explicitly not trying to be.

Pinned ladders

ladders/*.json are checked into this repo at min-len=5, min-zipf=3 (the config used for the results above). build_ladder() caches to a ladders/ directory next to fingerprint.py, so as long as these files stay in the repo, results are reproducible across machines and over time — independent of transformers/wordfreq version drift or a stray --refresh.