letterpress - build your own GPT, one letter at a time

A character-level GPT you train from scratch on public-domain books - built up from Karpathy's lecture, and runnable on a laptop or a rented cloud GPU.

Skip the training: five ready-made models (0.8M to 202M params) are one download away at huggingface.co/disco-jack-basement/letterpress - grab one and jump straight to chatting with it or grading it.

This project teaches a tiny computer program to write. You feed it a big pile of old, copyright-free books, and it learns - one letter at a time - what letter tends to come next. Do that a few million times and it starts producing writing that looks like the books it read: a Shakespeare-style speech, a 19th-century novel paragraph, a stretch of philosophy.

It is built from scratch in PyTorch, following Andrej Karpathy's lecture Let's build GPT: from scratch, in code, spelled out, and then grown into something you can actually train on a real, half-a-billion character library of public-domain books.

A few words you'll see throughout, in plain terms:

• model - the little program that does the writing. It starts out knowing nothing and gets better as it trains.
• character / token - here, one letter (or space, or punctuation mark). This model works one character at a time. "Token" is the jargon word for it.
• training - the slow process of showing the model lots of text so it learns.
• checkpoint - a saved model, frozen to a file (ending in .pt) so you can use it later without retraining.
• GPU - the fast chip (on a Mac, the built-in Apple graphics chip) that makes training go quicker. You do not need a separate one; it uses whatever you have.

Here is the kind of thing it produces, after training on Shakespeare:

First Citizen:
Before we proceed any further, hear me speak.

The grammar is real, the names are real, some of the longer words are invented but plausible. That is exactly what a small character model does.

One thing to set straight up front: this is a base model. It continues whatever text you give it, in the style of the books it read. It is not a chatbot like ChatGPT. If you type a question, it will not answer - it will just keep writing in book-style. Give it "ROMEO:" and you get a speech, not a reply.

It is NOT a chatbot - it continues your text in-style, it does not answer questions.

---

The whole project at a glance

Everything in this repo is one pipeline: get some books, turn them into a form the trainer reads fast, train a model, then use that model. Here is the map.

The rest of this README walks each step, fastest-win first. You can stop after the quickstart and still have seen the model work.

---

1. Setup (do this once)

Step 0: get the code onto your machine

Open the Terminal app, then download a copy of this project ("clone" it) and step into its folder:

git clone https://github.com/Novotarskyi/letterpress.git
cd letterpress

Every command in this README is run from inside this letterpress folder.

Step 1: install the software

You need Python 3.11 or newer (this project uses 3.14). First check what you have:

python3 --version          # should print 3.11 or higher

If that errors or prints something older, install a current Python from python.org, then try again.

Now make a venv - a private box for this project's software so it cannot clash with anything else on your machine - and install what it needs into it:

python3 -m venv .venv                        # makes the box (a new .venv/ folder appears - that is expected)
.venv/bin/pip install -r requirements.txt    # fills it: PyTorch (the math engine), NumPy, huggingface_hub

That is all you need to run a model. Every command below calls the venv's Python directly as .venv/bin/python ..., so it works in any terminal window and you never have to "activate" anything.

The program auto-detects your hardware: it uses an NVIDIA GPU (CUDA) if present, else an Apple-Silicon GPU (MPS), else the plain CPU. There is nothing to configure.

Optional - only if you want to run the test suite later:

.venv/bin/pip install -r requirements-dev.txt   # adds pytest

---

2. Quickstart: see it work in two commands (no training)

A tiny pre-trained model ships inside this repo: model_history/lecture-nano/v1/shakespeare-nano.pt (0.8 million parameters, about 4 MB). It already knows Shakespeare, so you can watch it write right now - no training, no downloads, no GPU needed.

Generate one block of text:

.venv/bin/python -m inference.sample --ckpt model_history/lecture-nano/v1/shakespeare-nano.pt --prompt "ROMEO:" --tokens 500

You should see roughly 500 characters of Shakespeare-shaped text scroll past: character names, lines of dialogue, the right rhythm. It will look a bit wild and garble some words - that is normal for a model this tiny.

Or chat with it live:

.venv/bin/python -m inference.interact --ckpt model_history/lecture-nano/v1/shakespeare-nano.pt

This opens a back-and-forth prompt. Type a line (for example JULIET:) and press Enter; the model continues it. A few things to know inside the chat:

• Pressing Enter on an empty line means "keep going" - it is not "do nothing".
• /reset clears the conversation, /show prints it, /settings shows the current knobs, /help lists everything.
• /quit exits (Ctrl-D also works).

Two important notes for a fresh checkout

1. Always pass --ckpt ... for the demo. If you run python -m inference.sample or python -m inference.interact with no flags, they look for a bigger trained model on disk that does not exist in a fresh clone - you'll get a "checkpoint not found" error. Pointing --ckpt at the shipped nano file (above) is the zero-training path.

2. sample and interact behave a little differently. sample is a one-shot dump and runs "hotter" (more random, more invented words). interact is tuned to run "cooler" and streams the text live, stopping at a natural blank line, so it tends to read more coherently. Both are correct - interact just has gentler defaults.

3. Generation uses a KV cache by default - it remembers each step's attention work instead of redoing it, which makes the exact same text come out faster (about 5x on a plain CPU; roughly neutral on an Apple-Silicon GPU, where these small models are launch-bound rather than math-bound). The output is byte-identical either way; --no-cache turns it off if you ever want to check.

A note on expectations

This is a base model: it continues text in the style of what it read. It is not a chatbot. It will not answer questions or follow instructions, because it was never taught to. Give it "ROMEO:" and you get a Shakespeare-shaped speech, not a reply. Turning a base model into an assistant (instruction-tuning, RLHF) is a whole separate stage this project does not attempt.

---

3. Walk the lecture, checkpoint by checkpoint (optional, for the curious)

The whole thing is built up in stages in Karpathy's lecture, and the lecture/ folder mirrors that. These three files are kept standalone and line-faithful to the video - read them top to bottom to see how a GPT is built from nothing:

File	Lecture section	What it shows
lecture/bigram.py	~0:00-0:42	The baseline: each character guesses the next from a lookup table alone. No context, no attention. This is the score to beat.
lecture/attention_steps.py	~0:42-1:11	The heart of it. Self-attention built four equivalent ways, from a slow loop up to real query/key/value attention, with assertions proving each version matches.
lecture/gpt.py	~1:11-1:55	The whole model in one file: Head -> MultiHeadAttention -> FeedForward -> Block -> GPTLanguageModel. The lecture's final artifact.

.venv/bin/python -m inference.selftest        # ~3s: quick correctness checks, no training needed
.venv/bin/python lecture/attention_steps.py   # prints nothing if all four attention versions agree
.venv/bin/python lecture/bigram.py            # trains the baseline - a few minutes
.venv/bin/python lecture/gpt.py               # trains the full model - tens of minutes

The same model, refactored into the reusable core/ package (configurable, checkpointed, with a command-line interface), is what the rest of this README uses. Keeping the standalone lecture/gpt.py next to the modular core/ is on purpose: you can compare the teaching version against the engineered one.

---

4. Get a real corpus the easy way

To train a model bigger than the toy demo, you need a big pile of books. You do not have to download them one by one. The whole library has been cleaned, frozen, and published - free - to a public Hugging Face dataset:

disco-jack-basement/byob-pd-book-corpus (license CC0, public-domain dedication)

Pulling it onto your machine is one command. No login or account is needed to pull a public dataset.

There are two flavours you can pull:

The raw books (.txt files, one per author) - use these to train with --data, or to re-prepare them yourself:

make hf-pull REPO=disco-jack-basement/byob-pd-book-corpus TIER=medium
# lands in data/medium/

The "prepared" version - already turned into the compact format the trainer reads fastest (train.bin / val.bin / meta.json). Use these with --data-bin and skip the preparation step:

make hf-pull REPO=disco-jack-basement/byob-pd-book-corpus TIER=medium PREPARED=1
# lands in data/prepared/medium_bin/

Swap TIER=large or TIER=xlarge for the bigger sets. The longhand form (same thing, spelled out) is:

.venv/bin/python -m corpus.hf_dataset pull --repo-id disco-jack-basement/byob-pd-book-corpus --tier medium
.venv/bin/python -m corpus.hf_dataset pull --repo-id disco-jack-basement/byob-pd-book-corpus --tier medium --prepared

After it lands, sanity-check what you got:

.venv/bin/python -m corpus stats   # prints per-tier authors, characters, size, vocab

REPO= names the dataset and is required in every pull command (it is the published corpus named above). And corpus stats and corpus.hf_dataset are both parts of the same corpus toolkit - the space-vs-dot is just how each piece is run, not a typo.

Tip for the big tiers. The xlarge prepared file is over 2 GB. Before pulling it, run export HF_XET_HIGH_PERFORMANCE=1 to speed up the download.

The five tiers

The corpus comes in five nested sizes. medium is a subset of large, which is a subset of xlarge, then 2xlarge, then 4xlarge - same authors, just more of them as you go bigger. Start with medium: it is the smallest and fastest to train.

(Sizes are in characters. Because every character fits in one byte, the character count is roughly the size on disk - so xlarge is about 2 GB of text and 4xlarge about 10 GB.)

What's in it, and the rules

• Public-domain only. Novels, philosophy, science, poetry - gathered from Project Gutenberg and Wikisource, cleaned, and released CC0 (no rights reserved).
• Multilingual but English-dominant - also French, German, Italian, Latin, Ancient Greek, and a curated Ukrainian collection.
• No Russian content, ever. This is a deliberate, code-enforced curation rule - Russian authors and Russia-themed works are refused on every harvesting path. (Curated Ukrainian and Polish authors are explicitly kept.)
• The text normalization is a derivative work and some Wikisource editorial apparatus may be CC-BY-SA, so please credit Project Gutenberg and Wikisource, and verify public-domain status in your own country.

The slow way (you almost certainly do not need this)

The books were originally harvested from Project Gutenberg and Wikisource. That path still exists for completeness, but it is slow (Gutenberg throttles heavy downloaders hard) and unnecessary now that the corpus is published - and all three tiers are locked read-only, so the harvest commands write nothing without an explicit unlock. For the record:

.venv/bin/python -m corpus add-author "Mark Twain"   # an author's complete Gutenberg works
make finalize                                        # clean -> dedup -> index -> stats

Prefer the Hugging Face pull above. It avoids all of this.

---

5. Train your own model (on your Mac)

This is the part where the model actually learns. You pick a size (a preset), point it at some text, and let it run. You do not need to understand the math: it prints a line every so often with two "loss" numbers (lower = the model is getting better) and saves itself to disk as it goes, so if your laptop sleeps or the run dies, nothing is lost - you just resume.

Here is what happens on each step of training:

Your first training run

To confirm everything works, run the smallest, fastest thing first - it finishes in seconds:

.venv/bin/python -m training --small

That is a tiny "smoke test" model. Then try a real preset:

.venv/bin/python -m training --preset nano   # smallest real preset, trains on the bundled toy Shakespeare
.venv/bin/python -m training                 # the default 'mini' preset (Karpathy's baseline)

Both train on lecture/input.txt (a small Shakespeare file that downloads automatically the first time). Results are saved under outputs/<preset>/.

The size presets

Bigger presets need a bigger corpus (section 4) - on tiny Shakespeare they just memorize it. Match the size to the data.

preset	params	layers x width	context	batch	learning rate	iterations	trains on
--small (smoke test)	~0.2M	3 x 64	32	16	1e-3	200	toy Shakespeare (seconds)
nano	0.82M	4 x 128	128	64	1e-3	3,000	lecture/input.txt
small	4.87M	6 x 256	128	64	5e-4	5,000	a small corpus
mini	10.89M	6 x 384	256	64	3e-4	5,000	lecture/input.txt (default; Karpathy's baseline)
medium	25.4M	8 x 512	256	40	3e-4	8,000	data/medium (~523M)
large	49.7M	10 x 640	384	16	2.5e-4	24,000	data/large (~1.08B)
xlarge	~100M	14 x 768	512	24	1.8e-4	48,000	data/xlarge (~2.05B)
2xlarge	202M	16 x 1024	512	16	1.5e-4	60,000	data/2xlarge (4.26B, locked; v1 trained)
4xlarge	~400M	20 x 1280	512	12	1.2e-4	80,000	data/4xlarge (10.18B, locked; model not yet trained)

The preset names past xlarge follow the AWS instance-size ladder (nano, small, medium, large, xlarge, 2xlarge, 4xlarge, ...). medium is trained and reaches a validation loss of 1.2285; large reaches 1.1111; xlarge reaches 0.9411; and 2xlarge (202M) is now trained too (val 0.8725, the strongest byob model), on a rented cloud H200 in about 12.6 hours for roughly $50 - see cloud_training/cloud_training_plan.md and cloud_training/runpod_runbook_2xlarge.md. You can train the small-to-large presets on a Mac; xlarge and up want a rented datacenter GPU. 4xlarge (~400M) is the one untrained rung left: its Chinchilla-scale corpus is finalized, locked, and published (10.18B chars, 4,132 authors, 33 categories), so training it is purely a cloud-GPU exercise - roughly a day on a single modern datacenter card (see cloud_training/runpod_runbook_4xlarge.md).

Order matters. Anything from medium up reads its books from data/<tier>/, so pull that tier first (section 4). Running --preset medium before data/medium/ exists stops with a file-not-found error. (nano/small/mini need no corpus - they train on the bundled toy Shakespeare.)

A real run on a real corpus

After pulling data/medium (section 4), train the medium model:

make train-medium

Always use the make train-* targets for long runs, not the bare python command. They wrap the run in caffeinate so macOS does not put the machine to sleep mid-training (which would kill the GPU process), and they save the log to outputs/medium/logs/train.log. The equivalent targets are make train-large, make train-xlarge, and so on.

If a run is interrupted, just resume it - it picks up from the latest checkpoint:

make resume-medium

Two ways to feed it text: --data vs --data-bin

• --data data/medium reads the raw .txt files and tokenizes them in memory. Simple, fine for the medium/large tiers on a Mac.
• --data-bin data/prepared/medium_bin reads a pre-tokenized .bin file (made by core.prepare, or pulled with PREPARED=1). It is memory-mapped, so it scales past your RAM - this is how the huge tiers and cloud runs work. --data-bin overrides --data.

.venv/bin/python -m training --preset medium --data-bin data/prepared/medium_bin

Overriding any single knob

Any one setting can be overridden on top of a preset; your flag wins:

.venv/bin/python -m training --preset mini --max-iters 8000 --block-size 384

What checkpoints are, and which one to use

A run writes two saved models under outputs/<preset>/, both self-contained (they carry the weights, the exact settings, and the vocabulary, so anything can rebuild the model from the file alone):

• outputs/<preset>/models/...pt - the latest model, rewritten as it trains (this is the crash-safe resume point).
• outputs/<preset>/best/...best.pt - the best (lowest validation loss) model so far. This is the one you want to sample from.

When you resume, the model size and vocabulary come from the checkpoint, so the data must be the identical corpus or the run aborts.

The two attention implementations (--attn-impl)

"Attention" is the part of the model that lets each character look back at the ones before it. This project ships it in two interchangeable forms, and you choose which one runs with --attn-impl:

• manual (the default) - the plain, lecture-faithful version: it works the attention out step by step by hand (the scores, the causal mask, the softmax, the weighted sum). It is the readable one, and it runs on any hardware.
• sdpa - calls PyTorch's built-in scaled_dot_product_attention, which uses the fused FlashAttention / cuDNN kernel on an NVIDIA GPU and the Metal kernel on a Mac. Same math, just faster and more memory-efficient.

The key point: both use the exact same weights (identical parameters and shapes), so a model trained with one loads and runs under the other with no conversion - it is purely a speed/kernel choice, and a test asserts the two produce the same output. You never need to touch this: manual is the default and just works. Reach for --attn-impl sdpa when you want the speed-up on a big GPU run.

.venv/bin/python -m training --preset medium --data-bin data/prepared/medium_bin --attn-impl sdpa

A few more advanced flags (mostly for cloud GPUs)

These exist for renting an NVIDIA GPU (section 6). On a Mac they are automatically off and make no difference - a plain make train-medium runs exactly as it always has:

• --lr-schedule {constant,cosine}, --warmup-iters N, --min-lr X - how the learning rate changes over the run.
• --amp / --no-amp, --no-compile - speed tricks that switch on only on CUDA.

Bigger is not better on tiny data. medium/large/xlarge only pay off on the big book corpus. On the toy Shakespeare text, anything past mini just memorizes. And remember: the result is a base model - prompt it with the start of something; it will not answer questions.

---

6. Train on a rented cloud GPU (for the big models)

Training the biggest models on a laptop would take days. The faster path is to rent an NVIDIA GPU by the second (this project documents RunPod), do the run in a ready-made Docker box, then stop the machine to stop paying. The full plan - costs, exact pod settings, the calibration run - is in cloud_training/cloud_training_plan.md. Here is the shape of it.

The same training code runs on both your Mac and a cloud GPU - the only difference is that on a real NVIDIA GPU a set of speed tricks (bf16, torch.compile, fused AdamW) switches on automatically; on a Mac they stay off and nothing changes.

The high-level recipe:

1. Prepare the corpus once on your laptop into a single compact file:

   make prepare DATA=data/xlarge OUT=data/xlarge_bin
   # = python -m core.prepare --data data/xlarge --out data/xlarge_bin

2. Get the .bin to the GPU. Either publish it to Hugging Face once and pull it on the pod, or pull the already-published one:

   # on the pod:
   .venv/bin/python -m corpus.hf_dataset pull --repo-id disco-jack-basement/byob-pd-book-corpus --tier xlarge --prepared --out data

3. Build the Docker image and run training, writing checkpoints to a persistent volume so they survive the machine being torn down:

   docker build -f Dockerfile.train -t byob-train .   # = make docker-train
   docker run --gpus all -v /workspace:/workspace byob-train \
       --preset xlarge --data-bin /workspace/xlarge_bin --device cuda \
       --out /workspace/outputs/xlarge/models/byob-lm.pt \
       --lr-schedule cosine --warmup-iters 2000 --batch-size 128 --max-iters <NUMBER>
   # replace <NUMBER> with a real iteration count - e.g. 48000; see cloud_training/cloud_training_plan.md for sizing

4. Do a short calibration run first (a small --max-iters) to check timing and cost, then do the full run. Stop the pod when done - per-second billing means an idle GPU still costs money.

A checkpoint trained on a cloud GPU loads on your laptop unchanged, so you can sample, chat, and benchmark it back home. Read cloud_training/cloud_training_plan.md before spending money - the cost and time tables there are floors, not promises.

---

7. Benchmark a model (how good is it?)

The bundled lm_bench subproject (in lm_bench/, with its own README) grades a model: it gives it text it never trained on and measures how well it predicts the next character. The headline number is bits-per-character - lower is better (around 6.0 means random guessing).

Its own environment. lm_bench keeps its own .venv inside lm_bench/ (set it up the same way as section 1; this repo's ../.venv/bin/python also works). Run all benchmark commands from inside the lm_bench/ folder.

The fastest first benchmark - score the shipped demo model on a held-out public-domain book that ships with the repo (offline, about 30 seconds):

# run from inside lm_bench/
.venv/bin/python -m lm_bench run \
    --model byob:../model_history/lecture-nano/v1/shakespeare-nano.pt \
    --tasks "bpc:test_data/medium/shelley-frankenstein.txt" --limit 20

Score a full tier with the standard scorecard (the byob:<tier> alias finds the checkpoint automatically). On a fresh clone only the nano demo ships - grab any trained tier from the model repo first (e.g. hf download disco-jack-basement/letterpress medium/byob-lm.best.pt --local-dir outputs/medium/best puts it where the alias looks):

.venv/bin/python -m lm_bench run --model byob:medium --tasks core
.venv/bin/python -m lm_bench run --model byob:large --tasks "bpc:test_data/medium"

The xlarge model scores about 1.31 bits-per-character on the held-out public-domain books in test_data/medium - the second-strongest byob model, after 2xlarge at 1.17 (large gets 1.57, medium 1.69; lower is better). You can also compare two saved runs into a plain-English report:

.venv/bin/python -m lm_bench compare benchmarks/runs/a.json benchmarks/runs/b.json --out comparison.md

How the five sizes compare

The presets tell one clean story: more scale buys better language modeling. Held-out bits-per-char (on public-domain books the models never saw) falls at every step - nano 2.68 -> medium 1.69 -> large 1.57 -> xlarge 1.31 -> 2xlarge 1.17 - and the out-of-domain WikiText number tracks it (4.09 -> 1.71). The 202M 2xlarge, trained on a rented cloud GPU over the 4.26B-char corpus, is the strongest model on every language-modeling metric.

What scale does not buy is world knowledge. On multiple-choice reasoning benchmarks (HellaSwag, ARC, PIQA, OpenBookQA) every size - 2xlarge included - scores at random chance: a character model has no facts to recall, only the shape of language. The one downstream task that does improve is LAMBADA (predict the last word of a passage), which is really a language-modeling test in disguise - it climbs from ~0% at nano through 14.8% at xlarge to 27.4% at 2xlarge as the bits-per-char drop. That is the honest ceiling of a character-level base model, and exactly what these numbers should show. Full scorecards live in lm_bench/benchmarks/.

---

8. Save a finished model so a retrain can't erase it

outputs/<preset>/ is a working directory - training a new model of the same preset overwrites it. Before you retrain, freeze the current model into a numbered, self-describing archive:

make version PRESET=medium       # = python -m core.version --preset medium

This creates model_history/<preset>/v<N>/ (auto-incrementing v1, v2, ...) containing:

• the model checkpoint (inference-ready; optimizer state stripped to save space);
• context.md - when it trained, how long it took, the exact settings, and the full validation-loss curve;
• corpus_index.md + corpus_stats.txt - the manifest of what it was trained on, so that exact corpus could be rebuilt.

So you can train, version, retrain on different data, and never lose the old model or the knowledge of how to reproduce it. The underlying command takes options like --note "first 1B run":

.venv/bin/python -m core.version --preset large --note "first 1B run"

Share the archives on Hugging Face

The versioned checkpoints are also published as a Hugging Face model repo - disco-jack-basement/letterpress - so anyone can grade or chat with a trained tier without retraining it (the "How to use" snippet is on the model card). To publish your own archives:

hf auth login                            # once; needs a WRITE token
.venv/bin/python -m core.hf_models publish --repo-id <you>/letterpress

---

9. Run the tests (a health check)

make test          # the pytest suite (175 tests) plus the end-to-end model selftest

make test runs .venv/bin/python -m pytest tests/ -q followed by .venv/bin/python -m inference.selftest. The pytest suite (175 tests) guards the things that matter: the tokenizer's round-trip, checkpoint save/load, the sampling loop, the model's shapes, the data loader, the corpus helpers, the no-Russian guard, the Hugging Face publish/pull paths, versioning, and that every package still imports. inference.selftest is a fast (about 3 seconds) end-to-end check that actually trains the model for a few steps and confirms the loss drops.

(Remember the test dependency: pip install -r requirements-dev.txt first.)

---

10. How the code is organized

The project is split into packages by job. There is one shared core that both training and inference build on, so the two can never disagree about how text is encoded or how the model is run.

In plain layout form:

core/        the shared library everything imports
               config.py      the model's settings (a dataclass) + size presets
               model.py       the transformer itself (attention, MLP, blocks)
               tokenizer.py   the char <-> integer codec (+ input cleanup)
               data.py        loads a text corpus into training batches (in-RAM or memmap .bin)
               prepare.py     pre-tokenize text -> train.bin / val.bin / meta.json
               checkpoint.py  save / load a model to / from a .pt file
               generate.py    the sampling loop (used by both sample and interact)
               paths.py       where each preset's files live
               version.py     archive a trained model (section 8)

training.py  the training loop (one module - run `python -m training`)
inference/   sample.py        one-shot text generation
             interact.py      the live REPL
             selftest.py      fast correctness checks on the model
corpus/      common.py        shared helpers (file IO, slugs, cleaning, no-Russian guard)
             manage.py        the corpus CLI: stats / clean / dedup / index
             hf_dataset.py    publish / pull the corpus to / from Hugging Face
             harvesting/      downloading books (Project Gutenberg, Wikisource)
lecture/     gpt.py, bigram.py, attention_steps.py (line-faithful demos) + input.txt (toy data)
perf_tooling/  train_timing.py, mem_probe.py, gen_timing.py    capacity-planning probes (time / RAM / generation)
lm_bench/      the bundled benchmark harness (own README + tests; section 7 - run from inside it)
tests/         the pytest suite (175 tests) + conftest.py fixtures

outputs/<preset>/        models/, best/, logs/   (working checkpoints + train logs; gitignored)
model_history/<preset>/  v1/, v2/ ...            (frozen archives; incl. lecture-nano/v1/shakespeare-nano.pt, the demo)
indices/                 medium.md ... 4xlarge.md   (lists of every book in each of the five tiers)
data/                    medium/ ... 4xlarge/ (+ prepared *_bin/ caches)   (corpus text; gitignored, it is large)

Dockerfile.train, Dockerfile.bench, cloud_training/cloud_training_plan.md   (the cloud-GPU path, section 6)

Every command is run with python -m <package>.<module> (no loose scripts at the top level). The Makefile wraps the common ones - run make help to see them all.

---

11. Good things to remember about attention

1. It is a communication mechanism on a graph. Here the graph is causal: each token may look at itself and earlier tokens only. Remove that mask and you have an encoder instead.
2. It has no built-in sense of order - it acts on a set. That is why we add positional embeddings; without them the model cannot tell token order apart.
3. Batch elements never interact. The batch dimension is just many independent sequences processed in parallel.

And about the project as a whole: this is the pretraining stage only - the model learns to babble in the style of its corpus. It is character-level (a real GPT uses sub-word tokens) and kept deliberately close to the lecture for readability, while adding the engineering (memmap data, a device-gated cloud-GPU recipe, corpus publishing) you need to take it past a toy.

---

12. License & credits

• Code: MIT (see LICENSE). Copyright (c) 2026 Kyrylo Novotarskyi.
• Built up from Andrej Karpathy's Let's build GPT lecture and nanoGPT - both MIT, (c) 2022 Andrej Karpathy. The lecture/ files are line-faithful to the video; attribution is in NOTICE.
• Corpus: CC0 (public-domain dedication), published separately at disco-jack-basement/byob-pd-book-corpus. Texts from Project Gutenberg and Wikisource, curated under a strict no-Russian-content rule. Code is MIT, data is CC0 - two different licenses.
• Contributing: see CONTRIBUTING.md.
• A note on names: this repo was formerly byob_llm ("build your own bot"). The model family keeps that historical name - checkpoints are byob-lm.*.pt, the benchmark adapter is byob:<tier>, and the published dataset is byob-pd-book-corpus. Old GitHub URLs redirect.