GPT from Scratch on OpenWebText

A compact decoder-only Transformer implemented directly in PyTorch and trained on a streamed OpenWebText sample. The implementation includes causal multi-head self-attention, pre-norm residual blocks, a four-times-width MLP, sinusoidal positions, tied token/output weights and autoregressive top-k sampling.

Recorded experiment

• 6 Transformer blocks, 6 attention heads, embedding width 384
• Context length 256 and GPT-2 tokenization (50,257 tokens)
• Approximately 29.9 million parameters
• 50 million streamed OpenWebText tokens
• 20,000 optimization steps on a GPU
• Training time: 52.14 minutes
• Training loss at step 20,000: 3.7404
• Held-out loss at step 20,000: 7.8685

The widening train/validation gap is retained in the report because it is an important diagnostic: the selected run overfit its fixed 9,000-window training sample even though generated text became more locally coherent.

Code

model.py   GPT configuration, attention, blocks, model and generation
train.py   streaming data preparation, training, evaluation and plots

Run

python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -r requirements.txt

# Small pipeline check
python train.py --max-tokens 100000 --steps 10 --batch-size 4

# Recorded-scale configuration
python train.py --max-tokens 50000000 --steps 20000 --batch-size 32

OpenWebText is downloaded by the Hugging Face datasets library and is not stored in this repository. Model weights, raw text, logs and generated samples are excluded from Git.

What I learned

• Implementing masking, head reshaping and weight tying without a high-level model class
• Building a bounded streaming data path for a dataset much larger than memory
• Separating optimization loss from held-out evaluation
• Diagnosing overfitting from both quantitative curves and generation behavior