Simple Reproduction of nanoGPT by Andrej Karpathy

This repository is a simple reproduction of the nanoGPT project, based on the well-known tutorial by Andrej Karpathy.

The model is a character-level, decoder-only Transformer (GPT) trained on a dataset of Harry Potter books.

Project Structure

• model.py: The main Python script containing the nanoGPT model architecture and text generation function.
• train.py: A second Python script containing the training loop and the hyperparameters.
• practice-dev-gpt.ipynb: A Jupyter Notebook used as a development and practice environment to build and test the model's concepts step-by-step, starting from a simple bigram model.

Model Architecture & Hyperparameters

The model in model.py is a decoder-only Transformer using the following main components:

• Token and Positional Embeddings
• Multiple Transformer Blocks, each consisting of:
  • Masked Multi-Head Self-Attention
  • A FeedForward Network
  • Layer Normalization and Residual Connections
• A final linear layer to predict the logits for the next character.

The key hyperparameters (defined in train.py) are:

• block_size: 128 (context length)
• batch_size: 64
• n_embed: 90 (embedding dimension)
• n_head: 6 (number of attention heads)
• n_layer: 2 (number of transformer blocks)
• epochs: 20000
• lr: 3e-4 (learning rate)
• dropout: 0.2

Setup and Execution

1. Dependencies

This project requires PyTorch.

pip install torch

2. Data

The script expects the training data at data/harry_potter.txt. This directory is included in the .gitignore and must be created manually. Download the dataset (e.g., from Kaggle) and place it in that location.

3. Training & Generation

Run the main script to train the model and subsequently generate 500 characters of text:

python train.py

The script will print the training and validation loss every eval_interval (200 steps) and print the generated text at the end. Additionally, the script saves a text-file of a longer output (10 000 characters) to example_out.txt.

4. Reproduction

The models' parameters are saved to nanogpt_model.pth.

License

This project is licensed under the MIT License. See the LICENSE file for details.