My Own LLM From Scratch 🧠

A complete, educational implementation of a GPT-style Large Language Model (LLM) built from the ground up using PyTorch. This project demonstrates the core components of modern LLMs, including Multi-Head Attention, Layer Normalization, and Feed-Forward Networks.

🚀 Project Overview

This repository contains the code to build, train, and generate text with a GPT (Generative Pre-trained Transformer) model. It follows a modular approach, breaking down the complex architecture into understandable chapters/modules.

Key Features:

• Custom Tokenizer: Implementation of a Byte Pair Encoding (BPE) style tokenizer.
• Multi-Head Attention: Causal self-attention mechanism implementation.
• Transformer Blocks: Full GPT architecture with residual connections and dropout.
• Training Loop: Custom training pipeline with loss calculation and optimization.
• Text Generation: Sampling strategies for generating coherent text.

📂 Project Structure

The codebase is organized into logical modules corresponding to the development stages:

File	Description
ch02_data.py	Data processing, dataset creation, and tokenizer implementation.
ch03_attention.py	Implementation of Self-Attention and Multi-Head Attention mechanisms.
ch04_model.py	The core GPTModel architecture definition.
ch05_train.py	Training functions, loss computation, and the main training loop.
generate.py	Script to load the trained model and generate text.
model.pth	Saved weights of the trained model.
the-verdict.txt	Sample training dataset.

🛠️ Installation & Requirements

Ensure you have Python installed. Install the required dependencies:

pip install torch tiktoken

(Note: tiktoken is used for the tokenizer logic)

💻 Usage

1. Training the Model

To train the model from scratch using the provided dataset (the-verdict.txt):

python ch05_train.py

This will initialize the model, train it for the specified number of epochs, and save the weights to model.pth.

2. Generating Text

Once the model is trained (or if you have a pre-trained model.pth), you can generate text using the generation script:

python generate.py "Your starting text here"

Example:

python generate.py "The verdict is"

📚 Acknowledgements

This project is based on the concepts and architecture detailed in "Build a Large Language Model (From Scratch)" by Sebastian Raschka. It serves as a practical implementation to understand the inner workings of LLMs.