Makemore Part 3 - Activations, Gradients & Training Dynamics

A character-level language model exploring neural network internals, activation distributions, and training dynamics. This project builds upon the MLP architecture from Part 2, focusing on understanding how neural networks learn through analysis of pre-activations, gradient flow, and learning rate scheduling.

Overview

Makemore Part 3 takes a deeper look under the hood of neural network training. While Part 2 introduced the multi-layer perceptron architecture, this project focuses on understanding the training process itself: how activations flow through the network, why certain initializations work better than others, and how learning rate scheduling affects convergence.

Features

• Pre-Activation Analysis: Examining hidden layer values before tanh activation
• Learning Rate Decay: Step-based scheduling (0.1 → 0.01) for fine-tuned convergence
• Loss Landscape Visualization: Tracking log-loss over 200k iterations
• Train/Dev/Test Evaluation: Proper split loss computation with gradient disabled
• Reproducible Training: Seeded random generators for consistent results
• Name Generation: Sampling from trained model with controlled randomness

Project Structure

makemore_3/
├── makemore_3.ipynb     # Main Jupyter notebook with training dynamics analysis
├── names.txt            # Dataset of 32,033 names
└── README.md            # This file

Dataset

The model is trained on names.txt, containing 32,033 names with varying lengths. The dataset is properly split into:

• Training set: 80% (182,580 examples) - for parameter optimization
• Development/Validation set: 10% (22,767 examples) - for hyperparameter tuning
• Test set: 10% (22,799 examples) - for final model evaluation

Technical Implementation

1. Character Vocabulary & Encoding

# Build vocabulary of characters and mappings to/from integers
chars = sorted(list(set(''.join(words))))
stoi = {s: i+1 for i, s in enumerate(chars)}
stoi['.'] = 0  # Special start/end token
itos = {i: s for s, i in stoi.items()}
vocab_size = 27  # 26 letters + special token

2. Context Window Construction

block_size = 3  # Context length: how many characters to predict next one

def build_dataset(words):
    X, Y = [], []
    for w in words:
        context = [0] * block_size
        for ch in w + '.':
            ix = stoi[ch]
            X.append(context)
            Y.append(ix)
            context = context[1:] + [ix]  # Sliding window
    return torch.tensor(X), torch.tensor(Y)

3. Neural Network Architecture

Embedding Layer:

• Character embeddings: 27 characters × 10-dimensional vectors
• Embedding matrix C: (27, 10)

Hidden Layer:

• 200 neurons with tanh activation
• Weight matrix W1: (30, 200) - maps concatenated embeddings to hidden
• Bias vector b1: (200,)

Output Layer:

• 27 output neurons (one per character)
• Weight matrix W2: (200, 27)
• Bias vector b2: (27,)
• Softmax activation for probability distribution

Total Parameters: 11,897 learnable parameters

Input: 3 characters (context window)
    ↓
Embedding Lookup: C[context] → (3, 10)
    ↓
Concatenate: view(-1, 30) → (30,)
    ↓
Pre-activation: embcat @ W1 + b1 → (200,)
    ↓
Activation: tanh(hpreact) → (200,)
    ↓
Output: h @ W2 + b2 → (27,)
    ↓
Softmax → Probability Distribution

4. Forward Pass with Pre-Activation Tracking

# Forward pass
emb = C[Xb]                           # Embed characters into vectors
embcat = emb.view(emb.shape[0], -1)   # Concatenate the vectors
hpreact = embcat @ W1 + b1            # Hidden layer pre-activation
h = torch.tanh(hpreact)               # Hidden layer activation
logits = h @ W2 + b2                  # Output logits
loss = F.cross_entropy(logits, Yb)    # Cross-entropy loss

5. Training Process with Learning Rate Decay

max_steps = 200000
batch_size = 32
lossi = []

for i in range(max_steps):
    # Mini-batch construction
    ix = torch.randint(0, Xtr.shape[0], (batch_size,), generator=g)
    Xb, Yb = Xtr[ix], Ytr[ix]

    # Forward pass
    emb = C[Xb]
    embcat = emb.view(emb.shape[0], -1)
    hpreact = embcat @ W1 + b1
    h = torch.tanh(hpreact)
    logits = h @ W2 + b2
    loss = F.cross_entropy(logits, Yb)

    # Backward pass
    for p in parameters:
        p.grad = None
    loss.backward()

    # Update with learning rate decay
    lr = 0.1 if i < 100000 else 0.01  # Step decay at 100k
    for p in parameters:
        p.data += -lr * p.grad

    # Track statistics
    lossi.append(loss.log10().item())

6. Evaluation with Gradient Disabled

@torch.no_grad()  # Decorator disables gradient tracking
def split_loss(split):
    x, y = {
        'train': (Xtr, Ytr),
        'val': (Xdev, Ydev),
        'test': (Xte, Yte),
    }[split]
    emb = C[x]
    embcat = emb.view(emb.shape[0], -1)
    h = torch.tanh(embcat @ W1 + b1)
    logits = h @ W2 + b2
    loss = F.cross_entropy(logits, y)
    print(split, loss.item())

Model Performance

Training Progress

      0/200000: 3.3148
  10000/200000: 2.6505
  20000/200000: 2.4705
  30000/200000: 2.0266
  ...
 100000/200000: 2.4564  ← Learning rate drops to 0.01
  ...
 190000/200000: 2.0372

Loss Metrics

• Initial Loss: ~3.31 (random initialization)
• Final Training Loss: ~2.04
• Convergence: Smooth descent with step decay at 100k iterations

Sample Generated Names

carlah
amorie
khirmin
rey
cassanden
jazhubedah
sart
kaeli
nellara
chaiir
kaleigh
ham
jore
quint
salin
alianni
wazthoniearyxi
jace
pirran
eddeci

Key Concepts Demonstrated

Training Dynamics

• Pre-Activation Distribution: Understanding hpreact = embcat @ W1 + b1 values
• Tanh Saturation: When pre-activations are too large (>|2|), gradients vanish
• Learning Rate Scheduling: Step decay for coarse-to-fine optimization
• Loss Landscape: Visualizing training progress with log-scale loss

Neural Network Internals

• Weight Initialization: Scaling W2 by 0.01, zeroing b2 for stable start
• Gradient Flow: How gradients propagate through tanh activation
• Mini-Batch Statistics: Loss variance across different batches
• Reproducibility: Using seeded generators for consistent experiments

PyTorch Techniques

• @torch.no_grad(): Disabling gradients for evaluation efficiency
• Generator Objects: torch.Generator().manual_seed() for reproducibility
• In-Place Updates: p.data += -lr * p.grad for parameter updates
• Log-Scale Tracking: loss.log10().item() for better visualization

Comparison with Previous Parts

Aspect	Part 1 (Bigram)	Part 2 (MLP)	Part 3 (This)
Context	1 character	3 characters	3 characters
Architecture	Single layer	MLP	MLP with analysis
Parameters	729	11,897	11,897
Focus	Probability tables	Embeddings & architecture	Training dynamics
Learning Rate	Fixed	Fixed 0.1	Decay 0.1 → 0.01
Analysis	Basic loss	Train/dev split	Pre-activation tracking
Key Insight	Bigram statistics	Learned embeddings	Gradient flow

Visualization

The notebook includes:

• Training Loss Curve: Log-scale loss over 200k iterations
• Learning Rate Impact: Visible inflection at step decay point
• Loss Smoothing: Understanding batch-to-batch variance

Requirements

• Python 3.7+
• PyTorch
• Matplotlib
• Jupyter Notebook

Installation

# Clone the repository
git clone https://github.com/Jaloch-glitch/makemore_3.git
cd makemore_3

# Install dependencies
pip install torch matplotlib jupyter

Usage

# Start Jupyter Notebook
jupyter notebook makemore_3.ipynb

Run all cells to:

1. Load and split the dataset (80/10/10)
2. Build character vocabulary and context windows
3. Initialize neural network with proper scaling
4. Train with learning rate decay over 200k steps
5. Visualize training loss curve
6. Evaluate on train/validation splits
7. Generate new names from the trained model

Future Enhancements

• Add batch normalization for stable activations
• Implement activation histograms at each layer
• Visualize gradient magnitudes during training
• Experiment with different learning rate schedules (cosine, exponential)
• Add dead neuron detection (tanh saturation analysis)
• Implement gradient clipping for stability
• Compare different weight initialization schemes
• Add residual connections for deeper networks
• Explore LayerNorm as alternative to BatchNorm

Educational Value

This project is ideal for:

• Understanding what happens inside neural networks during training
• Learning about activation saturation and vanishing gradients
• Practicing learning rate scheduling techniques
• Gaining intuition for weight initialization importance
• Visualizing training dynamics and loss landscapes
• Building debugging skills for neural network training
• Understanding the @torch.no_grad() decorator

Series Progress

• Part 1: Bigrams - Character-level bigram model
• Part 2: MLP - Multi-layer perceptron with embeddings
• Part 3: Activations - Training dynamics and gradient flow (this repo)
• Part 4: Batch Normalization - Normalizing activations for stable training
• Part 5: Wavenet - Dilated causal convolutions

References

This implementation follows Andrej Karpathy's "makemore" tutorial series, Part 3, which focuses on understanding neural network training dynamics. Andrej Karpathy is a renowned AI researcher, former Director of AI at Tesla, and founding member of OpenAI.

Resources:

• Andrej Karpathy's YouTube: Neural Networks: Zero to Hero
• Building makemore Part 3: Activations & Gradients
• Original makemore repository

Author

Felix Onyango

• GitHub: @Jaloch-glitch
• Location: Kenya, East Africa

License

This project is open source and available for educational purposes.

Acknowledgments

• Andrej Karpathy: This implementation follows his excellent "makemore" tutorial series (Part 3), focusing on training dynamics and neural network internals
• Dataset of names from public sources
• Built as part of deep learning fundamentals education
• Thanks to the PyTorch team for an excellent deep learning framework

---

Note: This is an educational project focusing on understanding neural network training dynamics. The emphasis is on why networks train the way they do, not just how to train them. For production systems, consider established frameworks with built-in normalization and optimization techniques.