No PyTorch. •
No TensorFlow. •
No Hidden Abstractions.
Just Mathematics → NumPy → Transformer.
Transformers From First Principles is an educational repository that explains the complete Transformer architecture by implementing every major component from scratch using only NumPy.
Unlike most repositories that rely on deep learning frameworks, this project focuses on how Transformers actually work internally, translating the mathematical equations directly into NumPy code.
Whether you're a student, researcher, or ML enthusiast, this repository is designed to help you understand the mechanics behind modern Transformer models.
Most tutorials teach how to use Transformers.
Very few teach how they work.
This repository bridges that gap by providing:
- Beginner-friendly documentation
- Mathematical derivations with equations
- Pure NumPy implementation
- Complete Encoder & Decoder implementation
- Manual forward and backward propagation
- Structured learning roadmap
- Visual explanations with diagrams
- Modular and easy-to-follow source code
This project aims to help readers:
- Understand every component of the Transformer architecture.
- Translate mathematical equations into NumPy code.
- Learn how forward and backward propagation work internally.
- Explore Encoder, Decoder, and Attention mechanisms.
- Gain intuition behind modern Large Language Models.
flowchart TD
subgraph Level_1["Level 1 — Foundations"]
A["Introduction"]
B["Why Transformers"]
C["Matrix Multiplication"]
end
subgraph Level_2[" Level 2 — Input Representation"]
D["Embeddings"]
E["Positional Encoding"]
end
subgraph Level_3[" Level 3 — Attention Mechanism"]
F["Attention"]
G["Scaled Dot-Product"]
H["Multi-Head Attention"]
end
subgraph Level_4[" Level 4 — Transformer Blocks"]
I["Residual + LayerNorm"]
J["Feed Forward Network"]
K["Encoder"]
L["Decoder"]
end
subgraph Level_5[" Level 5 — Complete Architecture"]
M["Full Transformer"]
N["Training"]
O["GPT vs BERT vs Transformer"]
end
A --> B --> C --> D --> E --> F --> G --> H --> I --> J --> K --> L --> M --> N --> O
A complete explanation of every file is available in :
The documentation gradually builds your understanding of the Transformer architecture from basic concepts to the complete model.
Topics include:
- Introduction
- Why Transformers
- Matrix Multiplication
- Embeddings
- Positional Encoding
- Attention
- Scaled Dot-Product Attention
- Multi-Head Attention
- Residual Connections & LayerNorm
- Feed Forward Network
- Encoder
- Decoder
- Full Transformer
- Training
- GPT vs BERT vs Transformer
The implementation is organized into modular components.
Basic neural network layers implemented using NumPy.
-
Linear Layer
-
Embedding Layer
-
Layer Normalization
-
Positional Encoding
-
Softmax
-
ReLU
Attention mechanisms.
-
Self Attention
-
Multi Head Attention
-
Masked Attention
-
Cross Attention
-
Scaled Dot Product Attention
Complete Encoder implementation.
Complete Decoder implementation.
Complete Encoder–Decoder Transformer architecture.
Clone the repository
git clone https://github.com/KARTHIK1749/transformers-from-first-principles.gitMove into the project
cd transformers-from-first-principlesInstall dependencies
pip install -r requirements.txtThis repository is intended for:
- Students learning Deep Learning
- Machine Learning enthusiasts
- AI Researchers
- Interview preparation
- Anyone curious about Transformer internals
- Developers who want to understand what happens under the hood of modern frameworks
Current progress can be found in :
Upcoming improvements include:
- Vision Transformer (ViT)
- Rotary Positional Embeddings (RoPE)
- Flash Attention
- KV Cache
- Tiny GPT implementation
- Additional worked mathematical examples
If you'd like to improve the repository, fix bugs, enhance documentation, or add educational examples, please read
before opening a Pull Request.
If you have suggestions, spot an issue, or simply want to discuss AI, feel free to open an issue or connect with me.