Build Deep Learning From Scratch

A project-based curriculum that teaches how PyTorch works internally by reimplementing it from the ground up. You start with scalar backpropagation, build a reverse-mode autodiff engine, grow it into an N-dimensional Tensor, then stack neural-network layers, optimizers, training loops, CNNs, attention, a full Transformer, a Vision Transformer, a small but real PyTorch-like framework, and finally capstone projects. The philosophy: you are the autodiff library. Nothing is imported that you haven't already built by hand, so every gradient and every chain-rule accumulation is code you wrote and understand.

The scalar autodiff engine in stages 01-05 is a from-scratch reimplementation of Andrej Karpathy's micrograd: Value(data) (stage 01) → computational graph (stage 02) → per-op _backward closures (stage 03) → the backward() reverse pass (stage 04) → the remaining ops tanh/exp/relu (stage 05). The same engine micrograd packs into one file, built up one concept per stage. Everything afterward generalizes it to tensors.

Tool restriction

The only permitted packages are NumPy (forward array math only — never to compute a derivative for you), Matplotlib (visualization), and pytest (tests). No torch, tensorflow, jax, autograd, tinygrad, micrograd, or any library that does autodiff/backprop for you — until you've built that concept by hand in an earlier stage. See requirements.txt.

How a stage works

Each stage_xx/ directory contains three source files:

• README.md — context, background (intuition + key equations/gradients), watch (videos), and a zero-ambiguity exercise.
• code.py — skeleton only: interfaces, signatures, docstrings, and TODOs. No working bodies.
• test.py — pytest tests, including numerical gradient checks (central differences) wherever gradients exist.

Implement code.py until the tests pass:

pytest stage_xx/test.py

Each stage builds on the code from prior stages (e.g. Value from stage_05, Tensor from stage_08).

One import convention to know: a later stage pulls a symbol from the latest stage that extended it, not from the stage that first created it. Tensor is created in stage_08 but extended in stage_11 (broadcasting) and stage_12 (sum/mean reductions), so stages 13+ do stage_import("stage_12", "Tensor"); likewise Dense is created in stage_10 but extended in stage_11, so later stages import it from stage_11.

Solutions

Reference implementations live on the solutions branch (currently solved through stage 17; later stages are being added as the course is finalized). Check it out (git checkout solutions) only after you've attempted a stage yourself — the whole point is that you write every gradient. The main branch ships skeletons only.

The 35 stages

#	Stage	What you build
01	Scalar Values	Value: wrap one number; forward arithmetic via operator overloading.
02	Computational Graph	Record each result's parents (_prev set), op (_op), and a no-op _backward hook.
03	Local Derivatives	Install per-op _backward closures (the local-derivative push).
04	Chain Rule	backward(): topological sort + seed grad=1 + reverse-walk the closures.
05	Backprop Engine	Add tanh, exp, relu on the scalar Value; the complete micrograd engine.
06	Vector Operations	A Vec container of Values: elementwise ops, dot, sum.
07	Matrix Operations	A Mat of Values: matmul/@, transpose, reshape, sum, mean.
08	Tensor Engine	Collapse scalar graphs onto one N-dim NumPy-backed autodiff Tensor.
09	Neuron	A single learnable neuron y = phi(x @ w + b) on the Tensor.
10	Dense Layer	Vectorized fully-connected layer Z = X @ W + b.
11	MLP	Stack Dense layers with activations between them.
12	Loss Functions	Single-example MSE/MAE/cross-entropy (+ stable softmax) and sum/mean reductions.
13	Loss Functions (Batched)	Lift softmax/cross-entropy to a (B, C) batch; mean over the batch.
14	SGD Optimizer	The Optimizer/SGD update step abstraction.
15	First Training Loop	Wire MLP + loss + SGD into the canonical learn loop.
16	Weight Initialization	Xavier/Glorot and He/Kaiming init, and why scale matters.
17	Momentum	SGD with momentum.
18	Adam	RMSProp/Adam with bias correction and weight decay.
19	Batch Training	Minibatching, epochs, shuffling; gradient-variance intuition.
20	DataLoader	Dataset/DataLoader batching abstraction.
21	Regularization	L2 / weight decay; train vs eval mode.
22	Dropout	Dropout forward + backward; inverted scaling.
23	BatchNorm	Batch normalization forward + backward by hand.
24	Conv2D Math	Convolution arithmetic and gradients via im2col.
25	Conv2D Implementation	Conv2D/pooling/flatten as Tensor layers.
26	CNN Project	Stack conv/pool/linear; train on image data.
27	Attention Math	Scaled dot-product attention forward + backward (pure NumPy).
28	Self-Attention	Self-attention on the Tensor autodiff engine.
29	Multi-Head Attention	Split/concat heads; the full MHA module.
30	Transformer	Residuals + LayerNorm + FFN; a full Transformer block.
31	Vision Transformer	Patch embeddings + Transformer for image classification.
32	Framework Refactor	Package it all into a clean PyTorch-like Module/Parameter API.
33	Capstone: MNIST	End-to-end MNIST classifier.
34	Capstone: CIFAR-10	End-to-end CIFAR-10 classifier.
35	Capstone: Transformer	End-to-end Transformer language model.

Expected effort

Roughly 150-250 hours end to end, depending on background.

AI / Agent setup

This repo ships a tutor prompt at tutor_skill.md — a Socratic study partner that guides your thinking with hints and questions but won't hand you the solution (the point is that you write every gradient). It asks which stage you're on, reads that stage's files, and helps you reason to the answer.

Works with any AI coding agent (Cursor, Copilot, Codex, Claude, Gemini, local models, …): give the agent tutor_skill.md as its system prompt / custom instructions, then tell it which stage you're stuck on. (For Claude Code, drop tutor_skill.md into .claude/skills/tutor/SKILL.md to run it as the /tutor slash command.)

Community

Questions, help, and updates: r/pytorch_from_scratch.