implementing dl from scratch using first principles.
This repository is a collection of deep learning concepts and architectures implemented completely from scratch using:
- Python
- NumPy
- Mathematics
- Matrix operations
- Basic deep learning fundamentals
No high-level training abstractions.
No black-box frameworks doing everything automatically.
The purpose of this repository is to deeply understand:
- how neural networks actually learn
- how gradients flow
- how backpropagation works
- why transformers work
- how generative models are trained
- how reinforcement learning algorithms optimize behavior
Each chapter is written as an educational notebook with:
- mathematical intuition
- step-by-step implementations
- visual explanations
- forward & backward passes
- training logic
- optimization details
The basic building block of neural networks.
- Binary classification
- Linear decision boundaries
- Weighted sums
- Threshold activation
- perceptron training
- weight updates
- prediction logic
- binary classification examples
Non-linear transformations used inside neural networks.
- Sigmoid
- Tanh
- ReLU
- Leaky ReLU
- Vanishing gradients
- activation visualizations
- forward computation
- derivative understanding
Measures how wrong a model’s predictions are.
- Mean Squared Error
- Cross Entropy Loss
- Binary classification loss
- Multi-class loss
- loss computation
- gradient intuition
- visualization of loss behavior
Core algorithm behind neural network learning.
- Chain rule
- Gradient computation
- Partial derivatives
- Computational graphs
- manual gradient calculations
- backward propagation
- parameter updates
Multi-layer neural network implementation from scratch.
- Hidden layers
- Forward propagation
- Neural network architecture
- dense layers
- forward pass
- backward pass
- training loop
Proper initialization for stable training.
- Xavier Initialization
- He Initialization
- Gradient stability
- initialization experiments
- training comparisons
Optimization algorithms used during training.
- SGD
- Momentum
- RMSProp
- Adam
- optimizer update rules
- parameter optimization
- learning rate behavior
Techniques to reduce overfitting.
- Dropout
- L2 Regularization
- Generalization
- dropout logic
- penalty terms
- regularized training
Convolutional Neural Networks for image understanding.
- Convolution operation
- Feature extraction
- Filters & kernels
- convolution operations
- feature maps
- CNN intuition
Dimensionality reduction in CNNs.
- Max Pooling
- Average Pooling
- Spatial reduction
- pooling operations
- feature reduction
- visualization
Recurrent Neural Networks for sequential data.
- Hidden states
- Sequence processing
- Temporal dependencies
- recurrent loops
- sequence prediction
- hidden state updates
Long Short-Term Memory networks.
- Forget gate
- Input gate
- Output gate
- Long-term dependencies
- gating mechanisms
- memory cell operations
- sequence learning
Simplified recurrent architecture.
- Update gate
- Reset gate
- Efficient sequence learning
- GRU cell mechanics
- hidden state transitions
Neural attention mechanisms.
- Query
- Key
- Value
- Context vectors
- attention score computation
- weighted representations
Core transformer mechanism.
- token relationships
- contextual understanding
- attention matrices
- self-attention blocks
- scaled dot-product attention
Injecting sequence order into transformers.
- sinusoidal encoding
- positional information
- positional vector generation
- transformer positional understanding
Dense vector representations of tokens.
- semantic representation
- embedding space
- vector similarity
- embedding lookup logic
- token vectorization
Neural network for representation learning.
- Encoder-decoder architecture
- Latent space
- Reconstruction loss
- encoding
- decoding
- reconstruction training
Probabilistic generative model.
- Latent distributions
- KL Divergence
- Variational inference
- probabilistic latent sampling
- VAE loss
- reparameterization trick
Generative Adversarial Networks.
- Generator
- Discriminator
- Adversarial training
- generator training
- discriminator optimization
- adversarial loss
Deep Convolutional GAN implementation.
- convolutional generators
- image synthesis
- stable GAN training
- DCGAN architecture
- MNIST experiments
- image generation pipeline
Value-based reinforcement learning.
- Q-table
- Bellman Equation
- Exploration vs Exploitation
- reward optimization
- policy updates
- environment interaction
Policy-based reinforcement learning.
- policy optimization
- reward maximization
- gradient-based RL
- policy learning
- reward-driven updates
- stochastic optimization
- Python
- NumPy
- Matplotlib
- Jupyter Notebook
dl-from-scratch/
│
├── 01. Perceptron/
├── 02. Activation Functions/
├── 03. Loss Functions/
├── 04. Backpropagation/
├── 05. Feedforward NN/
├── 06. Weight Initialization/
├── 07. Optimizers/
├── 08. Regularization/
├── 09. CNN/
├── 10. Pooling/
├── 11. RNN/
├── 12. LSTM/
├── 13. GRU/
├── 14. Attention/
├── 15. Self-Attention/
├── 16. Positional Encoding/
├── 17. Embeddings/
├── 18. Autoencoder/
├── 19. Variational Autoencoder/
├── 20. GAN/
├── 21. DCGAN/
├── 22. Q-Learning/
└── 23. Policy Gradient/Most people use deep learning frameworks like this:
model = NeuralNetwork()
model.fit(X, y)without fully understanding:
- how gradients are computed
- how backpropagation works
- why transformers use attention
- how GANs actually learn
- how recurrent networks remember sequences
- how optimization updates parameters
This repository focuses on:
- intuition first
- implementation second
- frameworks later
Clone the repository:
git clone https://github.com/piyushdev04/dl-from-scratch.gitMove into the project:
cd dl-from-scratchInstall dependencies:
pip install numpy matplotlib notebookRun Jupyter Notebook:
jupyter notebookIf this repository helped you learn something, consider giving it a ⭐