ML From Scratch

implementing ml from scratch — just to actually understand how it works from first principles.

i think it’s important to reason from first principles rather than by analogy. ~ elon musk

---

About

This repository is a collection of machine learning algorithms implemented completely from scratch using only:

• Python
• NumPy
• Matplotlib
• Basic mathematics

No scikit-learn training APIs.
No black-box abstractions.

The goal of this project is simple:

understand what actually happens inside machine learning algorithms instead of only importing libraries.

Each implementation is written in a clean and educational notebook format with:

• mathematical intuition
• step-by-step implementation
• visualizations
• training logic
• loss calculations
• optimization process

---

Algorithms Implemented

Supervised Learning

1. Linear Regression

Predicts continuous numerical values using a linear relationship.

Concepts Covered

• Mean Squared Error (MSE)
• Gradient Descent
• Parameter optimization
• Loss minimization
• Regression line visualization

What’s Implemented

• synthetic data generation
• forward propagation
• loss calculation
• manual gradient updates
• training loop
• plotting predictions vs actual values

---

2. Logistic Regression

Binary classification using probability estimation.

Concepts Covered

• Sigmoid activation
• Cross Entropy Loss
• Binary classification
• Decision boundary

What’s Implemented

• probability prediction
• gradient descent optimization
• classification accuracy
• visualization of predictions
• loss curve plotting

---

3. Softmax Regression

Extension of logistic regression for multi-class classification.

Concepts Covered

• Softmax function
• Multi-class probabilities
• Cross entropy loss

What’s Implemented

• multi-class prediction
• probability distributions
• training process
• visualization

---

4. K-Nearest Neighbors (KNN)

Simple distance-based classification algorithm.

Concepts Covered

• Euclidean distance
• Majority voting
• Lazy learning

What’s Implemented

• nearest neighbor search
• customizable K value
• classification visualization

---

5. Naive Bayes

Probabilistic classification using Bayes’ theorem.

Concepts Covered

• Conditional probability
• Gaussian distributions
• Bayesian inference

What’s Implemented

• probability calculations
• feature likelihood estimation
• prediction pipeline

---

6. Decision Trees

Tree-based classification using recursive splitting.

Concepts Covered

• Entropy
• Information Gain
• Recursive tree construction

What’s Implemented

• best feature selection
• node creation
• recursive splitting
• prediction traversal
• tree structure logic

---

7. Random Forest

Ensemble learning using multiple decision trees.

Concepts Covered

• Bagging
• Ensemble learning
• Variance reduction

What’s Implemented

• multiple tree training
• aggregated predictions
• robust classification

---

8. Support Vector Machine (SVM)

Classification using optimal separating hyperplanes.

Concepts Covered

• Margin maximization
• Hyperplanes
• Linear classification

What’s Implemented

• SVM optimization logic
• training procedure
• decision boundary visualization

---

Unsupervised Learning

9. Principal Component Analysis (PCA)

Dimensionality reduction technique.

Concepts Covered

• Covariance matrix
• Eigenvalues & Eigenvectors
• Variance preservation

What’s Implemented

• feature normalization
• covariance computation
• principal component extraction
• data projection
• variance visualization

---

10. K-Means Clustering

Centroid-based clustering algorithm.

Concepts Covered

• Cluster assignment
• Centroid optimization
• Iterative convergence

What’s Implemented

• centroid initialization
• cluster updates
• convergence detection
• cluster visualization

---

11. DBSCAN

Density-based clustering algorithm.

Concepts Covered

• Core points
• Border points
• Noise detection
• Density connectivity

What’s Implemented

• epsilon neighborhood search
• density expansion
• arbitrary-shaped clustering
• noise identification

---

Probabilistic Models

12. Hidden Markov Model (HMM)

Sequence modeling using hidden states.

Concepts Covered

• Hidden states
• Transition probabilities
• Emission probabilities
• Sequential prediction

What’s Implemented

• probability calculations
• sequence inference
• forward-style computations
• visualization and examples

---

Tech Stack

• Python
• NumPy
• Matplotlib
• Jupyter Notebook

---

Project Structure

ml-from-scratch/
│
├── 01. Linear Regression/
├── 02. Logistic Regression/
├── 03. Softmax Regression/
├── 04. KNN/
├── 05. Naive Bayes/
├── 06. Decision Trees/
├── 07. Random Forest/
├── 08. SVM/
├── 09. PCA/
├── 10. K-Means/
├── 11. DBSCAN/
└── 12. Hidden Markov Model/

---

Why This Repository Exists

Most people learn ML like this:

from sklearn import ...
model.fit(X, y)

but never really understand:

• how gradients are computed
• why loss decreases
• how optimization works
• what the model is mathematically learning
• why algorithms behave differently

This repository focuses on:

• intuition first
• implementation second
• libraries later

---

How to Run

Clone the repository:

git clone https://github.com/piyushdev04/ml-from-scratch.git

Move into the project:

cd ml-from-scratch

Install dependencies:

pip install numpy matplotlib notebook

Run Jupyter Notebook:

jupyter notebook

---

Connect

If this repository helped you learn something, consider giving it a ⭐

GitHub: https://github.com/piyushdev04/ml-from-scratch