Statistical Machine Learning Showcase

This repository contains from-scratch implementations of fundamental statistical machine learning algorithms. The implementations are evaluated primarily on the MNIST digit classification dataset and synthetic 2D datasets.

Directory Structure

• Discriminant_Analysis/
  • Linear Discriminant Analysis (LDA) and Quadratic Discriminant Analysis (QDA) from scratch.
  • Dimensionality visualization using t-SNE.
• Dimensionality_Reduction/
  • Principal Component Analysis (PCA) for variance maximization and sample reconstruction.
  • Fisher Discriminant Analysis (FDA) to maximize class separability in a 2D projection space.
• Ensembles_and_Regularization/
  • Regularized Linear Regression: Ridge Regression (L2) from scratch and Lasso Regression (L1) via scikit-learn.
  • Decision Trees, Bagging, and Random Forests classification from scratch.
  • Regression Decision Stumps and Bagging regression from scratch.
• Boosting_and_Perceptrons/
  • AdaBoost classification using decision stumps from scratch.
  • Gradient Boosting (GBM) minimizing L1 loss using decision stumps.
  • Perceptron Learning Algorithm for linearly separable and non-separable synthetic 2D data.

Summary of Results

Model / Subspace	Task / Classes	Configuration	Metric (Accuracy / MSE)
LDA	MNIST (0, 1, 2)	Raw pixel space	96.0% accuracy
QDA	MNIST (0, 1, 2)	Raw pixel space	97.6% accuracy
PCA + QDA	MNIST (0, 1, 2)	75% variance subspace	97.7% accuracy
FDA + QDA	MNIST (0, 1, 2)	2D projection subspace	83.3% accuracy
Random Forest	MNIST (0, 1, 2)	5 Trees (From scratch)	84.6% accuracy
AdaBoost	MNIST (4 vs 9)	300 Stumps (From scratch)	96.8% accuracy
Gradient Boosting	MNIST (4 vs 9)	L1 loss, learning rate 0.01	0.14 test MSE
Perceptron	Synthetic (Separable)	Linearly separable data	100% convergence

Installation and Setup

Follow these steps to clone, configure, and run the project locally on your machine.

Prerequisites

Make sure you have Python 3.8 or higher installed. You can check your Python version by running:

python --version

1. Clone the Repository

Clone the repository using Git and navigate into the project directory:

git clone <repository-url>
cd statistical-ml-showcase

2. Create and Activate a Virtual Environment (Recommended)

To prevent system-wide package conflicts, create and activate a Python virtual environment:

• On macOS / Linux:

  python -m venv .venv
  source .venv/bin/activate

• On Windows (PowerShell):

  python -m venv .venv
  .venv\Scripts\Activate.ps1

• On Windows (Command Prompt / CMD):

  python -m venv .venv
  .venv\Scripts\activate.bat

3. Install Dependencies

Install all required libraries specified in the dependency file:

pip install --upgrade pip
pip install -r requirements.txt

4. Run the Streamlit Dashboard

Launch the interactive Streamlit application:

streamlit run app.py

Note: On the first execution, the app will automatically download the 11MB MNIST dataset (mnist.npz) from a public Google Storage bucket and save it to the ./data/ folder (which is ignored by Git). No manual download is required.

Once running, the dashboard will open automatically in your web browser at http://localhost:8501.