Two complementary approaches to a hard real-world ML problem: (A) supervised classification under 14:1 class imbalance, and (B) active learning that reaches comparable performance with a fraction of the labels.
A two-part Machine Learning term project at King Fahd University of Petroleum and Minerals (KFUPM). The dataset has 11 features (f1–f11), one of them categorical, and a heavily imbalanced binary target (square vs circle).
The target is 93.3% majority class / 6.7% minority class — a 14:1 split.
A model that always predicts "majority" gets 93% accuracy and zero useful behavior. So the F1-score on the minority class is used as the primary metric throughout.
A clean preprocessing pipeline applied identically to every model:
- Numerical features (
f1–f10): mean imputation → standard scaling - Categorical feature (
f11): most-frequent imputation → one-hot encoding - All steps are fit on the training set only (no leakage)
- Stratified train/validation split
Each model wrapped in a scikit-learn Pipeline so preprocessing and the classifier are inseparable. Logistic Regression and Decision Tree were grid-searched; MLP and Random Forest were used with their tuned defaults from the assignment.
| Model | Minority-class F1 | Notes |
|---|---|---|
| Logistic Regression | 0.36 | Linear boundary not enough; grid search didn't help (folds with 0 minority samples gave NaN F1). |
| Decision Tree | 0.58 | Big jump — captures non-linearity. Best params were defaults. |
| Neural Network (MLP) | 0.66 | Selected as final model — best minority recall while staying balanced. |
| Random Forest | 0.62 | High precision, lower recall → conservative predictions. |
The MLP was used to generate predictions on the held-out test set and submitted to the course Kaggle leaderboard.
Setup: start with a pool of unlabeled samples. Label only 50 of them initially. Then iteratively pick which samples to label next, training a Logistic Regression model after each round. The question: which sampling strategy reaches good F1 with the fewest labels?
- Least Confidence sampling — query the unlabeled samples the model is most unsure about (lowest top-class probability).
- Entropy-based sampling — query the unlabeled samples with the highest predictive entropy across the class distribution.
Both were run with the same Logistic Regression base model, same starting seed, and same labeling budget steps (50 → 80 → 110 → 140 → 170 labels).
- Least Confidence improves steadily from the first iteration.
- Entropy sampling starts weaker but overtakes Least Confidence around 110 labeled samples and stays ahead, peaking near F1 ≈ 0.40 with only 110–170 labels.
- The fully supervised model in Part A still achieves higher absolute F1 (0.66 with the MLP), but it sees ~4 500 labeled training samples vs. ≤170 here. Active learning gets much closer per label.
When labeling is costly (medical, legal, expert review), entropy-based active learning is a strong default in the later stages of a labeling budget. Least Confidence is the safer pick for the very first batches.
ml-imbalanced-active-learning/
├── part_a_classification.ipynb # full Part A: EDA, preprocessing, 4 models, model selection
├── part_b_active_learning.ipynb # full Part B: 50-sample seed, 2 strategies, comparison plot
├── assets/
│ ├── class_distribution.png # 93/7 imbalance bar chart
│ └── active_learning_curves.png # LC vs Entropy F1 curves
├── requirements.txt
├── LICENSE
└── README.md
git clone https://github.com/Fai1Dude/ml-imbalanced-active-learning.git
cd ml-imbalanced-active-learning
pip install -r requirements.txt
jupyter notebookOpen either notebook and run all cells. The dataset CSVs (dataset-train-vf.csv, dataset-test-vf.csv) come from the course assignment and are not included in this repo; the notebook expects them at BASE_PATH = '/content/sample_data' (the path used in Google Colab). Update that line to point to wherever your copy of the data lives.
Note: because the dataset is imbalanced and the active-learning seed pool is small, results will vary slightly run-to-run. The qualitative pattern (Entropy overtaking LC after ~110 labels) is stable.
Python · scikit-learn · pandas · NumPy · matplotlib · Jupyter
Released under the MIT License. See LICENSE.
Faisal Alhamdi — B.Sc. Software Engineering, KFUPM LinkedIn · GitHub