Predictive Risk Model & ML Feature Pipeline

A complete ML pipeline for clinical risk prediction: PySpark feature extraction from Delta Lake datasets, stratified train/test partitioning, hyperparameter tuning (GridSearchCV), batch scoring deployed as a scheduled Databricks notebook, with outputs served to Power BI dashboards.

Architecture

Delta Lake (Clinical Data) → PySpark Feature Engineering → Scikit-learn Model → Batch Scoring → Power BI Dashboard

Key Results

Metric	Score
F1 Score	0.78
Precision	0.81
Recall	0.75
AUC-ROC	0.84
Forecast Accuracy Improvement	+15%

Tech Stack

• PySpark (feature engineering at scale)
• Scikit-learn (model training & tuning)
• Delta Lake (data storage)
• Databricks (notebook deployment)
• Power BI (output dashboards)

Project Structure

predictive-risk-ml-pipeline/
├── README.md
├── requirements.txt
├── notebooks/
│   └── risk_model_training.py       # Full training pipeline
├── src/
│   ├── feature_engineering.py        # PySpark feature extraction
│   ├── model_training.py             # Scikit-learn model with GridSearchCV
│   ├── model_evaluation.py           # Metrics, confusion matrix, ROC
│   └── batch_scoring.py              # Production batch scoring
├── config/
│   └── model_config.yaml
└── tests/
    └── test_features.py

Setup

git clone https://github.com/Mayur97V/predictive-risk-ml-pipeline.git
cd predictive-risk-ml-pipeline
pip install -r requirements.txt
python -m src.model_training

Author

Mayur Gudala - LinkedIn | GitHub