ECG Data Exploration, Preprocessing and Machine Learning

This project focuses on exploring/analyzing ECG data with literature-backed methods with the MIT-BIH Arrhythmia Database (PhysioNet). There is also focus on making these scripts more production ready in terms of how other teams can use the code developed.

NOTE: This project is now concluded. For a deep dive into methods and insights please visit this article.

Index

Project Information

1. Preprocessing
2. Data Analysis and Machine Learning
3. MLOps
4. Visuals

Setup and Usage

1. Setup Instructions
   • Option A: Docker (Recommended)
   • Option B: Local Setup
2. Usage
   • Using Docker
   • Using Local Setup
3. Contact

---

Preprocessing

• 0.5 - 30 Hz Filtering
• Notch filtering
• Detrend and baseline correction
• R-peak Detection
  • Write modules for Pam-Tompson QRS complex detection
  • Calibrate peaks after detection
  • Validate detected peaks with annotations as ground truth

Literature used:

• Rahul Kher (2019) Signal Processing Techniques for Removing Noise from ECG Signals. J Biomed Eng 1: 1-9
• https://martager.github.io/bbsig/ecg-preprocessing/

Feature Extraction and Engineering

• Extract R peak waveforms and other key features
• Build table with labels from annotations
• Extract RR interval features: pre/post RR, delta RR, RR ratio, local RR mean, deviation from local mean
• Extract QRS morphology features: Q-R interval, R-S interval, Q-R amplitude, R-S amplitude
• Extract wavelet features and use them with XGBoost

Literature used:

• https://inass.org/wp-content/uploads/2022/12/2023043016-2.pdf

data Analysis and Machine Learning

• Logistic regression with R waveforms only: Macro F1 = 0.41
• Logistic regression with waveforms + RR features: Macro F1 = 0.44
• Gradient boosting with waveforms + RR features: Macro F1 = 0.529
• SMOTE oversampling: Macro F1 = 0.520 (dropped: slowed training, no improvement)
• Feature ablation: waves + RR + QRS outperforms subsets; QRS features add value
• XGBoost with waveforms + RR + QRS features: Macro F1 = 0.577
• XGBoost hyperparameter optimization: randomized search with patient-wise CV
• RR irregularity feature (coefficient of variation of RR intervals)

ML Ops

• MLflow experiment tracking: all runs tracked with metrics, parameters, and artifacts for reproducibility and comparison across experiments
• Config-driven experiments: new experiments require only a config.yaml entry, no code changes
• Label encoding and artifact logging: LabelEncoder saved as MLflow artifact for consistent eval
• Finalized config file for ablation use in app

Visuals

• Create initial relevant visuals for N vs A vs V beats:
  • Waveform shape
  • pre_rr vs post_rr intervals scatter distribution
  • rr_irregularity violin plot
  • QRS beat variation violin
  • Change of rr_irregularity over time
  • Feature correlation matrix to show redundancy

---

Setup

Option A: Docker (Recommended)

No Python, no uv, no dependency setup required :just Docker.

1. Install Docker Desktop for your platform
2. Pull the image:

docker pull theopensourceguy/arrhythmia_ml

3. Download the raw data from PhysioNet and extract it into ./data/raw in whatever folder you'll be running commands from

That's it. See Using Docker for how to run scripts.

---

Option B: Local Setup

1. Install the following before going further:

   1. Git
   2. uv
   3. Raw data from PhysioNet — extract to data/raw

2. Clone the repository:

git clone https://github.com/ShekharNarayanan/arrhythmia_ml.git

3. Navigate to the repository:

cd arrhythmia_ml

4. Install dependencies:

uv sync

---

Usage

Using Docker

Always run Docker commands from the same directory. Your data should be in ./data/raw and MLflow runs will be saved to ./mlruns in that directory.

NOTE: Experiment names are specified in config.yaml

Train a model:

docker run -v ./data:/app/data -v ./mlruns:/app/mlruns theopensourceguy/arrhythmia_ml uv run python -m scripts.train --exp <exp_name>

Evaluate results:

docker run -v ./data:/app/data -v ./mlruns:/app/mlruns theopensourceguy/arrhythmia_ml uv run python -m scripts.eval --exp <exp_name>

Track experiments in MLflow UI:

docker run -v ./mlruns:/app/mlruns -p 5000:5000 theopensourceguy/arrhythmia_ml uv run mlflow ui --host 0.0.0.0

Then open http://localhost:5000 in your browser.

Validate peak detection:

docker run -v ./data:/app/data -v ./peak_detection_plots:/app/peak_detection_plots theopensourceguy/arrhythmia_ml uv run python validate_detected_peaks.py

Visualize features:

docker run -v ./data:/app/data -v ./feature_plots:/app/feature_plots theopensourceguy/arrhythmia_ml uv run python visualize_features.py

---

Using Local Setup

1. Make changes in config.yaml to specify your settings (beats to classify, preprocessing parameters, ML experiments etc.)

2. Train a model:

uv run python -m scripts.train --exp <insert exp from config>

This will create the mlruns folder. MLflow will track your experiments using it.

3. Evaluate results:

uv run python -m scripts.eval --exp <insert exp from config>

4. Track experiments in MLflow UI:

uv run mlflow ui

5. Validate peak detection:

uv run python validate_detected_peaks.py

6. Visualize features:

uv run python visualize_features.py

---

Contact

For any questions, you can contact me on LinkedIn.