Airflow ML Pipeline 🚀

End-to-end machine learning pipeline built with Apache Airflow for automated model training and prediction.

📌 Project Overview

This project demonstrates how to build and automate an ML workflow using Apache Airflow.

The pipeline:

• loads and prepares data
• trains a machine learning model
• saves the trained model
• performs predictions on new data
• organizes ML workflow using Airflow DAG

🏗️ Project Architecture

Data
↓
Data preprocessing
↓
Model training
↓
Model serialization (.pkl)
↓
Prediction pipeline
↓
Airflow DAG orchestration

📸 Airflow Pipeline Demo

Airflow DAG

Prediction Results

🛠️ Technologies

- Python 3.12
- Apache Airflow
- Docker & Docker Compose
- Scikit-learn
- Pandas
- Dill
- PostgreSQL
- Redis

📂 Project Structure

airflow-ml-pipeline/

├── dags/
│ └── hw_dag.py

├── modules/
│ ├── pipeline.py
│ └── predict.py

├── data/
│ ├── test/
│ └── models/

├── docker-compose.yml
├── requirements.txt
├── LICENSE
└── README.md

🤖 Machine Learning

The pipeline trains a machine learning model and stores it as a serialized file.

Steps:

1. Data loading
2. Feature preprocessing
3. Model training
4. Model saving
5. Prediction generation

⚙️ Workflow

Airflow DAG consists of tasks:

1. Model training
2. Saving trained model
3. Loading test data
4. Generating predictions

🚀 Running the project

Clone repository:

git clone https://github.com/lllda06/airflow-ml-pipeline.git

Run Airflow environment:

docker compose up -d

Open Airflow UI:

http://localhost:8080

📊 Results The pipeline successfully automates the ML lifecycle from training to prediction.

👩‍💻 Author Bahdanovich Lada