A production-style, end-to-end ETL pipeline that ingests data from the GitHub API, stores raw data in object storage, and transforms it into an analytics-ready dimensional data model. The pipeline is designed with reliability, idempotency, and scalability in mind and is orchestrated using Apache Airflow.
GitHub API ↓ Python Ingestion Layer ↓ Object Storage (MinIO / S3) — Raw Layer ↓ Apache Airflow Orchestration ↓ PostgreSQL — Analytics Warehouse
This architecture follows a modern data lake + warehouse pattern, where raw ingestion is decoupled from transformation and analytics. Apache Airflow manages scheduling, dependencies, retries, and observability to ensure production-grade reliability.
- Language: Python 3.9+
- Orchestration: Apache Airflow
- Object Storage (Raw Layer): MinIO (S3-Compatible)
- Data Warehouse: PostgreSQL
- Containerization: Docker & Docker Compose
- Version Control: Git
- Extract data from the GitHub API using Python, handling pagination and API rate limits.
- Store raw API responses as JSON files in object storage, partitioned by execution date.
- Load raw JSON data into a staging table in PostgreSQL while preserving full data fidelity.
- Transform staged data into analytics-ready fact and dimension tables using a star schema.
- Validate data post-load to ensure completeness and consistency.
- The pipeline is orchestrated using a modular Apache Airflow DAG.
- Tasks are designed to be independently retryable.
- Dependencies enforce correct execution order.
- Retries and failure handling are configured at the task level.
- DAG runs are safe to re-execute due to idempotent load logic.
This design enables operational safety and production readiness.
- Incremental extraction is driven by the Airflow execution date.
- Raw data is partitioned by date in object storage (
dt=YYYY-MM-DD). - Load steps implement idempotent logic by removing existing records for a partition before inserting new data.
- Supports safe re-runs, historical backfills, and recovery from partial failures.
| Column | Type | Description |
|---|---|---|
issue_id |
BIGINT | GitHub Issue ID (Primary Key) |
user_id |
BIGINT | Foreign Key to dim_users |
state |
TEXT | Issue state (open/closed) |
comments_count |
INT | Number of comments |
execution_date |
DATE | Partition Key |
| Column | Type | Description |
|---|---|---|
user_id |
BIGINT | GitHub User ID (Primary Key) |
login |
TEXT | GitHub username |
type |
TEXT | User or Organization |
- Schema validation during ingestion
- Idempotent load logic to prevent duplicate records
- Controlled retries for transient API and infrastructure failures
- Separation of raw and analytics layers to support reprocessing
These practices align with production data engineering standards.
Prerequisites: Docker Desktop installed
- Clone the repository
git clone <repo-url> cd Production-Grade-ETL-Pipeline-Tasks
- Configure environment variables
GITHUB_TOKEN=your_token_here
- Start infrastructure
docker compose up -d --build
- Access UIs
Airflow: http://localhost:8080 (admin / admin)
MinIO: http://localhost:9001 (minioadmin / minioadmin)
🧠 Key Design Decisions
Object storage decouples ingestion from transformation for scalability and fault tolerance.
PostgreSQL simulates a cloud-based analytical warehouse suitable for dimensional modeling.
Apache Airflow provides scheduling, retries, observability, and dependency management.
Idempotent pipelines ensure safe reprocessing and production reliability.
🔮 Future Improvements
Migrate analytics warehouse to Amazon Redshift
Add automated data quality checks (e.g., Great Expectations)
Implement monitoring and alerting for DAG failures
Add CI/CD validation for Airflow DAGs