A collection of foundational labs and projects from my Massive Data Engineering course. This repository demonstrates my progression from SQL basics to orchestrated ETL pipelines.
- Tech: SQL (PostgreSQL/MySQL)
- Focus: Data modeling, complex joins, aggregations, and query optimization.
- Goal: Solving business intelligence questions using structured data.
- Tech: Python, Requests, JSON
- Focus: Programmatic data extraction, handling rate limits, and parsing nested JSON structures.
- Project: Extracting and structuring data from the PokeAPI.
- Tech: Python, MongoDB, Docker Compose, Pandas
- Focus: Building modular ETL pipelines, containerizing databases, and NoSQL data modeling.
- Project: Streaming USGS Earthquake data into a MongoDB instance with automatic cleaning and enrichment.
- Languages: Python, SQL
- Databases: MongoDB (NoSQL), Relational SQL
- Tools: Docker, Docker Compose, Pandas, Requests
- Concepts: ETL (Extract, Transform, Load), Data Cleaning, Schema Mapping, Containerization
Each folder contains its own README.md and requirements.txt (or docker-compose.yml) with specific instructions on how to run the labs.