Data Engineering Portfolio

Portfolio of data engineering projects covering batch ingestion, streaming, data modeling, data quality, cloud architecture, NoSQL, analytical pipelines, and reproducible local infrastructure.

The repository is structured as a set of self-contained projects. Each project documents its architecture, implementation status, technical decisions, limitations, and reproducible execution path.

Portfolio principle: implemented capabilities are separated from planned work. Architecture targets are not presented as completed engineering evidence.

Recruiter Snapshot

Project	What it demonstrates	Core technologies	Status
Euro Air Quality Pipeline	End-to-end batch + streaming pipeline, multi-source ingestion, Kafka, Spark Structured Streaming, Medallion Architecture, Parquet, data quality	Python, Jupyter, Apache Kafka, Apache Spark, PostgreSQL, Parquet, Docker Compose	Implemented academic pipeline
CloudOps Insight Lake	AWS-oriented data-platform architecture for cost, usage, governance, and reliability analytics	Python; planned AWS S3, Glue, Athena, CloudWatch, CUR 2.0, Parquet, IAM	Active development — Phase 0
Weather Air Vienna	Reproducible API-to-NoSQL pipeline with Raw → Staging → Daily aggregation and analytical output	Python, Jupyter, MongoDB, pandas, Docker Compose, Open-Meteo	Implemented academic pipeline

Strongest Technical Evidence

1. Euro Air Quality Pipeline — Batch + Streaming Data Engineering

Best starting point for reviewing implemented pipeline engineering.

The project combines three different source types into one reproducible analytical pipeline:

• EEA data for historical air-quality measurements
• Wikipedia scraping for city metadata
• Open-Meteo REST API for current air-quality data
• Kafka producer for live-event ingestion
• Spark Structured Streaming for Kafka consumption and transformation
• Bronze → Silver → Gold data layers
• Parquet persistence for analytical datasets
• explicit schemas, deterministic event IDs, deduplication, validation, and data-quality checks
• local reproducibility through Docker Compose

EEA batch data ───────────────┐
Wikipedia scraping ───────────┼─> Bronze / Silver ─┐
Open-Meteo REST -> Kafka -> Spark ─> Silver ───────┼─> Gold -> Analysis
                                                  └─> Data Quality

Engineering topics demonstrated:

Batch ingestion · Streaming · Kafka · Spark Structured Streaming · Data Cleaning · Schema Validation · Deduplication · Medallion Architecture · Parquet · PostgreSQL · Docker

→ Open project

---

2. CloudOps Insight Lake — AWS Data Platform Engineering

An actively developed serverless data-platform project focused on the question:

Which AWS Lambda workloads show a notable divergence between cost development and actual usage, and where do governance or reliability issues require review?

The current repository evidence focuses on architecture and scope discipline before infrastructure implementation:

• bounded Lambda-first analytical scope
• Architecture Decision Records (ADRs)
• canonical cross-source resource identity
• explicit fact/dimension grain
• source-specific contracts before normalization
• incremental-processing design using object identity and ETags
• planned pipeline audit model and data-quality gates
• explicit separation of AWS runtime observability from data-pipeline observability
• least-privilege IAM design and negative permission-test strategy
• cost-aware serverless architecture

Planned implementation path:

AWS CUR 2.0 / Data Exports
Lambda Inventory
CloudWatch Metrics
        ↓
      S3 Raw
        ↓
   Glue ETL + DQ
        ↓
  Curated Parquet
        ↓
 Glue Catalog / Athena

Important: S3, Glue, Athena, collectors, IAM controls, and Infrastructure as Code are architecture targets and are not yet represented as completed implementation.

→ Open project

---

3. Weather Air Vienna — NoSQL Analytical Pipeline

A reproducible pipeline for weather and air-quality analysis in Vienna.

The project implements:

• REST ingestion from Open-Meteo
• raw persistence in MongoDB
• cleaning, normalization, staging, and deduplication
• Python Map / Shuffle / Reduce processing for daily aggregation
• persisted daily analytical results
• Jupyter-based analysis and visual storytelling
• Docker Compose for reproducible infrastructure

Open-Meteo API
      ↓
MongoDB Raw
      ↓
Cleaning + Staging
      ↓
Python MapReduce
      ↓
Daily Aggregate
      ↓
Analysis + Visualisation

Engineering topics demonstrated:

REST ingestion · MongoDB · NoSQL data modeling · ETL · Data Cleaning · MapReduce concepts · Reproducibility · Docker

→ Open project

---

Capability Matrix

Capability	Evidence in this repository
Python data engineering	ingestion, transformation, validation, aggregation, analysis
Batch pipelines	historical EEA ingestion and transformation
Streaming pipelines	Open-Meteo → Kafka → Spark Structured Streaming
REST API ingestion	Open-Meteo weather and air-quality sources
Web scraping	Wikipedia city metadata extraction
Relational storage	PostgreSQL
NoSQL storage	MongoDB
Analytical file formats	Apache Parquet
Data architecture	Bronze / Silver / Gold layering, Raw / Staging / Aggregate patterns
Data quality	schema checks, validity filters, deduplication, explicit missing-value handling
Data modeling	canonical identifiers, explicit dataset grain, fact/dimension design
Containerization	Docker Compose-based local infrastructure
Architecture documentation	Mermaid diagrams, ADRs, architecture and limitation documents
Cloud architecture	AWS serverless data-platform design in CloudOps Insight Lake
Governance & security thinking	tagging policy, lineage/audit concepts, IAM least-privilege design, negative permission tests

Engineering Approach

Across the portfolio, the projects follow several consistent principles:

1. Business question before technology selection
2. Explicit data grain and identifiers before aggregation
3. Raw data preservation before transformation
4. Source-specific ingestion before canonical modeling
5. Deterministic data-quality checks instead of assuming successful execution means correct data
6. Reproducible environments and documented execution paths
7. Architecture decisions and limitations documented explicitly
8. Implemented evidence separated from planned architecture

Repository Structure

data-engineering-projects/
│
├── euro-air-quality-pipeline/
│   ├── notebooks/
│   ├── docs/
│   ├── docker/
│   ├── presentation/
│   └── README.md
│
├── cloudops-insight-lake/
│   ├── docs/
│   ├── src/
│   ├── tests/
│   └── README.md
│
└── weather-air-vienna/
    ├── notebooks/
    ├── docs/
    ├── images/
    └── README.md

Each subproject contains its own detailed README, dependencies, architecture documentation, and execution instructions.

Suggested Review Path

For a quick technical review:

1. Euro Air Quality Pipeline — strongest implemented end-to-end data-engineering evidence
2. CloudOps Insight Lake — current cloud architecture and engineering-depth project
3. Weather Air Vienna — earlier NoSQL / ETL / MapReduce pipeline

Scope and Limitations

This repository contains both completed academic projects and an actively developed portfolio project. They are not presented as production systems.

The focus is on demonstrating engineering reasoning, reproducibility, pipeline structure, data-quality thinking, and technical progression rather than claiming production-scale workload volume where it has not been measured.

Author

Developed and maintained as a Data Engineering portfolio by Georg Wiesmüller.