A production-grade e-commerce data lakehouse built as a Senior Data Engineer portfolio and learning project.
StreamHouse demonstrates end-to-end real-time and batch data engineering: from transactional source systems through CDC, stream processing, analytical transformations, and cloud-scale warehousing.
PostgreSQL → Debezium → Kafka → Flink → PostgreSQL → dbt → Parquet → DuckDB/GCS → BigQuery → dbt Marts → BI
Airflow is the orchestration and control plane.
For the full architecture document, see docs/architecture.md.
| Layer | Technology | Role |
|---|---|---|
| Source | PostgreSQL | OLTP database |
| CDC | Debezium | WAL-based change data capture |
| Streaming | Kafka (KRaft) | Event backbone |
| Stream Processing | Flink (SQL-first) | Real-time stateful processing |
| Batch Transforms | dbt | Staging → Intermediate → Pre-Aggregated models |
| Interchange Format | Parquet | Analytical file format |
| Local OLAP | DuckDB | Validation, profiling, reconciliation |
| Cloud Storage | GCS | Object storage / lake layer |
| Cloud Warehouse | BigQuery | Analytical serving layer |
| Cloud Transforms | dbt | Mart models (business layer) |
| Orchestration | Airflow | Pipeline orchestration and control plane |
The following technologies are explicitly excluded from StreamHouse V1:
- MotherDuck
- Spark
- Kubernetes
- Snowflake
- Great Expectations
- Prometheus / Grafana
- ELK Stack (Elasticsearch, Logstash, Kibana)
Re-introduction of any excluded technology requires an Architecture Decision Record. See docs/decisions/.
Phase 1 and beyond. Setup instructions will be added as infrastructure is implemented.
├── .agents/ AI development guardrails
├── docs/ Architecture documentation and ADRs
├── src/ Application source code
├── Makefile Developer convenience commands
├── .env.example Environment variable template
└── README.md This file
This project is developed for educational and portfolio purposes.