Skip to content

hhoangdo/ecom-data-platform

BranchesTags

Repository files navigation

Vina Bim Shop Data Platform Coursework

vina-bim-shop is a staged, runnable local data engineering platform for a Shopee-inspired Vietnamese marketplace. It demonstrates how a modern e-commerce platform can support both realtime operational analytics and batch-reconciled historical analytics through a Lambda-style architecture.

The platform is runnable locally, but it should be operated through staged Docker Compose profiles. A broad full-stack startup attempt can cause Docker instability and API inspection failures on constrained machines, so this evidence pass treats staged startup as the supported workflow rather than relying on a monolithic all startup.

Table of Contents

System Architecture

Vina Bim Shop system architecture

The detailed architecture view shows the project as a staged local platform: synthetic source contracts feed Kafka and local batch snapshots, Spark and Flink process the two analytical paths, Trino and Pinot expose different serving surfaces, and Airflow, Great Expectations, DataHub, and evidence artifacts make the run auditable.

Related diagrams:

Tech Stack

Layer Technology Purpose
Source generation Python, Faker, pandas, PyArrow Generate Vietnamese marketplace snapshots, event streams, and evidence.
Ingestion Kafka KRaft, Schema Registry, Kafka Connect, Kafka UI Run event-log ingestion, JSON Schema registration, Bronze landing, and topic inspection.
Lakehouse storage MinIO, Hive Metastore, Postgres, Apache Iceberg Store Bronze replay logs, Silver/Gold tables, checkpoints, and catalog metadata.
Batch processing Apache Spark, dbt-DuckDB Build reconciled Silver/Gold tables and maintain a local parity oracle.
Streaming processing Apache Flink Produce event-time realtime metrics, alerts, and derived Kafka topics.
Serving Trino, Apache Pinot, DuckDB Serve canonical Gold SQL, provisional realtime OLAP, and portable local analysis marts.
Orchestration and quality Apache Airflow, Great Expectations Trigger control-plane workflows and publish validation evidence.
Governance DataHub, OpenSearch Emit metadata, tags, glossary terms, quality assertions, and lineage evidence.
Runtime Docker Compose profiles, uv, pytest Run the local stack, Python tooling, and verification suite.

Introduction

vina-bim-shop models the data platform needs of a Vietnamese e-commerce marketplace with customers, sellers, products, promotions, orders, payments, shipments, inventory, and user behavior events. The project is coursework-oriented, but it is intentionally built around realistic data engineering concerns: source contracts, replayable event logs, medallion lakehouse storage, quality gates, serving-layer truth policies, and evidence packaging.

The main design choice is a Lambda architecture because the platform needs two complementary analysis streams:

  • Realtime analysis for low-latency operational views such as traffic bursts, payment issues, live conversion, and alert candidates.
  • Batch analysis for reconciled executive KPIs, dimensional modeling, financial formulas, and repeatable historical reporting.

Kafka, Flink, and Pinot provide the fresh but provisional speed path. MinIO, Spark, Iceberg, Hive Metastore, Trino, and DuckDB provide the reconciled batch path. Airflow, Great Expectations, and DataHub make the platform inspectable for coursework evidence.

Problem Definition

The platform answers a practical e-commerce data problem: how can a marketplace support live operational monitoring while still preserving a reliable historical source of truth for business reporting?

The source design deliberately includes both event envelopes and table-state exports:

Source stream Answers Used by
Kafka-shaped JSON event envelopes What happened now? Flink realtime metrics, Pinot dashboards, replay logs, behavior analysis.
Periodic table-state snapshots What state is reliable at checkpoint? Spark batch reconciliation, Gold dimensions/facts, Trino SQL, DuckDB evidence.

The overlap is intentional. Orders, payments, shipments, and customer behavior can appear in both streams because events provide immediacy while snapshots provide checkpointed source-of-record state.

The generator also injects intentional data errors and contract drift so the platform can demonstrate realistic handling instead of only happy-path data:

  • duplicate order-item payloads and duplicate commerce-event envelopes
  • late-arriving commerce events
  • missing values in fields such as shipping_method, brand, and device_type
  • schema evolution where older slices miss newer optional fields
  • malformed event and snapshot examples routed through dead_letter_events, raw_bad_events, and raw_bad_snapshots
  • traffic burst and schema-version signals in operational events

Coursework Status

Sections 01 and 02 are fulfilled for the mini-coursework phase.

For the original Sections 01/02 submission boundary, Spark, Flink, Apache Pinot, and Trino are architectural target contracts, while dbt-DuckDB is the runnable local implementation. The broader repository now also includes the staged runnable platform implementation for ingestion, lakehouse, batch, streaming, serving, orchestration, and governance.

View Meaning
Runnable locally dbt-DuckDB, the final dataset package, and the documented evidence artifacts can be reproduced on one machine.
Architectural contract The staged Kafka, Spark, Flink, Pinot, Trino, Airflow, and DataHub stack documents the full target platform through service-level deliverables and evidence.

Mini-coursework artifacts and evidence:

Platform Implementation Overview

The root README is the entrypoint. The deeper service-by-service explanations live in deliverables and evidence folders, so the descriptions below are intentionally concise.

Stage Profile or local path Services and assets Deep dive
Data generation local Python Synthetic snapshots, Kafka-shaped JSONL, bad records, quality evidence. 01 data generator
Ingestion ingestion Kafka KRaft, Schema Registry, Kafka Connect, Kafka UI. 03 Kafka ingestion
Lakehouse lakehouse MinIO, Hive Metastore, Trino, shared Postgres. 04 lakehouse
Batch batch plus dbt-DuckDB Spark master, worker, history server, Iceberg Gold, dbt parity. 05 Spark batch
Streaming streaming Flink JobManager, TaskManager, job submitter, derived Kafka topics. 06 Flink streaming
Serving serving plus Trino/DuckDB Apache Pinot realtime OLAP, Trino canonical SQL, DuckDB local marts. 07 Pinot serving
Local analytics local DuckDB files plus dbt dbt-DuckDB parity oracle and DuckDB Executive Mart export. 10 DuckDB/dbt local analytics
Orchestration orchestration Airflow webserver, scheduler, init, GX Data Docs. 08 Airflow + GX
Governance governance DataHub GMS, frontend, actions, OpenSearch, metadata recipes. 09 DataHub governance, DataHub evidence

Data Generation

Data generation is the source-system simulator for the platform. It produces offline Parquet snapshots, Kafka-shaped JSONL topics, bad-record fixtures, and committed evidence that feed the batch, streaming, and local analytics paths.

Data generation overview

Key docs:

Ingestion

The ingestion profile turns the generated event contracts into a real Kafka surface. Kafka topics, Schema Registry subjects, Kafka UI inspection, and Kafka Connect Bronze landing together make the source-event path replayable.

Ingestion overview

Key docs:

Lakehouse

The lakehouse profile is the shared storage and SQL foundation for the platform. MinIO, Hive Metastore, shared Postgres, Iceberg, and Trino give Spark and Kafka Connect a common Bronze-to-Gold boundary.

Lakehouse overview

Key docs:

Batch

Spark owns the reconciled Bronze-to-Silver-to-Gold batch path over MinIO and Iceberg. Trino serves Spark Gold as the canonical SQL surface, while dbt-DuckDB parity and Executive Mart exports provide local verification and packaging.

Batch overview

Key docs:

Streaming

Flink consumes Kafka source topics with event-time logic and emits narrow serving contracts for metrics, corrections, and operational alerts. The streaming layer is intentionally fresh and provisional, not the final financial truth.

Streaming overview

Key docs:

Serving

Serving is split by freshness and truth role. Pinot answers low-latency questions over Flink-derived topics, while Trino answers canonical SQL over Spark-written Gold tables and can export portable DuckDB copies.

Serving overview

Schema design is part of the serving implementation, not just documentation. The committed model artifacts are:

Key docs:

Local Analytics

DuckDB provides the portable local analytics layer in two forms: the dbt-DuckDB parity oracle rebuilt from generated raw data, and the Executive Mart exported from Trino Gold. These files support regression checks, offline inspection, and coursework evidence without requiring the full distributed stack to stay online.

Local analytics overview

Key docs:

Orchestration

Airflow owns the control-plane workflows for staged local runs: bootstrap, batch windows, reconciliation, evidence packaging, and DataHub ingestion. Great Expectations applies the project quality policy and publishes GX Data Docs, while Flink runtime remains outside Airflow supervision in v1.

Orchestration overview

Key docs:

Governance

DataHub catalogs the same assets the platform actually produces: Kafka topics, MinIO prefixes, Trino/Iceberg tables, dbt models, Spark and Flink lineage, and GX assertions. Governance proof relies on GMS and GraphQL verification plus committed evidence artifacts, not only the frontend UI.

Governance overview

Key docs:

Repository Structure

coursework/
|-- architecture/             # Domain contracts, PlantUML, DBML, and Excalidraw architecture assets
|-- compose/                  # Domain Compose files included by the root compatibility entry point
|-- configs/                  # Generator, scenario, and pipeline configuration
|-- data/                     # Gitignored local raw and Gold outputs, with folder-intent .gitignore files
|-- deliverables/             # Official coursework writeups and service-level documentation
|-- evidence/                 # Committed evidence packages, screenshots, manifests, reports, and query outputs
|-- infra/                    # Docker images, service config, Airflow DAGs, and dbt project assets
|-- scripts/                  # CLI entrypoints for generation, bootstrap, smoke tests, evidence, reset, and exports
|-- src/vina_bim_shop/        # Python packages for generation, Kafka, lakehouse, Flink, Pinot, quality, and governance
|-- tests/                    # Unit and integration checks for contracts, runtime helpers, evidence, and docs
|-- docker-compose.yml        # Root Docker Compose compatibility entry point
|-- pyproject.toml            # Python project metadata and dependencies
`-- uv.lock                   # Locked Python dependency graph

See the Script inventory for the official command surface and the debug/manual/destructive helpers queued for later cleanup. Official evidence capture commands on main generate machine-verifiable artifacts only; committed screenshot files remain historical review evidence and are not regenerated by the official capture scripts.

Prerequisites

Tool Recommended version Purpose
Docker Desktop / Docker Compose Docker 24+ Run staged service profiles.
Python 3.12+ Run generator, scripts, tests, dbt-DuckDB path.
uv Current stable Install and run the Python environment.
Git Current stable Clone and inspect the coursework repo.
DBeaver or DuckDB CLI Optional Inspect local DuckDB evidence files.
PlantUML / DBML tooling Optional Preview architecture and ERD files locally.

The repo includes .env.example with local defaults. Copy it to .env if you want explicit environment settings.

Installation & Setup

This project is designed to demonstrate the platform end to end through staged local execution, with each subsystem being verifiable. It is not reliably optimized for a single frictionless full-stack-at-once startup on a constrained machine.

Common Commands

The repository ships a root Makefile that wraps the most common commands under a make + verb convention. Run make help at any time to see the full catalog. Recipes are shell-agnostic: they delegate to uv run python and scripts/ctl.py, whose compose lifecycle expands documented profile bundles and prebuilds shared images where needed.

Target What it does
make install uv sync to install Python dependencies.
make generate Run the data generator with default options (SCALE=medium, MODE=full, SEED=42). Override with make generate SCALE=smoke MODE=streaming SEED=7.
make build-dbt Run dbt build against the local DuckDB profile.
make test Run pytest.
make finalize Produce the final Section 01/02 evidence package.
make reset Run scripts/qa/reset_all.py (destructive; forwards RESET_FLAGS=..., e.g. --dry-run, --force, --force --clean-local-data).
make up-<profile> / make down-<profile> Start or stop one of the supported compose profiles; dependent profiles expand to the documented staged bundles.

The PowerShell snippets below remain the authoritative command surface; the Makefile is a thin, discoverable shortcut on top of them.

1. Install Python dependencies

uv sync

2. Run the fast local compatibility path

Use this path when you want reproducible Section 01/02 evidence without starting the distributed services.

uv run python scripts/generate/run_generator.py --scale medium --mode full --clean --seed 42
uv run dbt build --project-dir infra/analytics/dbt --profiles-dir infra/analytics/dbt
uv run pytest

3. Start the distributed platform in stages

Start each stage in order. All services share the same Docker network.

docker compose --profile ingestion up -d
docker compose --profile lakehouse up -d
docker compose --profile batch up -d
docker compose --profile ingestion --profile lakehouse --profile streaming up -d
docker compose --profile ingestion --profile lakehouse --profile streaming --profile serving up -d
docker compose --profile orchestration up -d
docker compose --profile ingestion --profile lakehouse --profile governance up -d

The all profile exists for best-effort demos, but staged startup is the supported evidence workflow.

4. Generate final Section 01/02 package evidence

uv run python scripts/qa/finalize_sections_01_02.py

5. Export the local executive mart

Run this after Spark Gold tables are available through Trino.

uv run python scripts/spark/export_executive_mart.py --duckdb-path data/gold/vina_bim_shop_executive.duckdb --evidence-root evidence/05_spark_batch

6. Reset local runtime state

# Preview what would be destroyed
uv run python scripts/qa/reset_all.py --dry-run

# Full reset with confirmation prompt
uv run python scripts/qa/reset_all.py

# Full reset without prompt
uv run python scripts/qa/reset_all.py --force

# Reset including gitignored local generated data
uv run python scripts/qa/reset_all.py --clean-local-data

The reset script is a documented needs-review destructive helper. It stops services, removes Docker volumes, and optionally cleans gitignored local data such as data/raw/, infra/analytics/dbt/target/, and evidence/runtime/. It does not remove committed source files or committed evidence.

Configuration

Local defaults are documented in .env.example. Most scripts work with defaults, but the variables below define the main configuration groups.

Group Variables Default intent
Runtime VBS_ENV, VBS_RANDOM_SEED, VBS_TAXONOMY_PATH Local execution with deterministic seed 42 and committed taxonomy snapshot.
Kafka VBS_KAFKA_BOOTSTRAP_SERVERS, VBS_SCHEMA_REGISTRY_URL, VBS_KAFKA_CONNECT_URL, VBS_KAFKA_UI_URL Local Kafka services on ports 9092, 8081, 8083, and 8084.
MinIO VBS_MINIO_ENDPOINT, VBS_MINIO_INTERNAL_ENDPOINT, VBS_MINIO_ROOT_USER, VBS_MINIO_ROOT_PASSWORD, VBS_MINIO_REGION Local object storage with internal Docker and external localhost endpoints.
Lakehouse catalog VBS_LAKEHOUSE_POSTGRES_*, VBS_HIVE_METASTORE_*, VBS_TRINO_* Shared Postgres, Hive Metastore, and Trino local serving defaults.
Spark VBS_SPARK_MASTER_URL, VBS_SPARK_MASTER_UI_URL, VBS_SPARK_HISTORY_URL Spark master, UI, and history server endpoints.
Flink VBS_FLINK_UI_URL, VBS_FLINK_JOBMANAGER_URL, VBS_FLINK_MAX_RUNTIME_MINUTES, VBS_FLINK_DISABLE_AUTO_STOP Flink UI/runtime settings and local safety timeout.
Pinot VBS_PINOT_CONTROLLER_URL, VBS_PINOT_BROKER_URL Pinot controller and broker endpoints for bootstrap and query scripts.
Orchestration VBS_AIRFLOW_UI_URL, VBS_AIRFLOW_ADMIN_USERNAME, VBS_AIRFLOW_ADMIN_PASSWORD, VBS_GX_DOCS_URL Airflow and GX Data Docs local URLs.
Data roots VBS_RAW_ROOT, VBS_BRONZE_*, VBS_*_BUCKET, VBS_DUCKDB_PATH Local raw output paths, MinIO bucket names, and DuckDB defaults.

Known Limits

  • The platform is runnable locally, but only if you follow staged profile startup and accept some UI/runtime constraints.
  • A broad full-stack startup attempt is known to cause Docker instability and API inspection failures on constrained machines; staged profiles are the normal workflow for this evidence pass.
  • The full all profile is high-resource and best-effort, not the primary acceptance path.
  • dbt-DuckDB is a local compatibility path; Spark/Iceberg/Trino is canonical for reconciled truth in the full platform evidence.
  • The DuckDB Executive Mart is a Trino Gold snapshot export; regenerate it after each Spark Gold refresh.
  • Pinot is fresh and provisional; Trino-served Gold tables are the official KPI source.
  • Airflow orchestrates batch and control-plane work; it does not supervise long-running Flink jobs in v1.
  • DataHub requires separate ingestion recipe runs after platform services are healthy.
  • Local DataHub UI capture did not render the fuller graph view, even though GMS/GraphQL evidence supports metadata, lineage, tag, and assertion emission.
  • Observability hardening, production security, CI/CD, and Section 03 drift scenarios are outside the current evidence boundary.

Data

Generated local data lives in data/ and is intentionally gitignored except for nested .gitignore files that preserve folder intent.

Path Role Git behavior
data/raw/ Generator-managed raw Parquet snapshots, Kafka-topic JSONL files, and bad-record examples. Gitignored local output.
data/gold/vina_bim_shop.duckdb dbt-DuckDB parity oracle rebuilt from local raw inputs. Gitignored local output.
data/gold/vina_bim_shop_executive.duckdb DuckDB Executive Mart exported from Trino Gold. Gitignored local output.
evidence/final_dataset/vina_bim_shop_medium_raw.zip Submitted medium raw dataset package. Committed evidence.
evidence/final_dataset/final_dataset_manifest.json Manifest for the final dataset package. Committed evidence.
evidence/final_integration/ Health JSON, historical screenshots, row counts, query outputs, reconciliation, and lineage evidence. Committed evidence.

Runtime service profiles:

Profile Services
ingestion Kafka KRaft, Schema Registry, Kafka Connect, Kafka UI
lakehouse MinIO, Hive Metastore, Trino, shared Postgres
batch Spark master, worker, history server
streaming Flink JobManager, TaskManager, job submitter
serving Pinot Zookeeper, controller, broker, server
orchestration Airflow webserver, scheduler, init, GX Data Docs
governance DataHub GMS, frontend, actions, OpenSearch
all Best-effort startup of all profiles; staged startup remains the supported workflow

About

A staged, locally runnable full-stack data platform for a synthetic Vietnamese e-commerce marketplace. Demonstrates Lambda architecture (batch + streaming) with a medallion lakehouse, data quality gates, and metadata governance.

Topics

python docker ecommerce apache-spark data-engineering minio apache-flink apache-kafka dbt data-quality lambda-architecture datahub trino apache-airflow data-governance apache-iceberg great-expectations lakehouse duckdb apache-pinot

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages