Spark Playground

A hands-on data engineering playground featuring Apache Spark 4.1, Delta Lake, Apache Iceberg, Unity Catalog, and Kafka. Includes production-grade scenarios covering data skew mitigation, SCD Type 2, streaming watermarks, fraud detection, and data quality validation.

Architecture

                    +-------------------+
                    |  Unity Catalog    |
                    |  :8080 (API)      |
                    |  :3000 (UI)       |
                    +--------+----------+
                             |
              +--------------+--------------+
              |        Spark Master         |
              |  :8088 (UI) :15002 (Connect)|
              +---------+--+---+------------+
                        |     |
               +--------+     +--------+
               |                       |
        +------+------+        +------+------+
        | Worker 1    |        | Worker 2    |
        | 2 cores/2GB |        | 2 cores/2GB |
        +-------------+        +-------------+
              |                       |
              +-------+-------+-------+
                      |
              +-------+-------+
              |    Kafka      |
              |  :9094 (ext)  |
              +---------------+

All services share a bridge network and access /data for datasets and /data/warehouse for Delta/Iceberg tables.

Quick Start

# Build and start the full stack
make setup

# Or step by step:
make build
make up
make init-uc

Access points after startup:

Service	URL
Spark Master UI	http://localhost:8088
Spark Application	http://localhost:4040
Unity Catalog API	http://localhost:8080
Unity Catalog UI	http://localhost:3000
Spark Worker 1	http://localhost:8081
Spark Worker 2	http://localhost:8082
Kafka (external)	localhost:9094

Prerequisites

• Docker and Docker Compose
• Python 3.10+
• uv (for local dependency management)

# Install Python dependencies locally (for IDE support / Marimo notebooks)
uv sync

Scenarios

Each scenario in src/scenarios/ models a real-world data engineering problem at scale:

Scenario	File	Problem
Data Skew	data_skew.py	4TB fact table join with skewed keys. Demonstrates AQE, salting, and broadcast joins.
Ride Sharing	ride_sharing.py	500GB/day trip data. Window functions to rank top drivers per city with deduplication.
SCD Type 2	scd_type_2.py	Product dimension tracking with SHA256 change detection and Delta MERGE.
Watermarks	watermarks.py	Real-time fraud detection at 180K events/sec with 14-day rolling windows and exactly-once semantics.
E-Commerce	e-commerce.py	5TB daily clickstream ETL: incremental ingestion, session aggregation, small file control.
Revenue Reconciliation	recompute_revenue.py	Multi-source financial reconciliation for 200M users with late-arriving data handling.

Data Quality

The src/spark_expectations_demo/ module demonstrates the Spark Expectations framework:

python -m src.spark_expectations_demo.demo

Three levels of validation rules:

• Row-level -- NULL checks, pattern matching, date validation
• Aggregate-level -- COUNT, COUNT DISTINCT thresholds
• Query-level -- Referential integrity, cross-table consistency

Spark Connect

The cluster exposes Spark Connect on port 15002, allowing you to run PySpark code from your local machine:

from src.utils import get_spark_session

spark = get_spark_session("my-app")
spark.sql("SHOW CATALOGS").show()

Or use the interactive Marimo notebook:

marimo edit spark.py

Running Tests

Tests run inside the Spark master container:

make test              # All tests
make test-unit         # Unit tests only
make test-delta        # Delta Lake tests
make test-iceberg      # Iceberg tests
make test-coverage     # With coverage report

Available pytest markers: unit, integration, performance, delta, iceberg, unity_catalog, slow.

Make Targets

make build       - Build Docker images
make up          - Start all services
make down        - Stop all services
make restart     - Restart all services
make setup       - Full setup (build + up + init Unity Catalog)
make dev         - Start services + init Unity Catalog
make shell       - Bash shell in Spark master
make pyspark     - PySpark REPL with Delta/Iceberg/Unity Catalog
make init-uc     - Initialize Unity Catalog
make test        - Run all tests
make clean       - Stop and remove containers + volumes
make clean-all   - Clean + remove images
make status      - Show service status
make logs        - Tail all service logs

Tech Stack

Component	Version	Purpose
Apache Spark	4.1.1	Distributed compute engine
Delta Lake	4.1.0	ACID transactions, time travel, CDC
Apache Iceberg	0.11.1	Table format with partition evolution
Unity Catalog	latest	Data governance and lineage
Apache Kafka	4.1.1	Event streaming
Spark Expectations	2.9.1	Data quality framework
Marimo	0.21+	Interactive notebooks
Polars	1.0+	Fast DataFrame operations
Great Expectations	1.15+	Data validation

Project Structure

spark-playground/
├── conf/                    # Spark and logging configuration
├── data/
│   ├── external/            # Source datasets (CSV, JSON, TXT)
│   ├── warehouse/           # Delta/Iceberg tables (generated)
│   └── catalogs/            # Unity Catalog metadata
├── scripts/                 # Init and utility scripts
├── src/
│   ├── scenarios/           # Production-grade Spark patterns
│   ├── spark_expectations_demo/  # Data quality validation
│   ├── spark-pipelines/     # Pipeline definitions
│   └── utils.py             # Spark session & utility functions
├── tests/                   # Pytest suite with markers
├── docker-compose.yaml      # Full stack definition
├── Dockerfile               # Spark cluster image
├── Makefile                 # Development workflow
├── spark.py                 # Marimo notebook entry point
└── pyproject.toml           # Python dependencies