Real-Time Kafka ➝ Iceberg Lakehouse (Nessie + MinIO + Trino)

Streaming reference stack that ingests JSON events from Kafka into an Apache Iceberg table using the Iceberg Kafka Connect Sink, stores data + metadata in MinIO (S3-compatible), versions tables through Nessie, and queries them with Trino.

🧩 Components

Service	Role
kafka	Event broker (single-node KRaft) hosting topics (session_topic, etc.)
kafka-connect	Runs Iceberg Sink Connector to persist Kafka records into Iceberg
catalog	Nessie catalog (Git-like versioned metadata for Iceberg)
storage	MinIO S3-compatible object storage (s3a://warehouse/)
trino, trino-worker	SQL query engine over Iceberg via Nessie
trino-init	Bootstraps schema + creates default.sessions_topic table
mc	MinIO client: creates buckets & policies
data-simulator	Python producer sending mock JSON records to Kafka

🔄 Data Flow

data-simulator ➝ Kafka topic (session_topic) ➝ Kafka Connect (Iceberg Sink) ➝ Iceberg table (default.sessions_topic) in Nessie catalog ➝ Parquet + metadata in MinIO ➝ Query with Trino.

📦 Iceberg Sink Connector

Configured in kafka-connect/iceberg-config/iceberg-sink-config.json:

• topics: session_topic
• iceberg.tables: default.sessions_topic
• Schema evolution: enabled (iceberg.tables.evolve-schema-enabled=true)
• Table must pre-exist (created by trino-init)
• Uses NessieCatalog + S3FileIO against MinIO endpoint

🚀 Quick Start

docker compose up -d