Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
74 lines (61 loc) · 3.46 KB

ARCHITECTURE.md

File metadata and controls

74 lines (61 loc) · 3.46 KB
Raw

Architecture

                 ┌──────────────┐
                 │   Airflow    │  Orchestration
                 │  (Celery)    │  Port 8081
                 └──────┬───────┘
                        │ triggers
                 ┌──────▼───────┐
                 │     dbt      │  Transformations
                 │  seed → run  │  landing → staging → curated
                 └──────┬───────┘
                        │ SQL
                 ┌──────▼───────┐
                 │    Trino     │  Query Engine
                 │ coord + 2w   │  Port 8080
                 └──────┬───────┘
                        │ reads/writes
          ┌─────────────┼─────────────┐
   ┌──────▼───────┐            ┌──────▼───────┐
   │    Nessie    │            │    MinIO     │
   │   Catalog    │            │   Storage    │
   │  Port 19120  │            │  Port 9001   │
   └──────────────┘            └──────────────┘
          Iceberg table format on S3-compatible storage

Data flow

  1. Airflow triggers the dbt_pipeline DAG daily
  2. dbt seed loads CSV files (AdventureWorks) into Iceberg landing schema via Trino
  3. dbt run transforms data through two layers:
    • staging — cleaned and typed models (stg_sales, stg_products, etc.)
    • curated — business-ready models (fact_sale, dim_product, dim_country)
  4. Trino executes all SQL against Iceberg tables stored in MinIO
  5. Nessie tracks table metadata with git-like versioning (branches, commits)

Components

Component Image Port Purpose
MinIO minio/minio 9000, 9001 S3-compatible object storage
Nessie projectnessie/nessie 19120 Iceberg catalog with versioning
Trino Coordinator trinodb/trino 8080 Query planning and coordination
Trino Worker x2 trinodb/trino — Query execution
Airflow API Server apache/airflow 8081 Web UI and REST API
Airflow Scheduler apache/airflow — DAG scheduling
Airflow Worker apache/airflow — Task execution (Celery)
PostgreSQL postgres — Airflow metadata
Redis redis — Celery message broker

Deployment options

Local (Docker Compose)

Three compose files manage isolated service groups:

  • docker-compose-lake.yaml — storage layer (MinIO + Nessie)
  • docker-compose-trino.yaml — query engine (coordinator + workers)
  • docker-compose-airflow.yaml — orchestration (full Airflow stack)

VK Cloud (Terraform + Packer)

  1. Packer builds a VM image with Docker and pre-copied project files
  2. Terraform provisions the VM and runs cloud-init bootstrap
  3. Cloud-init generates credentials and starts all services automatically

Kubernetes (Helm)

  1. Helm chart deploys all components as Deployments/StatefulSets
  2. Secrets auto-generated with random passwords
  3. Init Jobs create MinIO bucket, Iceberg schemas, and Airflow DB/user
  4. PVCs provide persistent storage for MinIO data and PostgreSQL metadata
  5. Custom Airflow image with dbt + DAGs baked in (optional)