DataHub Production Setup with Aiven Integration

This folder contains the production-ready DataHub Docker Compose setup with Aiven PostgreSQL and Kafka integration.

📁 Files Included

🔧 Core Files

• docker-compose.yml - Main Docker Compose configuration (production-ready)
• .gitignore - Security protection for sensitive files

🔐 Certificate & Configuration

• aiven-ca.pem.template - Template for Aiven CA certificate
• aiven-postgres-working.yaml - Working PostgreSQL connector configuration
• aiven-kafka-working-final.yaml - Working Kafka connector configuration

🛠 Helper Scripts

• setup-aiven-ca.sh - Interactive CA certificate setup script
• start-datahub-docker.sh - DataHub startup script with pre-checks

🚀 Quick Start

1. Setup Aiven Certificate

# Option A: Interactive setup
./setup-aiven-ca.sh

# Option B: Manual setup
cp aiven-ca.pem.template aiven-ca.pem
# Edit aiven-ca.pem with your actual Aiven CA certificate

2. Start DataHub

# Using the helper script (recommended)
./start-datahub-docker.sh

# Or directly with docker-compose
docker-compose up -d

3. Access DataHub

• Frontend: http://localhost:8092
• Backend API: http://localhost:8093
• Default Login: datahub / datahub

🔌 Aiven Connectors

PostgreSQL Ingestion

# Test the PostgreSQL connector
docker exec datahub-datahub-actions datahub ingest run -c /tmp/aiven-postgres-working.yaml

Kafka Ingestion

# Test the Kafka connector (requires CA certificate setup)
docker exec datahub-datahub-actions datahub ingest run -c /tmp/aiven-kafka-working-final.yaml

✅ What's Included

This setup solves all the common DataHub + Aiven integration issues:

• ✅ PostgreSQL psycopg2 dependency auto-installed
• ✅ Memory limits optimized (4GB for actions container)
• ✅ Port conflicts resolved (8092/8093/9201)
• ✅ Aiven SSL/CA certificate integration
• ✅ Neo4j health check issues fixed
• ✅ Security best practices (.gitignore)

📞 Support

If you encounter issues, check the logs:

docker-compose logs -f datahub-gms
docker-compose logs -f datahub-actions

Status: Production Ready! 🎉