Aiven → DataHub Connector

A Streamlit web application to easily connect Aiven PostgreSQL and ClickHouse services to DataHub for metadata discovery and governance.

Features

• 🔐 Secure token-based authentication for both Aiven and DataHub
• 📋 Automatic discovery of PostgreSQL and ClickHouse services from Aiven
• 🗄️ Database selection for PostgreSQL services (supports custom databases)
• 📊 Schema selection and filtering for PostgreSQL
• 🚀 One-click connector creation in DataHub
• ⚙️ Configurable schema and table patterns
• 👁️ Recipe preview before ingestion
• 📜 Real-time status and results display

Prerequisites

• Python 3.8 or higher
• Aiven account with API token
• DataHub instance with Personal Access Token (PAT)
• PostgreSQL services on Aiven (optional: ClickHouse services)

Installation

1. Clone this repository:

git clone <your-repo-url>
cd aiven_datahub_pg

2. Create a virtual environment (recommended):

python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install dependencies:

pip install -r requirements.txt

Usage

1. Start the Streamlit app:

streamlit run app.py

2. The app will open in your browser (typically at http://localhost:8501)

3. Configure your tokens in the sidebar:

   • Aiven API Token: Your Aiven API token (obtain from Aiven Console → Account → API Tokens)
   • DataHub GraphQL URL: Your DataHub GraphQL API endpoint (e.g., http://your-datahub-host:8080/api/graphql)
   • DataHub Access Token: Your DataHub Personal Access Token (generate in DataHub UI → Settings → Access Tokens)

4. Select a project and service:

   • Choose an Aiven project from the dropdown
   • Select PostgreSQL or ClickHouse services to connect

5. For PostgreSQL services:

   • Fetch Databases: Click to discover available databases (or manually enter database name)
   • Select Database: Choose which database to ingest
   • Fetch Schemas: Discover available schemas in the selected database
   • Select Schemas: Choose specific schemas to include (or use regex patterns)
   • Configure Filters:
     • Exclude system schemas (recommended)
     • Optionally exclude 'public' schema
     • Set table pattern filters
   • Preview Recipe: Review the generated DataHub recipe before ingestion

6. Click "🚀 Create DataHub Connectors" to create and execute ingestion jobs

How It Works

1. Service Discovery: Uses Aiven API to fetch all PostgreSQL and ClickHouse services across your projects
2. Connection Info Extraction: Retrieves connection details (host, port, credentials) from Aiven service details
3. Database/Schema Discovery: For PostgreSQL, connects to the database to discover available databases and schemas
4. Recipe Generation: Builds DataHub ingestion recipes with your selected configurations
5. Connector Creation: Creates ingestion sources in DataHub using the GraphQL API
6. Job Execution: Automatically triggers ingestion jobs for each created connector

Configuration Options

PostgreSQL

• Database Selection: Choose from discovered databases or manually enter a database name
• Schema Selection:
  • Select specific schemas by name (exact match)
  • Or use regex patterns (e.g., .* for all, public|analytics for specific schemas)
• System Schema Exclusion: Automatically exclude PostgreSQL system schemas (pg_catalog, information_schema, etc.)
• Table Patterns: Use regex to filter tables (e.g., .* for all, ^prod_.* for production tables)

ClickHouse

• Uses default settings (all tables included)
• Automatically configures SSL connections via uri_opts

Troubleshooting

No Services Found

• Verify your Aiven API token is correct
• Check that you have PostgreSQL/ClickHouse services in your Aiven projects
• Ensure the token has proper permissions

Database Fetching Fails

• pg_hba.conf errors: Aiven PostgreSQL may require IP whitelisting. You can manually enter the database name instead.
• Database 'postgres' does not exist: Aiven uses defaultdb as the default database name. Try fetching databases or manually enter defaultdb.

Connection Errors

• Verify DataHub URL is correct and accessible
• Check that DataHub authentication is enabled
• Ensure the DataHub token is valid and not expired

Schema Fetching Fails

• Verify database name is correct
• Check network connectivity to PostgreSQL
• Ensure SSL is properly configured (Aiven requires SSL)

Security Notes

• Tokens are stored only in session state (cleared on refresh)
• Never commit tokens to version control
• Use environment variables for production deployments
• Consider using Streamlit secrets management for sensitive data

Advanced Usage

Running with Custom Port

streamlit run app.py --server.port 8502

Using Environment Variables

You can modify the app to read from environment variables:

export DATAHUB_URL="http://your-datahub:8080/api/graphql"
export DATAHUB_TOKEN="your-token"
export AIVEN_TOKEN="your-aiven-token"

API Reference

The app uses:

• Aiven API: https://api.aiven.io/v1/project/{project}/service
• DataHub GraphQL API: {datahub_url}/api/graphql

License

MIT License - feel free to modify and use as needed.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.