DataHub Impact Analyzer

Know what breaks before you break it.
AI-powered schema change impact analysis using DataHub Lineage and LLMs.

The Problem

Schema evolution is one of the most common causes of production incidents in modern data platforms.

Data ecosystems are deeply interconnected — a single table feeds dashboards, ETL pipelines, ML models, and downstream datasets. When an engineer modifies a schema, they often have zero visibility into what will break:

Without Impact Analysis	With DataHub Impact Analyzer
"Which dashboards will break?"	Instantly identifies all affected dashboards
"Which ETL pipelines will fail?"	Maps every downstream pipeline dependency
"Who do I notify?"	Surfaces asset owners for immediate outreach
"How severe is this change?"	Classifies risk into Critical / High / Low
"What should I do next?"	AI-generated recommendations & action steps

The Solution

DataHub Impact Analyzer connects to your DataHub instance, traverses the lineage graph for any table, and produces a comprehensive impact report — complete with severity classifications and AI-generated explanations — in seconds.

Engineer submits schema change  →  Lineage graph traversed  →  Impact classified  →  AI explains risks

Works without any API keys. If no OpenAI key is configured, the system gracefully falls back to template-based explanations with zero loss of functionality.

Demo

Try it yourself in 60 seconds:

git clone https://github.com/Siteshcodes/DataHub-Impact-Analyzer.git
cd DataHub-Impact-Analyzer
docker compose up --build
# Open http://localhost:8000

Sample Input:

Field	Value
Table URN	`snowflake:prod.analytics.user_events`
Schema Change	`DROP COLUMN user_id`

Sample Output:

┌────────────────────────────────────────────────────────────────┐
│  CRITICAL ×2   │  HIGH ×1          │  LOW ×1                   │
│  Dashboards    │  Pipeline         │  Staging Table            │
├────────────────────────────────────────────────────────────────┤
│  AI Explanation:                                               │
│  "Dropping user_id will break primary key joins in the         │
│   Executive Analytics dashboard and User Retention dashboard.  │
│   The Daily User ETL pipeline will fail on its lookup step."   │
│                                                                │
│  Recommendation:                                               │
│  "Notify analytics-team@company.com before proceeding.         │
│   Consider a staged rollout with monitoring."                  │
└────────────────────────────────────────────────────────────────┘

Features

Feature	Description
Schema Change Parsing	Supports `DROP COLUMN`, `ADD COLUMN`, and `ALTER COLUMN` with regex-based extraction
DataHub Lineage Traversal	Fetches all downstream dependencies directly from DataHub's lineage graph
Severity Classification	Auto-classifies impact as Critical (dashboards), High (pipelines), or Low (tables)
AI-Powered Explanations	GPT-4o-mini generates clear, contextual summaries with actionable recommendations
Template Fallback	Full functionality without an API key — template engine produces structured explanations
Asset Ownership Detection	Identifies owners of every affected downstream asset for notification
Dual-Mode DataHub	Seamlessly switches between live DataHub GMS and high-fidelity mock data
Pydantic Validation	Strict request/response validation with detailed error messages
Containerized Deployment	One-command start with Docker Compose
Comprehensive Test Suite	Full pytest coverage across all services
🆕 DataHub Skills Integration	Five integrated skills — Search, Lineage, Enrich, Quality, Setup — with a premium dark-mode dashboard

DataHub Skills Integration

This project integrates all five DataHub Skills directly into the web UI, transforming the Impact Analyzer into a full-featured DataHub agent dashboard.

Skill	Endpoint	What it does
Search	`POST /api/skills/search`	Find data assets using descriptions, glossary terms, ownership, usage, and quality signals
Lineage	`POST /api/skills/lineage`	Trace upstream sources, transformations, and downstream consumers with interactive graph
Enrich	`POST /api/skills/enrich`	Add descriptions, tags, glossary terms, owners, domains with before/after diff preview
Quality	`POST /api/skills/quality`	Monitor assertions (freshness, volume, schema), investigate incidents, view health scores
Setup	`POST /api/skills/setup`	Configure and test your DataHub connection (Cloud or self-hosted)

Skills vs. Tools: Skills give agents instructions — how to chain multiple API calls into multi-step workflows with the right judgment at each step. The Skills dashboard lets you run these same workflows from a visual interface.

Architecture

flowchart TB
    subgraph Client
        User([👤 Engineer]) --> Frontend[🖥️ Skills Dashboard]
    end

    subgraph Backend ["FastAPI Backend"]
        direction TB
        API["/api/analyze"] --> Parser["Schema Parser"]
        Parser --> DHClient["DataHub Client"]
        DHClient --> Analyzer["Impact Analyzer"]
        Analyzer --> LLM["LLM Agent"]
        LLM --> Response["Response Builder"]

        SkillsAPI["/api/skills/*"] --> SkillsSvc["Skills Service"]
    end

    subgraph External ["External Services"]
        DataHub[(DataHub GMS)]
        OpenAI[OpenAI API]
    end

    Frontend -- "POST {table_urn, schema_change}" --> API
    Frontend -- "POST skills/search|lineage|enrich|quality|setup" --> SkillsAPI
    DHClient -- "Lineage Query" --> DataHub
    SkillsSvc -- "Search / Lineage / Enrich / Quality" --> DataHub
    LLM -- "Prompt" --> OpenAI
    Response -- "JSON Impact Report" --> Frontend

Request Lifecycle

1. Submit    →  User enters table URN + schema change (e.g., DROP COLUMN user_id)
2. Validate  →  Pydantic validates URN format and operation syntax
3. Parse     →  Regex parser extracts operation type, target column, and types
4. Traverse  →  DataHub client walks the lineage graph for downstream assets
5. Classify  →  Impact analyzer assigns severity by asset type
6. Explain   →  LLM (or template fallback) generates human-readable summary
7. Respond   →  Structured JSON response rendered on the interactive dashboard

Quick Start

Prerequisites

Docker & Docker Compose
(Optional) An OpenAI API key for AI-powered explanations

1. Clone & Configure

git clone https://github.com/Siteshcodes/DataHub-Impact-Analyzer.git
cd DataHub-Impact-Analyzer
cp .env.example .env

2. Launch

docker compose up --build

3. Open

Navigate to http://localhost:8000 and run your first analysis.

Tip

The app ships with mock DataHub data enabled by default (USE_MOCK_DATA=true), so you can explore the full workflow immediately — no DataHub instance required.

Configuration

All configuration is managed via environment variables in .env:

Variable	Default	Required	Description
`DATAHUB_SERVER`	`http://localhost:8080`	No	DataHub GMS server URL
`DATAHUB_TOKEN`	—	No	Authentication token for DataHub
`OPENAI_API_KEY`	—	No	OpenAI API key for AI explanations (falls back to templates)
`USE_MOCK_DATA`	`true`	No	Use built-in mock lineage data instead of a live DataHub instance

API Reference

`POST /api/analyze`

Analyze the downstream impact of a schema change.

Request Body

{
  "table_urn": "snowflake:prod.analytics.user_events",
  "schema_change": "DROP COLUMN user_id"
}

Field	Type	Validation	Description
`table_urn`	`string`	URN format: `urn:li:dataset:...` or `platform:db.schema.table`	Target table identifier
`schema_change`	`string`	Must start with `DROP`, `ADD`, or `ALTER`	Schema modification statement

Success Response (200)

{
  "table_urn": "snowflake:prod.analytics.user_events",
  "schema_change": "DROP COLUMN user_id",
  "total_affected": 4,
  "severity_counts": {
    "critical": 2,
    "high": 1,
    "low": 1
  },
  "affected_assets": [
    {
      "urn": "urn:li:dashboard:(tableau,user_analytics)",
      "name": "Executive User Analytics Dashboard",
      "type": "dashboard",
      "severity": "CRITICAL",
      "owner": "analytics-team@company.com",
      "reason": "May reference column 'user_id'"
    }
  ],
  "explanation": "Dropping 'user_id' will break 2 downstream dashboards..."
}

Error Response (400 / 500)

{
  "error": "Invalid table URN format",
  "details": "Expected format: 'urn:li:dataset:...' or 'platform:database.schema.table'"
}

`GET /health`

Returns service health status.

{ "status": "healthy", "version": "1.0.0" }

Testing

Run the full test suite:

cd backend
pytest -v

Test Coverage

Module	Tests Cover
API Endpoints	Request validation, error handling, response structure
Schema Parser	`DROP`, `ADD`, `ALTER` operations + edge cases
DataHub Client	Lineage resolution with mock & live modes
Impact Analyzer	Severity classification, affected asset detection
LLM Agent	Prompt construction, template fallback, error handling

Technology Stack

Layer	Technology	Why
API Framework	FastAPI	Async-first, auto-generated OpenAPI docs, Pydantic integration
Runtime	Python 3.11	Modern type hints, performance improvements
Metadata Platform	DataHub (acryl-datahub SDK)	Industry-standard lineage graph and metadata catalog
AI / LLM	OpenAI GPT-4o-mini	Cost-effective, fast, high-quality natural language generation
Validation	Pydantic v2	Runtime type checking with detailed error messages
Frontend	HTML5 / CSS3 / JavaScript	Lightweight, zero-dependency interactive UI
Deployment	Docker Compose	Single-command multi-container orchestration
Testing	pytest + pytest-asyncio + httpx	Async-compatible test framework with HTTP client

Project Structure

DataHub-Impact-Analyzer/
├── backend/
│   ├── app/
│   │   ├── api/                  # FastAPI route handlers
│   │   ├── models/
│   │   │   └── schemas.py        # Pydantic request/response models
│   │   ├── services/
│   │   │   ├── schema_parser.py  # SQL schema change parser
│   │   │   ├── datahub_client.py # DataHub lineage client (mock + live)
│   │   │   ├── impact_analyzer.py# Severity classification engine
│   │   │   └── llm_agent.py      # OpenAI integration + template fallback
│   │   ├── utils/                # Logger, config, helpers
│   │   └── main.py               # FastAPI app entry point
│   ├── tests/                    # Comprehensive pytest suite
│   ├── requirements.txt
│   ├── Dockerfile
│   └── pytest.ini
│
├── frontend/
│   ├── index.html                # Main UI
│   ├── styles.css                # Styling
│   └── app.js                    # Client-side logic
│
├── ARCHITECTURE.md               # Architecture design document
├── .env.example                  # Environment template
├── docker-compose.yml            # Container orchestration
├── LICENSE
└── README.md

Reliability & Resiliency

The application is engineered for production-grade resilience, even in a hackathon context:

Capability	How It Works
Dual-Mode DataHub	Seamless switching between live GMS and high-fidelity mock lineage — no config changes needed
AI Fallback System	Automatic failover to template-based explanations when OpenAI is unavailable or rate-limited
Strict Validation	Pydantic v2 validates every request and response at runtime with descriptive error messages
Modular Architecture	Decoupled services with dependency injection for isolated testing and easy extension
Cache-Control Headers	Static assets served with proper `Cache-Control` for optimal browser caching
Structured Logging	Consistent `timestamp - logger - level - message` format across all components
Health Endpoint	`GET /health` for container orchestration readiness checks

Hackathon Submission

Built for Build with DataHub: The Agent Hackathon

Criteria	How We Deliver
Use of DataHub	Deep integration with DataHub's lineage graph API for automated downstream impact discovery
Technical Execution	Full-stack implementation — async backend, interactive frontend, comprehensive test suite
Real-World Value	Prevents breaking schema changes in production data platforms before they reach deployment
AI Integration	GPT-4o-mini generates contextual, actionable impact explanations with graceful fallback

Future Roadmap

✅ Interactive Lineage Graph — Visual graph exploration embedded in the frontend (shipped!)
✅ DataHub Skills Integration — Search, Lineage, Enrich, Quality, Setup (shipped!)
⬜ Column-Level Lineage — Pinpoint exact field usage in SQL views and dashboard queries
⬜ MCP Server Integration — Enable AI agents (Claude Desktop, Cursor) to run impact analysis
⬜ Slack & Email Alerts — Automated notification dispatch to asset owners via webhooks
⬜ Multi-Change Analysis — Batch analysis of multiple schema changes in a single request
⬜ Historical Impact Tracking — Audit log of past analyses for compliance and review

Contributing

Contributions are welcome! Here's how to get started:

Fork the repository
Create a feature branch (git checkout -b feature/your-feature)
Commit your changes (git commit -m 'Add your feature')
Push to the branch (git push origin feature/your-feature)
Open a Pull Request

Please ensure all tests pass before submitting:

cd backend && pytest -v

License

Distributed under the Apache License 2.0. See LICENSE for details.

Author

Sitesh Gupta

_{Built with ☕ and a passion for preventing production incidents.}

Navigation Menu

Repository files navigation

DataHub Impact Analyzer

The Problem

The Solution

Demo

Features

DataHub Skills Integration

Architecture

Request Lifecycle

Quick Start

Prerequisites

1. Clone & Configure

2. Launch

3. Open

Configuration

API Reference

`POST /api/analyze`

`GET /health`

Testing

Test Coverage

Technology Stack

Project Structure

Reliability & Resiliency

Hackathon Submission

Future Roadmap

Contributing

License

Author

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 27 Commits
backend		backend
frontend		frontend
.env.example		.env.example
.gitignore		.gitignore
ARCHITECTURE.md		ARCHITECTURE.md
Architecture.png		Architecture.png
LICENSE		LICENSE
README.md		README.md
docker-compose.yml		docker-compose.yml

Folders and files

Latest commit

History

Repository files navigation

DataHub Impact Analyzer

The Problem

The Solution

Demo

Features

DataHub Skills Integration

Architecture

Request Lifecycle

Quick Start

Prerequisites

1. Clone & Configure

2. Launch

3. Open

Configuration

API Reference

POST /api/analyze

GET /health

Testing

Test Coverage

Technology Stack

Project Structure

Reliability & Resiliency

Hackathon Submission

Future Roadmap

Contributing

License

Author

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`POST /api/analyze`

`GET /health`