Skip to content
BranchesTags

Repository files navigation

Agentic ML Audit Copilot Banner

Agentic ML Audit Copilot

Human-in-the-loop ML audit system before model training

A deterministic-first ML engineering project that audits tabular datasets before model development, surfaces data risks, pauses risky workflows for human review, benchmarks baseline models, tracks experiments, explains results, and generates grounded audit reports.

Python Streamlit FastAPI LangGraph scikit-learn MLflow SHAP Docker License

Live Streamlit App YouTube Demo Docker Hub

Overview

Most ML projects start by training models too early.

In real projects, poor data quality, possible target leakage, class imbalance, and unsuitable metrics can make a model look stronger than it really is.

Agentic ML Audit Copilot solves this by auditing a dataset before model training. It behaves like a junior ML reviewer: it profiles the data, detects common ML risks, routes risky cases through a human review gate, and only then continues to baseline modeling, tracking, explainability, and report generation.

This is not an AutoML tool. The goal is not to train the best possible model. The goal is to decide whether the dataset is ready for responsible baseline experimentation.

Live Demo

Streamlit Application

https://shivamrajput-ds-agentic-ml-audit-copilo-appstreamlit-app-joxap5.streamlit.app/

YouTube Walkthrough

https://youtu.be/kFzNam74QBc

Docker Hub

https://hub.docker.com/r/shivamrajput130/agentic-ml-audit-copilot

Recommended Docker image:

shivamrajput130/agentic-ml-audit-copilot:v1.1.0

Demo Preview

Agentic ML Audit Copilot Demo

What It Checks

Area What the system does
Dataset Profiling Rows, columns, data types, memory usage, missing values, duplicate rows, and target summary
Problem Detection Detects classification or regression setup
Data Quality Finds missing values, duplicates, constant columns, near-constant columns, ID-like columns, high-cardinality columns, infinite values, and outliers
Leakage Risk Flags target-like columns, suspicious names, proxy features, and suspicious correlations
Class Imbalance Measures class distribution, minority class, imbalance ratio, and severity
Risk Aggregation Combines deterministic risk signals into review items
Decision Routing Decides whether the workflow can continue or needs human review
Human Review Allows reviewer decisions before modeling continues
Metric Recommendation Suggests suitable metrics for the detected problem type
Preprocessing Builds scikit-learn preprocessing pipelines
Baseline Models Trains baseline models for comparison
MLflow Tracking Logs baseline experiment metadata and metrics
Explainability Generates built-in feature importance and SHAP-based summaries when available
Reports Exports Markdown and JSON audit reports
API Provides FastAPI endpoints for programmatic audit access
UI Provides a Streamlit dashboard for interactive review

Core Design Principles

  • Deterministic-first: Python performs ML computation, data checks, and risk detection.
  • Human-in-the-loop: Risky datasets can be paused before modeling.
  • LLM is not the judge: The LLM is used only for explanations, Q&A, and report writing.
  • Baseline-first: The system trains simple baselines instead of pretending to be AutoML.
  • Transparent workflow: Every major decision is visible in the dashboard and API response.
  • Audit before training: The project focuses on responsible pre-training review.

System Architecture

System Architecture

The architecture separates the Streamlit UI, FastAPI backend, LangGraph workflow, audit modules, risk routing, human review, modeling, tracking, explainability, and reporting layers.

Human-in-the-Loop Workflow

Human-in-the-Loop Workflow

The workflow can pause at the Human Review Gate when important risks are detected.

Reviewer decisions include:

  • Accept risk and continue
  • Accept flag and fix later
  • Mark false positive
  • Needs data fix
  • Reject modeling

If the final human decision approves modeling, the workflow continues to metric recommendation, preprocessing, baseline models, MLflow, explainability, and final report generation.

If the final human decision rejects modeling, the workflow stops so the dataset can be fixed first.

FastAPI Workflow

FastAPI Workflow

The API supports both direct audit runs and a human-review-first workflow.

Dashboard Screenshots

Streamlit Home

Streamlit Home

Human Review Gate

Human Review Gate

Executive Dashboard

Executive Dashboard

FastAPI Docs

FastAPI Docs

Workflow

User
  |
  v
Streamlit UI / FastAPI API
  |
  v
CSV Upload + Target Selection
  |
  v
LangGraph Audit Workflow
  |
  v
Dataset Profiler
  |
  v
Problem Type Detector
  |
  v
Parallel Audit Layer
  |-- Data Quality Audit
  |-- Leakage Detection
  |-- Class Imbalance Detection
  |
  v
Risk Aggregator
  |
  v
Decision Router
  |
  v
Human Review Gate
  |-- Stop / Fix Dataset
  |-- Human Approved
          |
          v
      Metric Recommender
          |
          v
      Preprocessing Pipeline
          |
          v
      Baseline Models
          |
          v
      MLflow Tracking
          |
          v
      Explainability / SHAP
          |
          v
      LLM Audit Report
          |
          v
      Audit Q&A
          |
          v
      Final Dashboard + JSON Report

Technology Stack

Layer Tools
Language Python
Data Processing Pandas, NumPy
Machine Learning scikit-learn
Workflow Orchestration LangGraph
API FastAPI
Dashboard Streamlit
Experiment Tracking MLflow
Explainability SHAP
LLM Provider Groq
Visualization Plotly
Testing pytest
Linting and Formatting Ruff
Packaging uv
Deployment Docker, Streamlit Community Cloud

Quick Start

1. Clone the repository

git clone https://github.com/shivamrajput-ds/Agentic-ML-Audit-Copilot.git
cd Agentic-ML-Audit-Copilot

2. Create a virtual environment

uv venv --python 3.12

Windows PowerShell:

.venv\Scripts\Activate.ps1

Windows Git Bash:

source .venv/Scripts/activate

Linux/macOS:

source .venv/bin/activate

3. Install dependencies

uv pip install -r requirements.txt
uv pip install -e .

4. Configure environment variables

Create a .env file locally:

GROQ_API_KEY=your_groq_api_key

Do not commit .env.

For deterministic audit-only usage, LLM features can be disabled if supported by the configuration:

export LLM_ENABLED=false

Windows PowerShell:

$env:LLM_ENABLED="false"

5. Run Streamlit

uv run streamlit run app/streamlit_app.py --server.port 8501

Open:

http://localhost:8501

6. Run FastAPI

Use a second terminal:

uv run uvicorn app.api:app --reload --host 127.0.0.1 --port 8000

Open:

http://127.0.0.1:8000/docs

FastAPI Endpoints

System

Method Endpoint Description
GET / API information
GET /health Health check
GET /metadata Project and runtime metadata
GET /workflow-guide Human review workflow guide

Audit

Method Endpoint Description
POST /audit Run audit workflow
POST /audit/summary Run audit and return lightweight summary
GET /audit/modes Show available audit modes

Human Review

Method Endpoint Description
POST /audit/review-gate Run audit until human review gate
GET /human-review/decision-template Return reviewer decision JSON template
POST /audit/after-human-approval Continue workflow after reviewer approval

Recommended Human Review API Flow

1. POST /audit/review-gate
2. Review human_review.review_items
3. GET /human-review/decision-template
4. Fill reviewer decision JSON
5. POST /audit/after-human-approval
6. Continue to metrics, baselines, MLflow, SHAP, and final report

MLflow Tracking

The project logs baseline experiment information with MLflow.

Tracked information may include:

  • Problem type
  • Baseline model names
  • Evaluation metrics
  • Best baseline model
  • Parameters
  • Runtime metadata

Run MLflow UI locally:

uv run mlflow ui --backend-store-uri mlruns --host 127.0.0.1 --port 5000

Open:

http://127.0.0.1:5000

Note: the Docker container runs FastAPI and Streamlit. MLflow tracking data is generated by the workflow, but the MLflow UI is usually inspected separately in local development.

Testing

Run the full test suite:

uv run pytest -q

Run a specific test file:

uv run pytest tests/test_data_quality.py -q

Code Quality

Run Ruff checks:

uv run ruff check . --fix --unsafe-fixes

Format the project:

uv run ruff format .

Docker

The Docker image runs both services:

  • Streamlit on port 8501
  • FastAPI on port 8000

Pull the stable release:

docker pull shivamrajput130/agentic-ml-audit-copilot:v1.1.0

Run the stable release:

docker run --rm \
  --name agentic-audit-copilot \
  -p 8501:8501 \
  -p 8000:8000 \
  -e GROQ_API_KEY="your_groq_api_key" \
  shivamrajput130/agentic-ml-audit-copilot:v1.1.0

One-line Git Bash command:

docker run --rm --name agentic-audit-copilot -p 8501:8501 -p 8000:8000 -e GROQ_API_KEY="your_groq_api_key" shivamrajput130/agentic-ml-audit-copilot:v1.1.0

Open:

Streamlit Dashboard: http://localhost:8501
FastAPI Docs:       http://localhost:8000/docs
Health Check:       http://localhost:8000/health

Build locally:

docker build -t agentic-ml-audit-copilot .

Run local image:

docker run --rm \
  --name agentic-audit-test \
  -p 8501:8501 \
  -p 8000:8000 \
  -e GROQ_API_KEY="your_groq_api_key" \
  agentic-ml-audit-copilot:latest

Tag and push:

docker tag agentic-ml-audit-copilot:latest shivamrajput130/agentic-ml-audit-copilot:latest
docker tag agentic-ml-audit-copilot:latest shivamrajput130/agentic-ml-audit-copilot:v1.1.0

docker push shivamrajput130/agentic-ml-audit-copilot:latest
docker push shivamrajput130/agentic-ml-audit-copilot:v1.1.0

More details:

DOCKER.md

Streamlit Community Cloud

Streamlit Community Cloud deploys from GitHub, not Docker Hub.

For Streamlit Cloud:

  • Push the latest code to GitHub.
  • Set GROQ_API_KEY in Streamlit Cloud secrets.
  • Do not upload .env.
  • Make sure requirements.txt, app/streamlit_app.py, src/, config.yaml, and project modules are pushed.

Example Streamlit secret:

GROQ_API_KEY = "your_groq_api_key"

Project Structure

Agentic-ML-Audit-Copilot/
├── app/
│   ├── api.py
│   └── streamlit_app.py
├── src/
│   ├── audit/
│   └── utils/
├── tests/
├── docs/
├── assets/
│   ├── architecture/
│   ├── branding/
│   ├── demo/
│   └── screenshots/
├── data/
│   └── sample/
├── reports/
├── artifacts/
├── logs/
├── .github/
├── .streamlit/
├── config.yaml
├── pyproject.toml
├── requirements.txt
├── pytest.ini
├── Dockerfile
├── DOCKER.md
├── README.md
├── CHANGELOG.md
├── CONTRIBUTING.md
├── CODE_OF_CONDUCT.md
├── SECURITY.md
└── LICENSE.md

Documentation

Document Purpose
docs/ARCHITECTURE.md System architecture and workflow design
docs/API.md FastAPI endpoint guide
docs/USAGE.md Streamlit app and API usage guide
docs/TESTING.md Testing strategy and commands
docs/KNOWN_LIMITATIONS.md Known limitations and scope boundaries
docs/ROADMAP.md Planned improvements
docs/PROJECT_REVIEW.md Portfolio-level project review
docs/ASSETS.md Asset and screenshot guide
DOCKER.md Docker build, run, and publish guide
CHANGELOG.md Release history
CONTRIBUTING.md Contribution guidelines
CODE_OF_CONDUCT.md Community rules
SECURITY.md Security policy
LICENSE.md License details

Engineering Highlights

  • LangGraph-based audit workflow
  • Parallel deterministic audit checks
  • Risk Aggregator and Decision Router
  • Human Review Gate for risky datasets
  • Reviewer decision export as JSON
  • FastAPI backend with Swagger documentation
  • Streamlit dashboard with audit tabs and downloads
  • Baseline model benchmarking
  • MLflow experiment tracking
  • SHAP and built-in feature importance support
  • LLM-based audit report and Q&A
  • JSON-safe API responses
  • Configuration-driven behavior
  • Centralized logging and exception handling
  • pytest test suite
  • Ruff linting and formatting
  • Dockerized local deployment
  • Streamlit Cloud deployment
  • GitHub Actions CI support

Current Limitations

This project currently focuses on:

  • CSV datasets
  • Tabular ML
  • Classification and regression
  • Single-machine execution
  • Baseline model benchmarking
  • Pre-training audit and review

It is not a replacement for:

  • Full enterprise data governance
  • Security review
  • Production monitoring
  • Fairness certification
  • Model approval boards
  • AutoML systems
  • Production model serving platforms

Roadmap

Planned improvements:

  • Data drift detection
  • Feature drift detection
  • Fairness and bias analysis
  • Hyperparameter optimization
  • PDF reports
  • HTML reports
  • Polars support
  • Dask support
  • Authentication
  • Team workspaces
  • Kubernetes deployment
  • Cloud deployment templates

Contributing

Contributions are welcome.

Before opening a pull request:

uv run ruff check . --fix --unsafe-fixes
uv run ruff format .
uv run pytest -q

Please read:

CONTRIBUTING.md
CODE_OF_CONDUCT.md

Security

Do not commit secrets.

Use:

.env.example

for documenting required variables.

Use environment variables or platform secrets for real API keys.

License

This project is released under the MIT License.

See:

LICENSE.md

Author

Shivam Rajput

Data Science | Machine Learning | MLOps | Agentic AI

  • Portfolio: https://shivamrajput-ds.github.io/portfolio-website/
  • GitHub: https://github.com/shivamrajput-ds
  • LinkedIn: https://www.linkedin.com/in/shivam-rajput-ds/
  • Docker Hub: https://hub.docker.com/r/shivamrajput130/agentic-ml-audit-copilot
  • YouTube: https://youtu.be/kFzNam74QBc
  • Kaggle: https://www.kaggle.com/shivamja
  • LeetCode: https://leetcode.com/u/ShivamSynapse/
  • X: https://x.com/ShivamR65014299

Agentic ML Audit Copilot

Python • scikit-learn • FastAPI • Streamlit • LangGraph • MLflow • SHAP • Docker • Groq

About

Human-in-the-loop Agentic ML audit system for tabular datasets — detects data risks, possible leakage, class imbalance, recommends metrics, benchmarks baseline models, tracks experiments, and generates grounded audit reports.

shivamrajput-ds-agentic-ml-audit-copilo-appstreamlit-app-joxap5.streamlit.app/

Topics

agentic-aiai-agentsdata-qualitydata-sciencedockerexplainable-aifastapihuman-in-the-looplangchainllmmachine-learningmlflowmlopsmodel-evaluationpandaspythonscikit-learnshapstreamlittarget-leakage

Resources

Readme
MIT license

Code of conduct

Code of conduct

Contributing

Contributing

Security policy

Security policy
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages