Skip to content

README.md

Latest commit

 

History

History
206 lines (165 loc) · 10.8 KB

README.md

File metadata and controls

206 lines (165 loc) · 10.8 KB
Raw

AutoEDA

Agentic Exploratory Data Analysis — from raw CSV to statistical hypotheses and cross-dataset memory in one pipeline.

Python 3.11+ FastAPI React FAISS CrewAI License

What it does

AutoEDA runs six specialized agents against a dataset. Four agents (Schema, Stats, Correlation, Quality) execute in parallel using a ThreadPoolExecutor to produce a structured profile of the data. A fifth agent generates testable hypotheses from the profile and selects the appropriate statistical test automatically — Shapiro-Wilk for distributions, Mann-Whitney U for two-group comparisons, Kruskal-Wallis for three or more groups, Chi-squared and Cramér's V for categorical associations, Spearman for monotonic relationships, and ADF for stationarity — then interprets the results in plain English via Groq's LLM API. A sixth agent embeds every finding using sentence-transformers and stores it in a FAISS vector index, so each new run can retrieve semantically similar findings from every prior dataset without any configuration.

Architecture

CSV / Parquet / JSON / Excel / SQLite
        │
        ▼
  DataLoader ──► DataValidator
        │
        ▼
  EDAOrchestrator
  ├── SchemaAgent       ─┐
  ├── CorrelationAgent   ├─ parallel (ThreadPoolExecutor)
  ├── QualityAgent      ─┘
  └── StatsAgent  (sequential, depends on schema)
        │
        ├── HypothesisAgent  (optional, run_hypothesis=True)
        │   ├── ObservationExtractor
        │   ├── TestSelector  (8 statistical tests)
        │   ├── TestRunner
        │   └── LLM Interpreter  (Groq)
        │
        └── MemoryAgent  (optional, run_memory=True)
            ├── Retrieve  ◄── FAISS IndexFlatIP + SQLite
            └── Store     ──► FAISS IndexFlatIP + SQLite
                │
                ▼
          EDAResult  (Pydantic v2, fully JSON-serializable)
                │
                ▼
          FastAPI  ──► React Dashboard

Quickstart

git clone <repo-url>
cd AutoEDA
python -m venv venv && source venv/bin/activate   # Windows: venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env          # Add your GROQ_API_KEY
python data/samples/generate_samples.py
python data/samples/generate_edge_cases.py

# Terminal 1 — API server
uvicorn autoeda.api.main:app --reload

# Terminal 2 — React frontend
cd frontend && npm install && npm run dev

# Open http://localhost:5173

Groq API key: Sign up at console.groq.com. The free tier is sufficient. Without a key, the four core EDA agents still run; only the Hypothesis Agent is skipped.

Configuration

Variable Default Description
GROQ_API_KEY "" Groq API key. Required for the Hypothesis Agent.
LLM_MODEL llama-3.3-70b-versatile Groq model ID for LLM calls.
DATA_DIR data/ Root directory for dataset files.
MEMORY_INDEX_PATH memory/faiss.index Path to the FAISS vector index file.
MEMORY_DB_PATH memory/autoeda_memory.db Path to the SQLite metadata database.
LOG_LEVEL INFO Loguru log level (DEBUG / INFO / WARNING / ERROR).

API reference

Method Endpoint Description Request Response
GET /health Liveness check and memory finding count — {"status":"ok","version":"1.0.0","memory_findings":42}
POST /analyse Upload dataset, start background analysis multipart/form-data: file (required), dataset_name (optional), run_hypothesis (bool), run_memory (bool) 202 {"run_id":"…","status":"pending","dataset_name":"…","created_at":"…"}
GET /results/{run_id} Poll job status and retrieve result — {"status":"complete","result":{…EDAResult…}} or {"status":"failed","error":"…"}
GET /memory/stats Memory store statistics — {"total_findings":155,"datasets":[…],"index_size_mb":0.012}
DELETE /memory/clear Wipe all stored findings — 204 No Content

Supported file formats: .csv, .parquet, .json, .xlsx, .xls, .sqlite, .db. Maximum file size: 50 MB.

Running tests

# Unit tests only (no Groq key needed)
pytest tests/ --ignore=tests/test_e2e.py -m "not integration" -v

# Integration tests (requires GROQ_API_KEY)
pytest tests/ -m integration -v

# End-to-end tests against a live server (no Groq needed for most)
pytest tests/test_e2e.py -v -m "e2e and not integration"

# Full suite
pytest tests/ -v

Current counts: 196 tests — 187 unit/API + 9 e2e. Integration tests are skipped automatically when GROQ_API_KEY is not set.

Project structure

AutoEDA/
├── autoeda/
│   ├── api/
│   │   ├── main.py              # FastAPI app with CORS and lifespan startup
│   │   ├── background.py        # In-memory job store + background pipeline runner
│   │   ├── routes/
│   │   │   ├── analysis.py      # POST /analyse, GET /results/{run_id}
│   │   │   ├── health.py        # GET /health
│   │   │   └── memory.py        # GET /memory/stats, DELETE /memory/clear
│   │   └── models/
│   │       └── api_models.py    # Pydantic request/response models
│   ├── agents/
│   │   ├── orchestrator.py      # Parallel agent runner (ThreadPoolExecutor)
│   │   ├── schema_agent.py      # Column types, nullability, uniqueness
│   │   ├── stats_agent.py       # Descriptive stats, normality, skewness
│   │   ├── correlation_agent.py # Pearson + Spearman pairwise correlations
│   │   ├── quality_agent.py     # Missing values, outliers, duplicates, cardinality
│   │   └── hypothesis_agent.py  # Observation → hypothesis → test → interpretation
│   ├── ingestion/
│   │   ├── loader.py            # Format auto-detection, routes to connectors
│   │   ├── validator.py         # Null, duplicate, schema warnings
│   │   └── connectors/          # CSV, Parquet, JSON, Excel, SQLite adapters
│   ├── memory/
│   │   ├── store.py             # FAISS IndexFlatIP + SQLite (thread-safe)
│   │   ├── embedder.py          # sentence-transformers singleton
│   │   ├── memory_agent.py      # Retrieve-before-store pipeline
│   │   └── schema.py            # SQLAlchemy findings table definition
│   ├── models/
│   │   ├── eda_result.py        # Top-level EDAResult + all sub-models
│   │   ├── hypothesis_result.py # HypothesisReport, Hypothesis, StatisticalTest
│   │   └── memory_result.py     # MemoryContext, AnalystNote
│   ├── tools/                   # Pure-function computation per agent
│   ├── utils/logger.py          # Loguru setup
│   └── config.py                # pydantic-settings, reads from .env
├── frontend/
│   ├── src/
│   │   ├── App.jsx              # Layout: sidebar + tab router
│   │   ├── styles.css           # Dark theme CSS variables, no framework
│   │   ├── components/          # UploadPanel, StatusPoller, six result tabs
│   │   ├── hooks/useAnalysis.js # Upload + 2s poll lifecycle hook
│   │   └── api/client.js        # Axios client, proxied to FastAPI
│   ├── vite.config.js           # Vite dev server, /api proxy to :8000
│   └── package.json
├── data/samples/
│   ├── generate_samples.py      # Titanic, sales, inventory, store samples
│   └── generate_edge_cases.py   # Edge + hypothesis + memory stress datasets
├── tests/
│   ├── test_ingestion.py        # DataLoader + DataValidator
│   ├── test_agents.py           # Schema, Stats, Correlation, Quality agents
│   ├── test_agents_edge.py      # Edge cases: all-null, single row, wide datasets
│   ├── test_hypothesis.py       # Hypothesis engine unit tests
│   ├── test_hypothesis_edge.py  # Adversarial hypothesis scenarios
│   ├── test_memory.py           # MemoryStore, FindingEmbedder, MemoryAgent
│   ├── test_memory_stress.py    # Persistence, similarity, concurrency, scale
│   ├── test_api.py              # FastAPI endpoints via TestClient
│   └── test_e2e.py              # Live Uvicorn process tests (httpx + asyncio)
├── conftest.py                  # Markers, live_server fixture
├── pytest.ini                   # asyncio_mode = auto
├── requirements.txt
└── .env.example

How the hypothesis engine works

The HypothesisAgent starts by extracting up to ten observations from the EDAResult: skewed columns, high-correlation pairs, group-separable categoricals, potential outlier columns, and constant features. For each observation it generates a formal hypothesis statement using the Groq LLM, then selects the correct statistical test: Shapiro-Wilk or Kolmogorov-Smirnov for distribution testing, Mann-Whitney U for two-group comparisons, Kruskal-Wallis for three or more groups, Chi-squared and Cramér's V for categorical associations, Spearman for monotonic relationships, and ADF for stationarity. The test runs against the raw DataFrame using scipy.stats, then the LLM writes a plain-English interpretation that includes the implication and one or two feature engineering suggestions. The full result is stored in a HypothesisReport Pydantic model that is JSON-serializable and included in the EDAResult.

How the memory works

Every finding extracted from an EDAResult — schema observations, skewness flags, high-correlation pairs, quality issues, and hypothesis conclusions — is embedded using sentence-transformers/all-MiniLM-L6-v2 (384-dimensional vectors, L2-normalized). The vectors are stored in a FAISS IndexFlatIP flat inner-product index on disk alongside a SQLite table that holds the text, dataset name, finding type, and suggested action for each vector. When a new dataset is analysed, the MemoryAgent first builds a query string from the most salient features of the current run (high-severity quality issues, skewed columns, high-correlation pairs), searches the index using cosine similarity, and returns only findings with a similarity score at or above the 0.4 noise floor. Retrieval always happens before storage, so a dataset never surfaces its own findings as analyst notes. The FAISS index and SQLite database survive server restarts and accumulate findings across sessions.

License

MIT