Agentic Exploratory Data Analysis — from raw CSV to statistical hypotheses and cross-dataset memory in one pipeline.
AutoEDA runs six specialized agents against a dataset. Four agents (Schema, Stats, Correlation, Quality) execute in parallel using a ThreadPoolExecutor to produce a structured profile of the data. A fifth agent generates testable hypotheses from the profile and selects the appropriate statistical test automatically — Shapiro-Wilk for distributions, Mann-Whitney U for two-group comparisons, Kruskal-Wallis for three or more groups, Chi-squared and Cramér's V for categorical associations, Spearman for monotonic relationships, and ADF for stationarity — then interprets the results in plain English via Groq's LLM API. A sixth agent embeds every finding using sentence-transformers and stores it in a FAISS vector index, so each new run can retrieve semantically similar findings from every prior dataset without any configuration.
CSV / Parquet / JSON / Excel / SQLite
│
▼
DataLoader ──► DataValidator
│
▼
EDAOrchestrator
├── SchemaAgent ─┐
├── CorrelationAgent ├─ parallel (ThreadPoolExecutor)
├── QualityAgent ─┘
└── StatsAgent (sequential, depends on schema)
│
├── HypothesisAgent (optional, run_hypothesis=True)
│ ├── ObservationExtractor
│ ├── TestSelector (8 statistical tests)
│ ├── TestRunner
│ └── LLM Interpreter (Groq)
│
└── MemoryAgent (optional, run_memory=True)
├── Retrieve ◄── FAISS IndexFlatIP + SQLite
└── Store ──► FAISS IndexFlatIP + SQLite
│
▼
EDAResult (Pydantic v2, fully JSON-serializable)
│
▼
FastAPI ──► React Dashboard
git clone <repo-url>
cd AutoEDA
python -m venv venv && source venv/bin/activate # Windows: venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env # Add your GROQ_API_KEY
python data/samples/generate_samples.py
python data/samples/generate_edge_cases.py
# Terminal 1 — API server
uvicorn autoeda.api.main:app --reload
# Terminal 2 — React frontend
cd frontend && npm install && npm run dev
# Open http://localhost:5173Groq API key: Sign up at console.groq.com. The free tier is sufficient. Without a key, the four core EDA agents still run; only the Hypothesis Agent is skipped.
| Variable | Default | Description |
|---|---|---|
GROQ_API_KEY |
"" |
Groq API key. Required for the Hypothesis Agent. |
LLM_MODEL |
llama-3.3-70b-versatile |
Groq model ID for LLM calls. |
DATA_DIR |
data/ |
Root directory for dataset files. |
MEMORY_INDEX_PATH |
memory/faiss.index |
Path to the FAISS vector index file. |
MEMORY_DB_PATH |
memory/autoeda_memory.db |
Path to the SQLite metadata database. |
LOG_LEVEL |
INFO |
Loguru log level (DEBUG / INFO / WARNING / ERROR). |
| Method | Endpoint | Description | Request | Response |
|---|---|---|---|---|
GET |
/health |
Liveness check and memory finding count | — | {"status":"ok","version":"1.0.0","memory_findings":42} |
POST |
/analyse |
Upload dataset, start background analysis | multipart/form-data: file (required), dataset_name (optional), run_hypothesis (bool), run_memory (bool) |
202 {"run_id":"…","status":"pending","dataset_name":"…","created_at":"…"} |
GET |
/results/{run_id} |
Poll job status and retrieve result | — | {"status":"complete","result":{…EDAResult…}} or {"status":"failed","error":"…"} |
GET |
/memory/stats |
Memory store statistics | — | {"total_findings":155,"datasets":[…],"index_size_mb":0.012} |
DELETE |
/memory/clear |
Wipe all stored findings | — | 204 No Content |
Supported file formats: .csv, .parquet, .json, .xlsx, .xls, .sqlite, .db. Maximum file size: 50 MB.
# Unit tests only (no Groq key needed)
pytest tests/ --ignore=tests/test_e2e.py -m "not integration" -v
# Integration tests (requires GROQ_API_KEY)
pytest tests/ -m integration -v
# End-to-end tests against a live server (no Groq needed for most)
pytest tests/test_e2e.py -v -m "e2e and not integration"
# Full suite
pytest tests/ -vCurrent counts: 196 tests — 187 unit/API + 9 e2e. Integration tests are skipped automatically when GROQ_API_KEY is not set.
AutoEDA/
├── autoeda/
│ ├── api/
│ │ ├── main.py # FastAPI app with CORS and lifespan startup
│ │ ├── background.py # In-memory job store + background pipeline runner
│ │ ├── routes/
│ │ │ ├── analysis.py # POST /analyse, GET /results/{run_id}
│ │ │ ├── health.py # GET /health
│ │ │ └── memory.py # GET /memory/stats, DELETE /memory/clear
│ │ └── models/
│ │ └── api_models.py # Pydantic request/response models
│ ├── agents/
│ │ ├── orchestrator.py # Parallel agent runner (ThreadPoolExecutor)
│ │ ├── schema_agent.py # Column types, nullability, uniqueness
│ │ ├── stats_agent.py # Descriptive stats, normality, skewness
│ │ ├── correlation_agent.py # Pearson + Spearman pairwise correlations
│ │ ├── quality_agent.py # Missing values, outliers, duplicates, cardinality
│ │ └── hypothesis_agent.py # Observation → hypothesis → test → interpretation
│ ├── ingestion/
│ │ ├── loader.py # Format auto-detection, routes to connectors
│ │ ├── validator.py # Null, duplicate, schema warnings
│ │ └── connectors/ # CSV, Parquet, JSON, Excel, SQLite adapters
│ ├── memory/
│ │ ├── store.py # FAISS IndexFlatIP + SQLite (thread-safe)
│ │ ├── embedder.py # sentence-transformers singleton
│ │ ├── memory_agent.py # Retrieve-before-store pipeline
│ │ └── schema.py # SQLAlchemy findings table definition
│ ├── models/
│ │ ├── eda_result.py # Top-level EDAResult + all sub-models
│ │ ├── hypothesis_result.py # HypothesisReport, Hypothesis, StatisticalTest
│ │ └── memory_result.py # MemoryContext, AnalystNote
│ ├── tools/ # Pure-function computation per agent
│ ├── utils/logger.py # Loguru setup
│ └── config.py # pydantic-settings, reads from .env
├── frontend/
│ ├── src/
│ │ ├── App.jsx # Layout: sidebar + tab router
│ │ ├── styles.css # Dark theme CSS variables, no framework
│ │ ├── components/ # UploadPanel, StatusPoller, six result tabs
│ │ ├── hooks/useAnalysis.js # Upload + 2s poll lifecycle hook
│ │ └── api/client.js # Axios client, proxied to FastAPI
│ ├── vite.config.js # Vite dev server, /api proxy to :8000
│ └── package.json
├── data/samples/
│ ├── generate_samples.py # Titanic, sales, inventory, store samples
│ └── generate_edge_cases.py # Edge + hypothesis + memory stress datasets
├── tests/
│ ├── test_ingestion.py # DataLoader + DataValidator
│ ├── test_agents.py # Schema, Stats, Correlation, Quality agents
│ ├── test_agents_edge.py # Edge cases: all-null, single row, wide datasets
│ ├── test_hypothesis.py # Hypothesis engine unit tests
│ ├── test_hypothesis_edge.py # Adversarial hypothesis scenarios
│ ├── test_memory.py # MemoryStore, FindingEmbedder, MemoryAgent
│ ├── test_memory_stress.py # Persistence, similarity, concurrency, scale
│ ├── test_api.py # FastAPI endpoints via TestClient
│ └── test_e2e.py # Live Uvicorn process tests (httpx + asyncio)
├── conftest.py # Markers, live_server fixture
├── pytest.ini # asyncio_mode = auto
├── requirements.txt
└── .env.example
The HypothesisAgent starts by extracting up to ten observations from the EDAResult: skewed columns, high-correlation pairs, group-separable categoricals, potential outlier columns, and constant features. For each observation it generates a formal hypothesis statement using the Groq LLM, then selects the correct statistical test: Shapiro-Wilk or Kolmogorov-Smirnov for distribution testing, Mann-Whitney U for two-group comparisons, Kruskal-Wallis for three or more groups, Chi-squared and Cramér's V for categorical associations, Spearman for monotonic relationships, and ADF for stationarity. The test runs against the raw DataFrame using scipy.stats, then the LLM writes a plain-English interpretation that includes the implication and one or two feature engineering suggestions. The full result is stored in a HypothesisReport Pydantic model that is JSON-serializable and included in the EDAResult.
Every finding extracted from an EDAResult — schema observations, skewness flags, high-correlation pairs, quality issues, and hypothesis conclusions — is embedded using sentence-transformers/all-MiniLM-L6-v2 (384-dimensional vectors, L2-normalized). The vectors are stored in a FAISS IndexFlatIP flat inner-product index on disk alongside a SQLite table that holds the text, dataset name, finding type, and suggested action for each vector. When a new dataset is analysed, the MemoryAgent first builds a query string from the most salient features of the current run (high-severity quality issues, skewed columns, high-correlation pairs), searches the index using cosine similarity, and returns only findings with a similarity score at or above the 0.4 noise floor. Retrieval always happens before storage, so a dataset never surfaces its own findings as analyst notes. The FAISS index and SQLite database survive server restarts and accumulate findings across sessions.
MIT