Smart data cleaning tool for large CSV/Parquet files using AI-powered rule discovery
- Fast profiling - Handles multi-GB datasets with hundreds of thousands of rows per second
- Smart sampling - Profile subset of rows for speed without losing accuracy
- Comprehensive statistics - Column types, nulls, ranges, uniqueness, and more
- Heuristic rules - Instant rule discovery without any LLM required
- AI-powered suggestions - Leverage Groq (cloud) or Ollama (local) for intelligent rule generation
- Batched LLM calls - Process all columns in one efficient API call
- Multi-mode execution - Supports Heuristic-only, Groq, and Ollama configurations
- Interactive Streamlit UI - Beautiful, intuitive interface for rule validation
- Accept/Reject rules - Review and select rules before applying
- Rule details - See type, description, action, and severity for each rule
- Vectorized operations - Fast batch fixes using Polars
- Multiple cleaning actions -
clip_range,fill_null,abs_value,drop_rows,cross_column_check, and more - Automatic chunking - Handles multi-GB files with automatic chunked processing
- Incremental cleaning - Apply rules progressively without losing previous transformations
- Progress tracking - Real-time progress bars during cleaning operations
- Before/after comparison - Comprehensive metrics showing data quality improvements
- Anomaly detection - Identifies and tracks fixed issues automatically
- Cumulative statistics - Running totals across multiple cleaning passes
- Cleaning logs - Detailed log of all applied rules with cumulative statistics
- Preview cleaned data - See results before exporting
- Export cleaned data - Save as CSV or Parquet (supports chunked export for large files)
- Export rules - Save discovered rules as YAML or JSON for reuse
- Comprehensive testing - Automated test suite covering all functionality
- Error handling - Robust handling of edge cases and invalid inputs
- Performance validation - Benchmarking and performance tracking
# Create virtual environment
python -m venv venv
source venv/bin/activate # or `venv/bin/activate.fish` for fish shell
# Install dependencies
pip install -r requirements.txt
# Generate sample dataset (optional)
python src/generate_datasets.pyGenerate test files of various sizes (25K to 20M records) for testing auto-chunking and large file handling:
# Generate test files in datasets/ directory
python src/generate_datasets.pyThis will create CSV and Parquet files of increasing sizes:
- Small: 25K, 50K rows
- Medium: 100K, 250K rows
- Large: 500K, 1M rows
- XLarge: 2.5M, 5M rows
- Huge: 10M, 20M rows (maximum)
Files are saved to the datasets/ directory. The script automatically:
- Creates both CSV and Parquet versions
- Adds intentional data quality issues for testing
- Limits maximum size to 20M records for safety
# Run complete workflow demo
python src/cli.py datasets/sample_rides_25k.csv 10000# Run all tests
python tests/run_tests.pyHeuristics work great without any LLM! But for AI-powered suggestions, choose one:
- Get free API key at: https://console.groq.com
- Create
.envfile:GROQ_API_KEY=gsk_your_key_here
- Select "groq" in the app - super fast (~5-10 seconds for all columns)
- Install Docker (if not already installed)
- Run Ollama container:
docker run -d \ --name ollama \ -p 11434:11434 \ -v ollama_data:/root/.ollama \ ollama/ollama # Pull a model (3GB download) docker exec -it ollama ollama pull llama3.2
- Select "ollama" in the app - runs locally (~15-30 seconds)
No LLM? No problem! Heuristics discover rules instantly without any AI.
# Activate venv first
source venv/bin/activate # or venv/bin/activate.fish for fish
# Run the app
venv/bin/python -m streamlit run src/app.pyOpen: http://localhost:8501
1. Configure (sidebar)
- Choose LLM provider:
groq(fast) orollama(local) - Toggle "Use LLM" checkbox (or use heuristics only)
- Set sample size (default 10000 rows, 0 = all rows)
2. Load Data (tab 1)
- Upload CSV/Parquet (up to 5GB supported)
- Click "Profile Dataset" - see statistics instantly
- Profiling uses Polars for speed
3. Review Rules (tab 2)
- See heuristic rules and AI rules
- Check boxes to accept/reject rules
- Each rule shows type, description, action, severity
4. Clean Data (tab 3)
- Click "Clean Data" to apply accepted rules
- View before/after comparison
- See cleaning metrics and statistics
- Preview cleaned data
- View detailed cleaning logs
5. Export (tab 4)
- Export accepted rules as YAML or JSON
- Export cleaned dataset as CSV or Parquet
- Ready to use in your data pipeline
DataMender supports 3 distinct runtime modes, verified across all configurations:
| Mode | Description | Configuration |
|---|---|---|
| Heuristic only | Rule discovery using heuristics only | export GROQ_API_KEY=''; unset DATAMENDER_TEST_OLLAMA |
| Groq (Cloud) | Uses Groq LLM API for AI rule suggestions | export GROQ_API_KEY=gsk_your_key_here; unset DATAMENDER_TEST_OLLAMA |
| Ollama (Local) | Uses local Ollama model (via Docker) | export DATAMENDER_TEST_OLLAMA=1; export GROQ_API_KEY='' |
Tip: If both Groq and Ollama are set, Groq takes priority and Ollama serves as fallback.
The test suite is organized into four comprehensive categories:
Individual component tests:
- Profiler initialization and column statistics
- Rule discovery initialization and heuristic rules
- Data cleaner initialization and action normalization
- Metrics initialization
Complete workflow tests:
- Complete workflow: Profile → Rules → Clean → Metrics
- Streamlit UI workflow simulation
Command-line interface tests:
- Profiler CLI execution
- Rule discovery CLI execution
- CLI script validation
Full test suite covering:
- Data Profiler
- Rule Discovery
- Human Validation & Export
- Data Cleaning Engine
- Metrics & Re-Profiling
- Integration & Performance
# Run all tests (Recommended)
python tests/run_tests.py
# Or run tests individually
python tests/test_unit.py # Unit tests (fast, isolated)
python tests/test_integration.py # Integration tests (workflows)
python tests/test_cli.py # CLI tests (command-line)
python tests/test_e2e.py # Comprehensive E2E tests (full coverage)
# Or using pytest (if installed)
pytest tests/Test Suite Status: All tests passing ✅
Total Tests: 119 (All passing ✅)
- Unit Tests: 35 tests (including error handling and edge cases)
- E2E Tests: 76 tests
- Integration Tests: 4 tests
- CLI Tests: 1 test (3 functions, but 1 runs standalone)
Overall Coverage: 84% (Core library modules, excluding Streamlit UI)
| Module | Statements | Coverage |
|---|---|---|
| profiler.py | 66 | 91% |
| rule_discovery.py | 116 | 89% |
| metrics.py | 138 | 88% |
| data_cleaner.py | 275 | 88% |
| llm_client.py | 47 | 70% |
| init.py | 1 | 100% |
Note: Missing coverage is primarily in error handling paths (Ollama connection failures, missing API keys) and edge cases that are difficult to test programmatically.
Note: Streamlit UI (
app.py) is not included in coverage metrics as UI components are tested manually through interactive use.
DataMender/
├── src/
│ ├── profiler.py # Fast data profiling with Polars
│ ├── rule_discovery.py # Heuristic + LLM rule generation
│ ├── llm_client.py # LLM abstraction (Groq/Ollama)
│ ├── data_cleaner.py # Vectorized data cleaning engine
│ ├── metrics.py # Before/after comparison metrics
│ ├── app.py # Streamlit UI with human-in-the-loop
│ ├── cli.py # Complete workflow CLI script
│ └── generate_sample_data.py # Generate test dataset
├── tests/ # Test suite directory
│ ├── __init__.py # Test package initialization
│ ├── test_unit.py # Unit tests
│ ├── test_integration.py # Integration tests
│ ├── test_cli.py # CLI tests
│ └── test_e2e.py # Comprehensive E2E tests
├── .streamlit/
│ └── config.toml # Streamlit config (5GB upload limit)
├── requirements.txt # Python dependencies
├── .env # API keys (gitignored)
└── README.md # This file
| Technology | Purpose |
|---|---|
| Polars | Fast DataFrame operations (10-100x faster than pandas) |
| Streamlit | Interactive web UI |
| Groq | Cloud LLM API (free tier, super fast) |
| Ollama | Local LLM server (Docker-based) |
| Python 3.13 | Modern Python with venv |
| PyYAML | Rule export and configuration |
| python-dotenv | Environment variable management for API keys |
| psutil | Performance monitoring |
✅ Complete
All core features have been fully implemented:
- ✅ Data Profiler - Fast CSV/Parquet analysis
- ✅ Rule Discovery - Heuristic + LLM rule generation
- ✅ Human Validation - Interactive Streamlit UI
- ✅ Data Cleaning Engine - Vectorized data transformation
- ✅ Re-Profiling & Metrics - Before/after comparison
- ✅ Preview & Export - Cleaned data preview and export
- ✅ Comprehensive Testing - Full test suite coverage
Made with ❤️ for data quality