nrchd — LLM Enrichment Pipeline for CSV Files

Give me any CSV, tell me what you want to know or generate per row, and I'll run it through an LLM at scale and give you back an enriched CSV.

nrchd is a local-first, resumable LLM enrichment pipeline with a Streamlit UI. Upload a CSV, define a task and output schema, optionally attach a knowledge base, and run structured LLM enrichment across every row — in parallel, with crash recovery, and with no data leaving your machine unless you choose a cloud LLM.

---

Features

• Any CSV → enriched CSV — define your own prompt and output schema per task
• Structured output — schema validation and type coercion per field (bool, string, int, float)
• Resumable — crash or stop mid-run; re-run picks up exactly where it left off
• Parallel + rate-limited — configurable concurrency and calls/minute
• RAG (Retrieval-Augmented Generation) — attach PDF/DOCX/TXT/MD/CSV knowledge bases; relevant chunks are injected per row
• Semantic analysis — local dedup, clustering, outlier detection, cross-file similarity, semantic search (no API needed)
• Multiple LLM backends — Claude, GPT-4o-mini, Groq Llama, Google Gemini, Ollama (local)
• Task templates — built-in templates for common tasks; save your own
• No cloud dependency — use Ollama + local sentence-transformers for a fully offline pipeline

---

Quick Start

Prerequisites

• Python 3.10+
• uv (recommended) — install with:

  curl -LsSf https://astral.sh/uv/install.sh | sh

• At least one API key (Anthropic, OpenAI, or Groq) — or Ollama for local inference

Setup

git clone https://github.com/mortalapps/nrchd
cd nrchd
make setup        # installs deps, creates .env from .env.example

Edit .env and add your API key(s):

ANTHROPIC_API_KEY=sk-ant-...
# OPENAI_API_KEY=sk-...
# GROQ_API_KEY=gsk_...

Run

make run          # opens http://localhost:8501

Docker (alternative)

docker compose up

---

Supported Models

LLM Backends

Provider	Model	Key needed	Notes
Claude (Haiku)	claude-haiku-4-5	ANTHROPIC_API_KEY	Fast, cost-effective
Claude (Sonnet)	claude-sonnet-4-6	ANTHROPIC_API_KEY	Best quality
GPT-4o-mini	gpt-4o-mini	OPENAI_API_KEY	Good balance
Groq Llama 3.1 70B	llama-3.1-70b	GROQ_API_KEY	Free tier, very fast
Ollama	any local model	None	Fully local

Embedding Models (local, no API)

Model	Speed	Quality
all-MiniLM-L6-v2	Fast	Good (default)
all-mpnet-base-v2	Medium	Better

---

Built-in Templates

Template	Description
question_validator	Validate MCQ questions for quality, clarity, difficulty
sentiment_analyser	Classify sentiment, extract themes, draft responses
support_ticket_triage	Categorise, prioritise, and draft replies to support tickets

Load templates from the Task tab. Save your own with the "Save as template" button.

---

Workflow Overview

1. Data tab     — upload CSV, select ID column, filter rows
2. Analyse tab  — dedup, cluster, find outliers, cross-file similarity (optional)
3. Knowledge tab — ingest documents into a named RAG knowledge base (optional)
4. Task tab     — write prompt, define output schema, test on 3 rows
5. Run tab      — start pipeline; watch real-time progress
6. Results tab  — view stats, filter, semantic search, download

For a full step-by-step guide, see HELP.md.

---

Use Cases

Input CSV	Knowledge Base	Task
Question bank	Textbook / lecture notes	Validate answers, generate hints
Product reviews	Product specs	Sentiment + suggested response
Support tickets	Policy docs	Categorise, prioritise, draft reply
Legal clauses	Regulation PDFs	Flag risk, plain-English summary
Research abstracts	Domain taxonomy	Tag topics, novelty score
Job descriptions	Skills ontology	Enrich with skills, seniority

---

Configuration

All settings can be overridden in .env:

ANTHROPIC_API_KEY=
OPENAI_API_KEY=
GROQ_API_KEY=
OLLAMA_BASE_URL=http://localhost:11434
DEFAULT_BATCH_SIZE=100
DEFAULT_MAX_CONCURRENT=10
DEFAULT_MAX_PER_MINUTE=10

---

Project Structure

nrchd/
├── app.py              # Streamlit entry point
├── tabs/               # UI tabs (one file per tab)
├── engine/             # Core logic (no Streamlit dependency)
│   ├── batch_runner.py # Async batch executor
│   ├── llm_clients.py  # Claude / OpenAI / Groq / Ollama wrappers
│   ├── rag.py          # ChromaDB ingest + retrieval
│   ├── semantic.py     # Sentence Transformer features
│   ├── response_parser.py
│   ├── rate_limiter.py
│   └── utils.py
├── templates/          # Built-in + user task templates (JSON)
├── knowledge_bases/    # Persisted ChromaDB indexes (gitignored)
├── outputs/            # Generated CSVs (gitignored)
└── tests/              # pytest test suite

---

Contributing

• To add a new LLM provider: implement async def complete(prompt, system, max_tokens, temperature) and add it to LLMClient in engine/llm_clients.py
• To add a built-in template: create a JSON file in templates/ following the schema in docs/technical_architecture.md
• To add a new document type to RAG: extend _read_file() in engine/rag.py

---

License

MIT with Attribution — free to use, copy, and modify. Any use or derivative work must credit:

nrchd by Mortalapps.com

See LICENSE for the full terms.