🧠 Lumina Data Pipeline

Lumina Data Pipeline is a modular ingestion & indexing pipeline for Lumina Bank's AI Assistant.
It takes raw documents (PDF, Markdown, FAQs…), validates them with strict typing, chunks them intelligently, and pushes embeddings into Qdrant for RAG-style search.

---

✨ What This Pipeline Does

• ✅ Strict typing with Pydantic – filenames and metadata are validated up front (no more "mystery dicts")
• 🧬 Pluggable chunking strategies – different strategies per doc type (contracts, guides, FAQs…)
• 🧮 Embedding + Qdrant integration – sends vectors + rich payloads into a single collection (lumina_knowledge_base)
• 🔍 Filterable search – Qdrant payload indexes for category, doc_type, language, etc.
• 📊 Observability hooks (planned) – processing stats per file (chunks, tokens, status)

This repo is the "data side" of the hackathon project: it prepares Lumina's knowledge so the AI assistant can answer questions reliably.

---

🧱 Tech Stack

Component	Technology
Language	Python 3.11+
Dependency / env	uv
Typing / validation	pydantic, pydantic-settings
Vector DB	qdrant-client
Embeddings / LLMs	langchain-openai, langchain-google-genai (configurable providers)
Tooling	mypy, flake8, black

---

🚀 Getting Started

We use uv for fast, reproducible environments.
You normally don't need to call uv directly – the Makefile wraps the common commands.

1. Prerequisites

• Python ≥ 3.11 installed
• uv installed

macOS / Linux:

curl -LsSf https://astral.sh/uv/install.sh | sh

Windows (PowerShell):

powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

Confirm installation:

uv --version

2. Clone the Repository

git clone <repo-url>
cd lumina-pipeline

3. Environment Configuration

Copy the example env file and fill in your keys:

cp .env.example .env

Edit .env and set at least:

# Core
APP_ENV="dev"

# Vector DB
QDRANT_URL="https://<your-cluster>.qdrant.cloud"
QDRANT_API_KEY="<your-qdrant-api-key>"
COLLECTION_NAME="lumina_knowledge_base"

# Providers
EMBEDDING_PROVIDER="openai"     # or "gemini"
EMBEDDING_MODEL_ID="text-embedding-3-small"

LLM_PROVIDER="gemini"           # or "openai"
LLM_MODEL_ID="gemini-flash-lite-latest"

# API keys
OPENAI_API_KEY="sk-..."
GOOGLE_API_KEY="..."

💡 Note: The pipeline will fail fast if required keys are missing for the chosen provider.

4. Install Dependencies

We rely on uv + pyproject.toml for dependency management.
The Makefile wraps this behind a friendly command:

make setup

What make setup typically does:

1. Creates an isolated env for the project
2. Installs all dependencies
3. Ensures .env exists

5. Run the Pipeline

To run the main ingestion entry point:

make run

By default, this will:

1. Scan the input directory (e.g. data_source/)
2. Parse filenames into DocumentMeta (category, doc_type, language…)
3. Load + chunk document content
4. Embed and upsert into Qdrant

Check the logs in your console for a per-file "receipt".

---

🧪 Developer Workflow

Common Commands

All wrapped in the Makefile:

# Static type checking
uv run mypy src/

# Lint (flake8 + black check)
uv run flake8 src/
uv run black --check src/

# Run tests (if/when added)
uv run pytest

Or via make if you've wired them:

make lint     # flake8 + black --check
make test     # pytest (optional)

---

📂 Project Structure

src/
├── domain/
│   ├── models.py          # Pydantic models (DocumentMeta, ProcessingStats, enums)
│   └── interfaces.py      # Abstract ChunkingStrategy interface
├── strategies/
│   ├── legal.py           # Chunking strategies for contracts / policies
│   ├── markdown.py        # Chunking for guides, Brochures, docs
│   └── faq.py             # FAQ-specific behavior Question-Answer
├── infrastructure/
│   ├── settings.py        # AppSettings (env-driven config)
│   ├── qdrant_client.py   # Qdrant connection + helpers
│   └── factories.py       # Embeddings / LLM factory functions
├── utils/
│   ├── logging.py         # Rich-powered logger
│   └── parsing.py         # Filename → DocumentMeta parsing logic
└── main.py                # Entry point for the ingestion pipeline

---

🔄 Typical Contributor Flow

For your teammates, the day-to-day looks like this:

1. Clone and Set Up

git clone <repo-url>
cd lumina-pipeline
make setup

2. Create a Feature Branch

git checkout -b feat/add-legal-strategy

3. Make Changes, Then Run Checks

make lint
uv run mypy src/

4. Commit and Push

git commit -am "feat: add legal chunking strategy"
git push origin feat/add-legal-strategy

5. Open a Pull Request

• Open a Pull Request on GitHub
• Wait for CI (lint + mypy) to go green
• Request review from another team member

---

Happy coding! 🚀