Production-style data ingestion and indexing pipeline for RAG systems — load, clean, chunk, embed, and index documents into Qdrant.
Table of Contents
rag-data-indexing-service is the data preparation layer of a production-style RAG system. It takes raw documents, cleans them, splits them into chunks, enriches each chunk with metadata, generates vector embeddings, and stores everything in Qdrant — ready for downstream retrieval.
This repository does not implement answer generation, LLM calls, or retrieval benchmarking. For the full project specification, see docs/Project_Goal.md.
ML engineers and backend developers who want to understand and implement the data preparation layer of a production RAG system. The pipeline is first explored step by step in Jupyter notebooks, then packaged as a reusable FastAPI service and Typer CLI.
- Loading
.txt,.md, and.pdfdocuments from disk - Cleaning text: unicode normalization, control-character removal, whitespace collapsing
- Three chunking strategies: character, recursive, and token-aware
- Generating local embeddings with
BAAI/bge-small-en-v1.5(free, no API key) - Storing and querying vectors in Qdrant with cosine similarity
- Packaging the pipeline as a FastAPI HTTP API and a Typer CLI
- Comfortable with Python
- Basic familiarity with Docker
- No prior RAG experience required
- Docker Desktop — Windows, macOS, or Linux
make— pre-installed on Linux/macOS; on Windows use WSL or Git Bash
# 1. Clone the repository
git clone <repository-url>
cd rag-data-indexing-service
# 2. Copy the environment file
cp .env.example .env
# 3. Build and start the containers (JupyterLab + Qdrant)
make up
# 4. Download the SciFact dataset inside Docker
make download-data
# 5. Open JupyterLab at http://localhost:8888 and run notebooks 01 → 07
# 6. Stop everything
make down| Service | URL |
|---|---|
| JupyterLab | http://localhost:8888 |
| Qdrant | http://localhost:6333 |
Hugging Face model cache and Qdrant data are stored in named Docker volumes and persist across restarts. To also remove volumes:
docker compose down -v.
Raw Documents
→ Document Loading app/pipeline/loaders/
→ Text Cleaning app/pipeline/cleaning.py
→ Chunking app/pipeline/chunking/
→ Metadata Enrichment app/pipeline/metadata.py
→ Embedding app/pipeline/embedding.py
→ Qdrant Indexing app/pipeline/indexing/
Each stage is explored in a numbered Jupyter notebook first, then implemented as a reusable Python module in app/pipeline/.
Folder PATH listing
+---app <-- Production service layer (FastAPI + CLI)
+---config <-- Reserved for YAML config files
+---data <-- Raw and processed data (git-ignored)
+---docker <-- Reserved for extra Docker config
+---docs <-- Project goal and implementation plan
+---notebooks <-- Step-by-step pipeline notebooks (01-07)
+---scripts <-- Reserved for utility scripts
+---tests <-- Unit and integration tests
│ .dockerignore <-- Docker build exclusions
│ .env.example <-- Environment variable template
│ .gitignore <-- Git exclusions
│ docker-compose.yml <-- Service definitions (Jupyter, Qdrant, API)
│ Dockerfile <-- Python + JupyterLab image definition
│ LICENSE <-- MIT License
│ Makefile <-- Developer shortcuts (up, test, lint...)
│ README.md <-- This file
│ requirements.txt <-- Python dependencies
│
Each notebook covers exactly one pipeline stage and reads the output of the previous one. Run them in order inside JupyterLab.
| Notebook | Phase | Description |
|---|---|---|
01_download_scifact_corpus.ipynb |
1 | Download and normalize the SciFact corpus from Hugging Face |
02_clean_corpus.ipynb |
2 | Clean text: unicode, control characters, whitespace |
03_chunk_corpus.ipynb |
3 | Split documents into overlapping chunks |
04_enrich_metadata.ipynb |
4 | Attach stable chunk_id and retrieval metadata |
05_generate_embeddings.ipynb |
5 | Embed chunks locally with BAAI/bge-small-en-v1.5 |
06_index_qdrant.ipynb |
6 | Create the Qdrant collection and upsert vectors |
07_index_health_report.ipynb |
7 | End-to-end query demo and index health report |
See notebooks/README.md for the input and output file of each notebook.
The same pipeline logic from the notebooks is packaged as reusable modules in app/, exposed through a FastAPI HTTP API and a Typer CLI. See app/README.md.
Start with docker compose --profile api up:
| Method | Path | Purpose |
|---|---|---|
GET |
/health |
Liveness check |
POST |
/ingest |
Run the full pipeline on a document folder |
GET |
/collections/{name}/status |
Collection statistics |
DELETE |
/collections/{name} |
Delete a collection for a clean rebuild |
Interactive docs: http://localhost:8000/docs
python -m app.cli ingest --input-dir ./data/raw --collection rag_scifact
python -m app.cli status --collection rag_scifact
python -m app.cli reset --collection rag_scifactmake testUnit tests run without a network connection or a running Qdrant server. Integration tests require the Qdrant container to be running. See tests/README.md for details.
Copy .env.example to .env and adjust values as needed:
| Variable | Default | Description |
|---|---|---|
QDRANT_URL |
http://qdrant:6333 |
Qdrant server URL |
QDRANT_COLLECTION |
rag_scifact |
Default collection name |
EMBEDDING_MODEL |
BAAI/bge-small-en-v1.5 |
Local Hugging Face embedding model |
CHUNK_SIZE |
800 |
Maximum chunk size in characters |
CHUNK_OVERLAP |
100 |
Overlap between consecutive chunks |
CHUNKING_STRATEGY |
recursive |
character, recursive, or token |
INPUT_DIR |
./data/raw |
Directory to load documents from |
For questions or collaboration, connect with Max Ghadri on LinkedIn.