Skip to content

Project_Goal.md

Latest commit

 

History

History
402 lines (282 loc) · 7.18 KB

Project_Goal.md

File metadata and controls

402 lines (282 loc) · 7.18 KB
Raw

rag-data-indexing-service

Project Goal

rag-data-indexing-service is a production-style data ingestion and indexing service for Retrieval-Augmented Generation (RAG) systems.

The goal of this project is to show that RAG quality starts before retrieval and generation. A reliable RAG system needs clean documents, well-designed chunks, useful metadata, consistent embeddings, and a searchable vector index.

This project focuses only on the data preparation and indexing layer of a RAG pipeline.

What This Project Does

The service takes raw documents, preprocesses them, splits them into chunks, enriches each chunk with metadata, generates embeddings, and stores the indexed chunks in Qdrant.

The output of this project is a ready-to-query vector index that can be used by downstream retrieval and answering services.

Core Pipeline

Raw Documents
→ Document Loading
→ Text Cleaning
→ Chunking
→ Metadata Enrichment
→ Embedding
→ Qdrant Indexing
→ Index Health Report

Main Features

1. Document Loading

Supported input formats:

.txt
.md
.pdf

The initial implementation should focus on .txt and .md files for simplicity. PDF support can be added through pypdf or unstructured.

2. Text Cleaning

The cleaning layer normalizes text before chunking.

It handles:

extra whitespace
broken line breaks
repeated empty lines
basic formatting artifacts
invalid or empty documents

The goal is not heavy NLP preprocessing. The goal is to make the text clean enough for reliable chunking and embedding.

3. Chunking Strategies

The service supports multiple chunking strategies so their impact can be compared later.

Supported strategies:

character chunking
recursive character chunking
token-aware chunking

Configurable parameters:

chunk_size
chunk_overlap
chunking_strategy

Example configuration:

chunking:
  strategy: recursive
  chunk_size: 800
  chunk_overlap: 100

4. Metadata Enrichment

Each chunk should include metadata that makes retrieval easier and more trustworthy.

Recommended metadata:

{
  "document_id": "string",
  "source_file": "string",
  "chunk_id": "string",
  "chunk_index": 0,
  "chapter": "string",
  "section": "string",
  "document_type": "book_chapter",
  "created_at": "datetime"
}

Metadata is important because later retrieval services can filter, rank, debug, and cite chunks more effectively.

5. Embedding Generation

The project should use free/local embedding models.

Recommended default model:

BAAI/bge-small-en-v1.5

Alternative models:

sentence-transformers/all-MiniLM-L6-v2
intfloat/e5-small-v2
intfloat/multilingual-e5-small

For this project, BAAI/bge-small-en-v1.5 is the best default because it is lightweight, free, strong enough for English technical documents, and easy to run locally.

6. Vector Indexing

The project uses Qdrant as the vector database.

Qdrant should run locally through Docker Compose.

The service should create or refresh a Qdrant collection and insert embedded chunks with metadata payloads.

The collection should support:

vector search
metadata filtering
collection reset
index refresh
basic health checks

Technology Stack

Core Language

Python 3.11+

API

FastAPI
Uvicorn

Used for exposing ingestion and health-check endpoints.

CLI

Typer

Used for local batch indexing from the terminal.

Example:

python -m app.cli ingest --input-dir ./data/raw --collection rag_book

Vector Database

Qdrant
qdrant-client

Qdrant runs locally with Docker Compose.

Embeddings

sentence-transformers
BAAI/bge-small-en-v1.5

No paid embedding API is required.

Document Parsing

pypdf
unstructured optional

Use pypdf for basic PDF support. Keep unstructured optional because it can add extra dependencies.

Chunking

LangChain text splitters
transformers tokenizer

Use LangChain splitters for fast implementation and tokenizer-based splitting where needed.

Configuration

pydantic-settings
.env
YAML config

Configuration should control:

Qdrant URL
collection name
embedding model
chunk size
chunk overlap
chunking strategy
input directory

Logging

structlog

Logs should include:

number of loaded documents
number of generated chunks
embedding time
indexing time
failed files
Qdrant collection status

Testing

pytest

Minimum tests:

text cleaning test
chunking test
metadata creation test
embedding interface test
Qdrant indexing smoke test

Code Quality

ruff
black
mypy optional

Deployment

Docker
Docker Compose

Services:

api
qdrant

API Endpoints

Health Check

GET /health

Returns service status.

Ingest Documents

POST /ingest

Indexes documents from a configured input directory.

Request example:

{
  "input_dir": "./data/raw",
  "collection_name": "rag_book",
  "chunking_strategy": "recursive",
  "chunk_size": 800,
  "chunk_overlap": 100
}

Collection Status

GET /collections/{collection_name}/status

Returns basic index statistics.

Example response:

{
  "collection_name": "rag_book",
  "vectors_count": 245,
  "indexed_documents": 6,
  "embedding_model": "BAAI/bge-small-en-v1.5"
}

Reset Collection

DELETE /collections/{collection_name}

Deletes a collection for clean re-indexing.

CLI Commands

make up
make ingest
make test
make lint
make reset-index

Expected commands:

python -m app.cli ingest --input-dir ./data/raw --collection rag_book
python -m app.cli status --collection rag_book
python -m app.cli reset --collection rag_book

What This Project Intentionally Does Not Do

This project does not implement answer generation.

It does not call an LLM.

It does not implement hybrid retrieval, reranking, query rewriting, or observability dashboards.

Those belong to the next repositories:

rag-retrieval-benchmark
production-rag-answering-api

Keeping this repository focused makes the system easier to understand, test, and reuse.

Success Criteria

The project is complete when:

documents can be loaded from a folder
text is cleaned and split into chunks
metadata is attached to every chunk
embeddings are generated locally for free
chunks are stored in Qdrant
the index can be reset and rebuilt
the service runs with Docker Compose
basic tests pass
README explains the pipeline clearly

Branding Message

This repository demonstrates the data engineering foundation of a production-style RAG system.

It shows how raw documents become clean, metadata-rich, embedded, and searchable chunks that downstream RAG services can use reliably.