Boogle - RAG System

A powerful Retrieval-Augmented Generation (RAG) system for document-based question answering.

Features

• Multiple Document Formats: Support for PDF, DOCX, TXT, MD, and HTML files
• Intelligent Chunking: Automatic text splitting with configurable chunk sizes and overlap
• Flexible Embeddings: Use either sentence-transformers (local) or OpenAI embeddings
• Vector Storage: ChromaDB for efficient similarity search
• AI-Powered Answers: Generate contextual answers using OpenAI GPT models
• CLI Interface: Easy-to-use command-line interface
• Interactive Mode: Query your documents interactively

Installation

1. Clone the repository:

git clone <repository-url>
cd boogle

2. Install dependencies:

pip install -r requirements.txt

3. Set up environment variables (optional, for OpenAI):

cp .env.example .env
# Edit .env and add your OPENAI_API_KEY

Quick Start

1. Ingest Documents

Ingest documents from a directory:

python main.py ingest --directory ./documents

Ingest specific files:

python main.py ingest --files doc1.pdf doc2.txt doc3.md

2. Query Your Documents

Ask a question:

python main.py query "What is the main topic of the documents?"

Interactive mode:

python main.py query --interactive

3. View Statistics

python main.py stats

4. Reset the System

python main.py reset

Usage Examples

Ingesting Documents

# Ingest all documents from a directory (recursive)
python main.py ingest --directory ./my_docs

# Ingest specific files
python main.py ingest --files report.pdf notes.txt

# Custom chunk settings
python main.py ingest --directory ./docs --chunk-size 500 --chunk-overlap 100

# Use OpenAI embeddings (requires API key)
python main.py ingest --directory ./docs --use-openai-embeddings

Querying

# Simple query
python main.py query "What are the key findings?"

# Retrieve more context documents
python main.py query "Summarize the methodology" --top-k 10

# Use a different model
python main.py query "What is the conclusion?" --model gpt-4

# Hide sources
python main.py query "What is the main argument?" --no-sources

# Interactive mode
python main.py query --interactive

Advanced Options

# Use a custom collection name
python main.py --collection my_collection ingest --directory ./docs

# Custom persist directory
python main.py --persist-dir ./my_vector_db ingest --directory ./docs

Architecture

The RAG system consists of several components:

1. DocumentLoader: Loads documents from various file formats
2. TextChunker: Splits documents into manageable chunks
3. EmbeddingGenerator: Generates vector embeddings for text
4. VectorStore: Stores and retrieves embeddings using ChromaDB
5. Retriever: Finds relevant documents for a query
6. Generator: Generates answers using LLM and context
7. RAGPipeline: Orchestrates all components

Configuration

Chunk Settings

• chunk_size: Maximum size of each text chunk (default: 1000 characters)
• chunk_overlap: Overlap between chunks (default: 200 characters)

Embedding Models

• Local (default): sentence-transformers/all-MiniLM-L6-v2
• OpenAI: text-embedding-3-small (requires API key)

Generation Models

• Default: gpt-3.5-turbo
• Alternative: gpt-4, gpt-4-turbo, etc.

API Usage

You can also use the RAG system programmatically:

from src.rag import RAGPipeline

# Initialize pipeline
pipeline = RAGPipeline(
    collection_name="my_docs",
    chunk_size=1000,
    top_k=5
)

# Ingest documents
pipeline.ingest_documents(directory_path="./documents")

# Query
result = pipeline.query("What is this about?")
print(result['answer'])
print(result['sources'])

# Get stats
stats = pipeline.get_stats()
print(f"Total documents: {stats['total_documents']}")

Project Structure

boogle/
├── src/
│   └── rag/
│       ├── __init__.py
│       ├── document_loader.py    # Document loading
│       ├── text_chunker.py       # Text chunking
│       ├── embeddings.py         # Embedding generation
│       ├── vector_store.py       # Vector database
│       ├── retriever.py          # Document retrieval
│       ├── generator.py          # Answer generation
│       └── rag_pipeline.py       # Main pipeline
├── main.py                       # CLI interface
├── requirements.txt              # Dependencies
├── .env.example                  # Environment variables template
└── README.md                     # This file

Requirements

• Python 3.8+
• See requirements.txt for package dependencies

Environment Variables

• OPENAI_API_KEY: Required for OpenAI embeddings and generation

Troubleshooting

No documents found

• Make sure the document directory exists and contains supported file types
• Check file permissions

OpenAI API errors

• Verify your OPENAI_API_KEY is set correctly in .env
• Check your OpenAI account has sufficient credits

Memory issues

• Reduce chunk_size for large documents
• Process fewer documents at once

License

MIT License

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.