RAG Search Methods Lab

A comprehensive demonstration and comparison of different information retrieval methods for RAG (Retrieval-Augmented Generation) systems. This project implements and compares TF-IDF, BM25, and hybrid search algorithms using sample corporate documentation.

📋 Overview

This project explores various search and ranking algorithms commonly used in RAG systems. It provides implementations of:

• TF-IDF (Term Frequency-Inverse Document Frequency): A classic information retrieval method that balances term frequency with document rarity
• BM25 (Best Matching 25): An advanced probabilistic ranking function with document length normalization
• Hybrid Search: A combination of TF-IDF and BM25 with configurable weights
• Method Comparison: Side-by-side comparison of different search approaches

🚀 Features

• Multiple Search Algorithms: Implementations of TF-IDF, BM25, and hybrid search
• Real-World Documents: Sample corporate documentation (TechCorp) for testing
• Performance Comparison: Compare different search methods on the same queries
• Configurable Hybrid Search: Adjust weights between TF-IDF and BM25
• Cross-Platform Support: Works on Windows, macOS, and Linux

📁 Project Structure

rag-project/
├── techcorp-docs/              # Sample documents for search
│   ├── employee-handbook.md
│   ├── health-insurance-benefits.md
│   ├── onboarding-guide.md
│   ├── pet-policy.md
│   ├── remote-work-policy.md
│   └── vacation-policy.md
├── tfidf_search.py             # TF-IDF search implementation
├── bm25_search.py              # BM25 search implementation
├── hybrid_search.py            # Hybrid search (TF-IDF + BM25)
├── compare_methods.py          # Comparison of all methods
├── utils.py                    # Common utilities
├── requirements.txt            # Python dependencies
└── README.md                   # This file

🛠️ Installation

Prerequisites

• Python 3.8 or higher
• pip (Python package manager)

Setup

1. Clone the repository (or navigate to the project directory):

   cd rag-project

2. Create a virtual environment (recommended):

   python -m venv venv

3. Activate the virtual environment:

   • Windows:

     venv\Scripts\activate

   • macOS/Linux:

     source venv/bin/activate

4. Install dependencies:

   pip install -r requirements.txt

📖 Usage

TF-IDF Search

Run the TF-IDF search demo:

python tfidf_search.py

This will:

• Load documents from techcorp-docs/
• Perform searches for sample queries
• Display top results with similarity scores

BM25 Search

Run the BM25 search demo:

python bm25_search.py

Hybrid Search

Run the hybrid search demo with different weight combinations:

python hybrid_search.py

This demonstrates how different weightings between TF-IDF and BM25 affect search results.

Compare All Methods

Compare all search methods side-by-side:

python compare_methods.py

This shows how different algorithms rank the same documents for a given query.

🔍 Example Queries

The scripts include example queries such as:

• "remote work policy"
• "health insurance benefits"
• "pet policy dogs"

You can modify the queries in each script to test different search scenarios.

📊 Search Methods Explained

TF-IDF (Term Frequency-Inverse Document Frequency)

TF-IDF measures how important a word is to a document in a collection of documents. It:

• Increases with the number of times a word appears in a document
• Decreases with the frequency of the word in the corpus
• Helps identify documents that are most relevant to a query

BM25 (Best Matching 25)

BM25 is a ranking function used by search engines to estimate the relevance of documents. It:

• Uses probabilistic information retrieval
• Normalizes for document length
• Handles term saturation (repeated terms have diminishing returns)
• Generally performs better than TF-IDF for information retrieval

Hybrid Search

Combines TF-IDF and BM25 scores with configurable weights:

• Allows fine-tuning based on your specific use case
• Can leverage strengths of both methods
• Default weights: 30% TF-IDF, 70% BM25

🧪 Customizing Search

Adding Your Own Documents

1. Add markdown files to the techcorp-docs/ directory
2. The scripts will automatically load all .md files

Modifying Queries

Edit the queries list in any of the search scripts:

queries = ["your query here", "another query"]

Adjusting Hybrid Search Weights

In hybrid_search.py, modify the weight combinations:

weight_combinations = [
    (0.5, 0.5, "Equal weights"),
    (0.3, 0.7, "BM25 favored"),
    (0.7, 0.3, "TF-IDF favored")
]

📦 Dependencies

• scikit-learn (>=1.4.0): TF-IDF vectorization and cosine similarity
• rank-bm25 (==0.2.2): BM25 ranking implementation
• pandas (>=2.2.0): Data manipulation (if needed)
• numpy (>=1.26.0,<2): Numerical operations
• matplotlib (>=3.8.0): Visualization (if needed)

🐛 Troubleshooting

Encoding Issues on Windows

If you encounter Unicode encoding errors, the scripts include a safe_print() function that handles Windows console encoding automatically.

Document Loading Issues

If documents aren't loading:

1. Ensure markdown files are in the techcorp-docs/ directory
2. Check that files have .md extension
3. Verify files are not empty

Import Errors

If you get import errors:

1. Ensure the virtual environment is activated
2. Run pip install -r requirements.txt again
3. Check Python version: python --version (should be 3.8+)

🤝 Contributing

Feel free to:

• Add more search algorithms
• Improve document processing
• Add visualization capabilities
• Enhance the comparison functionality

📝 License

This project is provided as-is for educational and demonstration purposes.

🔗 References

• TF-IDF on Wikipedia
• BM25 on Wikipedia
• scikit-learn Documentation
• rank-bm25 Library

💡 Key Insights

• Grep: Simple exact matching, good for specific terms
• TF-IDF: Balances term frequency with document rarity
• BM25: Advanced ranking with document length normalization, generally better for information retrieval
• Hybrid: Combines strengths of multiple methods for optimal results

---

Note: This project is designed for educational purposes to understand different search and ranking algorithms in the context of RAG systems.