A diagnostic toolkit that pinpoints exactly why your RAG pipeline is producing bad answers.

Quick Start • Features • Examples • Contributing

What is this?

RAG Debugger Pro is a command‑line toolkit that isolates faults in retrieval‑augmented generation pipelines by analyzing embeddings, prompts, and retrieval logic. It is aimed at developers who need fast, actionable insights to improve answer quality.

Example usage:

$ rag-debugger analyze-embeddings tests/fixtures/test_embeddings_healthy.json
Embedding quality: GOOD
Mean cosine similarity: 0.87
Anomalies detected: 0

Problem

When RAG systems fail, developers waste time guessing whether the problem lies in embedding quality, retrieval logic, or prompt engineering. Without a systematic way to diagnose which component of the RAG pipeline is broken, debugging cycles become inefficient.

Features

Feature	Description
Embedding quality analysis	Computes similarity statistics and flags anomalous vectors that may degrade retrieval.
Prompt auditing	Scans prompt templates for common issues such as vague instructions, token overflow, or missing context placeholders.
Retrieval effectiveness check	Measures hit‑rate and relevance scores for a query set against the vector store.
CLI sub‑commands	Provides analyze-embeddings, audit-prompt, and check-retrieval commands with help and version flags.
Test fixtures	Includes healthy and anomalous embedding samples, clean and problematic prompts, and passing/failing query sets for verification.
Logging & statistics	Centralized logger and utility functions for consistent output and metric calculation across modules.
Extensible design	Separate modules for each pipeline component make it easy to add new diagnostics.
Automated testing	Pytest suite validates each analyzer against the supplied fixtures.

Quick Start

1. Clone the repository:

   git clone https://github.com/m2ai-portfolio/rag-debugger-pro.git
   cd rag-debugger-pro

2. Install in development mode:

   pip install -e .

3. Run a basic embedding analysis on the healthy fixture:

   $ rag-debugger analyze-embeddings tests/fixtures/test_embeddings_healthy.json
   Embedding quality: GOOD
   Mean cosine similarity: 0.87
   Anomalies detected: 0

Examples

Healthy embedding validation
Verify that a set of embeddings falls within expected similarity bounds.

$ rag-debugger analyze-embeddings tests/fixtures/test_embeddings_healthy.json
Embedding quality: GOOD
Mean cosine similarity: 0.87
Anomalies detected: 0

Prompt issue detection
Identify vague language and token‑limit risks in a prompt template.

$ rag-debugger audit-prompt tests/fixtures/test_prompt_issues.txt
Prompt clarity: LOW – contains ambiguous phrases
Estimated token count: 420 (limit: 512)
Suggestions: Replace "maybe" with specific criteria; add explicit context placeholder.

Retrieval failure diagnosis
Measure hit‑rate for a query set known to produce bad answers.

$ rag-debugger check-retrieval tests/fixtures/test_queries_failing.json
Hit rate: 0.32 (3/10 queries returned relevant docs)
Mean reciprocal rank: 0.21
Recommendation: Increase top‑k from 5 to 15 or revisit embedding model.

File Structure

rag-debugger-pro/
  assets/                 # Infographic used in README
  rag_debugger/           # Core source code
    __init__.py
    cli.py                # Command‑line interface entry point
    embedding_analyzer.py # Embedding quality diagnostics
    prompt_auditor.py     # Prompt template analysis
    retrieval_checker.py  # Retrieval effectiveness evaluation
    models/               # Report data structures
      __init__.py
      embedding_report.py
      prompt_report.py
      retrieval_report.py
    tests/                # Test suite
      __init__.py
      test_embedding_analyzer.py
      test_prompt_auditor.py
      test_retrieval_checker.py
      fixtures/           # Sample data for tests
        test_embeddings_anomalous.json
        test_embeddings_healthy.json
        test_prompt_clean.txt
        test_prompt_issues.txt
        test_queries_failing.json
        test_queries_passing.json
    utils/                # Shared helpers
      __init__.py
      logger.py
      stats.py
  screenshots/           # CLI output captures for reference
  rag_debugger.egg-info/ # Packaging metadata (generated)

Tech Stack

Technology	Purpose
Python 3.8+	Core language
Click	Building the CLI sub‑commands
Pytest	Running the test suite
Standard Library (json, os, statistics)	Data handling and metrics

Contributing

Fork the repo, make changes, run pytest to verify, and submit a pull request. Keep updates focused and well documented.

License

MIT

Author

Matthew Snow -- M2AI | @m2ai-portfolio