Awesome RAG Debugging
Debugging, evaluating, and tracing retrieval-augmented generation systems.
RAG systems rarely fail in just one place. They drift across ingestion, chunking, retrieval, reranking, prompting, evaluation, tracing, and infrastructure.
This list curates practical open-source tools and reference resources for finding those failures faster. It focuses on debugging value, not on collecting every RAG-related repository.
This list generally prioritizes repositories with 100+ GitHub stars, active maintenance, and clear practical value for diagnosing or fixing broken RAG behavior. GitHub star badges update automatically over time.
Failure Maps & Root-Cause Guides Resources that help name the failure before you try to fix it.
WFGY
Tools for traces, spans, evaluation runs, prompt inspection, and production debugging visibility.
Phoenix Langfuse OpenLIT Weave MLflow
Evaluation & Test Generation Frameworks for measuring answer quality, retrieval quality, hallucination risk, and regression over time.
Promptfoo DeepEval Ragas TruLens Giskard
Retrieval Quality & Reranking Projects that help diagnose retrieval misses, ranking weakness, and document processing problems.
RAGatouille Haystack Unstructured
Vector Databases & Indexing Core retrieval stores and indexing layers that commonly affect recall, latency, filtering, and ranking behavior.
Qdrant Milvus Weaviate Chroma LanceDB
Guardrails & Prompt Defense Projects that help detect or reduce unsafe context handling, injection, and policy failures around RAG systems.
Guardrails NeMo Guardrails LLM Guard garak
Neighbor lists that overlap with retrieval, production ML, or broader RAG infrastructure.
Awesome RAG Awesome RAG Awesome Production Machine Learning
Please read CONTRIBUTING.md before opening a pull request.
Suggested inclusion criteria:
The resource is directly useful for debugging, evaluating, tracing, or hardening RAG systems.
The repository is public, documented, and maintained.
The project has clear practical value beyond generic AI marketing.
The repository generally has 100+ GitHub stars.
The same project should not appear in multiple sections unless there is a very strong reason.
Suggested entry format:
- Reasoning engine repository that includes a structured problem map for recurrent RAG and AI pipeline failures.
- AI observability and evaluation platform for tracing, experimentation, and troubleshooting LLM applications.
- Open-source LLM engineering platform for traces, evals, prompts, datasets, and debugging workflows.
- OpenTelemetry-native observability platform for LLM and agent systems with evaluation and guardrail hooks.
- Toolkit for developing AI applications with logging, tracing, and evaluation support.
- Experiment tracking and observability platform with support for modern LLM and agent workflows.
- Evaluation and red-teaming framework for prompts, agents, and RAG applications.
- LLM evaluation framework for testing outputs, workflows, and application behavior.
- Evaluation toolkit for LLM applications with strong support for RAG metrics and test generation.
- Instrumentation and evaluation toolkit for identifying failure modes in LLM applications.
- Evaluation and testing framework for AI systems, including quality, bias, and security checks.
- Retrieval toolkit built around late-interaction methods such as ColBERT for stronger ranking quality.
- Transparent AI orchestration framework for retrieval pipelines, search systems, and modular RAG workflows.
- Document ingestion and preprocessing toolkit for turning messy source files into cleaner RAG-ready inputs.
- Vector database focused on similarity search with filtering and production-ready APIs.
- Cloud-native vector database built for large-scale ANN search and AI retrieval workloads.
- Vector database that combines semantic search, keyword filtering, and reranking in one query layer.
- Open-source search and retrieval engine for AI applications.
- Embedded retrieval database for vector search and multimodal AI workloads.
- Framework for input and output validation in LLM applications.
- Programmable guardrails toolkit for conversational and LLM-based systems.
- Security toolkit for filtering, scanning, and hardening LLM interactions.
- Vulnerability scanner for probing LLM failure behavior such as prompt injection and hallucination.
- Retrieval-augmented generation applications and ecosystem resources.
- Technical resources focused on retrieval-augmented generation for LLM systems.
- Production tooling and ecosystem references for deploying, monitoring, and scaling ML systems.
- Short objective description.
