RAGOps: Enterprise-Style RAG Assistant on BEIR SciFact

RAGOps is an end-to-end Retrieval-Augmented Generation system over the BEIR SciFact corpus.
It is built to be production-like: hybrid retrieval, reranking, citation-grounded generation, safe fallback behavior, offline evaluation, and per-query observability.

Highlights

• Ingests SciFact corpus, queries, and qrels directly from Hugging Face
• Hybrid retrieval: BM25 + dense embeddings
• Reranking with cross-encoder (with fallback reranker if needed)
• Multi-category routing: rag, sql, graph, auto
• Citation-grounded answering with confidence-aware no-answer behavior
• LLM provider abstraction: huggingface, openai, ollama, bytez
• Evaluation harness for:
  • Retrieval quality: Recall@k, MRR@k, nDCG@k
  • Generation quality: Groundedness, FactualAccuracy (citation-precision proxy), answer/no-answer rates
  • System efficiency: retrieval/rerank/generation/total latency
• Trace logging for every evaluated or served query (retrieval hits, reranker scores, latency, answer metadata)
• FastAPI service with /ask, /health, and /metrics

System Architecture

1. Data ingestion
   • Download SciFact corpus/queries and qrels
   • Normalize to local artifacts
2. Indexing
   • Build BM25 index
   • Build dense embedding index
3. Retrieval
   • Retrieve BM25 and dense candidates
   • Fuse with weighted reciprocal rank fusion (RRF)
4. Reranking
   • Cross-encoder rerank of fused candidates
5. Generation
   • LLM grounded synthesis with citation checks and overlap guardrails
   • Safe fallback to no-answer when evidence is insufficient
6. Evaluation + tracing
   • Batch query execution over qrels
   • Metrics computation and JSON report output
   • Per-query trace file for debugging/regression analysis

Project Structure

ragops/
  cli.py                 # Typer CLI
  pipeline.py            # Retrieval + rerank + generation orchestration
  generation.py          # Extractive + LLM grounded generators
  llm_providers.py       # Provider adapters (HF, OpenAI, Ollama, Bytez)
  model_eval.py          # Model-level eval (retrieval/generation/latency)
  server.py              # FastAPI app
  sql_engine.py          # SQL category
  graph_engine.py        # Graph category
experiments/
  scifact_experiments.yaml
tests/

Prerequisites

• Python >= 3.11
• Internet connection for first-time dataset/model downloads
• Optional provider credentials for hosted LLMs

Installation

python -m pip install -e .

Quickstart

1. Prepare data:

python -m ragops.cli prepare-data

2. Build indexes:

python -m ragops.cli build-index

3. Build SQL + graph artifacts:

python -m ragops.cli build-sql-db
python -m ragops.cli build-graph

4. Ask a question:

python -m ragops.cli ask "Does aspirin reduce heart attack risk?"

Category Routing

• Auto route:

python -m ragops.cli ask "How many documents are in the corpus?"

• Force category:

python -m ragops.cli ask "How many documents are in the corpus?" --category sql
python -m ragops.cli ask "Show most connected docs in the graph" --category graph
python -m ragops.cli ask "Does aspirin reduce heart attack risk?" --category rag

LLM Providers

Examples:

python -m ragops.cli ask "Does aspirin reduce heart attack risk?" --enable-llm --llm-provider huggingface
python -m ragops.cli ask "Does aspirin reduce heart attack risk?" --enable-llm --llm-provider openai --llm-model gpt-4.1-mini
python -m ragops.cli ask "Does aspirin reduce heart attack risk?" --enable-llm --llm-provider ollama --llm-model llama3.1
python -m ragops.cli ask "Does aspirin reduce heart attack risk?" --enable-llm --llm-provider bytez --llm-model openai/gpt-4.1-mini

Environment variables:

• OPENAI_API_KEY for OpenAI provider
• BYTEZ_API_KEY for Bytez provider

Notes:

• If your hosted model quota is exhausted, generation can fail or fallback behavior may be triggered.
• The CLI prints runtime generator metadata in evaluate-model so fallback usage is visible.

Evaluation

Retrieval-only evaluation

python -m ragops.cli evaluate --mode hybrid_rerank

Supported retrieval modes:

• bm25
• dense
• hybrid
• hybrid_rerank

Model evaluation (retrieval + generation + latency)

python -m ragops.cli evaluate-model --sample-size 50 --llm-provider bytez --llm-model openai/gpt-4.1-mini

Current tuned defaults in this repository:

• fused_top_k=30
• no_answer_threshold=0.35
• llm_min_grounding_overlap=0.15

You can override directly:

python -m ragops.cli evaluate-model \
  --sample-size 50 \
  --fused-top-k 30 \
  --answer-top-k 5 \
  --no-answer-threshold 0.35 \
  --llm-min-grounding-overlap 0.15

Output:

• outputs/eval/*.json: aggregated evaluation reports
• outputs/traces/*.jsonl: per-query traces

Experiment Sweeps

Run multi-config experiments from YAML:

python -m ragops.cli run-experiments --config-file experiments/scifact_experiments.yaml

Useful fields in experiment configs:

• retrieval: retrieval hyperparameters
• model: model/reranker settings
• llm: provider/model settings
• answer_eval: enable/disable generation scoring
• answer_eval_sample_size: subsample for faster iteration

API Service

Start server:

python -m ragops.cli serve --host 0.0.0.0 --port 8000

Endpoints:

• GET /health
• GET /metrics
• GET /metrics/json
• POST /ask

Example:

curl -X POST "http://localhost:8000/ask" \
  -H "Content-Type: application/json" \
  -d "{\"query\":\"Does aspirin reduce heart attack risk?\",\"category\":\"rag\"}"

Testing

python -m pytest -q

Security Notes

• Do not hardcode API keys in source files.
• Use environment variables for credentials.
• Large artifacts and outputs are excluded via .gitignore.

Typical Workflow

python -m ragops.cli prepare-data
python -m ragops.cli build-index
python -m ragops.cli build-sql-db
python -m ragops.cli build-graph
python -m ragops.cli evaluate --mode hybrid_rerank
python -m ragops.cli evaluate-model --sample-size 50 --llm-provider bytez --llm-model openai/gpt-4.1-mini
python -m ragops.cli ask "Can vitamin D reduce respiratory infections?"
python -m ragops.cli serve