Atlas — Agentic RAG Platform

Ingest a knowledge base, then ask an agent that retrieves with hybrid search + reranking + a knowledge graph, reasons with tool-calling over MCP, routes across Claude / OpenAI / local models, is wrapped in guardrails (prompt-injection defense, PII redaction), and is measured by a first-class evaluation harness — with Langfuse observability, Redis semantic caching, and full Docker / Kubernetes / CI-CD.

See the whole system in 20 seconds — no keys, no services:
pip install pydantic pydantic-settings fastapi "uvicorn[standard]" httpx rich && python -m atlas.cli demo

Atlas is a single, coherent agentic-RAG system rather than a folder of demos. Its subsystems are decoupled through Python Protocol interfaces, so every hosted dependency has a lightweight local fallback: the default configuration runs fully offline with no API keys and no external services (in-memory vector store, a deterministic embedder, and a mock model). Hosted model providers, Redis and Langfuse activate the moment you configure a key or URL; the pgvector and Qdrant stores are selected explicitly with ATLAS_RETRIEVAL__STORE (see Configuration).

---

Architecture

flowchart LR
    subgraph Ingest["atlas.ingestion"]
      SRC[Docs / URLs / PDFs] --> ETL[ETL + clean]
      ETL --> CHUNK[Chunking]
      CHUNK --> EMB[Embeddings]
      EMB --> VS[(Vector store)]
      CHUNK --> KG[Entity/Relation extract] --> GRAPH[(Knowledge Graph)]
      GOWORKER[Go ingestion worker] -. high-throughput fetch .-> ETL
    end

    subgraph Retrieve["atlas.retrieval"]
      Q[Query] --> HS[Hybrid search: vector + BM25]
      VS --> HS
      GRAPH --> GR[Graph retrieval]
      HS --> RRF[Reciprocal-rank fusion]
      GR --> RRF
      RRF --> RERANK[Cross-encoder rerank]
    end

    subgraph Agent["atlas.agent (LangGraph)"]
      RERANK --> PLAN[Plan / route]
      PLAN --> ROUTER{Model router}
      ROUTER -->|synthesis| CLAUDE[Claude]
      ROUTER -->|cheap triage| GPT[OpenAI]
      ROUTER -->|local/private| OLLAMA[Ollama / vLLM]
      PLAN --> MCP[[MCP tool server]]
    end

    subgraph Safety["atlas.guardrails"]
      GIN[Injection + PII in] --> Agent
      Agent --> GOUT[PII + content filter out]
    end

    U[User] --> API[FastAPI + Next.js UI]
    API --> GIN
    GOUT --> API
    Agent --> OBS[(Langfuse traces)]
    Agent --> CACHE[(Redis semantic cache)]
    Agent --> EVAL[atlas.evals: LLM-judge + A/B]

Loading

New to RAG? docs/PRIMER.md explains this project and every concept it is built from — embeddings, hybrid search, RRF, grounding, guardrails, eval metrics — assuming no prior knowledge. See docs/ARCHITECTURE.md for the request lifecycle and design deep-dive.

---

Features

Retrieval

• Hybrid search (dense vector + BM25 keyword) fused with reciprocal-rank fusion
• A knowledge-graph layer: entities/relations extracted at ingest feed a multi-hop graph retriever whose hits join the RRF fusion as a third, graph_weight-ed ranking — able to surface a document neither dense nor lexical search found, which the fusion arithmetic previously made impossible. It earns no recall on the bundled single-topic corpus; see Does it actually work?
• Cross-encoder reranking of the fused candidates, active when ATLAS_RETRIEVAL__RERANKER=cross-encoder. It defaults to none — chosen explicitly rather than by "is the library importable?", since sentence-transformers is a core dependency and auto-selecting would make every full install download a model on first query. On the default the stage is a pass-through and reports itself as one rather than claiming a rerank that did not happen
• Pluggable vector stores behind one VectorStore interface — in-memory (offline default), Postgres/pgvector, and Qdrant
• Configurable, token-aware chunking with markdown/heading awareness
• One Unicode-aware tokenizer shared by every stage that compares text, so Cyrillic, Greek, accented Latin and CJK corpora are retrievable rather than silently ingesting to an all-zero vector (CJK is segmented into characters plus adjacent-pair bigrams; no morphological stemming, so inflected languages still match on surface forms)

Agent

• A LangGraph state machine: guard_input → retrieve → plan → act → synthesize → guard_output
• Function/tool calling with Pydantic-derived JSON schemas; a safe AST calculator
• Model Context Protocol (MCP) — an MCP server exposing the KB as a tool, and an MCP client for consuming external ones (atlas/mcp). Both are standalone and exercised by tests; neither is wired into the default agent loop, which uses in-process tools
• Cost-aware multi-model routing across Claude, OpenAI, Ollama, vLLM, and Bedrock. The planner turn asks for the cheap tier and synthesis for the default one, and ChatRequest.routing_hint ("cheap" / "local" / "strong", or an explicit provider:model) overrides per request, exposed as a tier selector in the web UI. The semantic cache is partitioned by tier, so a request that asks for the local/private model is never served the cloud tier's cached answer
• Token-budgeted context packing with near-duplicate dedupe, and grounded answers that cite every claim back to a source chunk — and return only the sources the answer actually cites, renumbered to stay dense, rather than the whole retrieved set
• Faithful by design — answers are grounded strictly in the retrieved context; when nothing relevant is found, a relevance gate makes the agent refuse honestly ("I don't have anything relevant in the current sources") and suppress citations, rather than hallucinating an answer from off-topic chunks

Safety & governance

• Guardrail pipeline on both input and output: prompt-injection detection, PII redaction (Presidio with a regex fallback), and content filtering
• Guards fail closed — a detector that crashes blocks, rather than silently degrading to "allow"
• A red-team / jailbreak-resistance suite (data/redteam) that reports two rates rather than one flattering number: 66.7% blocked at the guardrail and 100% resisted end-to-end. The gap is the honest measure of how much work grounding does as a second layer — see Responsible AI. Read the second number narrowly: it is measured against the offline extractive provider, which can only restate retrieved passages and so cannot follow an injected instruction even if the guardrail misses it. It bounds this configuration, not a hosted model
• atlas redteam exits non-zero when resistance drops below --min-resistance (default 1.0), so the CI step is a gate rather than a report

Evaluation & LLMOps

• Eval harness with RAG metrics (faithfulness, answer relevancy, context recall) via an LLM-as-judge that degrades to deterministic heuristics offline
• Reproducible by construction — ids are content-addressed (BLAKE2b over the content, not uuid4) and every ranking breaks ties on stable content coordinates rather than dict/set iteration order, so the same corpus and suite score identically on every run and every machine
• A quality baseline as a CI gate — atlas eval --baseline diffs the current run against a committed data/evals/baseline.json and exits non-zero if any metric drops beyond a tolerance. Reproducibility is what makes this meaningful: a diff in the numbers is a real behaviour change, not sampling noise
• A/B testing across agent variants
• Every chat turn — REST, SSE, GraphQL and the CLI — runs through one seam (deps.run_chat, or deps.stream_chat for SSE, which shares its cache and cost steps): semantic-cache lookup (Redis, in-memory offline) → agent → cost accounting — so GET /metrics reports live hit rates and spend, and repeated questions come back in milliseconds without touching a model. The cache keys on the question's meaning-bearing tokens in order and its polarity, not on embedding cosine: over a bag-of-words embedder "does cover" and "does not cover" sit at cosine 0.96, so any threshold loose enough to catch paraphrases also served the opposite answer. Set ATLAS_REDIS__SEMANTIC_CACHE_REQUIRE_TOKEN_MATCH=false to use vector similarity once a genuinely semantic embedder is configured
• Langfuse tracing on the same seam. The Claude provider marks the system prompt as a cacheable prefix, but that prefix is ~355 tokens — below every model's minimum (512 on Opus 5, 1024 on Sonnet 5), so caching does not engage on the bundled prompt. The API ignores a short prefix silently, so the provider logs a warning with the measured size rather than letting a counter sit at zero unexplained; it starts paying off once your system prompt or tool set is large enough to clear the bar

Interfaces & infra

• FastAPI backend (REST + genuine token-level SSE streaming off the provider's own stream, plus GraphQL via the optional graphql extra — pip install -e ".[graphql]") and a Next.js/TypeScript chat UI
• A concurrent Go ingestion worker for high-throughput URL fetching
• Docker Compose for local dev; Kubernetes manifests and AWS/GCP/Azure deploy notes for production; GitHub Actions CI
• An optional LoRA/PEFT fine-tuning module (Hugging Face Transformers + PEFT)

---

Does it actually work? — the retrieval ablation

Most RAG projects assert that hybrid search and a knowledge graph help. This one measures it, and publishes the result even where it is unflattering.

python -m atlas.cli eval --ablation

Each rung adds exactly one signal to the one above, so the gap between two rows is that signal's contribution. No judge model is called: the metrics score retrieval directly against each golden case's gold contexts, which makes them deterministic and impossible to flatter with an eloquent answer. On the default configuration nothing is downloaded either.

retrieval configuration	recall@5	MRR	nDCG@5	Δ recall
dense vector only	0.800	0.758	0.525	—
BM25 keyword only	0.900	0.933	0.613	+0.100
hybrid — RRF(vector, BM25)	0.900	0.803	0.553	+0.000
+ knowledge graph	0.900	0.745	0.526	+0.000
+ reranker (off — reranker=none)	0.900	0.745	0.526	+0.000

10 cases over the 27-chunk bundled corpus, offline defaults. Byte-identical on every run — see Reproducibility.

Two of these columns used to read better than they were, and the table was republished when they were fixed.

Δ recall was differenced against the first row rather than the row above, so BM25's +0.100 was silently re-credited to the knowledge graph and to the disabled reranker. Read as documented — "the gap between two rows is that signal's contribution" — the old table claimed both bought +0.100 recall. They buy +0.000, which is what the prose under it already said.

nDCG@5 built its ideal ranking from the chunks the run actually retrieved, so a missing gold passage dropped out of the numerator and the denominator alike and the metric could not see recall at all: one of two required passages, ranked first, scored a perfect 1.000. It was a rank-compactness score wearing an nDCG label — note how closely it used to track MRR (0.933 vs 0.934 on the BM25 rung). Gains are now counted in gold-passage units on both sides, which is why every value in the column dropped by roughly 0.2–0.3.

The last rung is a pass-through here because ATLAS_RETRIEVAL__RERANKER defaults to none. That is deliberate: the ablation reports the configuration you are actually running rather than quietly constructing a cross-encoder the rest of the stack would not use. Turn it on and the rung does real work:

ATLAS_RETRIEVAL__RERANKER=cross-encoder python -m atlas.cli eval --ablation

retrieval configuration	recall@5	MRR	nDCG@5	Δ recall
+ reranker (ms-marco-MiniLM-L-6-v2)	0.950	1.000	0.682	+0.050

Needs sentence-transformers and a one-time model download, so it is not the default and CI does not depend on it.

The Δ here is +0.050, not the +0.150 this row used to claim: that was the old first-row baseline again, crediting the reranker with BM25's gain on top of its own. +0.050 recall and a perfect MRR is the reranker's actual contribution, and it is the only rung besides BM25 that earns one.

BM25 alone has the best MRR of any keyless rung, and that is the honest headline. The ablation's first run was worse: hybrid lost recall against plain BM25 (0.800 vs 0.900) and every added signal cost MRR. Reading why turned up two real defects, both since fixed — which is the point of building the instrument.

• RRF was partitioning, not ranking. The damping constant k=60 is calibrated for TREC-scale runs of thousands. Over a 20-candidate list it flattens rank so far that positions 1–20 span only 1.31×, narrower than the 1.67× ratio between two ranking weights. Once weights outweigh rank, every member of the heavier ranking beats every member of the lighter one whatever their positions — so a graph-only chunk at rank 1 scored 0.3/61 = 0.0049, below the worst dense hit at 0.5/80 = 0.0063, and could never enter the top-k. The graph could reorder, but never contribute a document: precisely the multi-hop capability it exists for. k is now a policy value the caller passes, set to the candidate depth.

  The first attempt at this fixed the wrong knob, and the regression test hid it. Clamping k to the observed depth made the fused score depend on how many candidates a given query happened to return, and at the real depth it still does not deliver: with 20 candidates a rank-1 graph hit at weight 0.3 scores 0.3/21 and loses to every dense rank 1–14. The test passed because its filler shared no vocabulary with the question, so the candidate lists came back three deep — a depth at which the clamp does help and the assertion is nearly free. With competing filler it fails. Weight is the lever, not damping; see the note below.

• Dense search had no relevance floor. It returned a full k whether or not anything matched, so chunks at the embedder's noise floor occupied every fusible rank and padded the answer's context with pure filler. It now applies the same score > 0 floor keyword_search always had. On an off-topic query against a 42-chunk corpus that cuts the candidate list from 20 hits to 4.

Together those took hybrid from 0.800 to 0.900 recall — no longer worse than the BM25 it fuses — and lifted end-to-end context_recall from 0.886 to 0.946. A regression test now pins the multi-hop case end to end: a chunk that neither dense nor lexical search can find is absent from the top-5 without the graph and present with it.

What the graph still costs, and what it cannot do at the default weight. It buys no recall on this corpus and gives up MRR (0.803 → 0.745). That is a property of the corpus and the offline extractor, not a bug left standing: relations here are co-occurrence edges between nearby capitalised phrases, and every document is about the TR-1, so the seed entities appear almost everywhere. Multi-hop retrieval pays off when a question must join facts across documents that do not share vocabulary — which the regression test demonstrates and this 10-case single-topic suite does not contain.

There is a sharper limit worth stating plainly. Under weighted RRF a chunk found only by the graph carries graph_weight alone, while one found by both dense and lexical carries the full 1.0 — so at the default graph_weight = 0.3 a graph-only chunk cannot reach a top-5 cut. At the shipped default the graph reranks candidates the other retrievers found; it does not add documents of its own. The regression test asserts the multi-hop capability at graph_weight=0.8 and states the weight explicitly, and a second test pins the limit at the default, so the two cannot drift apart.

Raising the default was measured and rejected, because it is worse on the only evidence available:

graph_weight	recall@5	MRR	nDCG@5
0.3 (default)	0.900	0.745	0.526
0.5	0.900	0.695	0.503
0.7	0.850	0.625	0.457
1.0	0.850	0.625	0.453

Monotonically down, and capping the graph's candidate list instead of reweighting it is worse still (a cap of 3 takes MRR to 0.440). Tuning the default to make one capability demonstrable, at the cost of every metric actually measured, would be tuning for the anecdote. Raise it to ~0.8 if your corpus has genuine multi-hop structure — and re-run the ablation to check that it pays.

Why dense underperforms. The zero-dependency default embedder is a hashed bag-of-words (HashingEmbedder), not a learned model — a lossy lexical matcher with no semantics. Fusing it with a strong lexical ranker cannot add meaning that was never there. Configure a real embedding model and this is the row that should move.

What was deliberately not done: the defaults were not re-tuned to these numbers. Ten cases on one 27-chunk single-topic corpus is far too small a sample to fit global configuration to; doing so would trade a real design for a better table. The ablation is shipped as the instrument to re-run when the corpus, embedder, or extractor changes — --ablation takes any suite with gold contexts.

---

Quickstart

Offline in under a minute (no API keys, no services)

python3 -m venv .venv && source .venv/bin/activate   # Python 3.11+
pip install pydantic pydantic-settings fastapi "uvicorn[standard]" httpx pytest pytest-asyncio rich

# 20-second guided tour of the whole platform (ingest → retrieve → guardrails → cache → eval)
python -m atlas.cli demo

# Ask the agent a question (auto-loads the bundled sample knowledge base)
python -m atlas.cli chat -m "How long is the TR-1 warranty?"

# Run the test suite (offline)
pytest -q

# Measure RAG quality — and export a shareable Markdown report
python -m atlas.cli eval --suite rag_golden --report report.md

# Gate on quality: diff against the committed baseline, exit 1 on regression
python -m atlas.cli eval --suite rag_golden --baseline

# Prove each retrieval signal earns its place (no model calls, no keys)
python -m atlas.cli eval --ablation

# Jailbreak resistance — exits non-zero if resistance drops
python -m atlas.cli redteam

# Serve the API + interactive docs (auto-loads the sample corpus in dev)
uvicorn atlas.api.main:app --port 8000   # -> http://localhost:8000/docs

In development the API loads the bundled sample corpus on startup, so POST /chat returns grounded, cited answers immediately — no manual ingest step. atlas demo walks through, in one command: sample-corpus ingestion, a grounded answer with citations, the three hybrid-retrieval signals side by side, a prompt-injection attack getting blocked, a semantic-cache MISS→HIT speedup, and an eval scorecard. The one-shot chat output is the screenshot at the top of this README; atlas eval --report produces a report like docs/sample-eval-report.md.

Ask the same question twice and the second answer is served by the semantic cache — check GET /metrics for live hit rates, chunk counts, and spend.

Run atlas eval twice and every score matches to the last decimal — retrieval is deterministic on purpose (see Reproducibility), which is what lets CI gate on --baseline instead of eyeballing noisy numbers.

Full stack (hosted models + real datastores)

cp .env.example .env          # add your API keys / DSNs
make up                       # Postgres+pgvector, Redis, Qdrant, Langfuse, Ollama
make install                  # full dependency set
make ingest                   # ETL -> chunk -> embed -> index -> graph
make api                      # http://localhost:8000/docs

Run make help to see all targets.

Configuration

All configuration is environment-driven (12-factor); see .env.example for the full surface. Nothing is required to run offline. Set any of the following to light up the corresponding integration:

Variable	Enables
ATLAS_PROVIDERS__ANTHROPIC_API_KEY	Routing to Claude
ATLAS_PROVIDERS__OPENAI_API_KEY	Routing to OpenAI
ATLAS_RETRIEVAL__STORE	Vector store backend: memory (default), pgvector, qdrant
ATLAS_RETRIEVAL__RERANKER	Rerank stage: none (default) or cross-encoder
ATLAS_DB__DSN	Postgres DSN, used when ATLAS_RETRIEVAL__STORE=pgvector
ATLAS_QDRANT__URL	Qdrant endpoint, used when ATLAS_RETRIEVAL__STORE=qdrant
ATLAS_REDIS__URL	Redis semantic cache
ATLAS_OBSERVABILITY__LANGFUSE_*	Langfuse tracing

The vector-store backend is an explicit choice rather than something inferred from "is a DSN set?" — ATLAS_DB__DSN ships with a working local default, so inferring would silently move a fresh checkout off the offline in-memory store the moment anyone copied .env.example. A hosted backend that fails to build logs a warning and falls back to memory rather than refusing to boot.

Repo layout

atlas/
  core/           shared Pydantic models, Protocol interfaces, config, logging
  ingestion/      ETL, chunking, embeddings, knowledge-graph extraction
  retrieval/      hybrid search, RRF, reranking, graph retrieval, vector stores
  providers/      Claude / OpenAI / Ollama / vLLM / Bedrock adapters
  agent/          LangGraph agent, model router, tools, prompts, context builder
  mcp/            MCP tool server + client
  guardrails/     prompt-injection, PII redaction, content filter pipeline
  evals/          eval harness, RAG metrics, A/B testing, red-team
  observability/  Langfuse tracing, Redis semantic cache, cost/latency tracking
  finetune/       LoRA/PEFT trainer for a reranker/intent classifier
  api/            FastAPI app (REST + GraphQL + streaming)
frontend/         Next.js + TypeScript chat UI
web/worker/       Go concurrent ingestion worker
infra/            Docker, Kubernetes, SQL, cloud (AWS/GCP/Azure) deploy configs
tests/            pytest suite
docs/             architecture and responsible-AI docs

Development

pip install -e ".[dev]"
make test        # pytest
make lint        # ruff rules + formatting (both gated in CI)
make typecheck   # mypy

Contributions welcome — see CONTRIBUTING.md. Security policy in SECURITY.md.

License

MIT.