groundwork

A hybrid-retrieval RAG service in FastAPI, with an evaluation lab that gates every retrieval change. Postgres does both halves of retrieval (pgvector for dense, full-text search for sparse), the two rankings are fused with Reciprocal Rank Fusion, optionally reranked, and the answer streams back over SSE with byte-offset citations into the source documents. It runs offline with zero API keys.

What it is. One service and one datastore. Ingestion parses a file, chunks it while recording each chunk's character offsets in the original document, embeds the chunks and indexes them; retrieval runs a dense kNN query and a Postgres FTS query in parallel, fuses them with RRF, optionally reranks the candidates, and hands the top chunks to an LLM as untrusted context. POST /query returns a grounded answer with a citation per chunk; GET /query/stream streams the same thing as SSE (retrieval_done → token* → usage), so a reader sees citations before the first token. Ingestion also has an async path: POST /ingest enqueues a job that the arq/Redis worker runs out of band, and GET /ingest/{job_id} reports its state. Everything is async end to end, typed under mypy --strict, and configured from GROUNDWORK_* env vars with working local defaults.

Why the eval lab exists. A retrieval change that nobody measured is a rumor. This repo treats evaluation as product surface, not garnish: a versioned golden set with per-question provenance, recall@k / MRR / nDCG@k, an ablation matrix over retrieval mode × rerank × chunk size, and a regression gate wired into make check that fails the build when recall or MRR drops more than two absolute points against the committed baseline. The whole pipeline runs on deterministic stub providers — a hash embedder and a template LLM — so every number is reproducible offline, in CI, by anyone, at a named commit. Real providers (Ollama, any OpenAI-compatible endpoint) are one env var away and are never required by the tests.

Honest scope — what this is not. It is a single-node portfolio project, not a system that has been run at production scale. The published quality numbers were measured on a synthetic 13-document fixture corpus with 15 golden questions using the stub providers, so they characterise the plumbing — chunking, fusion, ranking, the gate — and are not a claim about real-world retrieval quality (with a hash embedder, the dense arm is close to noise; see RESULTS.md, which says so at length). The published latency numbers come from the same stub providers, where generation is essentially free, so they measure the service's own overhead and predict nothing about latency behind a real model. There is no authentication and no multi-tenancy. Rate limiting is a single-process, per-IP token bucket (ADR 0031) — it does not survive a reverse proxy that hides the real client IP; see Limitations. Real-corpus evaluation is still queued work. The repo was built task-by-task by autonomous agent cycles under docs/AUTONOMY.md — that is a feature of the project, and state/journal.md is its build log. For the architecture and key-decisions essay — hybrid+RRF, deterministic stubs, the regression gate, and more, each cited back to the module or ADR that backs it — see docs/DESIGN.md.

Architecture

flowchart LR
  subgraph ingest["Ingestion (CLI or arq worker)"]
    FILES["corpus files<br/>md, html, txt"] --> PARSE["parse<br/>size-capped, sanitized"]
    PARSE --> CHUNK["chunk<br/>char-offset provenance"]
    CHUNK --> EMBED["embed<br/>384-d vectors"]
  end

  subgraph store["Index (Postgres 16)"]
    VECTORS["chunk_embeddings<br/>pgvector, HNSW cosine"]
    FTS["chunks<br/>tsvector full-text"]
  end

  EMBED --> VECTORS
  CHUNK --> FTS

  subgraph retrieval["Hybrid retrieval"]
    DENSE["dense<br/>vector kNN"]
    SPARSE["sparse<br/>Postgres FTS"]
    FUSE["RRF fusion"]
    RERANK["rerank<br/>optional"]
    DENSE --> FUSE
    SPARSE --> FUSE
    FUSE --> RERANK
  end

  VECTORS --> DENSE
  FTS --> SPARSE

  ASK["POST /query<br/>GET /query/stream"] --> DENSE
  ASK --> SPARSE
  RERANK --> GENERATE["generate<br/>stub | Ollama | OpenAI-compatible"]
  GENERATE --> ANSWER["SSE stream<br/>citations, tokens, usage"]

  subgraph evallab["Eval lab (first-class)"]
    GOLDEN["golden set<br/>15 questions, provenance"]
    METRICS["recall@k, MRR, nDCG@k"]
    GATE["regression gate<br/>runs in make check"]
    MATRIX["ablation matrix<br/>RESULTS.md"]
    GOLDEN --> METRICS
    METRICS --> GATE
    METRICS --> MATRIX
  end

  RERANK --> METRICS
  OBS["OTel spans<br/>Prometheus /metrics"]
  GENERATE --> OBS

Loading

Every stage of the query path — dense, sparse, fuse, rerank, the opt-in post-rerank mmr diversification stage, the pre-retrieval expand stage (multi-query expansion or HyDE), generate, and the end-to-end query stage — emits a Prometheus histogram (groundwork_stage_duration_seconds), and every stage but query also opens an OpenTelemetry span when tracing is enabled: query is plain wall-clock accounting around the whole request body, timed separately so it never reparents the span tree (ADR 0037). The three ingestion stages (parse, chunk, embed) emit spans only: they run in the arq worker, which serves no HTTP and is therefore never scraped, so exporting their durations needs a worker-side exporter and is deliberately out of scope (ADR 0024). Between them, the eval lab measures what the pipeline retrieves and the metrics measure how long the query path took to do it.

Two pre-retrieval stages are opt-in and reachable from /query and /query/stream directly — not just the eval CLI — behind their own flags, both off by default (identical behavior to before either existed): multi-query expansion (GROUNDWORK_QUERY_EXPANSION_ENABLED, GROUNDWORK_ QUERY_EXPANSION_N) fans the query out into LLM-paraphrased variants, retrieves each, and fuses the rankings by the same RRF fuse dense/sparse already use (ADR 0028); HyDE (GROUNDWORK_HYDE_ENABLED) drafts a hypothetical passage answering the query with the configured LLM, retrieves it densely, and fuses that ranking with the normal query ranking (ADR 0032). If both flags are set, HyDE takes priority — the two are not composed. A dead expansion/HyDE backend degrades to one plain retrieval rather than failing the request; either way the request is answered, just without the stage's benefit. Because both routes through the exact function the eval CLI already used to measure their delta (retrieve_expanded/ retrieve_hyde), a stage's eval numbers and its /query behavior are always the same code path, never a reimplementation that could drift.

An opt-in Redis read-through cache sits in front of whichever retrieval stage is selected above (GROUNDWORK_RETRIEVAL_CACHE_ENABLED, default off): a hit skips dense/sparse/fuse/rerank — and expansion's or HyDE's own extra retrievals — entirely, keyed on the resolved query params, the selected stage, and a digest over every (document id, content hash) pair, so ingesting, re-ingesting, or deleting a document invalidates every derived key with no explicit eviction path. Off is pure delegation — no Redis client is even constructed — and a Redis outage while on degrades to a live retrieval, never a failed request (ADR 0030).

An opt-in MMR (maximal-marginal-relevance) diversification stage sits at the end of the candidate pipeline (GROUNDWORK_MMR_ENABLED, GROUNDWORK_ MMR_LAMBDA, default off / 1.0): it greedily re-selects the final top-k, trading relevance for diversity so a pool full of near-duplicate chunks does not surface k near-copies of the same passage. It lives inside hybrid. retrieve itself, so /query, /query/stream, and the eval CLI all reach it the same way expansion/HyDE do — no separate wiring. lambda=1.0 (the default) makes the diversity term vanish, reproducing the plain top-k cut byte-for-byte — a second way to get pre-MMR behavior besides leaving the flag off (ADR 0043).

GET /status (task 110) is a read-only graceful-degradation snapshot: which providers the chaos fault-injection seam (ADR 0041) currently has forced down, full-hybrid vs degraded (dense/sparse-only) retrieval mode, the retrieval-cache hit rate, and concurrency/namespace-quota pressure. Every field is derived from Settings and the same collectors GET /metrics scrapes — it makes no database, Redis, or provider call of its own, so it can never itself become another thing to fail. Example response (default settings, nothing armed, cache disabled):

{
  "status": "ok",
  "retrieval_mode": "hybrid",
  "retrieval_degraded": false,
  "providers": {"embedding": "up", "llm": "up", "database": "up", "cache": "disabled"},
  "cache_hit_ratio": 0.0,
  "cache_hits": 0,
  "cache_misses": 0,
  "quota": {
    "concurrency_in_flight": 0,
    "concurrency_max_in_flight": 1000,
    "concurrency_shed_total": 0,
    "namespace_quota_rejected_total": 0
  },
  "chaos": {
    "provider_unavailable": false,
    "redis_outage": false,
    "db_unavailable": false,
    "nondeterministic_embedding": false
  }
}

Quickstart

Needs Docker, uv, and Python 3.12+. No API key, no model download — the default provider is the offline stub.

uv sync                 # install pinned dependencies from uv.lock
make db-up              # Postgres (pgvector) + Redis, bound to 127.0.0.1 only
make seed               # migrate, then ingest the fixture corpus (idempotent)
uv run uvicorn groundwork.api.app:create_app --factory   # serves on :8000

Then, in another terminal, ask it something:

curl -N -H 'Accept: text/event-stream' \
  'http://127.0.0.1:8000/query/stream?question=How%20do%20I%20undo%20the%20most%20recent%20vpk%20transaction%3F'

The real response, elided at the … marks — 5 citations arrive and 1 is shown, and the answer streams one token event per word. The chunk ids and token counts are stable across runs (stub providers are deterministic); only the wall-clock timings move:

event: retrieval_done
data: {"citations": [{"doc_id": 7, "chunk_id": 23, "char_start": 214, "char_end": 970, "quote": "## Everyday commands\n\n| Command             | Effect …"}, …]}

event: token
data: {"text": "(stub:stub-llm-v1) "}

event: token
data: {"text": "Answer "}

event: token
data: {"text": "to: "}

…

event: usage
data: {"retrieval_ms":33.47,"generate_ms":1.22,"chunks_considered":5,"prompt_tokens":930,"completion_tokens":77}

That answer text comes from the template stub LLM: it echoes the question, the top chunk and the citation ids, which is exactly what a deterministic provider should do. Point GROUNDWORK_PROVIDER_MODE at a real model to get prose.

Other things worth running:

make check       # ruff + mypy --strict + pytest + the eval regression gate
make eval        # score the golden set, write results/*.json
make ablations   # the full matrix (rebuilds the index; regenerates RESULTS.md)
make dashboard   # render results/dashboard.html: one self-contained HTML file
make obs-up      # Prometheus + Grafana with the committed dashboard
make loadtest    # 20 users / 2 min against a running API
make openapi     # write the OpenAPI schema to docs/openapi.json
make sbom        # write a CycloneDX-style dependency manifest to docs/sbom.json
make audit-deps  # fail if a direct dependency is outside the CLAUDE.md allowlist

Results

An excerpt of RESULTS.md, measured at commit ac1c6bf: the six retrieval configurations at the default chunk size (800) — the index make seed builds and the service ships. The full table adds the same six configurations at chunk 256 and 512, and carries the Run column this excerpt folds into this sentence.

Config	Chunk	Golden	Provider	Queries	recall@k	MRR	nDCG@k
dense_k10_norerank	800	fixture_v1	stub	15	0.1667	0.0207	0.0512
dense_k10_rerank	800	fixture_v1	stub	15	0.1667	0.1667	0.1496
sparse_k10_norerank	800	fixture_v1	stub	15	0.4000	0.4000	0.4000
sparse_k10_rerank	800	fixture_v1	stub	15	0.4000	0.4000	0.4000
hybrid_k10_norerank	800	fixture_v1	stub	15	0.5000	0.3474	0.3827
hybrid_k10_rerank	800	fixture_v1	stub	15	0.7333	0.6500	0.6462

Hybrid + rerank wins on recall@10 (0.7333). Read that with the caveats attached: dense retrieval here is a hash-embedding stub whose recall tracks the random baseline, so the hybrid rows are really "sparse plus noise", and one question is worth 6.7 recall points on a 15-question set — differences below ~0.07 are within the resolution of the set. The interpretation and the full limitations live in RESULTS.md; the numbers are regenerated by make ablations and refuse to be published from a dirty tree.

Latency

From the load test at commit 448588f: 20 users for 2 minutes against one uvicorn process, 7647 requests, 0 failures, 63.7 req/s, streaming time-to-first-token p50 9.6 ms / p95 21.5 ms / p99 30.7 ms. That is the service's overhead — retrieval, fusion, reranking, SSE framing — measured with the stub LLM, where generation costs nothing. It says nothing about latency with a real model, where TTFT is dominated by the model itself. It is a smoke-scale load test on a single node with the load generator on the same host, not a capacity study.

Eval methodology

• Golden set. evals/golden/fixture_v1.yaml: 15 questions, each carrying its provenance — the source document, the section, and a verbatim quote that is resolved to chunk ids at load time, so ground truth survives a change in chunk size. Questions are human-written against the corpus; generated answers never feed back into them (no eval contamination).
• Metrics. recall@k, MRR, nDCG@k, defined in ADR 0020 and computed by make eval over every golden query. Results are JSON artifacts stamped with the git sha, the config, and the provider mode.
• Regression gate. make check runs the gate (ADR 0021): it re-scores the golden set and fails the build if recall@k or MRR fell more than 2 absolute points below evals/baseline.json. The gate never writes its own baseline — promotion is the separate, explicit make eval-baseline. With no database reachable it reports SKIPPED, loudly; a skip is never a pass.
• Ablations. make ablations (ADR 0022) sweeps {dense, sparse, hybrid} × {rerank on/off} × chunk {256, 512}, restores the default index, and regenerates the table in RESULTS.md.
• A/B comparison. python -m groundwork compare <runA.json> <runB.json> loads any two committed eval result JSONs — no database, no services — and prints a per-metric table of A, B, delta, and the 95% bootstrap CI of the delta, paired by query id and reusing evals/bootstrap.py unmodified (the same primitive eval-gate uses). Only recall@k and MRR are gated; a gated metric whose delta's CI lies entirely below zero exits 1 (a real regression from A to B), a drop within the CI's noise exits 0. The table also lists which config fields differ between the two runs, so an ablation cell can be checked against another (or a PR's run against main's) without touching the database.
• Dashboard. make dashboard (ADR 0045) renders results/dashboard.html: one self-contained HTML file — inline CSS, hand-written inline SVG, no JS, no external asset, no new dependency — with the metric trend per golden set (recall@k/MRR/nDCG@k plus each run's own bootstrap 95% CI), the per-category breakdown, and the quality/cost table, all traced straight from results/*.json. Not committed (like the JSON itself); regenerate it any time with make dashboard.
• Determinism. Stub providers, a frozen fixture corpus, seeded randomness, pinned model names, and no wall clock in the pipeline. Every published row names the commit it was produced at, and a row from an uncommitted tree is refused rather than published.
• Cache warmup. python -m groundwork warmup (ADR 0062) pre-populates the opt-in retrieval-result cache (ADR 0030, ADR 0046) from the committed query set at evals/golden/warmup_queries_v1.yaml, so a query in that set is a cache hit the first time real traffic asks it instead of paying a cold miss. It only ever warms committed/eval query text — never live request text — is namespace-scoped, idempotent, and a no-op (no database touched) unless GROUNDWORK_RETRIEVAL_CACHE_ENABLED is set and the query set is non-empty.
• Corpus stats. python -m groundwork corpus-stats (ADR 0063) reports, per namespace, doc/chunk/token counts, a chunk-size histogram, embedding coverage, and FTS coverage, flagging orphan chunks (no embedding), zero-chunk documents, and oversized chunks. Read-only, deterministic for a fixed index, --format {markdown,json} and --out FILE selectable — a diagnostic, never a gate.

Providers

Real providers are opt-in and are never required by the test suite: make check runs entirely offline on the stubs.

GROUNDWORK_PROVIDER_MODE	Embeddings	LLM	Needs	Used by tests
stub (default)	hash embedder, 384-d, deterministic	template LLM, streams word by word	nothing — no key, no network	yes, always
ollama	<base>/api/embed	<base>/api/chat, streamed	Ollama running at GROUNDWORK_OLLAMA_BASE_URL; an embedding model that emits 384-d vectors (the column width is fixed)	no
openai_compat	<base>/v1/embeddings	<base>/v1/chat/completions, streamed	any OpenAI-shaped server at GROUNDWORK_OPENAI_COMPAT_BASE_URL (self-hosted vLLM, a vendor, …); if that server demands a key, both GROUNDWORK_EMBEDDING_API_KEY (the embeddings call) and GROUNDWORK_LLM_API_KEY (the chat call) — they are separate settings	no

Both real-provider base URLs default to loopback on purpose: a misconfigured deployment fails to connect rather than quietly shipping corpus text to a third party. See .env.example for every setting, and ADR 0004, 0017, 0018.

Limitations

• Single node, single process. One uvicorn process, one Postgres, one Redis, no replicas, no load balancer. /metrics is single-process only (prometheus_client's multiprocess mode is not configured), so multiple workers must be scraped separately.
• The published quality numbers are stub-provider numbers on a fixture corpus. 13 synthetic documents, 15 golden questions. The dense arm is a hash embedder, so its recall is near the random baseline and the hybrid rows are sparse-plus-noise; the stub reranker scores lexical overlap, which is what FTS already ranks by, so it barely moves sparse results. A real embedding model would change that arm — and the fusion trade-off — entirely. Evaluation over a real, curated corpus is queued, not done.
• The published latency numbers are stub-provider numbers too. Generation is free in that setup. Nothing in this repo benchmarks a real LLM, and 20 users for 2 minutes establishes no throughput ceiling.
• No authentication, authorization, or multi-tenancy. Any caller who can reach the port can query the whole corpus. /metrics is unauthenticated by design (it is a scrape surface holding no user content) — do not expose it publicly. The compose stack binds to 127.0.0.1 only, and its dev credentials are dev credentials.
• Rate limiting is in-process and keyed on the connecting IP only — it does not survive a reverse proxy that hides the real client address. TokenBucketLimiter (src/groundwork/api/ratelimit.py, ADR 0031) deliberately refuses to trust any client-supplied header (including X-Forwarded-For), since this service has no inbound authentication and an unauthenticated header is free for any caller to spoof — that refusal is correct, not a gap. The consequence: deployed behind a reverse proxy or load balancer that does not forward the original peer address into the ASGI scope, every request arrives from the proxy's own IP, so all callers behind it collapse onto a single shared bucket instead of one each. If you put groundwork behind a proxy, configure it to preserve the real client address (e.g. PROXY protocol or an ASGI server flag that trusts your specific, known proxy) — do not "fix" this by making the limiter trust X-Forwarded-For from an arbitrary caller.
• Ingestion trusts nothing, but it is also not a crawler. Local files only (md, html, txt), size-capped, path-traversal-guarded against GROUNDWORK_INGEST_ROOT, and chunk text is passed to the LLM as data with an explicit instruction to disregard instruction-like content. There is no URL fetching, no PDF, no OCR.
• No deployment story. No Dockerfile for the service itself, no Helm chart, no migrations-on-boot. The API runs under uvicorn on the host, by hand.
• Security posture is documented, not certified. A full-tree security audit was run (zero open CRITICAL/HIGH) and SECURITY.md states the threat model, enforced input caps, injection framing, and dependency policy. This is a single-node portfolio service with no authentication and no formal third-party assessment — read the disclosure section of SECURITY.md before relying on it.

Repository layout

src/groundwork/     package: api/, ingest/, retrieval/, providers/, evals/, db/
tests/              pytest suite; db-marked tests skip without Postgres
evals/              golden sets + the committed regression baseline
docs/adr/           one ADR per architectural decision (0001-0025)
docs/AUTONOMY.md    how the agent loop that built this is driven
docs/DESIGN.md      architecture + key-decisions essay, cited to modules/ADRs
loadtest/           locust load generator + published reports
ops/                Prometheus config + the Grafana dashboard JSON
tasks/, state/      the agent queue and its build journal

How this was built

Autonomously, by scheduled Claude Code agent cycles: one task per cycle, TDD, an adversarial review pass, one commit on auto/dev, all under the constitution in CLAUDE.md and the controls in docs/AUTONOMY.md. state/journal.md records every task, including the ones that went wrong.

License

MIT © Arshiya Shafizade