RAG8s — End-to-End RAG Platform for Internal Knowledge Bases

RAG8s is a production-oriented Retrieval-Augmented Generation (RAG) platform on Kubernetes (AWS EKS). It is built as 8 independently deployable microservices and covers the full RAG lifecycle: document ingestion and chunking, hybrid retrieval with conditional reranking, and streaming LLM inference. Citations are assigned by the application rather than the model, linking answers to retrieved passages and to the original documents through pre-signed S3 URLs.

RAG8s is designed for internal knowledge bases — company documentation, technical manuals, policies, and operational runbooks — where content is curated and updates periodically. It is not built for real-time ingestion of continuously changing data streams

The platform also provides OIDC authentication, per-user rate limiting, GitOps deployments, Infrastructure as Code, Karpenter Spot capacity for stateless, and end-to-end observability.

Key Features

Citation Grounding — The application assigns citation IDs to retrieved passages, which the LLM references. Invalid IDs are stripped; valid citations deep-link authenticated users to the exact document page and region through pre-signed S3 URLs.
Offline Evaluation — 75 curated test records tracked in MLflow. Latest results: groundedness 0.95, citation integrity 0.89, and recall@5 0.72.
Cost and Access Control — Exact and semantic caching reduce redundant model calls. Rate limiting defaults to 5 requests/minute per authenticated user, keyed by JWT subject rather than IP.
Observability — Prometheus metrics, 24 alerts with inhibition rules, structured logs retained in ClickHouse for 30 days, and retrieval metadata for debugging.

Explore the Platform

RAG Pipeline — ingestion, retrieval, reranking, and generation.
Offline Evaluation — metrics, methodology, and limitations.
Deployment Guide — infrastructure setup and deployment.

Demo: Grounded Answers & Citations

The embedded GIF is a preview Open the full demo which covers semantic caching, groundedness, rate limiting, and end-to-end latency.

Architecture

The RAG lifecycle is separated into two independent execution planes:

Batch Indexing Plane — An incremental, idempotent CronJob pipeline that ingests raw documents (PDF, DOCX, audio, images, CSV, Markdown, HTML, …) from S3, normalizes and OCRs them, splits them into traceable chunks, generates dense and sparse embeddings via stateless FastEmbed microservices, and upserts into Qdrant with full positional metadata. Backups are triggered automatically by configurable thresholds.

Online Inference Plane — A low-latency streaming request path that authenticates users via OIDC, performs exact and semantic cache lookups, embeds the query (dense and sparse in parallel), executes hybrid Qdrant search with Reciprocal Rank Fusion, optionally re-ranks with a cross-encoder, builds a strictly-grounded numbered prompt, and streams the answer via AWS Bedrock. Every response is citation-validated — hallucinated references are stripped, and users can open original documents with one-click pre-signed S3 URLs.

RAG Pipeline Overview

The operational heart of the system. Everything else — Kubernetes, IAM, observability — exists to make this pipeline reliable, debuggable, and cheap to run.

Inference control flow: exact cache → parallel dense+sparse embed → semantic cache → hybrid Qdrant + RRF → conditional cross-encoder rerank → numbered prompt → Bedrock with guardrails → structural citation validation → background cache write-back. Every external dependency has an independent circuit breaker and fallback, so the failure mode is a worse answer, not an error.

1. Document Extraction & Chunking (e.g., PDF)

PDF extraction uses PyMuPDF for text, layout, and images, and pdfplumber for tables. Column layouts are reconstructed from text-block coordinates. Captions are tied to figures, and images larger than 3 KB are processed with RapidOCR at 300 DPI, falling back to Tesseract. Font artifacts are stripped during cleaning.

Chunking preserves sentence boundaries and source traceability:

spaCy sentence segmentation, with fallback strategies.
Token counting with tiktoken (cl100k_base).
Defaults: 512-token max chunk, 2-sentence overlap, 100-token minimum (smaller chunks merge into the previous).
Oversized sentences are split to prevent content loss.

Each chunk stores a deterministic ID, source and page metadata, parser version, semantic region, and any associated figure, OCR, caption, or table content.

2. Retrieval

Retrieval follows a fixed workflow, not an agent loop.

Exact cache: Returns cached answers keyed on the normalized query, corpus/prompt/retrieval versions, model, tenant, top_k, and fetch_k.
Parallel embeddings: Dense and sparse query embeddings are generated concurrently, each behind its own circuit breaker.
Semantic cache: Searches at a strict similarity threshold of ~0.84, then a relaxed 0.78.
Hybrid search: Qdrant retrieves dense and sparse results in parallel and fuses them with Reciprocal Rank Fusion (RRF).
Conditional reranking: A cross-encoder reranks when retrieval confidence is low or the top two candidates score similarly.
Prompt construction: Retrieved passages are numbered and inserted into a grounded prompt.
Generation: Amazon Bedrock streams the answer with AWS Guardrails enabled.
Citation validation: Generated citation references are checked against the passages supplied in the prompt.
Cache write-back: Results are cached in a background task, off the response stream.

Hybrid ranking: Reciprocal Rank Fusion (RRF)

RRF merges dense and sparse results by rank rather than raw score, so the two methods' score scales don't need to match.

RRF(d) = Σ [1 / (60 + rank_r(d))]
         r ∈ R

Where:

d is a document.
R is the set of ranked result lists.
rank_r(d) is the position of document d in list r.
60 is the RRF constant, which dampens the influence of lower-ranked results.

Reranking

Reranking runs only when needed. Normalized reranker and fusion scores are blended by weight α; candidates outside the rerank pool keep their order and follow the reranked results. On failure, the system falls back to the original fusion ranking.

3. Citation Grounding

Citations are assigned by the application, not invented by the LLM. Passages receive numbered identifiers ([1], [2], …) before entering the prompt; the LLM cites those identifiers.

After generation, the application checks every citation against the supplied passages and removes invalid references. If validation leaves the answer empty, deterministic_summarize provides a fallback.

This catches invalid IDs but does not verify that a cited passage actually supports its claim.

4. Caching and Resilience

Exact cache: Avoids repeated retrieval and generation for identical requests.
Semantic cache: Reuses answers for sufficiently similar queries; cache hits are promoted to the exact cache.
Expiration: Entries expire per CACHE_TTL_SECONDS and are removed by a background cleanup loop.
Fault tolerance: Independent circuit breakers and fallbacks allow degraded operation when external components fail.
Observability: Responses expose retrieval mode, dense/sparse/fused result counts, and reranking metadata.

5. Embedding and Generation Models

Embeddings are self-hosted, because changing the embedding model means re-embedding the corpus and rebuilding the Qdrant index. Generation uses Amazon Bedrock, so the LLM can change without re-indexing.

Cloud Infrastructure

Infrastructure is managed with OpenTofu across AWS and Cloudflare.

AWS Infrastructure — Runs applications on EKS, with on-demand nodes for core services and Karpenter-managed Spot instances for scalable, stateless inference workloads. Qdrant stores persistent vector data on EBS, S3 stores infrastructure state, and ECR hosts container images. GitHub Actions publishes images using OIDC authentication, avoiding long-lived credentials.
Cloudflare Infrastructure — Secures external access through a single Cloudflare Tunnel with strict SSL and DDoS protection, without publicly exposed AWS IP addresses or load balancers.

Microservices

Source / Service	Role	Key Details
Frontend	Authentication and chat UI	Google/Microsoft OAuth, short-lived JWTs, streaming request proxy
Retriever	RAG orchestration	Caching → hybrid retrieval → conditional reranking → Bedrock streaming → citation validation
Dense Embedder	Dense vector generation	CPU by default, FastEmbed, `BAAI/bge-small-en-v1.5`, 384-dimensional normalized vectors
Sparse Embedder	Sparse vector generation	CPU by default, FastEmbed, `Qdrant/minicoil-v1`, stateless
Reranker	Document relevance scoring	CPU by default, cross-encoder `Xenova/ms-marco-MiniLM-L-6-v2`, conditionally triggered
Indexing CronJob	Batch document ingestion	Document conversion → chunking → embedding → Qdrant upserts → conditional backups
Valkey	Distributed rate limiting	Shared rate-limit counters across Retriever replicas; NetworkPolicy isolation
Cloudflared	Secure external connectivity	Cloudflare Tunnel to internal services; observability endpoints blocked at edge; Prometheus metrics

Note: The 8 entries above are the deployable units counted by this platform — six application services and two infrastructure components (Valkey, Cloudflared). Stateful stores (Qdrant, ClickHouse) and the observability stack are provisioned separately and are not counted as microservices.

Connectivity & Auth

External traffic enters through a single Cloudflare Tunnel with strict SSL and no publicly exposed origin IPs or load balancers. Google and Microsoft OAuth provide authentication, with short-lived JWTs and domain allowlists.

DDoS mitigation and rate limiting:

Cloudflare edge: DDoS mitigation before traffic reaches the cluster.
Per-user: 5 requests/minute at the frontend, keyed by JWT subject, to limit LLM spend.
Per-IP: 60 requests/minute on /generate/stream and 30/minute on /presign at the Retriever.

Observability

Built-in, no external SaaS required:

Prometheus + Alertmanager — 24 alert rules, Slack notifications with inhibition rules
Grafana — auto-discovered dashboards via ConfigMap sidecar
Vector + ClickHouse — structured JSON log aggregation with 30-day retention

Security

Layered across the full stack:

Edge: Cloudflare Tunnel with strict SSL and endpoint filtering
Auth: OIDC with PKCE and CSRF state protection, ES256 JWTs (15-min TTL), per-provider domain/org/tenant allowlists
Network: Kubernetes NetworkPolicies isolating namespaces by function
IAM: IRSA for least-privilege AWS access, GitHub Actions OIDC with per-repo roles
Runtime: Read-only root filesystems, non-root containers, no privileged pods
CI/CD: Pre-commit Gitleaks hook + CI-side scanning (Gitleaks, Trivy, OpenGrep) on every commit
GitOps: Argo CD reconciles cluster state from Git, self-heals drift, rollbacks via git revert

Offline Evaluation

Offline evaluation against a 75-record golden dataset, tracked in MLflow.

cat src/offline_eval/offline_eval_artifacts/summary.json

{
  "meta": {
    "records": 75
  },
  "performance": {
    "success_rate": 1.0
  },
  "retrieval": {
    "recall_at_k": 0.72,
    "mrr": 0.5793
  },
  "generation": {
    "fact_coverage": 0.7769,
    "groundedness": 0.9459,
  },
  "citations": {
    "citation_integrity": 0.8933
  },
  "errors": {
    "rate": 0.0
  }
}

Golden set composition

Each record defines:

query — a realistic employee question about the indexed corpus
expected_chunk_id — the chunk that should be retrieved
reference_answer — a hand-written ground-truth answer
expected_facts[] — the atomic facts that must appear in a complete answer

The 75-record evaluation provides an initial signal across four metrics but may miss rare failure modes.

Limitations

Not designed for real-time ingestion — Documents are indexed through a scheduled batch pipeline, making RAG8s better suited to internal knowledge bases, technical documentation, policies, and runbooks that update periodically rather than continuously.
AWS vendor lock-in — The platform is tightly coupled to AWS services, including Amazon Bedrock for LLM inference, EKS for orchestration, S3 for document storage, and AWS IAM for identity and access management.
Retrieval quality on a heterogeneous corpus — Documents are a random collection with very different, unpredictable structure, the main driver of the current recall@5 of 0.72: no single chunking, embedding, or retrieval configuration fits all of them. Gains will come from a representative evaluation dataset, systematic retrieval experiments, and iterative tuning.
No per-document access filtering — Authenticated users can query the full corpus. Query-time group-based filtering is required for enterprise access control.
No deletion reconciliation — Deleting a source from S3 does not remove its Qdrant chunks. Soft tombstones with query-time filtering are planned.

Step-by-Step Deployment Guide

Prerequisites

Docker installed and running without sudo access. Non-root is required for devcontainer stability. Run sudo usermod -aG docker $USER && newgrp docker if not already.
Visual Studio Code with the Dev Containers extension for a reproducible dev environment.
An AWS account with sufficient IAM permissions (AdministratorAccess or equivalent) to manage:
- Amazon EKS (Elastic Kubernetes Service)
- EC2, VPCs, Subnets, and Security Groups
- Amazon S3
- IAM Roles, Policies, and Instance Profiles AWS Free Tier is sufficient for development and testing purposes.
A Cloudflare account with a registered domain, with permissions to manage DNS records and create Cloudflare Tunnels (cloudflared).

PHASE 0.1 Clone the repo and build the devcontainer (Reproducible).

cd $HOME && rm -rf RAG8s && git clone https://github.com/Athithya-Sakthivel/RAG8s.git
cd RAG8s && code .
# Ctrl + Shift + P → Paste `Dev containers: Rebuild Container Without Cache` and Enter.

First-time build takes 5–15 minutes depending on network speed. You may create a GitHub codespace instead if network is slow.

PHASE 0.2 Open a new terminal and log in to your `gh` account as shown below

git config --global user.name "Your Name"
git config --global user.email you@example.com
gh auth login

? What account do you want to log into? GitHub.com
? What is your preferred protocol for Git operations? SSH
? Generate a new SSH key to add to your GitHub account? No
? How would you like to authenticate GitHub CLI? Login with a web browser

! First copy your one-time code: <code>
- Press Enter to open github.com in your browser...
✓ Authentication complete. Press Enter to continue...

PHASE 0.3 Create a new private repo in your GitHub account

export REPO_NAME="RAG8s" # or any name
git remote remove origin 2>/dev/null || true
gh repo create "$REPO_NAME" --private >/dev/null 2>&1
REMOTE_URL="https://github.com/$(gh api user | jq -r .login)/$REPO_NAME.git"
git remote add origin "$REMOTE_URL" 2>/dev/null || true
git branch -M main 2>/dev/null || true
git push -u origin main
git pull
git remote -v
echo "[INFO] A private repo '$REPO_NAME' created and pushed. Only visible from your account."

Phase 1 — Infrastructure Foundation

1.1 Provision AWS Infrastructure

Creates the VPC, EKS cluster, S3 buckets, ECR repositories, and all IAM roles. Uses OpenTofu (Terraform-compatible).

export TF_VAR_region="ap-south-1"
export TF_VAR_github_repository=<user_name/<repo_name>"    # replace with your username and $REPO_NAME
bash src/infra/terraform/aws/run.sh --create --env staging

1.2 Connect to the New EKS Cluster

aws eks update-kubeconfig --region ap-south-1 --name rag-eks-staging

Phase 2 — Container Images (CI/CD)

2.1 Trigger Image Builds to ECR

Replaces account IDs and region in CI workflow files so GitHub Actions can push images to your ECR. After running, open your repo's Actions tab — all 6 service images will build and push in ~5 minutes.

bash src/scripts/replace.sh

Phase 3 — GitOps Controller & Auto-Scaling

3.1 Install Argo CD

Deploys the GitOps controller that will sync all applications from this repo. Requires a GitHub personal access token for private repo access. The secret shown is temporary.

export GIT_PAT=ghp_   # Visit https://github.com/settings/tokens/new
bash src/infra/core/argo_setup.sh --rollout

3.2 Bootstrap Karpenter for the stateless workloads

Karpenter provisions EC2 Spot nodes labeled node-type=compute with a NoSchedule taint; the stateless workloads carry matching tolerations to schedule there, and a Pod Disruption Budget keeps Cloudflare tunnel available. Stateful stores and the observability stack stay on the on-demand EKS node group, and underutilized Spot nodes consolidate after 15 minutes via the WhenEmptyOrUnderutilized policy.

export GH_REPO= # replace with your full repo url (https://github.com/<USER_NAME/$REPO_NAME.git)
export GH_BRANCH="main"
export AWS_REGION="ap-south-1"
bash src/scripts/eks/bootstrap_karpenter.sh --rollout

Phase 4 — Data Ingestion & Vector Storage: Deploy the Indexing Pipeline

This phase sets up the complete document processing stack. It deploys Qdrant cluster, FastEmbed services, and finally the indexing CronJob that runs on a schedule.

Once deployed, this pipeline automatically handles the full document lifecycle: upload a few PDFs and HTMLs to S3, ingest raw files from S3, convert them to text (including OCR for pages/slides with images), split them into smaller chunks, generate embeddings for each chunk, index them into Qdrant for fast retrieval and conditionally performs qdrant backups.

See the indexing pipeline documentation for configuration details.

export HF_TOKEN=   # Hugging Face token for faster model downloads (optional)
bash src/scripts/eks/run_indexing_pipeline.sh

NOTE: Karpenter may take 5–15 minutes to provision EC2 instances if the cheapest matching instance type is unavailable. It retries with other c-family types automatically. Pods will stay Pending until a compatible instance launches.

Phase 5 — External Access & DNS: Set Up Cloudflare Tunnel and DNS

Creates DNS records and a Cloudflared/Argo tunnel that securely routes traffic to your cluster — no LoadBalancers or public IPs needed. The script waits for you to authorize Cloudflare access.

export CLOUDFLARE_ACCOUNT_ID=      # Cloudflare dashboard > Account Home > Search and enter "Copy account ID".
export CLOUDFLARE_GLOBAL_API_KEY=  # https://dash.cloudflare.com/profile/api-tokens > API Keys
export CLOUDFLARE_EMAIL=           # example: "athithya651@gmail.com"
export DOMAIN=                     # example: "athithya.site"

bash src/infra/terraform/cloudflare/run.sh --apply

# Export tunnel credentials
export CLOUDFLARE_TUNNEL_TOKEN="$(tofu -chdir=src/infra/terraform/cloudflare output -raw cloudflare_tunnel_token)"
export CLOUDFLARE_TUNNEL_NAME="$(tofu -chdir=src/infra/terraform/cloudflare output -raw cloudflare_tunnel_name)"
export CLOUDFLARE_TUNNEL_ID="$(tofu -chdir=src/infra/terraform/cloudflare output -raw cloudflare_tunnel_id)"

# Deploy cloudflared with secrets
python3 src/infra/core/cloudflared_setup.py --write

Phase 6 — Query Engine & User-Facing Services: Deploy the Inference Stack

Launches the retriever, Chat UI with OIDC authentication, Valkey for per-user rate limiting, and a Cloudflare Tunnel. Configure Google OAuth, Microsoft OAuth, or both; either provider is sufficient for authentication.

Create OAuth Credentials Google | Microsoft

export DOMAIN=                 # Use the same $DOMAIN
export GOOGLE_CLIENT_ID=             # Google OAuth client ID
export GOOGLE_CLIENT_SECRET=          # Google OAuth client secret
export GOOGLE_ALLOWED_DOMAINS="company.com,gmail.com"        # Comma-separated allowed email domains
export MS_CLIENT_ID=                       # Azure AD application (client) ID
export MS_CLIENT_SECRET=                   # Azure AD client secret
export MICROSOFT_ALLOWED_TENANT_IDS=       # Primary tenant ID (single-tenant or common)
export MICROSOFT_ALLOWED_DOMAINS="outlook.com,company.com"   # Comma-separated allowed email domains

bash src/scripts/eks/run_inference_pipeline.sh

Phase 7 — Observability

7.1 Deploy Monitoring, Logging, and Alerting

Sets up Prometheus (metrics + alerts), Grafana (dashboards), ClickHouse (log storage), and Vector (log collector). Optionally connect Slack and PagerDuty for alerts.

export CLICKHOUSE_USER=vector
export CLICKHOUSE_PASSWORD=vectorpass
export SLACK_WEBHOOK_URL=   # Optional: Slack alerting
export PAGERDUTY_ROUTING_KEY=   # Optional: PagerDuty escalation
export ADMIN_PASSWORD=grafana # set strong password
bash src/scripts/eks/observability_setup.sh

End-to-End System Deployed

The deployment reproduces the ground-truth configuration as shown in the full demo, including semantic caching, OAuth-authenticated per-user rate limiting, hybrid retrieval, and the complete RAG inference pipeline.

URL	Service
`https://rag.<DOMAIN>`	RAG Chat UI (Google/Microsoft sign-in)
`https://argocd.<DOMAIN>`	Argo CD (GitOps dashboard)
`https://grafana.<DOMAIN>`	Grafana (observability dashboards)

Note: OIDC allows all Google and Microsoft accounts by default. For production, restrict access with GOOGLE_ALLOWED_DOMAINS and MICROSOFT_ALLOWED_TENANT_IDS.

Optional — Test Alerting and Disaster Recovery

Simulate a complete Qdrant outage to verify Slack alerts fire and data can be restored from S3 backups.

# 1. Pause ArgoCD sync to prevent auto-healing during the test
kubectl patch application qdrant -n argocd --type='merge' \
  -p '{"spec": {"syncPolicy": null}}'

# 2. Delete Qdrant and its persistent volumes
kubectl delete pvc -n qdrant --all --ignore-not-found=true
kubectl delete namespace qdrant --ignore-not-found=true

# 3. Wait for the QdrantDown alert to trigger (~3 minutes)
echo "Waiting for QdrantDown alert..."
sleep 180

# 4. Re-enable ArgoCD — it recreates Qdrant with fresh volumes
kubectl patch application qdrant -n argocd --type='merge' \
  -p '{"spec": {"syncPolicy": {"automated": {"prune": true, "selfHeal": true}}}}'
echo "Waiting for ArgoCD to recreate Qdrant..."
sleep 180

# 5. Confirm Qdrant is back but empty
kubectl port-forward -n qdrant svc/qdrant 6333:6333 &>/dev/null & sleep 3
echo "Points before restore:"
curl -s http://localhost:6333/collections/default_rag_collection1 | jq '.result.points_count // 0'
kill %1 2>/dev/null || true

# 6. Restore from the latest S3 backup
export DATA_S3_BUCKET=$DATA_S3_BUCKET
export QDRANT_BACKUP_S3_PREFIX="qdrant/backups/"
bash src/scripts/backups_and_restore.sh restore

# 7. Verify data recovery
sleep 10
kubectl port-forward -n qdrant svc/qdrant 6333:6333 &>/dev/null & sleep 3
echo "Points after restore:"
curl -s http://localhost:6333/collections/default_rag_collection1 | jq '.result.points_count // 0'
kill %1 2>/dev/null || true

What this validates:

Alertmanager fires QdrantDown and notifies Slack
ArgoCD self-heals infrastructure when re-enabled
S3 backups are restorable with point-count parity

Cleanup

To tear down the entire infrastructure and avoid ongoing costs:

Note: Karpenter manages AWS resources (EC2 instances, VPC tags) outside Terraform state. If terraform destroy hangs, manually delete the VPC and verify no orphaned EC2 instances remain.

# 1. Remove workloads to trigger Karpenter scale-down
kubectl delete ns inference --ignore-not-found
kubectl delete ns fastembed --ignore-not-found
sleep 600 

# 2. Destroy Cloudflare resources
bash src/infra/terraform/cloudflare/run.sh --destroy

# 3. Destroy AWS infrastructure
bash src/infra/terraform/aws/run.sh --destroy --env staging --yes-delete

Post-cleanup verification:

Confirm no RAG8s-specific EC2 instances remain in the TF_VAR_region
Verify the EKS cluster and associated security groups are removed
Run python3 src/scripts/eks/force_delete.py as a last resort for orphaned resources

Name		Name	Last commit message	Last commit date
.devcontainer		.devcontainer
.github/workflows		.github/workflows
src		src
.dockerignore		.dockerignore
.gitignore		.gitignore
.gitleaks.toml		.gitleaks.toml
.pre-commit-config.yaml		.pre-commit-config.yaml
.trivyignore		.trivyignore
README.md		README.md
ruff.toml		ruff.toml

Latest commit

History

Folders and files

Repository files navigation