A production-oriented, containerized Retrieval-Augmented Generation (RAG) platform featuring strict multi-tenant isolation, role-based access control (RBAC), hybrid dense-sparse vector search, self-correcting agentic workflows, OWASP defensive guardrails, and real-time distributed tracing.
Figure 1: Production Operator Dashboard featuring tenant-scoped streaming chat, role-based document ingestion, and live telemetry.
This platform is engineered to solve the most critical enterprise challenges in Generative AI: cross-tenant data leakage, unauthorized privilege escalation, hallucination in knowledge retrieval, and adversarial prompt injection attacks.
| Capability | Enterprise Implementation | Production Impact |
|---|---|---|
| Multi-Tenant Isolation | Composite key (tenant_id, username) in PostgreSQL + Mandatory metadata filters in Qdrant |
Zero cross-tenant data leakage at database, vector, and cache layers. |
| Granular RBAC | Role-filtered retrieval (admin, analyst, viewer) |
Restricted users cannot retrieve confidential chunks even within the same tenant. |
| Credential Lifecycle | Tenant-scoped failed-login lockout/reset, active-user checks, and Redis-backed access-token revocation via /auth/logout |
Limits brute-force attempts without account-enumeration details; revoked tokens stop working across replicas when Redis is available. |
| Defensive AI & NeMo Guardrails | NVIDIA NeMo Guardrails (Colang input/output rails) + OWASP injection regexes + Presidio PII Redaction | Malicious prompts are blocked and PII is redacted before touching downstream LLMs or vector stores. |
| Hybrid Search (Dense + Sparse) | FastEmbed BGE (384d) + BM25 Sparse Tokenizer + Qdrant Reciprocal Rank Fusion (RRF) |
Superior retrieval accuracy for both semantic intent and exact hardware SKUs / legal codes. |
| Cross-Encoder Reranking | Cohere Rerank API (rerank-v3.5) cross-encoder re-ordering |
Re-scores hybrid retrieval candidates with high semantic precision before passing to LLM. |
| Corrective Agentic RAG (CRAG) | Stateful LangGraph workflow with LLM-as-a-Judge grading and autonomous query reformulation | Bounded self-correction prevents hallucinations and loops when retrieval is weak. |
| Universal Parser & OCR | PDF, DOCX, XLSX, CSV, HTML, Code + Pytesseract OCR for scanned/image-based PDFs | Seamless multi-format knowledge extraction with automatic OCR fallback. |
| Async Background Ingestion | Dedicated POST /api/v1/ingest/async & GET /api/v1/ingest/status/{task_id} polling |
Non-blocking ingestion for large documents without HTTP gateway timeouts. |
| Stateful Persistence | PostgreSQL Checkpointing via AsyncPostgresSaver |
Multi-turn chat state persists safely across container restarts and horizontal scale. |
| Version-Scoped Caching | SHA-256 digested Redis keys with Tenant Version Epochs (tenant_ver:<tenant_id>) |
Sub-5ms cache hits; instant tenant-wide cache invalidation upon uploading new documents. |
| High-Throughput Inference | Groq Cloud integration with asynchronous SSE token streaming | Token generation at ultra-low latency. |
| Durable Ingestion Queue | Redis-backed queue with processing retention and acknowledgement | Async ingestion can be consumed across backend replicas instead of relying only on process memory. |
| Distributed Observability | In-memory span tracker + REST API telemetry + Native Langfuse Cloud sync | Real-time visibility into node-by-node latencies (retrieve_hybrid, rerank_cohere, grade, generate_llm). |
For the complete topology, data-isolation model, CRAG state graph, security controls, deployment design, and latency analysis, see ARCHITECTURE.md.
+---------------------------------------------+
| Streamlit Operator Dashboard (UI) |
| (Chat Interface + Observability Tabs) |
+---------------------------------------------+
│ (HTTP / SSE Stream)
▼
+─────────────────────────────────────────────────────────────────────────────────────────────────────────────────+
| FastAPI Gateway (Port 8000) |
| |
| [Security Headers] ──► [JWT / RBAC Auth] ──► [NVIDIA NeMo Guardrails / OWASP Regex] ──► [Presidio PII Redaction]|
+───────────────────────────────────────────────────┬─────────────────────────────────────────────────────────────+
│
┌────────────────────────────────┴────────────────────────────────┐
▼ ▼
+──────────────────────────+ +──────────────────────────+
| Redis Response Cache | | Distributed Tracing |
| (Scoped by Tenant+Role) | | (Spans & Langfuse Sync) |
+──────────────────────────+ +──────────────────────────+
│ (Cache Miss) │
└────────────────────────────────┬────────────────────────────────┘
│
▼
+──────────────────────────────────────────────────+
| LangGraph Corrective Agentic Engine |
| |
| [retrieve_hybrid] ──► [Cohere Reranker (v3.5)] |
| ▲ │ |
| │ ▼ |
| │ [grade_documents (LLM)] |
| │ │ |
| │ (Low Relevance) ▼ (Relevant) |
| [rewrite_query] ◄────── [decide_to_generate] |
| │ |
| ▼ |
| [generate_llm] |
+────────────────────────┬─────────────────────────+
│
┌────────────────────────────────┴────────────────────────────────┐
▼ ▼
+──────────────────────────+ +──────────────────────────+
| Qdrant Hybrid Vector | | PostgreSQL Checkpoints |
| Dense + BM25 Sparse RRF | | & Tenant User Registry |
+──────────────────────────+ +──────────────────────────+
To customize the documentation with your live screenshots, save your images in
docs/screenshots/matching the filenames below:
Figure 2: Administrator document upload with multi-select role permissions and automatic hybrid indexing.
Figure 3: Immediate pre-flight interception of adversarial prompt injection and DAN jailbreak attempts.
Figure 4: Real-time telemetry dashboard detailing sub-operation execution spans and latency percentiles.
- Docker Engine 24.0+ and Docker Compose v2
- A Groq API Key (https://console.groq.com)
- Python 3.11+ (if running tests locally)
git clone https://github.com/your-org/multi-tenant-agentic-rag-platform.git
cd multi-tenant-agentic-rag-platform
# Copy the environment template
cp .env.example .envEdit .env and set your secrets:
SECRET_KEY="generate-a-secure-random-32-char-secret-key"
GROQ_API_KEY="gsk_your_groq_api_key_here"
GROQ_MODEL="llama-3.3-70b-versatile"
BOOTSTRAP_ADMIN_USERNAME="admin_user"
BOOTSTRAP_ADMIN_PASSWORD="your-strong-password-under-72-characters"
BOOTSTRAP_ADMIN_TENANT_ID="tenant_alpha"
# Optional: Langfuse Cloud or Self-Hosted Telemetry Sync
LANGFUSE_PUBLIC_KEY=""
LANGFUSE_SECRET_KEY=""
LANGFUSE_HOST="https://cloud.langfuse.com"docker compose up --build -d| Service | URL | Purpose |
|---|---|---|
| Streamlit Dashboard | http://localhost:8501 |
Multi-tenant operator interface & tracing viewer |
| FastAPI Gateway | http://localhost:8000/docs |
Interactive Swagger API documentation |
| Readiness Probe | http://localhost:8000/ready |
Orchestrator health check (PostgreSQL, Qdrant, Redis) |
Docker Compose is the supported deployment for this repository. It runs one backend worker locally to avoid loading duplicate embedding models into memory. The root Dockerfile also runs the FastAPI service on the platform-provided PORT for container hosts such as Render.
The primary answer-generation model is Groq Cloud llama-3.3-70b-versatile. NeMo Guardrails is optional by configuration and, when enabled, uses a separate evaluator; it is not the primary generation model. The configured NeMo provider must have access to its model credentials. MFA and refresh-token flows are not exposed by the current API.
You can deploy the platform in either of two hosting models:
The repository includes a multi-service render.yaml Blueprint that automatically provisions both the FastAPI Backend Gateway and the Streamlit Frontend Operator UI within the same Render environment:
- Connect your repository to Render as a Blueprint.
- Render automatically builds both services using the multi-stage
Dockerfile. - The frontend service automatically discovers and links to the backend via
API_BASE_URL(fromService: host).
- Frontend: Deploy the repository on Streamlit Community Cloud with entrypoint
frontend/app.pyand requirements filefrontend/requirements.txt. - Backend: Deploy on Render (or any container host) as a Docker web service using the root
Dockerfileand readiness probe/ready. - Environment & Secrets Synchronization:
- In Streamlit Cloud settings or secrets, configure
API_BASE_URL=https://<your-render-backend-domain>/api/v1(the frontend automatically normalizes protocols and paths). - In Render backend environment variables, set
CORS_ORIGINS=https://<your-streamlit-app>.streamlit.app. - Provide managed datastores (Neon/Supabase PostgreSQL
POSTGRES_*, UpstashREDIS_URL, Qdrant CloudQDRANT_URL+QDRANT_API_KEY). - Configure model secrets (
GROQ_API_KEY,SECRET_KEY, and optionalCOHERE_API_KEY/LANGFUSE_*).
- In Streamlit Cloud settings or secrets, configure
For production environments, set REQUIRE_REDIS=true. The frontend includes live backend reachability and latency checks in the sidebar with automatic status notices during cold starts.
The platform includes a unit and regression test suite verifying tenant boundaries, RBAC isolation, and guardrails:
# Run unit tests
python -m pytest -q
# Output:
# 24 passedThe built-in evaluation harness uses Ragas for context precision and context recall, with measured mean and P95 latency reported alongside the quality scores. The bundled smoke fixture is deterministic in its inputs but requires a configured Groq key because Ragas evaluates retrieval quality with an LLM:
python -m scripts.evaluate_ragFor credible hiring or production evidence, replace the fixture with labeled tenant-safe queries and record Ragas results for dense-only, sparse-only, hybrid RRF, and hybrid-plus-reranking configurations. Report the dataset size, K, hardware, provider, and whether latency includes network time. Langfuse remains the request-level observability layer for tracing retrieval, reranking, grading, and generation latency.
The platform intentionally accepts additional latency for stronger security, retrieval quality, tenant isolation, and observability. Each component is measured separately so the critical path can be optimized without claiming an artificially low end-to-end number.
For a warm request with external reranking disabled, the current portfolio reference is approximately 2.2 seconds average for the retrieval-to-first-response path:
| Stage | Local target | Main variable |
|---|---|---|
| Request validation, JWT, and rate-limit lookup | ~0.05 s | Redis and database connection reuse |
| Dense + sparse query embedding | ~0.35 s | Warm CPU models and query length |
| Tenant-filtered Qdrant hybrid retrieval | ~0.45 s | Collection size and network round trip |
| Relevance grading | ~0.01 s | Heuristic grading; LLM grading is slower |
| Checkpoint and cache persistence | ~0.20 s | PostgreSQL/Redis health and connection reuse |
| Groq generation to first visible response | ~1.14 s | Provider queue, prompt size, and network |
| Warm-path average | ~2.20 s | Excludes cold starts and optional Cohere reranking |
The 2.2-second figure is a warm-path reference budget, not a universal guarantee. Cold starts, hosted database poolers, Qdrant Cloud, NeMo checks, Cohere reranking, model queueing, and long completions can increase total latency. The telemetry API reports each span and should be used to replace this reference with measurements from the target deployment.
| Test Scenario | Query / Action | Expected Result | Live Result | Status |
|---|---|---|---|---|
| Admin Login | Auth token request (tenant_alpha, admin) |
JWT token with tenant/role claims | Token issued successfully | ✅ PASS |
| Cross-Tenant Guard | Tenant Alpha provisions user for Tenant Beta | 403 Forbidden | Blocked with 403 status | ✅ PASS |
| Confidential Ingest | Upload admin-only financial memo | Ingested with allowed_roles=['admin'] |
Indexed with Dense + BM25 | ✅ PASS |
| Prompt Injection | Submit DAN / override instructions | 400 Bad Request / Policy block | Intercepted pre-flight | ✅ PASS |
| PII Redaction | Query with SSN and Email | In-memory redaction before LLM | PII masked, clean answer streamed | ✅ PASS |
| RBAC Isolation | Viewer queries confidential admin memo | 0 chunks returned / safe fallback | "No authorized records found" |
✅ PASS |
| RBAC Access | Admin queries confidential admin memo | Relevant chunks retrieved & synthesized | Accurate answer generated | ✅ PASS |
| Cross-Tenant Isolation | Tenant Beta queries Tenant Alpha data | Complete data isolation (Zero leak) | "No authorized records found" |
✅ PASS |
| Hybrid Search (SKU) | Exact keyword search #K8S-9921 |
Dense + BM25 RRF fusion match | Retrieved exact SKU specifications | ✅ PASS |
| Response Cache | Repeat identical query in same thread | Version-scoped Redis cache hit (cached: true) |
Covered by implementation and runtime smoke checks | ✅ PASS |
[ Incoming Request ]
│
▼
[ NVIDIA NeMo Guardrails / OWASP Pattern Inspector ]
│ ──► Matches Jailbreak / Policy Violation? ──► YES ──► Raise HTTP 400 & Log Security Audit Event
│ (Clean)
▼
[ Presidio PII Redaction Engine ] ──► Masks Emails, Phones, SSNs, Credit Cards, JWTs, API Keys
│ (Sanitized)
▼
[ Hybrid Vector Retrieval (Qdrant) ──► Cohere Cross-Encoder Reranker ──► LLM-as-a-Judge Grading ]
│ (Answer Generated)
▼
[ NeMo Output Guardrails & PII Scrubbing ] ──► Final SSE Token Stream
- NVIDIA NeMo Guardrails (Input & Output Rails): Configured via Colang flows in
guardrails/rails.coto enforce conversational boundaries, prevent prompt injections, stop roleplaying jailbreaks, and sanitize output. - Deterministic & Presidio PII Sanitization: Redacts sensitive patterns in-memory (
[REDACTED_EMAIL],[REDACTED_SSN],[REDACTED_CARD], etc.) using Microsoft Presidio and optimized regex scrubbers. - Structured Audit Events: Emits immutable JSON security logs (
auth.login,document.ingest) recording timestamps, IP, tenant ID, and user ID without logging raw credentials or prompts.
Every request automatically creates a distributed trace containing detailed sub-operation spans:
{
"trace_id": "df1202c0-c7a0-4434-a9fb-f6e6cd18ba5d",
"tenant_id": "tenant_alpha",
"user_id": "admin_user",
"total_duration_ms": 2233.86,
"cache_hit": false,
"spans": [
{ "name": "retrieve_hybrid", "duration_ms": 699.25, "status": "ok", "metadata": { "retrieved_count": 8 } },
{ "name": "rerank_cohere", "duration_ms": 182.40, "status": "ok", "metadata": { "reranked_count": 5 } },
{ "name": "grade_documents", "duration_ms": 110.15, "status": "ok", "metadata": { "grader_type": "llm_as_judge", "is_relevant": true } },
{ "name": "generate_llm", "duration_ms": 1404.70, "status": "ok", "metadata": { "response_length": 1686 } }
]
}- View live trace breakdowns and P95 latency percentiles directly inside the Streamlit UI or query
/api/v1/telemetry/traces. Trace deletion is restricted to tenant administrators. - Set
LANGFUSE_PUBLIC_KEYandLANGFUSE_SECRET_KEYto seamlessly stream traces to Langfuse Cloud or self-hosted Langfuse.
├── app/
│ ├── agents/ # LangGraph workflow, nodes, and LLM routers
│ │ ├── graph.py # Corrective Agentic RAG graph with LLM grading & Cohere rerank
│ │ └── llm.py # Groq Cloud chat model client
│ ├── api/ # FastAPI routes and schemas
│ │ └── routes.py # Auth, Ingest (Sync & Async), Streaming Chat, Telemetry
│ ├── core/ # Core security, auth, and telemetry infrastructure
│ │ ├── audit.py # Structured JSON audit logging
│ │ ├── auth.py # Multi-tenant JWT auth & OIDC JWKS validator
│ │ ├── rerank.py # Cohere cross-encoder reranker client
│ │ ├── security.py # NVIDIA NeMo Guardrails, OWASP injection & PII redaction
│ │ └── tracing.py # Distributed span collector & Langfuse export
│ ├── db/ # Qdrant hybrid vector store integration
│ │ └── qdrant.py # Named dense+sparse collection & RRF hybrid search
│ ├── ingestion/ # Multi-format document parser
│ │ └── parser.py # PDF, DOCX, XLSX, HTML, Code & Pytesseract OCR
│ ├── migrations/ # Versioned PostgreSQL schema migrations
│ ├── config.py # Pydantic v2 application settings
│ ├── database.py # Async PostgreSQL connection pool & checkpointer
│ ├── main.py # FastAPI application & lifespan pre-warming
│ └── redis_client.py # Async Redis client & tenant cache versioning
├── frontend/
│ └── app.py # Streamlit operator dashboard & telemetry viewer
├── scripts/
│ ├── evaluate_rag.py # Ragas retrieval evaluation runner
│ └── __init__.py
├── scratch/
│ └── check_documents.py # Local document inspection utility
├── guardrails/ # NVIDIA NeMo Guardrails configuration
│ ├── config.yml # Model & rail definitions
│ ├── prompts.yml # Self-check prompt templates
│ └── rails.co # Colang flow security definitions
├── tests/ # Automated unit and regression test suite
├── .github/
│ └── workflows/ # CI and container delivery workflows
├── docs/
│ └── screenshots/ # Optional documentation screenshots
├── docker-compose.yml # Supported local deployment stack
├── Dockerfile # Multi-stage non-root container build
├── ARCHITECTURE.md # In-depth architectural design specification
└── requirements.txt # Version-constrained runtime dependencies
Distributed under the Apache 2.0 License. See LICENSE for the complete terms.