MemoraAI

AI systems forget everything.

MemoraAI is a memory layer that helps AI assistants build long-term understanding of users by extracting, storing, retrieving, and injecting relevant context into future interactions.

The goal is not to store more information. The goal is to surface the right information at the right time.

---

⚠️ The Problem

Stateless Large Language Models (LLMs) treat every interaction as if it is the first time they are meeting the user. Existing solutions try to solve this by:

1. Shoving everything into the context window: This leads to exponential token costs, latency spikes, and the "lost in the middle" retrieval degradation where LLMs fail to recall facts placed in large contexts.
2. Naive RAG (Retrieval-Augmented Generation): Standard semantic retrieval frequently fetches irrelevant noise or fails to resolve exact keywords, acronyms, or references.
3. Flat chat logs: Treating memory merely as a raw database of past messages without structural representation, classification, or long-term user preferences extraction.

---

🏛️ System Architecture

MemoraAI is deployed as a serverless, decoupled microservices suite designed for cloud elasticity, low-latency execution, and granular observability.

📘 System Design & Deep Dive: For a comprehensive, pin-to-pin breakdown of the system components, data flow, memory lifecycle, and hybrid retrieval specifications, refer Below Diagrams

🧩 Functional System Architecture

graph TB
    classDef client fill:#3b82f6,stroke:#1d4ed8,stroke-width:2px,color:#fff
    classDef frontend fill:#10b981,stroke:#047857,stroke-width:2px,color:#fff
    classDef gateway fill:#f59e0b,stroke:#b45309,stroke-width:2px,color:#fff
    classDef ai fill:#8b5cf6,stroke:#6d28d9,stroke-width:2px,color:#fff
    classDef db fill:#ec4899,stroke:#be185d,stroke-width:2px,color:#fff
    classDef ext fill:#6b7280,stroke:#374151,stroke-width:2px,color:#fff

    User["Client Browser"]:::client
    NextJS["Next.js Frontend - Vercel"]:::frontend
    NestJS["NestJS API Gateway - GCP Cloud Run"]:::gateway
    FastAPI["FastAPI AI Service - GCP Cloud Run"]:::ai

    User -->|HTTPS| NextJS
    NextJS -->|"REST + x-request-id"| NestJS

    subgraph GW_TASKS["NestJS Gateway Middleware"]
        LogMW["LoggingMiddleware - AsyncLocalStorage"]:::gateway
        ZodVal["Zod Schema Validation"]:::gateway
        MetricsAuth["Metrics Token Check"]:::gateway
    end

    NestJS --> LogMW
    NestJS --> ZodVal
    NestJS --> MetricsAuth
    NestJS -->|"Axios REST + x-request-id"| FastAPI

    subgraph AI_CORE["FastAPI AI Engine"]
        Router["Adaptive Router Classifier"]:::ai
        Reformulate["Query Reformulator - Pronoun Resolution"]:::ai
        DirectLLM["Direct Prompt Generator"]:::ai
        MemorySearch["Memory-Aware Retrieval"]:::ai
    end

    FastAPI --> Router
    Router -->|"factual / analytical"| Reformulate
    Router -->|conversational| DirectLLM
    Router -->|memory_related| MemorySearch

    subgraph RAG_PIPE["Hybrid RAG Pipeline"]
        VectorSearch["pgvector Cosine Search"]:::ai
        BM25Search["PostgreSQL BM25 FTS"]:::ai
        RRF["Reciprocal Rank Fusion"]:::ai
        CrossEncoder["Cross-Encoder Reranker - Top 5 Cap"]:::ai
    end

    Reformulate -->|"Standalone Query"| VectorSearch
    Reformulate -->|"Standalone Query"| BM25Search
    VectorSearch --> RRF
    BM25Search --> RRF
    RRF --> CrossEncoder

    subgraph DATA["Data Layer - Supabase PostgreSQL"]
        VectorDB[("pgvector Chunks DB")]:::db
        STM[("short_term_memory")]:::db
        LTM[("long_term_memory")]:::db
        EM[("episodic_memory")]:::db
    end

    VectorSearch -->|"Cosine similarity"| VectorDB
    BM25Search -->|"GIN ts_rank"| VectorDB

    subgraph EXTERNAL["External Services"]
        Gemini["Google Gemini API"]:::ext
        RerankerModel["Cross-Encoder Model Cache"]:::ext
    end

    CrossEncoder --> RerankerModel
    FastAPI -->|"Fetch last 5 turns"| STM
    FastAPI -->|"Async Fact Extraction"| LTM
    FastAPI -->|"Session Summarization"| EM

    DirectLLM --> Gemini
    CrossEncoder -->|"Top chunks + context"| Gemini
    MemorySearch --> Gemini
    Gemini -->|"Stream tokens"| FastAPI
    FastAPI -->|"Stream chunks"| NestJS
    NestJS -->|"SSE Stream"| NextJS

    subgraph OBS["Observability"]
        MetricsDash["Telemetry Dashboard - /metrics"]:::frontend
        StructLog["structlog Correlation Tracing"]:::gateway
    end

    MetricsDash -->|"x-metrics-token auth"| NestJS
    LogMW --> StructLog
    FastAPI --> StructLog

Loading

🛰️ Request Context Flow & Distributed Tracing

1. Trace Propagation: Client requests hit the Next.js Frontend hosted on Vercel. A unique x-request-id transaction trace is generated (or captured) and forwarded in all API requests to the NestJS Backend Gateway running on GCP Cloud Run.
2. NestJS Context Tracking: The NestJS Gateway handles global rate-limiting (via @nestjs/throttler mapping 30 req/min per IP), request schema validation, and metrics dashboard token validation. Using AsyncLocalStorage, NestJS propagates the trace token downstream to the AI microservice.
3. FastAPI Observability Middleware: The core FastAPI AI Service running on GCP Cloud Run extracts the x-request-id header in its ObservabilityMiddleware, binding it to python structlog contexts. Every database query, exception, or external LLM request is logged alongside this trace ID.
4. Data Layer Ingress: Queries are whitelisted inside FastAPI's DB connector (db/connection.py) against known database tables (chunks, documents) to block SQL injection vectors on dynamic queries, before executing against Supabase PostgreSQL.

🔄 End-to-End Hybrid RAG & Memory Sequence

sequenceDiagram
    autonumber
    actor User as Client Browser
    participant FE as Next.js Frontend
    participant GW as NestJS Gateway
    participant AI as FastAPI AI Service
    participant DB as Supabase PostgreSQL
    participant LLM as Google Gemini API

    User->>FE: Submits query ("What did he do there?")
    FE->>GW: POST /api/v1/query (JSON)
    Note over GW: LoggingMiddleware extracts/generates x-request-id<br/>Runs in AsyncLocalStorage context
    
    GW->>AI: POST /query (Forward x-request-id header)
    Note over AI: ObservabilityMiddleware binds request_id to structlog
    
    AI->>AI: AdaptiveRouter classifies query as FACTUAL
    
    Note over AI: Query Reformulator resolves pronouns ("he", "there")<br/>Fetches last 4 messages from STM history
    AI->>DB: Query last 4 messages (short_term_memory)
    DB-->>AI: Conversation history turns
    AI->>LLM: Rewrite follow-up query into standalone form
    LLM-->>AI: Standalone query ("What did Ganesh do at Annam.AI?")
    
    par Vector Search
        AI->>LLM: Generate query embeddings (text-embedding-004)
        LLM-->>AI: Vector array (3072 dims)
        AI->>DB: Query cosine similarity (pgvector threshold > 0.7)
        DB-->>AI: Top 10 chunks with Cosine Scores and Vector Rank
    and BM25 Search
        AI->>DB: Query FTS ts_rank (GIN Index search)
        DB-->>AI: Top 10 chunks with BM25 Scores and lexical Rank
    end
    
    AI->>AI: Execute Reciprocal Rank Fusion (RRF)<br/>Fuse Vector and BM25 positions
    
    AI->>AI: Feed fused candidates to Local Cross-Encoder<br/>Rerank based on joint token attention (Capped at top 5)
    
    par Memory Context Retrieval
        AI->>DB: Select last 5 messages (Short-Term Memory)
        DB-->>AI: Active conversational log window
        AI->>DB: Query matching long-term user facts
        DB-->>AI: High-confidence preference records
    end

    AI->>AI: Assemble final grounded prompt<br/>(Context Chunks + STM + LTM + System Rules)
    
    AI->>LLM: Request completion (Gemini 2.5 Flash)
    LLM-->>AI: Streamed tokens response
    
    alt Asynchronous Memory Job
        Note over AI: Every 5 messages, spawn BG task<br/>Extract fresh facts, upsert to DB
        AI->>DB: INSERT ... ON CONFLICT DO UPDATE (long_term_memory)
    end

    AI-->>GW: Forward chunk stream + x-request-id
    GW-->>FE: Stream Server-Sent Events (SSE)
    FE-->>User: Render stream in Chat and populate Explainability Panels

Loading

---

🔍 Multi-Stage Hybrid Search & Reranking Pipeline

To maximize both recall (finding all candidate matches) and precision (ensuring the most relevant chunks are at the top), MemoraAI utilizes a multi-stage retrieval pipeline (detailed in the End-to-End Hybrid RAG & Memory Sequence diagram above):

1. Query Reformulation: If the user query contains ambiguous pronouns (e.g., "he", "that", "there"), the system extracts context from the last 4 messages in the short-term memory window and triggers a lightweight Gemini API call to rewrite the query into a self-contained, noun-based search query.
2. pgvector Semantic Search: Embeds the query into a 3072-dimensional vector using text-embedding-004. Runs cosine distance (<=> operator) query against the database, filtering chunks above the configured SIMILARITY_THRESHOLD.
3. Lexical Full-Text Search (FTS): Formats the query into English lexical tokens and executes a GIN-indexed keyword search using ts_rank (equivalent to BM25 relevance ranking) to match exact terminology, acronyms, or numbers.
4. Reciprocal Rank Fusion (RRF): Blends the ranks of the vector and lexical search runs using the formula: $$\text{RRF Score} = \sum_{m \in {\text{Vector}, \text{FTS}}} \frac{1}{60 + \text{Rank}_m}$$
5. Cross-Encoder Reranking (CPU-Optimized): Takes the RRF candidates and processes them together with the query through the ms-marco-MiniLM-L-6-v2 transformer. The model uses self-attention to calculate deep token-level similarity. To optimize CPU latency on Cloud Run:
   • Reranking is skipped entirely if only 1 candidate is retrieved.
   • The reranker input is capped at the top 5 candidates.
   • This optimization reduces RAG request latency from ~4.5 seconds down to 1.2 seconds.
   • Docker Optimization: The 150MB Reranker weights are cached directly inside the Docker image during the build stage, reducing cold-start container initialization in GCP Cloud Run to 1.5s.

---

🧠 Memory Lifecycle

MemoraAI models human memory using a three-tier system: Short-Term, Long-Term, and Episodic.

┌────────────────────────────────────────────────────────┐
│                    MEMORY LIFECYCLE                    │
├────────────────────────────────────────────────────────┤
│                                                        │
│  [Ingestion / Input]                                  │
│         │                                              │
│         ▼                                              │
│  [Short-Term Memory] ──► (Active Session Context)      │
│         │                                              │
│         ├─► (Every 5 messages: Extract facts)          │
│         ▼                                              │
│  [Long-Term Memory] ──► (Persistent User Facts)        │
│         │                                              │
│         ├─► (On New Session/Clear: Summarize)          │
│         ▼                                              │
│  [Episodic Memory]  ──► (Chronological Session Events) │
└────────────────────────────────────────────────────────┘

1. Ingestion (Active Interaction):
   • Files (PDFs), URL contents, or plain texts are parsed, chunked via a token-based sliding window (700 tokens, 100 overlap), vectorized, and saved globally in the documents and chunks tables.
   • User chat inputs are immediately committed to the short_term_memory table.
2. Short-Term Context Retention:
   • On every message query, the last 5 conversation pairs for the specific session_id are injected into the prompt context to maintain immediate dialog flow.
3. Long-Term Fact Extraction (LTM):
   • For every 5 messages exchanged, a background extraction event executes. The recent chat logs are evaluated by Gemini using structured JSON output schemas to distill personal user facts (e.g., "user works as a backend developer", "user prefers Python") into the long_term_memory table.
   • Relevant facts are retrieved and prepended to the system prompt dynamically on all future interactions.
4. Episodic Summarization:
   • When a session is closed (via clicking "New Session" or "Clear Chat" in the UI), the system calls POST /api/v1/memory/episodic/summarize.
   • The full chat log is summarized, and key topics and sentiment metrics are extracted and saved to episodic_memory before clearing short-term tables.

---

⚖️ Tradeoffs

Engineering a production memory system requires balancing performance, cost, and complexity:

• Periodic vs. Inline Extraction: We run Long-Term Memory (LTM) fact extraction asynchronously in the background every 5 messages rather than inline during every user query. Inline extraction increases user latency by ~1.5s and bloats token expenses, whereas periodic asynchronous extraction keeps user interactions sub-second while maintaining semantic updates.
• Bi-Encoder Embeddings vs. Cross-Encoder Rerankers: Bi-encoder embeddings represent documents as static vectors for fast vector indexing, but cannot model complex query-context interactions. Cross-Encoders calculate deep cross-attention at runtime for superior accuracy but are compute-heavy. We run Vector+FTS first to retrieve 10 candidates, then run the Cross-Encoder only on those 10 candidates to maintain sub-100ms pipeline latencies while achieving high precision.
• Global Supabase SQL Database vs. Session-Isolated Storage: Ingested documents are stored globally in PostgreSQL rather than inside session scopes. This allows cross-session discovery (any session can query any past document) but requires extra care to prevent cross-user data leakage by enforcing user-id filters at the database query level.

---

💡 Lessons Learned

• LLM JSON Extraction Fragility: LLMs occasionally output unescaped quotes or nested markdown lists that crash standard parsers. In the offline evaluation engine (evaluator.py), we built an index-slicing backup parser (_parse_judge_json) that isolates brackets and keys to robustly extract scores and reasoning even if the JSON is malformed.
• Vector Dimension Scaling Bottleneck: Upgrading the embedding model to a 3072-dimensional output caused Supabase pgvector indexing scripts to crash because pgvector ivfflat/hnsw indexes restrict vector dimensions to a maximum of 2000. We learned to disable the vector indexes to allow 3072-dimension vectors while using exact nearest-neighbor search, which remains sub-millisecond for our database size.
• Port Collision and Proxy Loops: Running NestJS and Next.js both on port 3000 on Windows caused an infinite proxy loop because Next.js bound to IPv6 (::1) while NestJS bound to IPv4 (0.0.0.0). Standardizing the frontend on port 3001 and configuring explicit health and proxy re-writes in next.config.js fixed this loop.

---

🔮 Future Improvements

1. Entity-Relation Graph Memory: Move LTM from flat key-value facts to a Neo4j Graph database representation. This would allow the agent to understand multi-degree relationships between concepts (e.g. knowing that "React" is related to "Next.js").
2. Relevance Decay & Memory Pruning: Implement a chronological decay index ($e^{-\lambda t}$) on user facts. Facts that haven't been referenced in months or are contradicted by newer inputs would automatically prune or archive themselves.
3. Adaptive Chunking: Replace fixed sliding-window chunking with semantic chunking that detects changes in topics, headings, or paragraph structures, preserving context bounds more cleanly.

---

⚙️ Service Ports & Verification Commands

Verify system health and metrics endpoints post-deployment:

Health Verification

# Core AI Service
curl http://localhost:8000/api/v1/health

# Backend Gateway
curl http://localhost:3000/health

Telemetry & Metrics Scrape

# Scrape FastAPI core metrics
curl http://localhost:8000/metrics

# Fetch aggregated dashboard JSON (requires Metrics Token)
curl -H "x-metrics-token: YOUR_METRICS_TOKEN" http://localhost:3000/api/v1/health/metrics/dashboard

Run RAG Pipeline Quality Evaluation

docker compose exec ai-service python run_eval.py

This command runs the automated quality suite (LLM-as-a-judge scorers for Faithfulness, Relevance, and Recall) and writes the results to docs/RAG_EVALUATION_REPORT.md.

---

⚙️ Tech Stack

Layer	Technology	Description
Frontend	Next.js 15, TypeScript, Tailwind CSS, shadcn/ui, Zustand, Framer Motion	Deployed on Vercel; includes a secure, authenticated /metrics latency dashboard
Backend Gateway	NestJS, TypeScript, Express, Zod, Throttler Guard, NestJS Axios	Deployed on GCP Cloud Run; orchestrates request tracing, rate-limiting, and secured proxy dashboard endpoints
AI Microservice	Python 3.11, FastAPI, asyncpg, Pydantic, Structlog, SentenceTransformers	Deployed on GCP Cloud Run; houses the hybrid search, CPU-optimized reranker, and async memory pipeline
Database	Supabase PostgreSQL + pgvector	Managed relational vector database; contains STM/LTM/EM tables and GIN indices for full-text search
AI Foundations	Google Gemini 2.5 Flash, text-embedding-004, ms-marco-MiniLM-L-6-v2	Advanced LLM generator, 3072-dimension embeddings, and cross-encoder rerank model

---