Micro RAG Demo: How to RAG Your RAG

TL;DR: Use RAG as a preprocessing tool to improve your main RAG system by removing redundancy and noise from documents before indexing.

The Problem

When building RAG (Retrieval-Augmented Generation) systems, we typically embed entire documents and store them in a vector database. But real-world documents have two major issues:

1. Low signal-to-noise ratio: Documents contain filler, repeated content, and redundant information
2. LLM token waste: We retrieve and process all that filler at inference time

This hurts both search quality and cost.

The Solution

Use RAG to improve your RAG! Before indexing documents into your main vector database, we:

1. Build a temporary vector index (FAISS) for each document's sentences
2. Construct a similarity graph to identify central vs. redundant sentences
3. Prune redundant content using greedy maximum coverage
4. Index only the information-dense cleaned documents

We pay a small upfront cost at indexing time for better retrieval quality and lower inference costs.

Results

Dataset: 1000 CNN/DailyMail articles
Compression: ~50% of sentences retained (e.g., 43 sentences → 16 kept)

Query: "Where were there magnitude 4 earthquakes?"

Approach	Top Result	Second Result
Naive RAG	Magnitude 4 quake (relevant)	Drifted to irrelevant content
Micro RAG	Magnitude 4 quake (relevant)	Second magnitude 4 quake (relevant)

The pruned index returns more focused, information-dense results.

---

Installation

pip install sentence-transformers faiss-cpu pandas datasets

Or run the first cell in the notebook to install in Google Colab:

!pip install -q sentence-transformers faiss-cpu pandas

---

Code Walkthrough

0. Setup and Imports

import numpy as np
import faiss
import pandas as pd
from sentence_transformers import SentenceTransformer
import re

We use:

• sentence-transformers: For embedding text
• FAISS: For fast similarity search (our "micro RAG")
• pandas: For result comparison
• datasets: To load real documents (CNN/DailyMail)

1. Text Splitting and Embedding

def split_into_sentences(text):
    """Simple sentence splitter using regex."""
    raw_sentences = re.split(r'(?<=[.!?])\s+', text.strip())
    sentences = [s.strip() for s in raw_sentences if len(s.strip()) > 0]
    return sentences

# Load embedding model
model_name = "sentence-transformers/all-MiniLM-L6-v2"
embedder = SentenceTransformer(model_name)

def embed_texts(texts, batch_size=32):
    """Embed texts and return numpy array."""
    return embedder.encode(texts, batch_size=batch_size, 
                          show_progress_bar=False, 
                          convert_to_numpy=True)

Key Point: We split documents into sentences because we want to identify and remove redundant sentences, not entire paragraphs.

2. Load Real Documents

from datasets import load_dataset

# Load CNN/DailyMail news articles
dataset = load_dataset("cnn_dailymail", "3.0.0", split="train[:1000]")
raw_docs = [x["article"] for x in dataset]

# Filter and clean documents
docs = []
for d in raw_docs:
    text = d.strip()
    if len(text) < 1000:   # Skip very short docs
        continue
    if len(text) > 8000:   # Truncate very long docs
        text = text[:8000]
    docs.append(text)

We load 1000 real news articles to demonstrate the technique on realistic data.

3. Build Naive RAG Index (Baseline)

# Embed full documents as-is
naive_embeddings = embed_texts(docs)

# Normalize embeddings for cosine similarity
naive_norms = np.linalg.norm(naive_embeddings, axis=1, keepdims=True) + 1e-10
naive_embeddings_norm = naive_embeddings / naive_norms

# Create FAISS index
d = naive_embeddings_norm.shape[1]
naive_index = faiss.IndexFlatIP(d)  # Inner product = cosine sim with normalized vectors
naive_index.add(naive_embeddings_norm.astype("float32"))

# Store metadata
naive_meta = [
    {"doc_id": i, "text": docs[i]}
    for i in range(len(docs))
]

This is the standard RAG approach: embed full documents and index them.

4. Build Micro RAG Index (The Magic!)

This is where we use RAG to improve RAG. For each document:

K_NEIGHBORS = 5            # top-K neighbors per sentence (local graph)
COVERAGE_BUDGET_PCT = 0.5  # Keep ~50% of sentences

cleaned_docs = []

for doc_id, doc in enumerate(docs):
    # Step 1: Split into sentences
    sentences = split_into_sentences(doc)
    n = len(sentences)
    
    # Step 2: Embed all sentences
    sent_embs = embed_texts(sentences)
    norms = np.linalg.norm(sent_embs, axis=1, keepdims=True) + 1e-10
    sent_embs_norm = sent_embs / norms

Key Insight: Instead of comparing every sentence to every other sentence (O(N²)), we use FAISS to find top-K neighbors (O(N)):

    # Step 3: Build transient FAISS index (THIS IS THE MICRO RAG!)
    dim = sent_embs_norm.shape[1]
    temp_index = faiss.IndexFlatIP(dim)
    temp_index.add(sent_embs_norm.astype("float32"))
    
    # Step 4: For each sentence, get top-K similar neighbors
    k = min(K_NEIGHBORS, n)
    sims, idxs = temp_index.search(sent_embs_norm.astype("float32"), k)
    
    # Build neighbor lists with similarity weights
    neighbors = []
    for i in range(n):
        nbrs = []
        for sim_val, j in zip(sims[i], idxs[i]):
            j = int(j)
            nbrs.append((j, float(sim_val)))
        neighbors.append(nbrs)

Now we have a similarity graph where each sentence knows its top-K most similar neighbors.

Greedy Maximum Coverage

We select sentences that best "cover" the document's semantic space:

    # Step 5: Greedy maximum coverage
    coverage_budget = max(1, int(COVERAGE_BUDGET_PCT * n))
    
    covered = np.zeros(n, dtype=bool)      # Which sentences are "covered"
    selected_mask = np.zeros(n, dtype=bool) # Which sentences we've selected
    selected_indices = []
    
    while len(selected_indices) < coverage_budget:
        best_i = None
        best_gain = -1.0
        
        # Find the sentence that covers the most uncovered content
        for i in range(n):
            if selected_mask[i]:
                continue
            
            gain = 0.0
            for j, w in neighbors[i]:
                if not covered[j]:
                    gain += w  # Weight by similarity
            
            if gain > best_gain + 1e-8:
                best_gain = gain
                best_i = i
        
        if best_i is None or best_gain <= 0:
            break
        
        # Select this sentence
        selected_indices.append(best_i)
        selected_mask[best_i] = True
        
        # Mark neighbors as covered
        for j, w in neighbors[best_i]:
            covered[j] = True

How it works:

• Each sentence "covers" itself and its similar neighbors
• We greedily select sentences that cover the most uncovered content
• Sentences similar to already-selected ones contribute less
• Redundant sentences naturally get pruned

    # Step 6: Rebuild document from selected sentences (in original order)
    keep_indices = sorted(selected_indices)
    cleaned_sentences = [sentences[i] for i in keep_indices]
    cleaned_doc = " ".join(cleaned_sentences)
    cleaned_docs.append(cleaned_doc)

Finally, index the cleaned documents:

# Embed cleaned documents
pruned_embeddings = embed_texts(cleaned_docs)
pruned_norms = np.linalg.norm(pruned_embeddings, axis=1, keepdims=True) + 1e-10
pruned_embeddings_norm = pruned_embeddings / pruned_norms

# Create FAISS index
pruned_index = faiss.IndexFlatIP(pruned_embeddings_norm.shape[1])
pruned_index.add(pruned_embeddings_norm.astype("float32"))

pruned_meta = [
    {"doc_id": i, "text": cleaned_docs[i]}
    for i in range(len(cleaned_docs))
]

5. Search and Compare

def search_index(query, index, embeddings_norm, meta, top_k=5):
    """Search a FAISS index and return top-k results."""
    q_emb = embed_texts([query])[0]
    q_emb = q_emb / (np.linalg.norm(q_emb) + 1e-10)
    sims, idxs = index.search(q_emb.reshape(1, -1).astype("float32"), top_k)
    
    results = []
    for rank, (i, s) in enumerate(zip(idxs[0], sims[0]), start=1):
        row = meta[int(i)].copy()
        row["rank"] = rank
        row["score"] = float(s)
        results.append(row)
    return results

def compare_naive_vs_pruned(query, top_k=5):
    """Compare naive and pruned retrieval side-by-side."""
    naive_results = search_index(query, naive_index, naive_embeddings_norm, 
                                 naive_meta, top_k=top_k)
    pruned_results = search_index(query, pruned_index, pruned_embeddings_norm, 
                                  pruned_meta, top_k=top_k)
    
    # Format results in a comparison table
    rows = []
    for r in range(top_k):
        naive_row = naive_results[r] if r < len(naive_results) else {}
        pruned_row = pruned_results[r] if r < len(pruned_results) else {}
        
        rows.append({
            "Rank": r + 1,
            "Naive_score": naive_row.get("score", None),
            "Naive_preview": naive_row.get("text", "")[:260] + "...",
            "Pruned_score": pruned_row.get("score", None),
            "Pruned_preview": pruned_row.get("text", "")[:260] + "...",
        })
    
    return pd.DataFrame(rows)

6. Run Queries

query = "where were there magnitude 4 earthquakes"
df_compare = compare_naive_vs_pruned(query, top_k=5)
display(df_compare)

You'll see that the pruned index returns more focused, relevant results!

---

Key Insights

Why This Works

1. RAG is a tool, not just a product: We use vector similarity search to solve the sentence deduplication problem
2. O(N) instead of O(N²): FAISS gives us approximate nearest neighbors efficiently
3. Pay upfront, save later: Small cost at indexing time → better retrieval + lower inference costs
4. Graph-based coverage: The similarity graph captures which sentences are redundant
5. Preserve order: We keep sentences in their original order so the document still reads naturally

When to Use This

• Large documents with redundancy (news, reports, legal docs)
• Cost-sensitive applications (fewer tokens to LLM)
• Quality-focused retrieval (better signal-to-noise)

Customization

Tune these parameters for your use case:

• K_NEIGHBORS: How many similar sentences to consider (default: 5)
• COVERAGE_BUDGET_PCT: What % of sentences to keep (default: 0.5 / 50%)
• Coverage algorithm: Try other graph algorithms (PageRank, clustering, etc.)

---

Usage

Run in Google Colab

1. Open the microrag.ipynb notebook in Colab
2. Run cells sequentially
3. Experiment with different queries and datasets!

Run Locally

# Clone the repo
git clone https://github.com/YOUR_USERNAME/microrag-demo.git
cd microrag-demo

# Install dependencies
pip install sentence-transformers faiss-cpu pandas datasets

# Open notebook
jupyter notebook microrag.ipynb

---

Philosophy

RAG is more than a corporate product; it's a tool for success.

Don't treat RAG as a monolithic architecture. It's semantic similarity search—use it wherever you need to compare embeddings. Here we used it to preprocess documents, but you could use "micro RAG" for:

• Deduplicating knowledge bases
• Finding similar code snippets within a function
• Clustering similar user queries
• Any scenario where you need to understand semantic relationships

---

Citation

If you use this technique, please link back to this repo or cite it in your work!

@misc{microrag2025,
  title={Micro RAG: Using RAG to Improve RAG},
  author={Your Name},
  year={2025},
  howpublished={\url{https://github.com/YOUR_USERNAME/microrag-demo}},
}

---

License

MIT License - feel free to use this in your projects!

---

Contributing

Found a better coverage algorithm? Have ideas for optimizations? PRs welcome!

---

Remember: Don't put baby in a corner. RAG is a versatile tool—use it creatively! 🚀