mtRAG_wMem0

Dataset sourced from mt-rag-benchmark.

Small testsets for multi-turn QA retrieval using Mem0 memory search.

This project does two things:

• addMemory.py: loads FiQA retrieval chunks into Mem0 with metadata.
• searchMemory.py: runs multi-turn questions and measures which chunks are retrieved and reused across turns.

The goal is to show cross-turn retrieval overlap, which is a good fit for CacheBlend use cases.

Project Files

• addMemory.py

  • Samples conversations from FiQA rewrite queries.
  • Finds turn-level ground-truth chunk IDs from dev.tsv.
  • Loads those chunks from fiqa.jsonl.
  • Adds each chunk to Mem0 with metadata={"chunk_id": ...}.
  • Writes test_config.json for evaluation.

• searchMemory.py

  • Reads test_config.json.
  • Runs Mem0 search for each turn in each conversation.
  • Tracks retrieved chunk IDs, reused chunks, and new chunks per turn.
  • Prints precision/recall against ground truth and overlap tables.

• test_config.json

  • Generated by addMemory.py.
  • Stores selected conversations, turns, ground truth, and chunk IDs used for testing.

Setup

1. Install dependencies:

uv sync

2. Set your Mem0 API key:

export MEM0_API_KEY="your-api-key"

Run

1. Add chunks into Mem0:

uv run python addMemory.py --num-convs 3 --user-id fiqa_test1 --seed 42

Change user-id to have a fresh memory store in mem0 cloud

2. Run multi-turn retrieval evaluation:

uv run python searchMemory.py --top-k 5

Example results

Each letter (A, B, C, ..., Z26, Z27, ...) is a short label for a unique corpus chunk ID (e.g. 272784-0-597). The same letter appearing across multiple turns means the same chunk was retrieved again

Retrieval order is not fixed: Mem0 returns chunks ranked by semantic similarity score, so the same chunk may appear at different positions in different turns. This non-fixed ordering is exactly why this data is a good fit for CacheBlend

(mtrag-wmem0) (base) joyce@joycedeMacBook-Pro mtRAG_wMem0 % uv run python searchMemory.py

======================================================================
Conversation: 4751cd8210b4adb8bce5cbc3fe913096  (8 turns)
======================================================================

  Turn 1: Do common stocks and preferred stocks have any differences in terms of percentage of the company per unit they represent?
    Retrieved (5): ['K', 'N', 'P', 'Q', 'W']
    Ground truth:              ['K', 'N', 'Q']
    Reused from prev turns:    (none — first turn)
    New this turn:             ['K', 'N', 'P', 'Q', 'W']
    Precision=0.60  Recall=1.00

  Turn 2: Where can I purchase common and preferred stocks?
    Retrieved (5): ['K', 'N', 'P', 'Q', 'Z33']
    Ground truth:              ['N', 'Z55']
    Reused from prev turns:    ['K', 'N', 'P', 'Q']
    New this turn:             ['Z33']
    Precision=0.20  Recall=0.50

  Turn 3: What are the tax implications for preferred stocks?
    Retrieved (4): ['F', 'P', 'Q', 'Z32']
    Ground truth:              ['F', 'P', 'Z32']
    Reused from prev turns:    ['P', 'Q']
    New this turn:             ['F', 'Z32']
    Precision=0.75  Recall=1.00

  Turn 4: Can preferred stocks be designated into a trust?
    Retrieved (4): ['K', 'P', 'Q', 'Z47']
    Ground truth:              ['T', 'X', 'Z47']
    Reused from prev turns:    ['K', 'P', 'Q']
    New this turn:             ['Z47']
    Precision=0.25  Recall=0.33

  Turn 5: What is the tax rate for dividends in the US?
    Retrieved (4): ['H', 'T', 'Z31', 'Z32']
    Ground truth:              ['C', 'D', 'H', 'U', 'Z31', 'Z41']
    Reused from prev turns:    ['Z32']
    New this turn:             ['H', 'T', 'Z31']
    Precision=0.50  Recall=0.33

  Turn 6: What about the tax implications for losses on stocks?
    Retrieved (5): ['F', 'I', 'P', 'Z32', 'Z46']
    Ground truth:              ['I', 'Z46', 'Z52', 'Z54']
    Reused from prev turns:    ['F', 'P', 'Z32']
    New this turn:             ['I', 'Z46']
    Precision=0.40  Recall=0.50

  Turn 7: What does FIFO or LIFO mean in stock trading?
    Retrieved (5): ['F', 'K', 'Z29', 'Z33', 'Z46']
    Ground truth:              ['Z27', 'Z33', 'Z37']
    Reused from prev turns:    ['F', 'K', 'Z33', 'Z46']
    New this turn:             ['Z29']
    Precision=0.20  Recall=0.33

  Turn 8: Is keeping track of FIFO and LIFO methods for tax purposes complicated?
    Retrieved (5): ['F', 'I', 'Z29', 'Z32', 'Z33']
    Ground truth:              ['Y', 'Z29', 'Z33']
    Reused from prev turns:    ['F', 'I', 'Z29', 'Z32', 'Z33']
    New this turn:             []
    Precision=0.40  Recall=0.67

  Pairwise chunk overlap:
    T1 ∩ T2: ['K', 'N', 'P', 'Q']
    T1 ∩ T3: ['P', 'Q']
    T1 ∩ T4: ['K', 'P', 'Q']
    T1 ∩ T5: (none)
    T1 ∩ T6: ['P']
    T1 ∩ T7: ['K']
    T1 ∩ T8: (none)
    T2 ∩ T3: ['P', 'Q']
    T2 ∩ T4: ['K', 'P', 'Q']
    T2 ∩ T5: (none)
    T2 ∩ T6: ['P']
    T2 ∩ T7: ['K', 'Z33']
    T2 ∩ T8: ['Z33']
    T3 ∩ T4: ['P', 'Q']
    T3 ∩ T5: ['Z32']
    T3 ∩ T6: ['F', 'P', 'Z32']
    T3 ∩ T7: ['F']
    T3 ∩ T8: ['F', 'Z32']
    T4 ∩ T5: (none)
    T4 ∩ T6: ['P']
    T4 ∩ T7: ['K']
    T4 ∩ T8: (none)
    T5 ∩ T6: ['Z32']
    T5 ∩ T7: (none)
    T5 ∩ T8: ['Z32']
    T6 ∩ T7: ['F', 'Z46']
    T6 ∩ T8: ['F', 'I', 'Z32']
    T7 ∩ T8: ['F', 'Z29', 'Z33']

  Chunk reuse grid (✓ = retrieved, · = not retrieved):
  Turn |    F    H    I    K    N    P    Q    T    W  Z29  Z31  Z32  Z33  Z46  Z47
  ---------------------------------------------------------------------------------
  T  1 |    ·    ·    ·    ✓    ✓    ✓    ✓    ·    ✓    ·    ·    ·    ·    ·    ·  
  T  2 |    ·    ·    ·    ✓    ✓    ✓    ✓    ·    ·    ·    ·    ·    ✓    ·    ·  
  T  3 |    ✓    ·    ·    ·    ·    ✓    ✓    ·    ·    ·    ·    ✓    ·    ·    ·  
  T  4 |    ·    ·    ·    ✓    ·    ✓    ✓    ·    ·    ·    ·    ·    ·    ·    ✓  
  T  5 |    ·    ✓    ·    ·    ·    ·    ·    ✓    ·    ·    ✓    ✓    ·    ·    ·  
  T  6 |    ✓    ·    ✓    ·    ·    ✓    ·    ·    ·    ·    ·    ✓    ·    ✓    ·  
  T  7 |    ✓    ·    ·    ✓    ·    ·    ·    ·    ·    ✓    ·    ·    ✓    ✓    ·  
  T  8 |    ✓    ·    ✓    ·    ·    ·    ·    ·    ·    ✓    ·    ✓    ✓    ·    ·  

...