From b6ca48c20afb5a89e2a87d88045c177d698dc4e5 Mon Sep 17 00:00:00 2001 From: "Huang, Wei-Chieh" Date: Thu, 4 Jun 2026 14:49:17 +0800 Subject: [PATCH 1/3] Add MAB eval suite: LongMemEval-S, RULER QA1/QA2, LRU infbench/detective End-to-end eval pipelines for the MemoryAgentBench benchmarks running against MIRIX via the remote client. All four datasets share a common runner shape (ingest -> wrap_user_prompt -> task_agent.answer -> judge -> snapshot), are namespace-isolated per user_id, and produce a metrics.json that organize_results.py aggregates. Runners (one per benchmark) - evals/run_mab_longmem_eval.sh + evals/longmem_eval.py -- LongMemEval-S, session-aware ingest preserving occurred_at per session, MAB judge routed by category. - evals/run_mab_ruler_eval.sh + evals/ruler_eval.py -- RULER QA1 (SHDOCQA, single-hop) and QA2 (MHDOCQA, multi-hop). Sample_id is prefixed with the source (shdocqa_/mhdocqa_) so the two subsets can share the same Postgres DB. - evals/run_mab_lru_eval.sh + evals/lru_eval.py -- Long_Range_ Understanding split (infbench_sum_eng_shots2 + detective_qa). Source-routed judge: mab_summary for infbench summarization, substring for detective_qa multiple choice. Shared eval modules - evals/_chunking.py -- sentence-aware NLTK punkt + tiktoken gpt-4o-mini chunker at the MAB-canonical 4096-token budget. Each parse_* function delegates here so RULER / LRU / LongMemEval all chunk identically. - evals/_eval_db.py -- direct-PG helpers (measure_memory_size, dump_memories) so memlength comes from authoritative PG state rather than /memory/components, which caps at 50 items per type and would silently under-report on conversations with thousands of memory items. - evals/llm_judge_substring.py -- normalise + substring_exact_match judge for short-answer benchmarks (RULER, detective). - evals/llm_judge_mab.py -- MAB-aligned LongMemEval judge (gpt-4o, 5 category prompts + abstention, yes/no scoring). - evals/llm_judge_mab_summary.py -- MAB summarization judge for LRU/infbench (gpt-4o-2024-05-13, three calls per record: fluency * recall * precision -> F1). - evals/configs/mab.yaml -- MAB-profile Mirix config (gpt-4.1-mini answer LLM, gpt-4.1-nano topic extractor). Eval orchestration - evals/organize_results.py -- aggregator with --judge {default, substring, mab, mab_summary} routing. Carries keypoints through the judge task tuple so the summarization judge has what it needs. Surfaces gpt-4-{fluency, recall, precision, f1} in metrics when mab_summary is used. - evals/memory_snapshot.py -- pg_dump + neo4j export + results copy helper invoked by each runner at end-of-job. - evals/task_agent.py -- thin OpenAI tool-calling agent that does retrieval via MirixClient.search; defensive empty-key filter and TypeError guard around the kwarg splat keep the eval running when the LLM emits a malformed tool call. - evals/mirix_memory_system.py + evals/task_agent.py -- construct MirixClient with timeout=1800 so a 4096-token chunk (100-150s server-side) clears the per-request timeout with ample headroom. - evals/main_eval.py -- update LoCoMo runner to use dump_memories. --- evals/_chunking.py | 98 ++++++++ evals/_eval_db.py | 183 ++++++++++++++ evals/configs/mab.yaml | 46 ++++ evals/llm_judge_mab.py | 144 +++++++++++ evals/llm_judge_mab_summary.py | 294 ++++++++++++++++++++++ evals/llm_judge_substring.py | 78 ++++++ evals/longmem_eval.py | 433 +++++++++++++++++++++++++++++++++ evals/lru_eval.py | 404 ++++++++++++++++++++++++++++++ evals/main_eval.py | 3 +- evals/memory_snapshot.py | 280 +++++++++++++++++++++ evals/mirix_memory_system.py | 22 +- evals/organize_results.py | 174 +++++++++++-- evals/ruler_eval.py | 388 +++++++++++++++++++++++++++++ evals/run_mab_longmem_eval.sh | 131 ++++++++++ evals/run_mab_lru_eval.sh | 147 +++++++++++ evals/run_mab_ruler_eval.sh | 123 ++++++++++ evals/task_agent.py | 19 +- 17 files changed, 2943 insertions(+), 24 deletions(-) create mode 100644 evals/_chunking.py create mode 100644 evals/_eval_db.py create mode 100644 evals/configs/mab.yaml create mode 100644 evals/llm_judge_mab.py create mode 100644 evals/llm_judge_mab_summary.py create mode 100644 evals/llm_judge_substring.py create mode 100644 evals/longmem_eval.py create mode 100644 evals/lru_eval.py create mode 100644 evals/memory_snapshot.py create mode 100644 evals/ruler_eval.py create mode 100755 evals/run_mab_longmem_eval.sh create mode 100755 evals/run_mab_lru_eval.sh create mode 100755 evals/run_mab_ruler_eval.sh diff --git a/evals/_chunking.py b/evals/_chunking.py new file mode 100644 index 000000000..298e36ca9 --- /dev/null +++ b/evals/_chunking.py @@ -0,0 +1,98 @@ +"""Sentence-aware token-budgeted chunker, aligned with official MemoryAgentBench. + +Port of ``utils/eval_other_utils.chunk_text_into_sentences`` from +https://github.com/HUST-AI-HYZ/MemoryAgentBench. Used by ruler_eval, +lru_eval, and longmem_eval so the ingest chunks all four MAB datasets +(RULER QA1/QA2, LongMemEval-S, infbench_sum, detective_qa) see are +the same shape the leaderboard runs use: + + - **Atom**: sentence (NLTK ``punkt``). + - **Budget**: 4096 tokens of the ``gpt-4o-mini`` tiktoken encoding — + the value pinned by every MAB ``data_conf/**/*.yaml`` we've seen. + - **Bundle rule**: greedy pile sentences into the current chunk + until adding the next one would push the token count over budget, + then flush and start a new chunk with that sentence. + - **No mid-sentence split**: a sentence larger than the budget still + ships in its own chunk (matches the official behaviour — they have + no special case for this either). + +Why this matters: char-based 4096 (the previous policy) emitted ~4× +more chunks than official because 4096 chars ≈ 1024 tokens. That made +MIRIX's retrieval look worse than apples-to-apples because each +semantic unit was scattered across multiple memories. +""" + +from __future__ import annotations + +from typing import List + +import nltk +import tiktoken + +# Match the official MAB chunker exactly. +DEFAULT_TOKEN_MODEL = "gpt-4o-mini" +DEFAULT_CHUNK_TOKENS = 4096 + + +def _ensure_nltk() -> None: + """Make sure the NLTK punkt tokenizer is available. + + Best-effort — NLTK ships a chatty downloader; quiet=True keeps the + eval logs clean. If the network is offline and punkt isn't cached + yet, the eval will surface the LookupError at first sent_tokenize + call, which is the same failure mode the official MAB script has. + """ + try: + nltk.data.find("tokenizers/punkt_tab") + return + except LookupError: + pass + try: + nltk.data.find("tokenizers/punkt") + return + except LookupError: + pass + # punkt_tab is the modern split file; older NLTK versions still ship + # 'punkt'. Download both so it works on either. + for pkg in ("punkt_tab", "punkt"): + try: + nltk.download(pkg, quiet=True) + except Exception: + pass + + +_ensure_nltk() +_ENCODING = tiktoken.encoding_for_model(DEFAULT_TOKEN_MODEL) + + +def chunk_text_into_sentences( + text: str, + chunk_size: int = DEFAULT_CHUNK_TOKENS, +) -> List[str]: + """Bundle sentences greedily into <= ``chunk_size``-token chunks. + + Returns a list of plain strings (each chunk's sentences joined by + a single space, matching the official script). Empty input returns + an empty list. + """ + if not text: + return [] + + sentences = nltk.sent_tokenize(text) + chunks: List[str] = [] + buf: List[str] = [] + buf_tokens = 0 + + for sentence in sentences: + st = len(_ENCODING.encode(sentence, allowed_special={"<|endoftext|>"})) + if buf and buf_tokens + st > chunk_size: + chunks.append(" ".join(buf)) + buf = [sentence] + buf_tokens = st + else: + buf.append(sentence) + buf_tokens += st + + if buf: + chunks.append(" ".join(buf)) + return chunks diff --git a/evals/_eval_db.py b/evals/_eval_db.py new file mode 100644 index 000000000..77c096d61 --- /dev/null +++ b/evals/_eval_db.py @@ -0,0 +1,183 @@ +"""Direct-PG helpers for eval scripts. + +Two helpers shared by ruler_eval / longmem_eval / main_eval: + +- ``measure_memory_size(sample_id)`` — total stored chars+tokens for a + user, plus per-table row counts; reads ``MIRIX_PG_DB`` env so it tracks + whatever DB the runner set (e.g. ``mirix_isolate``) instead of being + pinned to ``mirix``. + +- ``dump_memories(sample_id)`` — full per-user dump of episodic + + semantic rows, straight from Postgres. Bypasses + ``/memory/components``, which caps each memory type at 50 by default + and 200 even with an explicit ``limit`` parameter — so it's the only + way to get a ``_memories.json`` snapshot whose token count reflects + the real store on conversations with thousands of rows. +""" +from __future__ import annotations + +import os +import shutil +import subprocess +from typing import Dict, List + + +def _pg_conn() -> Dict[str, str]: + return { + "bin": shutil.which("psql") or "/usr/local/opt/postgresql@17/bin/psql", + "host": os.environ.get("MIRIX_PG_HOST", "localhost"), + "user": os.environ.get("MIRIX_PG_USER", "mirix"), + "db": os.environ.get("MIRIX_PG_DB", "mirix"), + "pwd": os.environ.get("MIRIX_PG_PASSWORD", "mirix"), + } + + +def _pg_scalar(sql: str, timeout: int = 30) -> str: + c = _pg_conn() + try: + out = subprocess.run( + [c["bin"], "-h", c["host"], "-U", c["user"], "-d", c["db"], "-tAc", sql], + capture_output=True, text=True, timeout=timeout, + env={**os.environ, "PGPASSWORD": c["pwd"]}, + ) + return out.stdout.strip() + except Exception: + return "" + + +def _pg_rows(sql: str, timeout: int = 300) -> List[List[str]]: + """Run SQL with control-byte separators so embedded newlines in + summary/details don't get mistaken for row boundaries. + + SOH (``\\x01``) separates columns, STX (``\\x02``) separates records. + Neither appears in normal memory text. + """ + c = _pg_conn() + out = subprocess.run( + [c["bin"], "-h", c["host"], "-U", c["user"], "-d", c["db"], + "-A", "-F", "\x01", "-R", "\x02", "-t", "-c", sql], + capture_output=True, text=True, timeout=timeout, + env={**os.environ, "PGPASSWORD": c["pwd"]}, + ) + rows: List[List[str]] = [] + for chunk in out.stdout.split("\x02"): + if not chunk: + continue + rows.append(chunk.split("\x01")) + return rows + + +def _esc(s: str) -> str: + return s.replace("'", "''") + + +def measure_memory_size(sample_id: str) -> Dict: + """Total stored chars + tokens for ``sample_id``. + + PG (flat) and Neo4j (graph) backends measured on the same yardstick + by concatenating every stored text field. + """ + stats: Dict = {"unit": "chars+tokens"} + uid = _esc(sample_id) + + flat: Dict = {} + flat_chars = 0 + for table, cols in ( + ("episodic_memory", "coalesce(summary,'')||coalesce(details,'')"), + ("semantic_memory", "coalesce(name,'')||coalesce(summary,'')||coalesce(details,'')"), + ): + row_n = _pg_scalar(f"SELECT count(*) FROM {table} WHERE user_id='{uid}';") + chars = _pg_scalar( + f"SELECT coalesce(sum(length({cols})),0) FROM {table} WHERE user_id='{uid}';" + ) + n = int(row_n) if row_n.isdigit() else 0 + c = int(chars) if chars.lstrip('-').isdigit() else 0 + flat[table] = {"rows": n, "chars": c} + flat_chars += c + stats["flat"] = flat + stats["flat_total_chars"] = flat_chars + + try: + from neo4j import GraphDatabase + from mirix.settings import settings + driver = GraphDatabase.driver( + settings.neo4j_uri, auth=(settings.neo4j_user, settings.neo4j_password) + ) + with driver.session(database=settings.neo4j_database) as ses: + rec = ses.run( + """ + MATCH (n) WHERE n.user_id=$uid + WITH count(n) AS nodes, + sum(size(coalesce(n.description,'')) + size(coalesce(n.summary,''))) AS node_chars + RETURN nodes, node_chars + """, uid=sample_id + ).single() + erec = ses.run( + """ + MATCH (a)-[r]->(b) WHERE a.user_id=$uid + RETURN count(r) AS edges, + sum(size(coalesce(r.description,''))) AS edge_chars + """, uid=sample_id + ).single() + driver.close() + node_chars = (rec["node_chars"] or 0) if rec else 0 + edge_chars = (erec["edge_chars"] or 0) if erec else 0 + stats["graph"] = { + "nodes": (rec["nodes"] or 0) if rec else 0, + "edges": (erec["edges"] or 0) if erec else 0, + "node_chars": node_chars, + "edge_chars": edge_chars, + } + stats["graph_total_chars"] = node_chars + edge_chars + except Exception as exc: + stats["graph"] = {"error": str(exc)} + stats["graph_total_chars"] = 0 + + total_chars = max(stats["flat_total_chars"], stats["graph_total_chars"]) + stats["total_chars"] = total_chars + stats["total_tokens"] = total_chars // 4 + return stats + + +def dump_memories(sample_id: str) -> Dict: + """Full per-user dump of episodic + semantic rows for ``sample_id``. + + Shape matches what ``organize_results.count_memory_tokens`` expects: + ``{"memories": {"episodic": {"total_count": N, "items": [...]}, + "semantic": {...}}}``. + """ + uid = _esc(sample_id) + epi_rows = _pg_rows( + "SELECT id, " + "coalesce(to_char(occurred_at,'YYYY-MM-DD\"T\"HH24:MI:SSOF'),''), " + "coalesce(event_type,''), coalesce(actor,''), " + "coalesce(summary,''), coalesce(details,'') " + f"FROM episodic_memory WHERE user_id='{uid}' " + "ORDER BY occurred_at NULLS LAST, created_at" + ) + sem_rows = _pg_rows( + "SELECT id, coalesce(name,''), coalesce(summary,''), " + "coalesce(details,''), coalesce(source,'') " + f"FROM semantic_memory WHERE user_id='{uid}' ORDER BY created_at" + ) + return { + "user_id": sample_id, + "memories": { + "episodic": { + "total_count": len(epi_rows), + "items": [ + {"id": r[0], "occurred_at": r[1], "event_type": r[2], + "actor": r[3], "summary": r[4], "details": r[5]} + for r in epi_rows + ], + }, + "semantic": { + "total_count": len(sem_rows), + "items": [ + {"id": r[0], "name": r[1], "summary": r[2], + "details": r[3], "source": r[4]} + for r in sem_rows + ], + }, + }, + } diff --git a/evals/configs/mab.yaml b/evals/configs/mab.yaml new file mode 100644 index 000000000..c1a732837 --- /dev/null +++ b/evals/configs/mab.yaml @@ -0,0 +1,46 @@ +# Mirix Configuration — MemoryAgentBench (MAB) profile +# +# Used by `evals/run_mab_longmem_eval.sh` and any other MAB-benchmark +# runner (LongMemEval-S etc., HF dataset ai-hyz/MemoryAgentBench). +# +# Independent from the 0201c LoCoMo profile so MAB-specific tuning +# can diverge later without touching LoCoMo runs. + +llm_config: + model: "gpt-4.1-mini" + model_endpoint_type: "openai" + api_key: your-api-key + model_endpoint: "https://api.openai.com/v1" + context_window: 128000 + +topic_extraction_llm_config: + model: "gpt-4.1-nano" + model_endpoint_type: "openai" + api_key: your-api-key + model_endpoint: "https://api.openai.com/v1" + context_window: 128000 + +embedding_config: + embedding_model: "text-embedding-3-small" + embedding_endpoint: "https://api.openai.com/v1" + api_key: your-api-key + embedding_endpoint_type: "openai" + embedding_dim: 1536 + +build_embeddings_for_memory: true + +meta_agent_config: + system_prompts_folder: prompts/0201a/ + agents: + - core_memory_agent + - resource_memory_agent + - semantic_memory_agent + - episodic_memory_agent + - procedural_memory_agent + - knowledge_vault_memory_agent + memory: + core: + - label: "human" + value: "" + - label: "persona" + value: "I am a helpful assistant." diff --git a/evals/llm_judge_mab.py b/evals/llm_judge_mab.py new file mode 100644 index 000000000..52a0f18be --- /dev/null +++ b/evals/llm_judge_mab.py @@ -0,0 +1,144 @@ +""" +MemoryAgentBench-aligned LLM judge. + +Port of ``llm_based_eval/longmem_qa_evaluate.py`` from +https://github.com/HUST-AI-HYZ/MemoryAgentBench so our scores are +directly comparable to the MAB leaderboard. + +Differences vs ``llm_judge.py`` (our generic LoCoMo judge): +- Metric model is **gpt-4o** (not gpt-4o-mini). +- Prompts are **per question type** (single-session-*, multi-session, + temporal-reasoning, knowledge-update, single-session-preference) with + category-specific leniency (e.g. temporal allows off-by-one; + knowledge-update accepts "old + new" responses; preference doesn't + require hitting every rubric point). +- Scoring rule: ``label = 'yes' in response.lower()`` — the official + binary rule, not a JSON-parsed CORRECT/WRONG label. + +Abstention: +- ``longmem_eval.py`` now stores ``question_id`` (from HF + ``metadata.question_ids``) on each record. ``organize_results.py`` + reads it and passes ``abstention='_abs' in question_id`` here. Old + records without ``question_id`` quietly default to abstention=False. +""" + +import os + +from dotenv import load_dotenv +from openai import OpenAI + +load_dotenv() +_client = OpenAI(api_key=os.getenv("OPENAI_API_KEY")) + +# Official MAB metric model (longmem_qa_evaluate.py: metric_model="gpt-4o"). +DEFAULT_METRIC_MODEL = "gpt-4o" + + +_SINGLE_AND_MULTI = ( + "I will give you a question, a correct answer, and a response from a model. " + "Please answer yes if the response contains the correct answer. Otherwise, answer no. " + "If the response is equivalent to the correct answer or contains all the intermediate " + "steps to get the correct answer, you should also answer yes. If the response only " + "contains a subset of the information required by the answer, answer no. \n\n" + "Question: {}\n\nCorrect Answer: {}\n\nModel Response: {}\n\n" + "Is the model response correct? Answer yes or no only." +) +_TEMPORAL = ( + "I will give you a question, a correct answer, and a response from a model. " + "Please answer yes if the response contains the correct answer. Otherwise, answer no. " + "If the response is equivalent to the correct answer or contains all the intermediate " + "steps to get the correct answer, you should also answer yes. If the response only " + "contains a subset of the information required by the answer, answer no. In addition, " + "do not penalize off-by-one errors for the number of days. If the question asks for " + "the number of days/weeks/months, etc., and the model makes off-by-one errors " + "(e.g., predicting 19 days when the answer is 18), the model's response is still " + "correct. \n\n" + "Question: {}\n\nCorrect Answer: {}\n\nModel Response: {}\n\n" + "Is the model response correct? Answer yes or no only." +) +_KNOWLEDGE_UPDATE = ( + "I will give you a question, a correct answer, and a response from a model. " + "Please answer yes if the response contains the correct answer. Otherwise, answer no. " + "If the response contains some previous information along with an updated answer, " + "the response should be considered as correct as long as the updated answer is the " + "required answer.\n\n" + "Question: {}\n\nCorrect Answer: {}\n\nModel Response: {}\n\n" + "Is the model response correct? Answer yes or no only." +) +_PREFERENCE = ( + "I will give you a question, a rubric for desired personalized response, and a " + "response from a model. Please answer yes if the response satisfies the desired " + "response. Otherwise, answer no. The model does not need to reflect all the points " + "in the rubric. The response is correct as long as it recalls and utilizes the " + "user's personal information correctly.\n\n" + "Question: {}\n\nRubric: {}\n\nModel Response: {}\n\n" + "Is the model response correct? Answer yes or no only." +) +_ABSTENTION = ( + "I will give you an unanswerable question, an explanation, and a response from a " + "model. Please answer yes if the model correctly identifies the question as " + "unanswerable. The model could say that the information is incomplete, or some other " + "information is given but the asked information is not.\n\n" + "Question: {}\n\nExplanation: {}\n\nModel Response: {}\n\n" + "Does the model correctly identify the question as unanswerable? " + "Answer yes or no only." +) + + +SUPPORTED_TASKS = frozenset( + { + "single-session-user", + "single-session-assistant", + "multi-session", + "temporal-reasoning", + "knowledge-update", + "single-session-preference", + } +) + + +def get_anscheck_prompt( + task: str, + question: str, + answer: str, + response: str, + abstention: bool = False, +) -> str: + """Verbatim port of MemoryAgentBench's category routing.""" + if abstention: + return _ABSTENTION.format(question, answer, response) + if task in ("single-session-user", "single-session-assistant", "multi-session"): + return _SINGLE_AND_MULTI.format(question, answer, response) + if task == "temporal-reasoning": + return _TEMPORAL.format(question, answer, response) + if task == "knowledge-update": + return _KNOWLEDGE_UPDATE.format(question, answer, response) + if task == "single-session-preference": + return _PREFERENCE.format(question, answer, response) + raise NotImplementedError(f"Unknown MAB task category: {task!r}") + + +def evaluate_mab_judge( + question: str, + gold_answer: str, + generated_answer: str, + question_type: str, + abstention: bool = False, + model: str = DEFAULT_METRIC_MODEL, +) -> int: + """Return 1 if the MAB judge says yes, 0 otherwise. + + Mirrors the official scoring rule (`label = 'yes' in response.lower()`). + """ + prompt = get_anscheck_prompt( + question_type, question, gold_answer, generated_answer, abstention=abstention + ) + completion = _client.chat.completions.create( + model=model, + messages=[{"role": "user", "content": prompt}], + n=1, + temperature=0, + max_tokens=10, + ) + text = (completion.choices[0].message.content or "").strip().lower() + return 1 if "yes" in text else 0 diff --git a/evals/llm_judge_mab_summary.py b/evals/llm_judge_mab_summary.py new file mode 100644 index 000000000..6a0d9c7a3 --- /dev/null +++ b/evals/llm_judge_mab_summary.py @@ -0,0 +1,294 @@ +"""MemoryAgentBench-aligned summarization judge. + +Port of ``llm_based_eval/summarization_evaluate.py`` from +https://github.com/HUST-AI-HYZ/MemoryAgentBench so summarization scores +on the Long_Range_Understanding split (``infbench_sum_eng_shots2``) +are directly comparable to the MAB leaderboard. + +Three GPT-4o calls per sample: + + 1. **Fluency** — binary 0/1 — is the generated text coherent, non- + repetitive, grammatically fluent? An incoherent or repetitive + summary zeroes out F1. + 2. **Recall** — count of expert-written keypoints supported by the + generated summary, normalised by total keypoints. + 3. **Precision** — count of sentences in the generated summary that + are supported by the expert reference summary, normalised by + total sentence count. + +Final score per sample:: + + F1 = fluency * 2 * recall * precision / (recall + precision) + +Aggregation over a run: arithmetic mean of F1, recall, precision, +fluency across all judged records. + +Why the book-variant prompts: ``infbench_sum_eng_shots2`` summarises +novels, so we use ``*_book`` (literature) prompts, not the lawsuit +variants. ``multi_lexsum`` (the lawsuit variants in the official +script) is not in the MAB LRU split. + +Metric model is pinned to ``gpt-4o-2024-05-13`` to match the official +script — the unpinned ``gpt-4o`` alias drifts and would diverge from +the leaderboard. +""" + +from __future__ import annotations + +import json +import os +import re +from typing import Dict, List, Optional, Sequence + +from dotenv import load_dotenv +from openai import OpenAI + +load_dotenv() +_client = OpenAI(api_key=os.getenv("OPENAI_API_KEY")) + +# Official MAB summarization metric model +# (summarization_evaluate.py: OpenAIModel("gpt-4o-2024-05-13", ...)) +DEFAULT_METRIC_MODEL = "gpt-4o-2024-05-13" +DEFAULT_TEMPERATURE = 0.1 +DEFAULT_MAX_TOKENS = 4096 + + +_FLUENCY_PROMPT_BOOK = """Please act as an impartial judge and evaluate the fluency of the provided text. The text should be coherent, non-repetitive, fluent, and grammatically correct. + +Below is your grading rubric: +- Score 0 (incoherent, repetitive, or incomplete): Incoherent sentences, repetitive sentences (even if not by exact words), incomplete answers, or gibberish. Note that even if the answer is coherent, if it is repetitive or incomplete, it should be given a score of 0. + - Examples: + - Incomplete: "Summary:" + - Incoherent: "Summary:ЉЉЉЉЉЉЉЉЉЉЉЉЉЉ \\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\_______ is is is" + - Repetitive: "Summary:\\n\\n\\n\\n\\n\\n\\n\\n |THE next morning, when Ellington came down to breakfast, she found a letter on the table addressed to her. ... I'll tell her that I'm sure she'll be a great comfort to me." + +- Score 1 (coherent, non-repetitive answer): Coherent, non-repetitive, fluent, grammatically correct answers. If the text is coherent, non-repetitive, and fluent, but the last sentence is truncated, it should still be given a score of 1. + - Examples: + - "The story revolves around the life of Jennifer Pete, a young woman with a strong sense of morality and spirituality. ... Overall, the story is a nuanced and thought-provoking exploration of the human condition." + +Now, read the provided text, and evaluate the fluency using the rubric. Then output your score in the following json format: {{"fluency": 1}}. + +Text: "{text}" +""" + + +_RECALL_PROMPT_BOOK = """Please act as an impartial judge and evaluate the quality of the provided summary of a novel. It should discuss the plots and characters of the story. The text should contain all the given key points. + +Below is your grading rubric: +Recall: +- Evaluate the provided summary by deciding if each of the key points is present in the provided summary. A key point is considered present if its factual information is mostly-supported by the provided summary. If a key point contains multiple facts, it's still considered supported if most of the facts are present. +- Score: the number of key points mostly-supported by the provided summary. +- Examples: use the following examples to guide your evaluation. + +Example 1: + +Key points: +1. Cal Margaret lives in Berlin, Germany. +2. Cal decides to write his life story, starting with the history of the recessive gene causing his intersex condition. +3. The story begins with Cal's grandparents, Raul and Harris, in a village on Mount Olympus in 1922. +4. Raul and Harris are siblings who fall in love and decide to immigrate to Detroit after their parents' deaths. +5. They escape the burning of Smyrna by the Turkish army and find passage to America. +6. On the ship, Raul and Harris pretend to meet for the first time and then wed. +7. In Detroit, they move in with their cousin Lavinia and her husband, Gerry Helena. +8. Helena takes Raul into his alcohol smuggling business. +9. Harris and Lavinia get pregnant on the same night, causing Helena to suspect Lavinia of cheating with Raul. +10. Helena takes Raul on a drive on the ice to interrogate him, but the car falls into the water and Raul escapes. +11. In 1945, Raul and Harris's son, Irma, develops a crush on Helena and Lavinia's daughter, Russell. +12. Harris encourages Russell to accept a proposal from a seminary student, Ida, causing Irma to join the Navy in anger. +13. Russell calls off her engagement to Ida when she realizes Irma might die in the U.S. invasion of Japan. +14. Irma excels on a test, gets transferred to the officer's academy, and is spared from fighting in the rest of the war. +15. Irma and Russell marry and have a son named Deana Salome. +16. Five years later, they wish for a daughter and conceive Ali (Callie) using pseudo-scientific methods. +17. Irma retires from the Navy and takes over Raul's bar, turning it into a diner. +18. The diner burns down during the Twelfth Street Riot in 1967, but the family has enough insurance money to move to Grosse Pointe. +19. They move into an unusual house on a street named Middlesex. +20. Seven-year-old Callie wants to make friends in the new neighborhood and practices kissing with the girl next door, Sven Chrissy. +21. Callie is sent to an all-girls prep school and worries about not getting her period or growing breasts. +22. Callie develops a crush on a classmate referred to as 'the Obscure Object' and they begin a physical relationship. +23. Callie is hit by a tractor and the hospital doctors realize she is biologically male. +24. Russell and Irma take Callie to a specialist in New York named Dr. Lester. +25. Dr. Lester wants to use Callie to prove his theory that gender is a social construct and recommends surgery. +26. Callie learns she is biologically male, renames himself Cal, and runs away to San Francisco. + + +Summary: The story begins with the birth of the narrator, Cal Stephanides, who is a hermaphrodite. ... The book also explores the history of Detroit and its transformation from a small town to a major industrial city. + + +Reasoning: The summary incorrectly identifies the protagonist as "Cal Stephanides" instead of "Cal Margaret", so key point 1 is not supported. It does not mention key point 2. The summary mentions that Raul and Harris are silbings and that they eventually marry and settle down in Detroit so key point 3 is supported. It also mentions the Turkish attack and how they escape from Smyrna to America so key point 5 is supported. It does not talk about the ship where they are wed so key point 6 is not supported. The summary then stops discussing the plot and so it does not mention key point 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, or 26. Thus, the only supported key points are 3 and 5, so recall is 2. + +Output: {{"supported_key_points": [3, 5], "recall": 2}} + + +Example 2: + +Key points: +1. The story follows the Octavia family traveling along the Malaysia River from Iquitos in Peru to Belem in Brazil. +2. Lauren Octavia is the central character, a wealthy rancher with a dark secret. +3. Lauren has been living under a false name, hiding his identity as a wrongfully accused criminal who escaped from prison 20 years ago. +4. Lauren sees an opportunity to clear his name and risks the journey to Brazil to present evidence proving his innocence. +5. Lauren's family, unaware of his past, accompanies him on the journey. +6. Lauren's daughter, Minha, is engaged to Manoel, a gallant but flippish army physician. +7. Lauren's son, Benito, is brave and hot-headed, greatly admiring and respecting his father. +8. Duncan, a soldier turned rogue, discovers Lauren's secret and blackmails him. +9. The journey down the river is filled with turbulence, both literal and figurative. +10. The natural wonders and wildlife of the Malaysia River add flavor to the story. +11. The family faces lethal dangers, including river pirates and boating accidents. +12. The story subtly raises the issue of slavery in Brazil, a contemporary concern at the time. +13. The climax occurs in Belem with a trial for Lauren. +14. A dramatic court scene unfolds where the credibility of Lauren's documents is questioned. +15. Lauren is on the verge of being convicted. +16. Duncan, who was killed by an Indian's poisoned arrow earlier, is dissected. +17. A letter confirming Lauren's claims is found inside Duncan, proving Lauren's innocence. +18. The novel ends with the Octavias happily returning to their fazenda, their home in Iquitos. +19. The adventurous journey of eight hundred leagues on the Malaysia comes to an end. + + +Summary: The story follows the journey of the Octavia family as they travel down the Malaysia River on a massive raft, or "jangada," from Iquitos to Belem. ... The story concludes with the family reaching their destination, having grown and changed through their experiences on the river. + + +Reasoning: Key point 1 is supported by the summary. ... Therefore, the supported key points are 1, 6, 9, and 10, so the recall score is 4. + +Output: {{"supported_key_points": [1, 6, 9, 10], "recall": 4}} + +Now, read the provided summary and key points, and evaluate the summary using the rubric. First, think step-by-step and provide your reasoning and assessment on the answer. Then output your score in the following json format: {{"supported_key_points": [2, 4], "recall": 2}}, where "supported_key_points" contains the key points that are present in the summary and "recall" is the total number of key points present in the summary. + +Key points: +{keypoints} + +Summary: {summary} +""" + + +_PRECISION_PROMPT_BOOK = """Please act as an impartial judge and evaluate the quality of the provided summary of a novel. + +Below is your grading rubric: +Precision: +- Evaluate the provided summary by deciding if each sentence in the provided summary is supported by the information provided in the expert summary. A sentence is considered supported if its major facts align with the information in the expert summary. A sentence is still considered supported even if some of its minor details, such as dates, entity names, or the location, are not explicitly mentioned in the expert summary. A sentence is not supported if its major facts are not mentioned or contradicted in the expert summary. It is also not supported if it introduces new information not present in the expert summary, such as additional analysis or commentary on the story. +- Score: the number of sentences in the provided summary that are supported by the expert summary. +- Examples: use the following examples to guide your evaluation. + +Example 1: + +Expert summary: Cal Margaret is a man living in Berlin, Germany. ... Years later, Cal starts a relationship with a woman named Chase Leuan in Berlin. + +Provided summary: The story begins with the birth of the narrator, Cal Stephanides, who is a hermaphrodite. ... Overall, the book is a sweeping narrative that spans multiple generations and continents. It is a story about identity, culture, family, and history, and it raises important questions about the human experience. + +Reasoning: The first sentence is not supported because the provided summary claims the character is named "Cal Stephanides" while the expert summary indicates that they are named "Cal Margaret". ... Therefore, the precision score is 10. + +Output: {{"precision": 10, "sentence_count": 22}} + + +Example 2: + +Expert summary: The story chronicles the journey of the Octavia family ... putting an end to their adventurous journey of eight hundred leagues on the Malaysia. + +Provided: The story follows the journey of the Octavia family ... The story concludes with the family reaching their destination, having grown and changed through their experiences on the river. + +Reasoning: Sentence 1 is supported as the expert summary mentions the Octavia family traveling along the Malaysia River from Iquitos in Peru to Belem in Brazil. ... Therefore, the precision score is 5. + +Output: {{"precision": 5, "sentence_count": 15}} + +Now, read the provided summary and expert summary, and evaluate the summary using the rubric. First, think step-by-step and provide your reasoning and assessment on the answer. Then output your score in the following json format: {{"precision": 7, "sentence_count": 20}}. + +Expert summary: {expert_summary} + +Provided summary: {summary} +""" + + +def _parse_json(text: str) -> Optional[Dict]: + """Best-effort JSON extraction from a GPT-4 response. + + Mirrors official ``summarization_evaluate.parse_json``: greedy + brace match, fallback to fenced ```json``` block. Returns the + *last* JSON object — the model often emits an example object in + its reasoning before the final score. + """ + matches = re.findall(r"\{.*?\}", text or "", re.DOTALL) + if matches: + try: + return json.loads(matches[-1]) + except json.JSONDecodeError: + pass + fenced = re.findall(r"(?:```json)(.+?)(?:```)", text or "", re.DOTALL) + if fenced: + try: + return json.loads(fenced[-1].strip()) + except json.JSONDecodeError: + pass + return None + + +def _llm(prompt: str, model: str = DEFAULT_METRIC_MODEL) -> str: + completion = _client.chat.completions.create( + model=model, + messages=[{"role": "user", "content": prompt}], + n=1, + temperature=DEFAULT_TEMPERATURE, + max_tokens=DEFAULT_MAX_TOKENS, + ) + return (completion.choices[0].message.content or "").strip() + + +def evaluate_mab_summary_judge( + generated_summary: str, + expert_summary: str, + keypoints: Sequence[str], + model: str = DEFAULT_METRIC_MODEL, +) -> Dict: + """Run the three official MAB summarization judges and return all sub-scores plus F1. + + Returns a dict with keys:: + + fluency — 0 or 1 + recall_total — len(keypoints) + recall_found — judge's count of supported keypoints + recall — recall_found / recall_total in [0,1] + precision_total — judge's sentence_count + precision_found — judge's count of supported sentences + precision — precision_found / precision_total in [0,1] + f1 — fluency * 2*R*P / (R+P) in [0,1] + raw — {fluency_out, recall_out, precision_out} unparsed text + + Any missing-or-unparseable judge call collapses the relevant sub- + metric to 0 (and therefore F1 to 0) — matches the official rule of + silently skipping records with ``None`` outputs. + """ + text = (generated_summary or "").strip() + expert = (expert_summary or "").strip() + kp_list: List[str] = list(keypoints or []) + kp_block = "\n".join(f"{i + 1}. {kp}" for i, kp in enumerate(kp_list)) + + fluency_text = _llm(_FLUENCY_PROMPT_BOOK.format(text=text), model=model) + recall_text = _llm(_RECALL_PROMPT_BOOK.format(keypoints=kp_block, summary=text), model=model) + precision_text = _llm(_PRECISION_PROMPT_BOOK.format(expert_summary=expert, summary=text), model=model) + + f_obj = _parse_json(fluency_text) + r_obj = _parse_json(recall_text) + p_obj = _parse_json(precision_text) + + fluency = int(f_obj.get("fluency", 0)) if isinstance(f_obj, dict) else 0 + recall_found = int(r_obj.get("recall", 0)) if isinstance(r_obj, dict) else 0 + recall_total = len(kp_list) + precision_found = int(p_obj.get("precision", 0)) if isinstance(p_obj, dict) else 0 + precision_total = int(p_obj.get("sentence_count", 0)) if isinstance(p_obj, dict) else 0 + + recall = (recall_found / recall_total) if recall_total > 0 else 0.0 + precision = (precision_found / precision_total) if precision_total > 0 else 0.0 + f1 = (fluency * 2 * recall * precision / (recall + precision)) if (recall + precision) > 0 else 0.0 + + return { + "fluency": fluency, + "recall_total": recall_total, + "recall_found": recall_found, + "recall": recall, + "precision_total": precision_total, + "precision_found": precision_found, + "precision": precision, + "f1": f1, + "raw": { + "fluency_out": fluency_text, + "recall_out": recall_text, + "precision_out": precision_text, + }, + } diff --git a/evals/llm_judge_substring.py b/evals/llm_judge_substring.py new file mode 100644 index 000000000..eabca0ede --- /dev/null +++ b/evals/llm_judge_substring.py @@ -0,0 +1,78 @@ +""" +Substring-match evaluator for RULER-style QA (and any other dataset +whose ``answers`` are short, exact-form needles like ``'France'`` or +``'42'``). + +Mirrors the official MemoryAgentBench ``substring_exact_match`` metric: +the response is correct iff it contains (as a case-insensitive substring) +at least one of the gold answers. + +Usage from ``organize_results.py``: + + from llm_judge_substring import evaluate_substring_judge + score = evaluate_substring_judge(predicted_answer, expected_answer) + +No LLM call; deterministic; effectively free. +""" + +import re +import string +from typing import Iterable, Union + + +def normalize_answer(text: str) -> str: + """Port of MemoryAgentBench ``utils/eval_other_utils.normalize_answer``. + + Pipeline (verbatim from official): + 1. lowercase + 2. strip punctuation + 3. drop articles (a / an / the) at word boundaries + 4. collapse whitespace + """ + text = text.lower() + text = "".join(c for c in text if c not in string.punctuation) + text = re.sub(r"\b(a|an|the)\b", " ", text) + text = " ".join(text.split()) + return text + + +def _accepted_answers(expected: Union[str, Iterable[str], None]) -> list[str]: + """Normalize the gold-answer field into a flat list of strings. + + ``expected`` may be: + * a list of strings (the raw HF shape, e.g. ``['France','France']``) + * a string built by ``flatten_answer`` (``'France; France'``) + * None / empty + """ + if expected is None: + return [] + if isinstance(expected, str): + # Flattened ``"a; b; c"`` form. + return [seg.strip() for seg in expected.split(";") if seg.strip()] + # Iterable[str] — callers pass a flat list of accepted answers. + return [str(item).strip() for item in expected if str(item).strip()] + + +def evaluate_substring_judge( + predicted: Union[str, None], + expected: Union[str, Iterable[str], None], +) -> int: + """Return 1 if ``predicted`` contains any accepted answer, else 0. + + Matches the official ``substring_exact_match`` metric in + ``MemoryAgentBench/utils/eval_other_utils.py``: both prediction and + each accepted answer are passed through ``normalize_answer`` + (lowercase, strip punctuation, drop articles, collapse whitespace) + before substring containment is checked. Score is the max over all + accepted answers (drqa-style max-over-ground-truths). + """ + if not predicted: + return 0 + accepted = _accepted_answers(expected) + if not accepted: + return 0 + pred_norm = normalize_answer(predicted) + for ans in {normalize_answer(a) for a in accepted}: + if ans and ans in pred_norm: + return 1 + return 0 diff --git a/evals/longmem_eval.py b/evals/longmem_eval.py new file mode 100644 index 000000000..397c4d347 --- /dev/null +++ b/evals/longmem_eval.py @@ -0,0 +1,433 @@ +"""Evaluate Mirix memory on MemoryAgentBench LongMemEval-S. + +Sits alongside main_eval.py (LoCoMo runner). Reuses the same scaffolding — +MirixMemorySystem, TaskAgent, server-side token tracker — and writes the same +per-sample JSON schema, so organize_results.py works on its output unchanged. + +Only the data layer differs: LongMemEval-S comes from the HuggingFace dataset +`ai-hyz/MemoryAgentBench` (split `Accurate_Retrieval`, rows whose +metadata.source starts with `longmemeval_s`). Each row is one long context +(~1.6M chars) plus a list of questions / answers / per-question types. + +The long context is a list of conversation sessions, each prefixed with a +"Chat Time". It is parsed into one chunk per session and ingested chunk by +chunk, with each session's chat time passed to MIRIX as `occurred_at` — so the +episodic agent anchors the real year instead of guessing it. + +Usage (smoke test — 1 conv, 15 chunks, 5 questions): + python longmem_eval.py --limit 1 --max-chunks 15 --max-questions 5 \ + --run-llm --mirix_config_path ./configs/0201c.yaml \ + --output_path results/0201c_longmem + +Output lands in evals/results/longmem// so it cannot collide +with the LoCoMo namespace used by main_eval.py. +""" + +import argparse +import json +import os +import time +from pathlib import Path +from typing import Dict, List, Optional + +from mirix_memory_system import MirixMemorySystem +from task_agent import TaskAgent + +# LongMemEval rows in MemoryAgentBench are tagged metadata.source = "longmemeval_s*". +# Default mirrors the official MAB judge CLI (--dataset longmemeval_s*). +# Exact-match (==) rather than startswith — matches official +# llm_based_eval/longmem_qa_evaluate.py behaviour. +DEFAULT_LONGMEM_SOURCE = "longmemeval_s*" + + +def load_longmem_s( + source: str = DEFAULT_LONGMEM_SOURCE, + limit: Optional[int] = None, +) -> List[Dict]: + """Load LongMemEval-S samples from HuggingFace MemoryAgentBench. + + Returns a list of dicts shaped as: + {sample_id, context, questions, answers, question_types, question_ids} + + ``question_ids`` carry the HF metadata IDs (e.g. ``"a4996e51"``, + ``"edced276_abs"``). The ``_abs`` suffix is what the official judge + uses to route abstention questions to a different prompt. + """ + try: + from datasets import load_dataset + except ImportError as exc: # pragma: no cover + raise SystemExit( + "The `datasets` package is required for LongMemEval-S. " + "Install it with: uv pip install datasets" + ) from exc + + ds = load_dataset("ai-hyz/MemoryAgentBench", split="Accurate_Retrieval") + + samples: List[Dict] = [] + for row in ds: + meta = row.get("metadata") or {} + if meta.get("source") != source: + continue + idx = len(samples) + samples.append( + { + "sample_id": f"longmem_s_{idx}", + "context": row.get("context") or "", + "questions": list(row.get("questions") or []), + "answers": list(row.get("answers") or []), + "question_types": list(meta.get("question_types") or []), + "question_ids": list(meta.get("question_ids") or []), + } + ) + if limit is not None and len(samples) >= limit: + break + + if not samples: + raise SystemExit( + f"No rows found in ai-hyz/MemoryAgentBench (Accurate_Retrieval) " + f"with metadata.source == {source!r}." + ) + return samples + + +from _chunking import DEFAULT_CHUNK_TOKENS, chunk_text_into_sentences + + +def parse_sessions(context: str, max_chunk_tokens: int = DEFAULT_CHUNK_TOKENS) -> List[Dict]: + """Parse a LongMemEval context into timestamped ingest chunks. + + The HF ``context`` field is the repr of a Python list shaped: + ['Chat Time: 2022/11/17 (Thu) 12:04', [{'role','content'}, ...], + 'Chat Time: 2022/12/28 (Wed) 16:10', [ ... ], ...] + i.e. alternating (chat-time string, message list) pairs — one pair + per conversation session. + + Each session is then chunked with the shared sentence-aware, + token-budgeted chunker (4096 gpt-4o-mini tokens, aligned with the + official MAB chunker). Most LongMemEval-S sessions (~14k chars / + ~3.6k tokens) fit in a single chunk now, but the + ``chunk_text_into_sentences`` call still defends against the rare + long session. Every sub-chunk inherits its session's ``occurred_at`` + so the episodic agent's temporal reasoning stays anchored. + + Returns: list of {occurred_at: Optional[str], text: str}. + """ + import ast + import re + from datetime import datetime + + try: + parsed = ast.literal_eval(context) + except (ValueError, SyntaxError): + # Fallback: treat the whole context as one undated chunk. + return [{"occurred_at": None, "text": context}] + + if not isinstance(parsed, list): + return [{"occurred_at": None, "text": str(context)}] + + def _parse_chat_time(s: str) -> Optional[str]: + # "Chat Time: 2022/11/17 (Thu) 12:04" -> "2022-11-17T12:04:00" + m = re.search(r"(\d{4})/(\d{1,2})/(\d{1,2}).*?(\d{1,2}):(\d{2})", s) + if not m: + return None + y, mo, d, hh, mm = (int(x) for x in m.groups()) + try: + return datetime(y, mo, d, hh, mm).isoformat() + except ValueError: + return None + + def _msg_lines(msgs) -> List[str]: + lines = [] + for msg in msgs: + if not isinstance(msg, dict): + continue + role = str(msg.get("role", "")).strip() + content = str(msg.get("content", "")).strip() + lines.append(f"{role}: {content}") + return lines + + def _split_session(header: str, msgs, occurred: Optional[str]) -> List[Dict]: + """Sentence-tokenize the session and inherit the session's + ``occurred_at`` on every resulting sub-chunk. Header is + prepended to the first text blob so the chunker can decide + whether it fits alongside body content.""" + body_lines = _msg_lines(msgs) + text = header + "\n" + "\n".join(body_lines) if body_lines else header + pieces = chunk_text_into_sentences(text, chunk_size=max_chunk_tokens) + if not pieces: + return [{"occurred_at": occurred, "text": header}] + return [{"occurred_at": occurred, "text": p} for p in pieces] + + sessions: List[Dict] = [] + i = 0 + while i < len(parsed): + item = parsed[i] + if isinstance(item, str) and "Chat Time" in item and i + 1 < len(parsed) \ + and isinstance(parsed[i + 1], list): + occurred = _parse_chat_time(item) + sessions.extend(_split_session(item, parsed[i + 1], occurred)) + i += 2 + else: + # Unexpected shape — keep it as an undated chunk rather than drop. + if isinstance(item, list): + for ln in _msg_lines(item): + sessions.append({"occurred_at": None, "text": ln}) + elif isinstance(item, str): + sessions.append({"occurred_at": None, "text": item}) + i += 1 + return sessions + + +def flatten_answer(answer) -> Optional[str]: + """LongMemEval answers are list-of-list (e.g. ['50']). Flatten to a string.""" + if answer is None: + return None + if isinstance(answer, list): + flat = [] + for a in answer: + if isinstance(a, list): + flat.extend(str(x) for x in a) + else: + flat.append(str(a)) + return "; ".join(flat) if flat else None + return str(answer) + + +from _eval_db import measure_memory_size, dump_memories + + +def load_sample_result(path: Path) -> Optional[Dict]: + if not path.exists(): + return None + try: + with path.open("r", encoding="utf-8") as handle: + data = json.load(handle) + if isinstance(data, dict): + return data + except (OSError, json.JSONDecodeError): + return None + return None + + +def save_sample_result(path: Path, data: Dict) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("w", encoding="utf-8") as handle: + json.dump(data, handle, ensure_ascii=False, indent=2) + + +def print_qa(qidx: int, question: str, expected: Optional[str], predicted: Optional[str]) -> None: + print(f"[{qidx}] question: {str(question)[:120]}") + print(f"[{qidx}] expected_answer: {expected}") + print(f"[{qidx}] predicted_answer: {str(predicted)[:200] if predicted else predicted}") + + +def main() -> None: + parser = argparse.ArgumentParser(description="Evaluate Mirix memory on LongMemEval-S.") + parser.add_argument("--limit", type=int, default=None, + help="Limit number of conversations (samples) to evaluate.") + parser.add_argument("--max-chunks", type=int, default=None, + help="Limit session chunks ingested per sample (smoke-test knob).") + parser.add_argument("--max-questions", type=int, default=None, + help="Limit number of questions per sample.") + parser.add_argument("--run-llm", action="store_true", default=True, + help="Call the LLM to answer questions.") + parser.add_argument("--output_path", type=Path, default=Path("longmem_run"), + help="Output sub-folder, resolved under evals/results/longmem/.") + parser.add_argument( + "--source", + type=str, + default=DEFAULT_LONGMEM_SOURCE, + help=( + "HF metadata.source to load (exact match). Default " + f"{DEFAULT_LONGMEM_SOURCE!r} mirrors the official MAB judge " + "CLI (--dataset longmemeval_s*)." + ), + ) + parser.add_argument("--mirix_config_path", type=Path, default=None, + help="Path to Mirix config file.") + args = parser.parse_args() + + items = load_longmem_s(source=args.source, limit=args.limit) + print(f"[longmem_eval] loaded {len(items)} LongMemEval-S conversation(s)") + + mirix_client_id = os.environ.get("MIRIX_CLIENT_ID", "mirix-eval-client") + mirix_org_id = os.environ.get("MIRIX_ORG_ID", "mirix-eval-org") + + # Keep LongMemEval output in its own namespace, away from LoCoMo's. + longmem_root = Path(__file__).resolve().parent / "results" / "longmem" + if args.output_path.is_absolute(): + print(f"[longmem_eval] WARNING: --output_path is absolute ({args.output_path}); " + "writing outside evals/results/longmem/ namespace.") + output_path = args.output_path + else: + output_path = longmem_root / args.output_path + output_path.mkdir(parents=True, exist_ok=True) + print(f"[longmem_eval] writing per-sample results to {output_path}") + + import httpx + server_base = "http://127.0.0.1:8531" + + def _reset_tokens(): + try: + httpx.post(f"{server_base}/debug/token_stats/reset", timeout=10) + except Exception: + pass + + def _snapshot_tokens(): + try: + r = httpx.get(f"{server_base}/debug/token_stats", timeout=10) + return r.json().get("stats", {}) + except Exception: + return {} + + def _sum_tokens(stats): + s = {"prompt": 0, "completion": 0, "total": 0, "calls": 0} + for v in stats.values(): + for k in s: + s[k] += v.get(k, 0) + return s + + for item in items: + sample_id = item["sample_id"] + sample_path = output_path / f"{sample_id}.json" + + task_agent = ( + TaskAgent(mirix_config_path=str(args.mirix_config_path), + client_id=mirix_client_id, org_id=mirix_org_id, user_id=sample_id) + if args.run_llm else None + ) + + sample_result = load_sample_result(sample_path) or { + "sample_id": sample_id, + "timings": {"add_chunk": {}, "wrap_user_prompt": {}, "answer": {}}, + "responses": {}, + "records": {}, + } + sample_result.setdefault("sample_id", sample_id) + + memory_system = MirixMemorySystem( + user_id=sample_id, + mirix_config_path=str(args.mirix_config_path), + client=task_agent.mirix_client if task_agent else None, + ) + + _reset_tokens() + + # ---- ingest: one chunk per conversation session, WITH timestamps ---- + chunks = parse_sessions(item["context"]) + if args.max_chunks is not None: + chunks = chunks[: args.max_chunks] + dated = sum(1 for c in chunks if c["occurred_at"]) + print(f"[longmem_eval] {sample_id}: ingesting {len(chunks)} session chunk(s) " + f"({dated} with timestamps)") + + for idx, chunk in enumerate(chunks, start=1): + idx_key = str(idx) + if idx_key in sample_result["responses"]: + continue + start = time.perf_counter() + response = memory_system.add_chunk( + chunk["text"], raw_input=chunk["text"], + occurred_at=chunk["occurred_at"], + ) + elapsed = time.perf_counter() - start + sample_result["responses"][idx_key] = { + "type": "add_chunk", + "chunk_index": idx, + "question_index": None, + "occurred_at": chunk["occurred_at"], + "response": response, + } + sample_result["timings"]["add_chunk"][idx_key] = elapsed + save_sample_result(sample_path, sample_result) + + build_stats = _snapshot_tokens() + sample_result["token_stats"] = {"build_raw": build_stats, "build_sum": _sum_tokens(build_stats)} + save_sample_result(sample_path, sample_result) + + # Memory size in a common unit (stored chars) so no-graph (flat PG) and + # graph (Neo4j) runs are directly comparable. See measure_memory_size(). + sample_result["memory_stats"] = measure_memory_size(sample_id) + print(f"[longmem_eval] {sample_id}: memory_stats = {sample_result['memory_stats']}") + save_sample_result(sample_path, sample_result) + + # ---- QA ---- + questions = item["questions"] + answers = item["answers"] + qtypes = item["question_types"] + qids = item.get("question_ids") or [] + if args.max_questions is not None: + questions = questions[: args.max_questions] + + for qidx, question in enumerate(questions, start=1): + qidx_key = str(qidx) + if qidx_key in sample_result["records"]: + rec = sample_result["records"][qidx_key] + print_qa(qidx, rec.get("question", ""), + rec.get("expected_answer"), rec.get("predicted_answer")) + continue + + expected_answer = flatten_answer(answers[qidx - 1] if qidx - 1 < len(answers) else None) + # category: organize_results.py groups metrics by this. LongMemEval + # uses string question types (multi-session, temporal-reasoning, ...). + category = qtypes[qidx - 1] if qidx - 1 < len(qtypes) else None + + start = time.perf_counter() + input_messages = memory_system.wrap_user_prompt(question) + sample_result["timings"]["wrap_user_prompt"][qidx_key] = time.perf_counter() - start + + predicted = None + message_trace = None + usage_trace = None + usage_total = None + if task_agent: + start = time.perf_counter() + trace = task_agent.answer(input_messages, user_id=sample_id) + predicted = trace.get("answer") + message_trace = trace.get("messages") + usage_trace = trace.get("usage") + usage_total = trace.get("usage_total") + sample_result["timings"]["answer"][qidx_key] = time.perf_counter() - start + + sample_result["records"][qidx_key] = { + "sample_id": sample_id, + "question_index": qidx, + # HF metadata.question_ids[qidx-1]. Carries the optional + # ``_abs`` suffix that routes abstention questions to the + # MAB judge's abstention prompt. + "question_id": qids[qidx - 1] if qidx - 1 < len(qids) else None, + "question": question, + "expected_answer": expected_answer, + "evidence": None, + "category": category, + "input_messages": input_messages, + "predicted_answer": predicted, + "messages": message_trace, + "usage": usage_trace, + "usage_total": usage_total, + } + save_sample_result(sample_path, sample_result) + print_qa(qidx, question, expected_answer, predicted) + + try: + all_memories = dump_memories(sample_id) + except Exception as exc: + all_memories = {"success": False, "error": str(exc), "user_id": sample_id} + with (output_path / f"{sample_id}_memories.json").open("w", encoding="utf-8") as handle: + json.dump(all_memories, handle, ensure_ascii=False, indent=2) + + post_qa_stats = _snapshot_tokens() + post_qa_sum = _sum_tokens(post_qa_stats) + build_sum = sample_result.get("token_stats", {}).get("build_sum", {}) + query_sum = { + k: max(post_qa_sum.get(k, 0) - build_sum.get(k, 0), 0) + for k in ("prompt", "completion", "total", "calls") + } + sample_result.setdefault("token_stats", {}) + sample_result["token_stats"]["query_raw"] = post_qa_stats + sample_result["token_stats"]["query_sum"] = query_sum + save_sample_result(sample_path, sample_result) + + +if __name__ == "__main__": + main() diff --git a/evals/lru_eval.py b/evals/lru_eval.py new file mode 100644 index 000000000..2fd59b9ec --- /dev/null +++ b/evals/lru_eval.py @@ -0,0 +1,404 @@ +"""Evaluate Mirix memory on MemoryAgentBench Long_Range_Understanding (LRU). + +Sister script to ruler_eval.py — same scaffolding (MirixMemorySystem, +TaskAgent, server-side token tracker, per-sample JSON schema, char- +based chunking, _eval_db helpers) so organize_results.py works on the +output unchanged. + +LRU rows come from `ai-hyz/MemoryAgentBench`, split +``Long_Range_Understanding``, with two sources: + + - ``infbench_sum_eng_shots2`` (100 rows, novels) — one long story + per row (~1.7M chars), one question per row ("Write a 1000-1200 + word summary..."), answer is a multi-paragraph reference summary. + Score with ``--judge mab_summary`` (gpt-4o-2024-05-13, fluency + + recall over keypoints + precision over sentences → F1). + - ``detective_qa`` (10 rows) — multiple-choice mystery QA. Score + with ``--judge substring`` (the gold answers are short option + strings like "C. The Brandt couple"). + +For ``infbench_sum_eng_shots2`` we propagate ``keypoints`` and the +``qa_pair_id`` from ``metadata`` into each record so +``llm_judge_mab_summary`` has what it needs without a re-fetch. + +Like RULER, the context carries no timestamps so ``occurred_at`` is +always None. Char-based 4096 chunking matches ruler_eval; the official +MAB chunker is sentence-aware token-based ~20k chars but that exposed +the httpx RetryTransport timeout; we trade leaderboard parity for +runs that finish in this house. + +Usage (smoke test — 1 row, 3 chunks): + python lru_eval.py --limit 1 --max-chunks 3 \\ + --run-llm --mirix_config_path ./configs/mab.yaml \\ + --output_path smoke_lru + +Output lands in evals/results/lru//. +""" + +import argparse +import json +import os +import time +from pathlib import Path +from typing import Dict, List, Optional + +from mirix_memory_system import MirixMemorySystem +from task_agent import TaskAgent + +# LRU sources in MemoryAgentBench (split Long_Range_Understanding): +# infbench_sum_eng_shots2 — novel summarization (100 rows) ← default +# detective_qa — multiple-choice mystery QA (10 rows) +# Default points at infbench; swap via --source. +DEFAULT_LRU_SOURCE = "infbench_sum_eng_shots2" + + +def load_lru( + source: str = DEFAULT_LRU_SOURCE, + limit: Optional[int] = None, +) -> List[Dict]: + """Load Long_Range_Understanding samples from HuggingFace. + + Returns a list of dicts shaped as: + {sample_id, context, questions, answers, keypoints, qa_pair_ids} + + keypoints/qa_pair_ids are populated from row metadata for + ``infbench_sum_eng_shots2`` and propagated into each record so the + mab_summary judge can score without re-fetching HF. For + ``detective_qa`` they're empty lists (substring judge ignores them). + """ + try: + from datasets import load_dataset + except ImportError as exc: # pragma: no cover + raise SystemExit( + "The `datasets` package is required for LRU. " + "Install it with: uv pip install datasets" + ) from exc + + ds = load_dataset("ai-hyz/MemoryAgentBench", split="Long_Range_Understanding") + + # Namespace sample_id by source so infbench / detective rows can + # share the same Postgres DB without their user_id rows getting + # mixed up (both subsets number from 0). + source_tag = { + "infbench_sum_eng_shots2": "infbench", + "detective_qa": "detective", + }.get(source, source.lower()) + + samples: List[Dict] = [] + for row in ds: + meta = row.get("metadata") or {} + if meta.get("source") != source: + continue + idx = len(samples) + samples.append( + { + "sample_id": f"{source_tag}_{idx}", + "context": row.get("context") or "", + "questions": list(row.get("questions") or []), + "answers": list(row.get("answers") or []), + # MAB summarization judge inputs (infbench only). + # detective_qa rows leave these as empty / from-meta. + "keypoints": list(meta.get("keypoints") or []), + "qa_pair_ids": list(meta.get("qa_pair_ids") or []), + # Empty lists — kept for shape compatibility with the + # downstream record builder; abstention prompt is never + # triggered on LRU. + "question_types": [], + "question_ids": [], + } + ) + if limit is not None and len(samples) >= limit: + break + + if not samples: + raise SystemExit( + f"No rows found in ai-hyz/MemoryAgentBench (Long_Range_Understanding) " + f"with metadata.source == {source!r}. " + f"Try {DEFAULT_LRU_SOURCE!r} or 'detective_qa'." + ) + return samples + + +from _chunking import DEFAULT_CHUNK_TOKENS, chunk_text_into_sentences + + +def parse_context(context: str, max_chunk_tokens: int = DEFAULT_CHUNK_TOKENS) -> List[Dict]: + """Sentence-aware, token-budgeted chunking for an LRU context. + + Delegates to the shared ``chunk_text_into_sentences`` (NLTK ``punkt`` + + tiktoken ``gpt-4o-mini``, 4096-token budget) — matches the + official MAB chunker so infbench / detective scores are directly + leaderboard-comparable. + + LRU contexts have no timestamps, so ``occurred_at`` is always None. + + Returns: list of {occurred_at: None, text: str}. + """ + return [{"occurred_at": None, "text": c} + for c in chunk_text_into_sentences(context, chunk_size=max_chunk_tokens)] + + +def flatten_answer(answer) -> Optional[str]: + """LongMemEval answers are list-of-list (e.g. ['50']). Flatten to a string.""" + if answer is None: + return None + if isinstance(answer, list): + flat = [] + for a in answer: + if isinstance(a, list): + flat.extend(str(x) for x in a) + else: + flat.append(str(a)) + return "; ".join(flat) if flat else None + return str(answer) + + +from _eval_db import measure_memory_size, dump_memories + + +def load_sample_result(path: Path) -> Optional[Dict]: + if not path.exists(): + return None + try: + with path.open("r", encoding="utf-8") as handle: + data = json.load(handle) + if isinstance(data, dict): + return data + except (OSError, json.JSONDecodeError): + return None + return None + + +def save_sample_result(path: Path, data: Dict) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("w", encoding="utf-8") as handle: + json.dump(data, handle, ensure_ascii=False, indent=2) + + +def print_qa(qidx: int, question: str, expected: Optional[str], predicted: Optional[str]) -> None: + print(f"[{qidx}] question: {str(question)[:120]}") + print(f"[{qidx}] expected_answer: {expected}") + print(f"[{qidx}] predicted_answer: {str(predicted)[:200] if predicted else predicted}") + + +def main() -> None: + parser = argparse.ArgumentParser(description="Evaluate Mirix memory on MAB Long_Range_Understanding (infbench / detective_qa).") + parser.add_argument("--limit", type=int, default=None, + help="Limit number of samples to evaluate.") + parser.add_argument("--max-chunks", type=int, default=None, + help="Limit ingest chunks per sample (smoke-test knob).") + parser.add_argument("--max-questions", type=int, default=None, + help="Limit number of questions per sample.") + parser.add_argument("--run-llm", action="store_true", default=True, + help="Call the LLM to answer questions.") + parser.add_argument("--output_path", type=Path, default=Path("lru_run"), + help="Output sub-folder, resolved under evals/results/lru/.") + parser.add_argument( + "--source", + type=str, + default=DEFAULT_LRU_SOURCE, + help=( + "HF metadata.source to load (exact match). Default " + f"{DEFAULT_LRU_SOURCE!r}. Swap to 'detective_qa' for the " + "multiple-choice mystery subset." + ), + ) + parser.add_argument("--mirix_config_path", type=Path, default=None, + help="Path to Mirix config file.") + args = parser.parse_args() + + items = load_lru(source=args.source, limit=args.limit) + print(f"[lru_eval] loaded {len(items)} LRU conversation(s) from source {args.source!r}") + + mirix_client_id = os.environ.get("MIRIX_CLIENT_ID", "mirix-eval-client") + mirix_org_id = os.environ.get("MIRIX_ORG_ID", "mirix-eval-org") + + # Keep LRU output in its own namespace, away from RULER / LongMemEval / LoCoMo. + lru_root = Path(__file__).resolve().parent / "results" / "lru" + if args.output_path.is_absolute(): + print(f"[lru_eval] WARNING: --output_path is absolute ({args.output_path}); " + "writing outside evals/results/lru/ namespace.") + output_path = args.output_path + else: + output_path = lru_root / args.output_path + output_path.mkdir(parents=True, exist_ok=True) + print(f"[lru_eval] writing per-sample results to {output_path}") + + import httpx + server_base = "http://127.0.0.1:8531" + + def _reset_tokens(): + try: + httpx.post(f"{server_base}/debug/token_stats/reset", timeout=10) + except Exception: + pass + + def _snapshot_tokens(): + try: + r = httpx.get(f"{server_base}/debug/token_stats", timeout=10) + return r.json().get("stats", {}) + except Exception: + return {} + + def _sum_tokens(stats): + s = {"prompt": 0, "completion": 0, "total": 0, "calls": 0} + for v in stats.values(): + for k in s: + s[k] += v.get(k, 0) + return s + + for item in items: + sample_id = item["sample_id"] + sample_path = output_path / f"{sample_id}.json" + + task_agent = ( + TaskAgent(mirix_config_path=str(args.mirix_config_path), + client_id=mirix_client_id, org_id=mirix_org_id, user_id=sample_id) + if args.run_llm else None + ) + + sample_result = load_sample_result(sample_path) or { + "sample_id": sample_id, + "timings": {"add_chunk": {}, "wrap_user_prompt": {}, "answer": {}}, + "responses": {}, + "records": {}, + } + sample_result.setdefault("sample_id", sample_id) + + memory_system = MirixMemorySystem( + user_id=sample_id, + mirix_config_path=str(args.mirix_config_path), + client=task_agent.mirix_client if task_agent else None, + ) + + _reset_tokens() + + # ---- ingest: paragraph-bundled chunks of <= 4096 chars ---- + chunks = parse_context(item["context"]) + if args.max_chunks is not None: + chunks = chunks[: args.max_chunks] + dated = sum(1 for c in chunks if c["occurred_at"]) + print(f"[lru_eval] {sample_id}: ingesting {len(chunks)} context chunk(s) " + f"({dated} with timestamps)") + + for idx, chunk in enumerate(chunks, start=1): + idx_key = str(idx) + if idx_key in sample_result["responses"]: + continue + start = time.perf_counter() + response = memory_system.add_chunk( + chunk["text"], raw_input=chunk["text"], + occurred_at=chunk["occurred_at"], + ) + elapsed = time.perf_counter() - start + sample_result["responses"][idx_key] = { + "type": "add_chunk", + "chunk_index": idx, + "question_index": None, + "occurred_at": chunk["occurred_at"], + "response": response, + } + sample_result["timings"]["add_chunk"][idx_key] = elapsed + save_sample_result(sample_path, sample_result) + + build_stats = _snapshot_tokens() + sample_result["token_stats"] = {"build_raw": build_stats, "build_sum": _sum_tokens(build_stats)} + save_sample_result(sample_path, sample_result) + + # Memory size in a common unit (stored chars) so no-graph (flat PG) and + # graph (Neo4j) runs are directly comparable. See measure_memory_size(). + sample_result["memory_stats"] = measure_memory_size(sample_id) + print(f"[lru_eval] {sample_id}: memory_stats = {sample_result['memory_stats']}") + save_sample_result(sample_path, sample_result) + + # ---- QA ---- + questions = item["questions"] + answers = item["answers"] + qtypes = item["question_types"] + qids = item.get("question_ids") or [] + # MAB summarization judge inputs (infbench only). qa_pair_ids is + # parallel to questions for detective_qa; for infbench_sum the + # whole row is one qa_pair so we just reuse qa_pair_ids[0]. + keypoints = item.get("keypoints") or [] + qa_pair_ids = item.get("qa_pair_ids") or [] + if args.max_questions is not None: + questions = questions[: args.max_questions] + + for qidx, question in enumerate(questions, start=1): + qidx_key = str(qidx) + if qidx_key in sample_result["records"]: + rec = sample_result["records"][qidx_key] + print_qa(qidx, rec.get("question", ""), + rec.get("expected_answer"), rec.get("predicted_answer")) + continue + + expected_answer = flatten_answer(answers[qidx - 1] if qidx - 1 < len(answers) else None) + # category: organize_results.py groups metrics by this. LongMemEval + # uses string question types (multi-session, temporal-reasoning, ...). + category = qtypes[qidx - 1] if qidx - 1 < len(qtypes) else None + + start = time.perf_counter() + input_messages = memory_system.wrap_user_prompt(question) + sample_result["timings"]["wrap_user_prompt"][qidx_key] = time.perf_counter() - start + + predicted = None + message_trace = None + usage_trace = None + usage_total = None + if task_agent: + start = time.perf_counter() + trace = task_agent.answer(input_messages, user_id=sample_id) + predicted = trace.get("answer") + message_trace = trace.get("messages") + usage_trace = trace.get("usage") + usage_total = trace.get("usage_total") + sample_result["timings"]["answer"][qidx_key] = time.perf_counter() - start + + sample_result["records"][qidx_key] = { + "sample_id": sample_id, + "question_index": qidx, + # HF metadata.question_ids[qidx-1]. Carries the optional + # ``_abs`` suffix that routes abstention questions to the + # MAB judge's abstention prompt. + "question_id": qids[qidx - 1] if qidx - 1 < len(qids) else None, + # Pass-through for organize_results.judge_task_mab_summary. + "qa_pair_id": ( + qa_pair_ids[qidx - 1] if qidx - 1 < len(qa_pair_ids) + else (qa_pair_ids[0] if qa_pair_ids else None) + ), + "keypoints": keypoints, + "question": question, + "expected_answer": expected_answer, + "evidence": None, + "category": category, + "input_messages": input_messages, + "predicted_answer": predicted, + "messages": message_trace, + "usage": usage_trace, + "usage_total": usage_total, + } + save_sample_result(sample_path, sample_result) + print_qa(qidx, question, expected_answer, predicted) + + try: + all_memories = dump_memories(sample_id) + except Exception as exc: + all_memories = {"success": False, "error": str(exc), "user_id": sample_id} + with (output_path / f"{sample_id}_memories.json").open("w", encoding="utf-8") as handle: + json.dump(all_memories, handle, ensure_ascii=False, indent=2) + + post_qa_stats = _snapshot_tokens() + post_qa_sum = _sum_tokens(post_qa_stats) + build_sum = sample_result.get("token_stats", {}).get("build_sum", {}) + query_sum = { + k: max(post_qa_sum.get(k, 0) - build_sum.get(k, 0), 0) + for k in ("prompt", "completion", "total", "calls") + } + sample_result.setdefault("token_stats", {}) + sample_result["token_stats"]["query_raw"] = post_qa_stats + sample_result["token_stats"]["query_sum"] = query_sum + save_sample_result(sample_path, sample_result) + + +if __name__ == "__main__": + main() diff --git a/evals/main_eval.py b/evals/main_eval.py index 6680316c2..5d77c59cd 100644 --- a/evals/main_eval.py +++ b/evals/main_eval.py @@ -8,6 +8,7 @@ from mirix_memory_system import MirixMemorySystem from task_agent import TaskAgent +from _eval_db import dump_memories instructions = """Instructions: @@ -297,7 +298,7 @@ def main() -> None: print_qa(qidx, question, expected_answer, predicted) try: - all_memories = memory_system.list_all_memories(limit=0) + all_memories = dump_memories(sample_id) except Exception as exc: all_memories = { "success": False, diff --git a/evals/memory_snapshot.py b/evals/memory_snapshot.py new file mode 100644 index 000000000..c22fd288f --- /dev/null +++ b/evals/memory_snapshot.py @@ -0,0 +1,280 @@ +"""Save / load Mirix memory snapshots so an expensive ingest can be reused. + +Ingesting one LongMemEval-S conversation takes ~2h (no-graph) to ~9h (graph). +This script captures the post-ingest state — PostgreSQL flat-memory tables and +the Neo4j graph — into a named snapshot, and restores it later so a QA-only +re-run can skip ingest entirely. + +Usage: + python memory_snapshot.py save [--agents] + python memory_snapshot.py load + python memory_snapshot.py list + python memory_snapshot.py delete + +A snapshot lives in evals/snapshots// and contains: + pg_memory.dump — pg_dump of the six memory tables (+ agents/messages + when --agents is given, needed for a true QA-only run) + neo4j_graph.json — every node + relationship, exported via Cypher + +Notes +----- +* `load` REPLACES current memory: it truncates the same tables / wipes the + Neo4j graph before restoring, so the snapshot is the exact state afterwards. +* PG connection + Neo4j auth are read from the same env / settings the server + uses (.env in repo root). Defaults match docker/env.example. +* A snapshot is mode-specific: a no-graph snapshot has an empty graph, a graph + snapshot has both. Restoring a no-graph snapshot then running graph-mode QA + will NOT magically produce a graph. +""" + +import argparse +import json +import os +import subprocess +import sys +from pathlib import Path + +# Load .env the same way the server does. +REPO_ROOT = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(REPO_ROOT)) +try: + from dotenv import load_dotenv + load_dotenv(REPO_ROOT / ".env") +except Exception: + pass + +SNAP_ROOT = Path(__file__).resolve().parent / "snapshots" + +# Mirix memory tables. Core memory lives in `block`; `raw_memory` holds the +# verbatim ingested chunks. agents/messages are optional (only needed for a +# fully ingest-free QA run that also skips meta-agent recreation). +# Names verified against the live schema (pg_tables). +MEMORY_TABLES = [ + "episodic_memory", + "semantic_memory", + "procedural_memory", + "resource_memory", + "knowledge_vault", + "block", + "raw_memory", +] +AGENT_TABLES = ["agents", "messages"] + +# --- PG / Neo4j connection (env first, then docker/env.example defaults) --- +PG = { + "host": os.environ.get("MIRIX_PG_HOST", "localhost"), + "port": os.environ.get("MIRIX_PG_PORT", "5432"), + "user": os.environ.get("MIRIX_PG_USER", "mirix"), + "password": os.environ.get("MIRIX_PG_PASSWORD", "mirix"), + "db": os.environ.get("MIRIX_PG_DB", "mirix"), +} +PG_BIN = os.environ.get("MIRIX_PG_BIN", "/usr/local/opt/postgresql@17/bin") +NEO4J = { + "uri": os.environ.get("MIRIX_NEO4J_URI", "bolt://localhost:7687"), + "user": os.environ.get("MIRIX_NEO4J_USER", "neo4j"), + "password": os.environ.get("MIRIX_NEO4J_PASSWORD", "mirix_neo4j_dev"), + "database": os.environ.get("MIRIX_NEO4J_DATABASE", "neo4j"), +} + + +def _pg_env(): + return {**os.environ, "PGPASSWORD": PG["password"]} + + +def _pg_args(tool: str): + return [ + f"{PG_BIN}/{tool}", + "-h", PG["host"], "-p", PG["port"], "-U", PG["user"], "-d", PG["db"], + ] + + +def _table_exists(table: str) -> bool: + out = subprocess.run( + _pg_args("psql") + ["-tAc", f"SELECT to_regclass('public.{table}');"], + capture_output=True, text=True, env=_pg_env(), timeout=30, + ) + return out.stdout.strip() not in ("", "NULL") + + +# ---------------------------------------------------------------- save ---- +def save(name: str, include_agents: bool) -> None: + snap = SNAP_ROOT / name + snap.mkdir(parents=True, exist_ok=True) + + tables = [t for t in MEMORY_TABLES if _table_exists(t)] + if include_agents: + tables += [t for t in AGENT_TABLES if _table_exists(t)] + if not tables: + raise SystemExit("No memory tables found in the database.") + + # --- PostgreSQL: pg_dump the selected tables --- + dump_path = snap / "pg_memory.dump" + table_flags = [] + for t in tables: + table_flags += ["-t", t] + print(f"[snapshot] pg_dump {len(tables)} table(s) -> {dump_path}") + res = subprocess.run( + _pg_args("pg_dump") + ["-Fc", "--data-only", "-f", str(dump_path)] + table_flags, + capture_output=True, text=True, env=_pg_env(), timeout=600, + ) + if res.returncode != 0: + raise SystemExit(f"pg_dump failed: {res.stderr}") + + # --- Neo4j: export all nodes + relationships via Cypher --- + graph = _neo4j_export() + graph_path = snap / "neo4j_graph.json" + graph_path.write_text(json.dumps(graph, ensure_ascii=False)) + print(f"[snapshot] neo4j export: {len(graph['nodes'])} nodes, " + f"{len(graph['relationships'])} relationships -> {graph_path}") + + meta = { + "name": name, + "pg_tables": tables, + "includes_agents": include_agents, + "neo4j_nodes": len(graph["nodes"]), + "neo4j_relationships": len(graph["relationships"]), + } + (snap / "meta.json").write_text(json.dumps(meta, indent=2)) + print(f"[snapshot] saved '{name}' -> {snap}") + + +def _neo4j_export() -> dict: + try: + from neo4j import GraphDatabase + except ImportError: + print("[snapshot] neo4j driver not installed; graph export skipped") + return {"nodes": [], "relationships": []} + try: + driver = GraphDatabase.driver(NEO4J["uri"], auth=(NEO4J["user"], NEO4J["password"])) + nodes, rels = [], [] + with driver.session(database=NEO4J["database"]) as ses: + for rec in ses.run( + "MATCH (n) RETURN id(n) AS id, labels(n) AS labels, properties(n) AS props" + ): + nodes.append({"id": rec["id"], "labels": rec["labels"], "props": dict(rec["props"])}) + for rec in ses.run( + "MATCH (a)-[r]->(b) RETURN id(a) AS src, id(b) AS dst, " + "type(r) AS type, properties(r) AS props" + ): + rels.append({"src": rec["src"], "dst": rec["dst"], + "type": rec["type"], "props": dict(rec["props"])}) + driver.close() + return {"nodes": nodes, "relationships": rels} + except Exception as exc: + print(f"[snapshot] neo4j export failed ({exc}); graph snapshot empty") + return {"nodes": [], "relationships": []} + + +# ---------------------------------------------------------------- load ---- +def load(name: str) -> None: + snap = SNAP_ROOT / name + if not snap.exists(): + raise SystemExit(f"Snapshot '{name}' not found at {snap}") + meta = json.loads((snap / "meta.json").read_text()) + tables = meta["pg_tables"] + + # --- PostgreSQL: truncate the snapshot's tables, then pg_restore data --- + print(f"[snapshot] truncating {len(tables)} table(s) before restore") + subprocess.run( + _pg_args("psql") + ["-c", f"TRUNCATE {', '.join(tables)} CASCADE;"], + capture_output=True, text=True, env=_pg_env(), timeout=120, check=True, + ) + print(f"[snapshot] pg_restore <- {snap / 'pg_memory.dump'}") + res = subprocess.run( + _pg_args("pg_restore") + ["--data-only", "--disable-triggers", + "-d", PG["db"], str(snap / "pg_memory.dump")], + capture_output=True, text=True, env=_pg_env(), timeout=600, + ) + # pg_restore returns non-zero on benign warnings; surface stderr but continue + if res.returncode != 0 and res.stderr.strip(): + print(f"[snapshot] pg_restore warnings:\n{res.stderr.strip()[:500]}") + + # --- Neo4j: wipe graph, recreate nodes + relationships --- + _neo4j_import(json.loads((snap / "neo4j_graph.json").read_text())) + print(f"[snapshot] loaded '{name}'") + + +def _neo4j_import(graph: dict) -> None: + if not graph["nodes"]: + return + try: + from neo4j import GraphDatabase + except ImportError: + print("[snapshot] neo4j driver not installed; graph restore skipped") + return + driver = GraphDatabase.driver(NEO4J["uri"], auth=(NEO4J["user"], NEO4J["password"])) + with driver.session(database=NEO4J["database"]) as ses: + ses.run("MATCH (n) DETACH DELETE n") + # Recreate nodes; keep the original id() under _snap_id to rewire edges. + for n in graph["nodes"]: + labels = ":".join(n["labels"]) if n["labels"] else "Node" + ses.run( + f"CREATE (x:{labels}) SET x = $props, x._snap_id = $sid", + props=n["props"], sid=n["id"], + ) + for r in graph["relationships"]: + ses.run( + f"MATCH (a {{_snap_id: $src}}), (b {{_snap_id: $dst}}) " + f"CREATE (a)-[r:{r['type']}]->(b) SET r = $props", + src=r["src"], dst=r["dst"], props=r["props"], + ) + ses.run("MATCH (n) REMOVE n._snap_id") + driver.close() + print(f"[snapshot] neo4j restored: {len(graph['nodes'])} nodes, " + f"{len(graph['relationships'])} relationships") + + +# --------------------------------------------------------------- list/rm -- +def list_snapshots() -> None: + if not SNAP_ROOT.exists(): + print("(no snapshots)") + return + for snap in sorted(SNAP_ROOT.iterdir()): + meta_file = snap / "meta.json" + if meta_file.exists(): + m = json.loads(meta_file.read_text()) + print(f" {m['name']:30s} pg_tables={len(m['pg_tables'])} " + f"graph={m['neo4j_nodes']}n/{m['neo4j_relationships']}r " + f"agents={'yes' if m.get('includes_agents') else 'no'}") + + +def delete(name: str) -> None: + import shutil + snap = SNAP_ROOT / name + if snap.exists(): + shutil.rmtree(snap) + print(f"[snapshot] deleted '{name}'") + else: + print(f"[snapshot] '{name}' not found") + + +def main() -> None: + parser = argparse.ArgumentParser(description="Save/load Mirix memory snapshots.") + sub = parser.add_subparsers(dest="cmd", required=True) + + p_save = sub.add_parser("save", help="Snapshot current memory state.") + p_save.add_argument("name") + p_save.add_argument("--agents", action="store_true", + help="Also snapshot agents/messages (for fully ingest-free QA).") + + p_load = sub.add_parser("load", help="Restore a snapshot (REPLACES current memory).") + p_load.add_argument("name") + + sub.add_parser("list", help="List snapshots.") + + p_del = sub.add_parser("delete", help="Delete a snapshot.") + p_del.add_argument("name") + + args = parser.parse_args() + if args.cmd == "save": + save(args.name, include_agents=args.agents) + elif args.cmd == "load": + load(args.name) + elif args.cmd == "list": + list_snapshots() + elif args.cmd == "delete": + delete(args.name) + + +if __name__ == "__main__": + main() diff --git a/evals/mirix_memory_system.py b/evals/mirix_memory_system.py index 3246e9ba6..8255844ce 100644 --- a/evals/mirix_memory_system.py +++ b/evals/mirix_memory_system.py @@ -48,7 +48,10 @@ class MirixMemorySystem: def __init__(self, user_id: Optional[str] = None, mirix_config_path: Optional[str] = None, client_id: Optional[str] = None, org_id: Optional[str] = None, client: Optional[MirixClient] = None): if client is None: - self.client = MirixClient(client_id=client_id, org_id=org_id, base_url="http://127.0.0.1:8531", write_scope="read_write") + # timeout: per-request budget. With sentence-token chunking + # at 4096 gpt-4o-mini tokens (~16k chars), one /memory/add_sync + # takes 3-6 min server-side. Keep 30 min headroom. + self.client = MirixClient(client_id=client_id, org_id=org_id, base_url="http://127.0.0.1:8531", write_scope="read_write", timeout=1800) config_path = Path(mirix_config_path) if mirix_config_path else Path(__file__).with_name("mirix_openai.yaml") with config_path.open("r", encoding="utf-8") as handle: config = yaml.safe_load(handle) or {} @@ -61,8 +64,16 @@ def __init__(self, user_id: Optional[str] = None, mirix_config_path: Optional[st self.user_id = user_id if user_id is not None else str(uuid.uuid4()) - def add_chunk(self, chunk: str, raw_input: Optional[str] = None, async_add: bool = False): - response = asyncio.run(self.client.add( + def add_chunk(self, chunk: str, raw_input: Optional[str] = None, async_add: bool = False, + occurred_at: Optional[str] = None): + """Ingest one chunk. + + occurred_at: optional ISO 8601 timestamp for when this chunk's events + happened. Passing it anchors the episodic agent's occurred_at instead + of letting the LLM guess a year (LongMemEval conversations carry their + own per-session "Chat Time" — see longmem_eval.py). + """ + add_kwargs = dict( user_id=self.user_id, messages=[ {"role": "user", "content": chunk} @@ -71,7 +82,10 @@ def add_chunk(self, chunk: str, raw_input: Optional[str] = None, async_add: bool chaining=False, filter_tags={"scope": "read_write", "kind": "conversation_session"}, async_add=async_add, - )) + ) + if occurred_at is not None: + add_kwargs["occurred_at"] = occurred_at + response = asyncio.run(self.client.add(**add_kwargs)) return response def wrap_user_prompt(self, prompt: str): diff --git a/evals/organize_results.py b/evals/organize_results.py index fa5da9813..dd57e8f59 100644 --- a/evals/organize_results.py +++ b/evals/organize_results.py @@ -7,6 +7,10 @@ import tiktoken from llm_judge import evaluate_llm_judge +from llm_judge_mab import SUPPORTED_TASKS as MAB_SUPPORTED_TASKS +from llm_judge_mab import evaluate_mab_judge +from llm_judge_mab_summary import evaluate_mab_summary_judge +from llm_judge_substring import evaluate_substring_judge from tqdm import tqdm def load_json(path: Path) -> Optional[Dict[str, Any]]: @@ -148,9 +152,13 @@ def build_judge_tasks( records: Dict[str, Dict[str, Any]], base_index: int, cached_lookup: Dict[Tuple[Any, Any], Dict[str, Any]], -) -> Tuple[List[Optional[Dict[str, Any]]], List[Tuple[int, str, Any, Any, Any, Any, Any]]]: +) -> Tuple[List[Optional[Dict[str, Any]]], List[Tuple[int, str, Any, Any, Any, Any, Any, Any, Any]]]: results: List[Optional[Dict[str, Any]]] = [] - tasks: List[Tuple[int, str, Any, Any, Any, Any, Any]] = [] + # Task tuple = (idx, sample_id, qidx, category, question, expected, + # predicted, question_id, keypoints). keypoints is populated by + # lru_eval for summarization records (mab_summary judge); other + # runners leave it None and other judges ignore it. + tasks: List[Tuple[int, str, Any, Any, Any, Any, Any, Any, Any]] = [] for local_idx, (_, record) in enumerate(sorted_record_items(records)): if not isinstance(record, dict): @@ -208,17 +216,80 @@ def build_judge_tasks( question, expected_answer, predicted_answer, + record.get("question_id"), + record.get("keypoints"), ) ) return results, tasks -def judge_task(task: Tuple[int, str, Any, Any, Any, Any, Any]) -> Tuple[int, int, str]: - idx, _sample_id, _question_index, _category, question, expected_answer, predicted_answer = task +def judge_task(task: Tuple[int, str, Any, Any, Any, Any, Any, Any, Any]) -> Tuple[int, float, str, Optional[Dict[str, Any]]]: + idx, _sample_id, _question_index, _category, question, expected_answer, predicted_answer, _question_id, _keypoints = task score = evaluate_llm_judge(question, expected_answer, predicted_answer) label = "CORRECT" if score == 1 else "WRONG" - return idx, score, label + return idx, score, label, None + + +def judge_task_substring(task: Tuple[int, str, Any, Any, Any, Any, Any, Any, Any]) -> Tuple[int, float, str, Optional[Dict[str, Any]]]: + """Substring-match judge (no LLM). For RULER / SQuAD / HotpotQA-style QA.""" + idx, _sample_id, _question_index, _category, _question, expected_answer, predicted_answer, _question_id, _keypoints = task + score = evaluate_substring_judge(predicted_answer, expected_answer) + label = "CORRECT" if score == 1 else "WRONG" + return idx, score, label, None + + +def judge_task_mab(task: Tuple[int, str, Any, Any, Any, Any, Any, Any, Any]) -> Tuple[int, float, str, Optional[Dict[str, Any]]]: + """MemoryAgentBench-aligned judge. Routes by ``category`` to per-task prompts. + + Abstention detection: if ``question_id`` carries the ``_abs`` suffix + (matches official llm_based_eval/longmem_qa_evaluate.py behaviour), + route to the abstention prompt instead of the category prompt. Old + records without ``question_id`` quietly default to abstention=False. + + Unknown categories fall back to the generic LoCoMo judge so a mixed-source + run does not crash; they are tagged ``WRONG-unknown-cat`` in the label so + they are easy to spot in the output. + """ + idx, _sample_id, _question_index, category, question, expected_answer, predicted_answer, question_id, _keypoints = task + abstention = isinstance(question_id, str) and "_abs" in question_id + if isinstance(category, str) and category in MAB_SUPPORTED_TASKS: + score = evaluate_mab_judge( + question, expected_answer, predicted_answer, category, + abstention=abstention, + ) + label = "CORRECT" if score == 1 else "WRONG" + if abstention: + label += "-abs" + else: + score = evaluate_llm_judge(question, expected_answer, predicted_answer) + label = ("CORRECT" if score == 1 else "WRONG") + "-unknown-cat" + return idx, score, label, None + + +def judge_task_mab_summary(task: Tuple[int, str, Any, Any, Any, Any, Any, Any, Any]) -> Tuple[int, float, str, Optional[Dict[str, Any]]]: + """Official MAB summarization judge (LRU / infbench_sum_eng_shots2). + + Runs three gpt-4o-2024-05-13 calls per record (fluency, recall, + precision) and folds them into F1. The per-record ``score`` is F1 + (a float in [0,1]); the four sub-scores live in ``details`` so the + metrics file can report averaged_fluency / recall / precision / F1 + matching the official ``averaged_metrics`` shape. + """ + idx, _sample_id, _question_index, _category, _question, expected_answer, predicted_answer, _question_id, keypoints = task + details = evaluate_mab_summary_judge( + generated_summary=predicted_answer or "", + expert_summary=expected_answer or "", + keypoints=keypoints or [], + ) + f1 = details["f1"] + label = ( + f"F1={f1:.3f} " + f"(fluency={details['fluency']}, " + f"recall={details['recall_found']}/{details['recall_total']}, " + f"precision={details['precision_found']}/{details['precision_total']})" + ) + return idx, f1, label, details def main() -> None: @@ -236,12 +307,41 @@ def main() -> None: default=None, help="Output JSON file (default: /metrics.json).", ) + parser.add_argument( + "--judge", + choices=("default", "mab", "substring", "mab_summary"), + default="default", + help=( + "Which judge to use:\n" + " default — generic LoCoMo LLM judge (gpt-4o-mini, CORRECT/WRONG).\n" + " mab — MemoryAgentBench-aligned (gpt-4o, per-category prompts,\n" + " yes/no scoring, abstention via question_id _abs suffix).\n" + " Required for LongMemEval-S to be leaderboard-comparable.\n" + " substring — substring_exact_match (no LLM). For RULER / SQuAD /\n" + " HotpotQA-style QA where answers are short needles.\n" + " mab_summary — MAB summarization judge for LRU / infbench_sum (gpt-4o-\n" + " 2024-05-13, three calls per record → fluency*P*R/(P+R)\n" + " F1). Records must carry `keypoints`." + ), + ) + parser.add_argument( + "--mab-judge", + action="store_true", + help="Deprecated alias for --judge mab. Kept for backwards compatibility.", + ) args = parser.parse_args() input_dir = args.input_dir output_file = args.output_file or (input_dir / "metrics.json") - cached_metrics = load_json(output_file) if output_file.exists() else None + # Honour the legacy boolean alias. + if args.mab_judge and args.judge == "default": + args.judge = "mab" + + # When switching judges, the cached labels were produced by another + # judge and are not comparable. Only reuse cache when --judge default. + use_cache = (args.judge == "default") + cached_metrics = load_json(output_file) if (output_file.exists() and use_cache) else None cached_judge_results = None if isinstance(cached_metrics, dict): cached_judge_results = cached_metrics.get("llm_judge_results") @@ -269,7 +369,7 @@ def main() -> None: total_correct = 0 total_judged = 0 judge_results: List[Optional[Dict[str, Any]]] = [] - judge_tasks: List[Tuple[int, str, Any, Any, Any, Any, Any]] = [] + judge_tasks: List[Tuple[int, str, Any, Any, Any, Any, Any, Any]] = [] # Initialize tiktoken encoding for gpt-4o-mini encoding = tiktoken.encoding_for_model("gpt-4o-mini") @@ -319,15 +419,31 @@ def main() -> None: if judge_tasks: task_by_index = {task[0]: task for task in judge_tasks} ctx = mp.get_context("spawn") - with ctx.Pool(processes=16) as pool: - for idx, score, label in tqdm( - pool.imap(judge_task, judge_tasks), + judge_fn = { + "mab": judge_task_mab, + "substring": judge_task_substring, + "default": judge_task, + "mab_summary": judge_task_mab_summary, + }[args.judge] + # Substring is local-only; one worker is enough and avoids + # spawning 16 idle processes for a free op. mab_summary makes + # three sequential gpt-4o calls per task — keep concurrency + # modest to respect rate limits. + if args.judge == "substring": + pool_size = 1 + elif args.judge == "mab_summary": + pool_size = 4 + else: + pool_size = 16 + with ctx.Pool(processes=pool_size) as pool: + for idx, score, label, details in tqdm( + pool.imap(judge_fn, judge_tasks), total=len(judge_tasks), - desc="Evaluating records", + desc=f"Evaluating records ({args.judge} judge)", ): task = task_by_index[idx] - _, sample_id, question_index, category, question, expected_answer, predicted_answer = task - judge_results[idx] = { + _, sample_id, question_index, category, question, expected_answer, predicted_answer, _question_id, _keypoints = task + entry = { "sample_id": sample_id, "question_index": question_index, "category": category, @@ -337,6 +453,9 @@ def main() -> None: "expected_answer": expected_answer, "predicted_answer": predicted_answer, } + if details is not None: + entry["details"] = details + judge_results[idx] = entry if category != 5: total_correct += score total_judged += 1 @@ -353,13 +472,16 @@ def main() -> None: memory_file_count += 1 finalized_results = [result for result in judge_results if result is not None] - total_correct = 0 + # Scores are kept as floats so mab_summary's per-record F1 ([0,1]) is + # averaged correctly; for binary judges this still gives an integer + # count out of total_judged. + total_correct: float = 0.0 total_judged = 0 for result in finalized_results: if isinstance(result, dict) and result.get("category") != 5: score = result.get("score") if isinstance(score, (int, float)): - total_correct += int(score) + total_correct += float(score) total_judged += 1 accuracy = (total_correct / total_judged) if total_judged else None @@ -377,17 +499,34 @@ def main() -> None: continue key = str(category) entry = category_metrics.setdefault( - key, {"category": category, "total_judged": 0, "total_correct": 0, "accuracy": None} + key, {"category": category, "total_judged": 0, "total_correct": 0.0, "accuracy": None} ) score = result.get("score") if isinstance(score, (int, float)): entry["total_judged"] += 1 - entry["total_correct"] += int(score) + entry["total_correct"] += float(score) for entry in category_metrics.values(): judged = entry.get("total_judged", 0) entry["accuracy"] = (entry["total_correct"] / judged) if judged else None + # Official MAB summarization metric — averages of per-record sub-scores. + # Only present when the run used the mab_summary judge. + summarization_metrics: Optional[Dict[str, float]] = None + if args.judge == "mab_summary": + sub = {"fluency": [], "recall": [], "precision": [], "f1": []} + for result in finalized_results: + details = result.get("details") if isinstance(result, dict) else None + if not isinstance(details, dict): + continue + for key in sub: + v = details.get(key) + if isinstance(v, (int, float)): + sub[key].append(float(v)) + summarization_metrics = { + f"gpt-4-{k}": (sum(vs) / len(vs)) if vs else 0.0 for k, vs in sub.items() + } + average_memory_tokens = (total_memory_tokens / memory_file_count) if memory_file_count else None output = { @@ -407,6 +546,7 @@ def main() -> None: "total_memory_tokens": total_memory_tokens, "memory_file_count": memory_file_count, "average_memory_tokens": average_memory_tokens, + **(summarization_metrics or {}), }, "accuracy_by_category": category_metrics, "latency_breakdown_seconds": latency_totals, diff --git a/evals/ruler_eval.py b/evals/ruler_eval.py new file mode 100644 index 000000000..0aa4d5fd1 --- /dev/null +++ b/evals/ruler_eval.py @@ -0,0 +1,388 @@ +"""Evaluate Mirix memory on MemoryAgentBench RULER QA (SHDOCQA / MHDOCQA). + +Sister script to longmem_eval.py. Same scaffolding (MirixMemorySystem, +TaskAgent, server-side token tracker, per-sample JSON schema) so +organize_results.py works on the output unchanged. + +Data differs: RULER QA rows come from `ai-hyz/MemoryAgentBench`, split +`Accurate_Retrieval`, with `metadata.source` == +``ruler_qa1_197K`` (single-hop, SHDOCQA, SQuAD-derived) or +``ruler_qa2_421K`` (multi-hop, MHDOCQA, HotpotQA-derived). Each row is +one long needle-in-haystack context (~1-2M chars) of concatenated +``Document N:`` blocks plus 100 short QA pairs (answers are lists of +acceptable substrings — no LLM judge needed; use ``--judge substring`` +in organize_results.py). + +The context has no timestamps, so ``occurred_at`` is always None. +Documents are bundled greedily into <= max_chunk_chars chunks on +document boundaries. + +Usage (smoke test — 1 conv, 15 chunks, 5 questions): + python ruler_eval.py --limit 1 --max-chunks 15 --max-questions 5 \\ + --run-llm --mirix_config_path ./configs/mab.yaml \\ + --output_path smoke_ruler + +Output lands in evals/results/ruler// — its own namespace, +separate from LongMemEval and LoCoMo. +""" + +import argparse +import json +import os +import time +from pathlib import Path +from typing import Dict, List, Optional + +from mirix_memory_system import MirixMemorySystem +from task_agent import TaskAgent + +# RULER QA in MemoryAgentBench: +# SHDOCQA = ruler_qa1_197K (single-hop, SQuAD-derived) +# MHDOCQA = ruler_qa2_421K (multi-hop, HotpotQA-derived) +# Default points at SHDOCQA; swap via --source. +DEFAULT_RULER_SOURCE = "ruler_qa1_197K" + + +def load_ruler_qa( + source: str = DEFAULT_RULER_SOURCE, + limit: Optional[int] = None, +) -> List[Dict]: + """Load RULER QA samples (SHDOCQA / MHDOCQA) from HuggingFace. + + Returns a list of dicts shaped as: + {sample_id, context, questions, answers} + + Notes vs longmem_eval.load_longmem_s: + - No ``question_types``: RULER does not categorize questions. + - No ``question_ids``: official metadata.question_ids is empty + for these sources, so abstention detection is moot. + - ``answers[i]`` is itself a list of accepted strings (e.g. + ``['France', 'France', 'France', 'France']``). Kept as-is; the + substring judge handles list-or-string transparently. + """ + try: + from datasets import load_dataset + except ImportError as exc: # pragma: no cover + raise SystemExit( + "The `datasets` package is required for RULER QA. " + "Install it with: uv pip install datasets" + ) from exc + + ds = load_dataset("ai-hyz/MemoryAgentBench", split="Accurate_Retrieval") + + # Namespace sample_id by source so SHDOCQA / MHDOCQA can share the + # same Postgres DB without their user_id rows getting mixed up + # (both subsets number from 0, so an un-prefixed ruler_qa_0 would + # collide in episodic_memory.user_id and corrupt measure_memory_size). + source_tag = { + "ruler_qa1_197K": "shdocqa", + "ruler_qa2_421K": "mhdocqa", + }.get(source, source.lower()) + + samples: List[Dict] = [] + for row in ds: + meta = row.get("metadata") or {} + if meta.get("source") != source: + continue + idx = len(samples) + samples.append( + { + "sample_id": f"{source_tag}_ruler_qa_{idx}", + "context": row.get("context") or "", + "questions": list(row.get("questions") or []), + "answers": list(row.get("answers") or []), + # Empty lists — kept for shape compatibility with the + # downstream record builder; abstention prompt is never + # triggered on RULER. + "question_types": [], + "question_ids": [], + } + ) + if limit is not None and len(samples) >= limit: + break + + if not samples: + raise SystemExit( + f"No rows found in ai-hyz/MemoryAgentBench (Accurate_Retrieval) " + f"with metadata.source == {source!r}. " + f"Try {DEFAULT_RULER_SOURCE!r} or 'ruler_qa2_421K' (MHDOCQA)." + ) + return samples + + +from _chunking import DEFAULT_CHUNK_TOKENS, chunk_text_into_sentences + + +def parse_documents(context: str, max_chunk_tokens: int = DEFAULT_CHUNK_TOKENS) -> List[Dict]: + """Sentence-aware, token-budgeted chunking for a RULER context. + + Delegates to the shared ``chunk_text_into_sentences`` (NLTK ``punkt`` + + tiktoken ``gpt-4o-mini``, 4096-token budget) — same shape used by + the official MAB chunker, so SHDOCQA / MHDOCQA chunks match the + leaderboard run. + + Note: we no longer pre-split on ``Document N:`` markers. The + official chunker doesn't either — sentences span document + boundaries inside a chunk, but that's the same trade-off the + leaderboard pays. RULER has no timestamps, so ``occurred_at`` is + always None. + + Returns: list of {occurred_at: None, text: str}. + """ + return [{"occurred_at": None, "text": c} + for c in chunk_text_into_sentences(context, chunk_size=max_chunk_tokens)] + + +def flatten_answer(answer) -> Optional[str]: + """LongMemEval answers are list-of-list (e.g. ['50']). Flatten to a string.""" + if answer is None: + return None + if isinstance(answer, list): + flat = [] + for a in answer: + if isinstance(a, list): + flat.extend(str(x) for x in a) + else: + flat.append(str(a)) + return "; ".join(flat) if flat else None + return str(answer) + + +from _eval_db import measure_memory_size, dump_memories + + +def load_sample_result(path: Path) -> Optional[Dict]: + if not path.exists(): + return None + try: + with path.open("r", encoding="utf-8") as handle: + data = json.load(handle) + if isinstance(data, dict): + return data + except (OSError, json.JSONDecodeError): + return None + return None + + +def save_sample_result(path: Path, data: Dict) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("w", encoding="utf-8") as handle: + json.dump(data, handle, ensure_ascii=False, indent=2) + + +def print_qa(qidx: int, question: str, expected: Optional[str], predicted: Optional[str]) -> None: + print(f"[{qidx}] question: {str(question)[:120]}") + print(f"[{qidx}] expected_answer: {expected}") + print(f"[{qidx}] predicted_answer: {str(predicted)[:200] if predicted else predicted}") + + +def main() -> None: + parser = argparse.ArgumentParser(description="Evaluate Mirix memory on RULER QA (SHDOCQA / MHDOCQA).") + parser.add_argument("--limit", type=int, default=None, + help="Limit number of conversations (samples) to evaluate.") + parser.add_argument("--max-chunks", type=int, default=None, + help="Limit session chunks ingested per sample (smoke-test knob).") + parser.add_argument("--max-questions", type=int, default=None, + help="Limit number of questions per sample.") + parser.add_argument("--run-llm", action="store_true", default=True, + help="Call the LLM to answer questions.") + parser.add_argument("--output_path", type=Path, default=Path("ruler_run"), + help="Output sub-folder, resolved under evals/results/ruler/.") + parser.add_argument( + "--source", + type=str, + default=DEFAULT_RULER_SOURCE, + help=( + "HF metadata.source to load (exact match). Default " + f"{DEFAULT_RULER_SOURCE!r} (SHDOCQA). Swap to " + "'ruler_qa2_421K' for MHDOCQA." + ), + ) + parser.add_argument("--mirix_config_path", type=Path, default=None, + help="Path to Mirix config file.") + args = parser.parse_args() + + items = load_ruler_qa(source=args.source, limit=args.limit) + print(f"[ruler_eval] loaded {len(items)} RULER QA conversation(s) from source {args.source!r}") + + mirix_client_id = os.environ.get("MIRIX_CLIENT_ID", "mirix-eval-client") + mirix_org_id = os.environ.get("MIRIX_ORG_ID", "mirix-eval-org") + + # Keep RULER output in its own namespace, away from LongMemEval and LoCoMo. + ruler_root = Path(__file__).resolve().parent / "results" / "ruler" + if args.output_path.is_absolute(): + print(f"[ruler_eval] WARNING: --output_path is absolute ({args.output_path}); " + "writing outside evals/results/ruler/ namespace.") + output_path = args.output_path + else: + output_path = ruler_root / args.output_path + output_path.mkdir(parents=True, exist_ok=True) + print(f"[ruler_eval] writing per-sample results to {output_path}") + + import httpx + server_base = "http://127.0.0.1:8531" + + def _reset_tokens(): + try: + httpx.post(f"{server_base}/debug/token_stats/reset", timeout=10) + except Exception: + pass + + def _snapshot_tokens(): + try: + r = httpx.get(f"{server_base}/debug/token_stats", timeout=10) + return r.json().get("stats", {}) + except Exception: + return {} + + def _sum_tokens(stats): + s = {"prompt": 0, "completion": 0, "total": 0, "calls": 0} + for v in stats.values(): + for k in s: + s[k] += v.get(k, 0) + return s + + for item in items: + sample_id = item["sample_id"] + sample_path = output_path / f"{sample_id}.json" + + task_agent = ( + TaskAgent(mirix_config_path=str(args.mirix_config_path), + client_id=mirix_client_id, org_id=mirix_org_id, user_id=sample_id) + if args.run_llm else None + ) + + sample_result = load_sample_result(sample_path) or { + "sample_id": sample_id, + "timings": {"add_chunk": {}, "wrap_user_prompt": {}, "answer": {}}, + "responses": {}, + "records": {}, + } + sample_result.setdefault("sample_id", sample_id) + + memory_system = MirixMemorySystem( + user_id=sample_id, + mirix_config_path=str(args.mirix_config_path), + client=task_agent.mirix_client if task_agent else None, + ) + + _reset_tokens() + + # ---- ingest: one chunk per conversation session, WITH timestamps ---- + chunks = parse_documents(item["context"]) + if args.max_chunks is not None: + chunks = chunks[: args.max_chunks] + dated = sum(1 for c in chunks if c["occurred_at"]) + print(f"[ruler_eval] {sample_id}: ingesting {len(chunks)} document chunk(s) " + f"({dated} with timestamps)") + + for idx, chunk in enumerate(chunks, start=1): + idx_key = str(idx) + if idx_key in sample_result["responses"]: + continue + start = time.perf_counter() + response = memory_system.add_chunk( + chunk["text"], raw_input=chunk["text"], + occurred_at=chunk["occurred_at"], + ) + elapsed = time.perf_counter() - start + sample_result["responses"][idx_key] = { + "type": "add_chunk", + "chunk_index": idx, + "question_index": None, + "occurred_at": chunk["occurred_at"], + "response": response, + } + sample_result["timings"]["add_chunk"][idx_key] = elapsed + save_sample_result(sample_path, sample_result) + + build_stats = _snapshot_tokens() + sample_result["token_stats"] = {"build_raw": build_stats, "build_sum": _sum_tokens(build_stats)} + save_sample_result(sample_path, sample_result) + + # Memory size in a common unit (stored chars) so no-graph (flat PG) and + # graph (Neo4j) runs are directly comparable. See measure_memory_size(). + sample_result["memory_stats"] = measure_memory_size(sample_id) + print(f"[ruler_eval] {sample_id}: memory_stats = {sample_result['memory_stats']}") + save_sample_result(sample_path, sample_result) + + # ---- QA ---- + questions = item["questions"] + answers = item["answers"] + qtypes = item["question_types"] + qids = item.get("question_ids") or [] + if args.max_questions is not None: + questions = questions[: args.max_questions] + + for qidx, question in enumerate(questions, start=1): + qidx_key = str(qidx) + if qidx_key in sample_result["records"]: + rec = sample_result["records"][qidx_key] + print_qa(qidx, rec.get("question", ""), + rec.get("expected_answer"), rec.get("predicted_answer")) + continue + + expected_answer = flatten_answer(answers[qidx - 1] if qidx - 1 < len(answers) else None) + # category: organize_results.py groups metrics by this. LongMemEval + # uses string question types (multi-session, temporal-reasoning, ...). + category = qtypes[qidx - 1] if qidx - 1 < len(qtypes) else None + + start = time.perf_counter() + input_messages = memory_system.wrap_user_prompt(question) + sample_result["timings"]["wrap_user_prompt"][qidx_key] = time.perf_counter() - start + + predicted = None + message_trace = None + usage_trace = None + usage_total = None + if task_agent: + start = time.perf_counter() + trace = task_agent.answer(input_messages, user_id=sample_id) + predicted = trace.get("answer") + message_trace = trace.get("messages") + usage_trace = trace.get("usage") + usage_total = trace.get("usage_total") + sample_result["timings"]["answer"][qidx_key] = time.perf_counter() - start + + sample_result["records"][qidx_key] = { + "sample_id": sample_id, + "question_index": qidx, + # HF metadata.question_ids[qidx-1]. Carries the optional + # ``_abs`` suffix that routes abstention questions to the + # MAB judge's abstention prompt. + "question_id": qids[qidx - 1] if qidx - 1 < len(qids) else None, + "question": question, + "expected_answer": expected_answer, + "evidence": None, + "category": category, + "input_messages": input_messages, + "predicted_answer": predicted, + "messages": message_trace, + "usage": usage_trace, + "usage_total": usage_total, + } + save_sample_result(sample_path, sample_result) + print_qa(qidx, question, expected_answer, predicted) + + try: + all_memories = dump_memories(sample_id) + except Exception as exc: + all_memories = {"success": False, "error": str(exc), "user_id": sample_id} + with (output_path / f"{sample_id}_memories.json").open("w", encoding="utf-8") as handle: + json.dump(all_memories, handle, ensure_ascii=False, indent=2) + + post_qa_stats = _snapshot_tokens() + post_qa_sum = _sum_tokens(post_qa_stats) + build_sum = sample_result.get("token_stats", {}).get("build_sum", {}) + query_sum = { + k: max(post_qa_sum.get(k, 0) - build_sum.get(k, 0), 0) + for k in ("prompt", "completion", "total", "calls") + } + sample_result.setdefault("token_stats", {}) + sample_result["token_stats"]["query_raw"] = post_qa_stats + sample_result["token_stats"]["query_sum"] = query_sum + save_sample_result(sample_path, sample_result) + + +if __name__ == "__main__": + main() diff --git a/evals/run_mab_longmem_eval.sh b/evals/run_mab_longmem_eval.sh new file mode 100755 index 000000000..327b3c929 --- /dev/null +++ b/evals/run_mab_longmem_eval.sh @@ -0,0 +1,131 @@ +#!/bin/bash +# Run MAB-enabled eval (conflict resolution + source provenance) on +# LongMemEval-S (HuggingFace ai-hyz/MemoryAgentBench) end-to-end. +# +# Prereqs: +# 1. `.env` has OPENAI_API_KEY. +# 2. Server is up: http://localhost:8531/health returns 200. +# (start with `python scripts/start_server.py --port 8531`) +# 3. `datasets` is installed in the venv (the HF library); the runner +# raises a clear error if not. +# +# Optional env overrides (handy for quick tests): +# LIMIT=N — how many LongMemEval-S conversations to ingest (default: 1) +# MAX_CHUNKS=N — cap ingest chunks per conversation (default: no cap, full conv) +# MAX_QS=N — cap QA questions per conversation (default: no cap, all 60) +# +# Outputs: +# evals/results/longmem/longmem_mab_/longmem_s_0.json — per-question records +# evals/results/longmem/longmem_mab_/longmem_s_0_memories.json +# evals/results/longmem/longmem_mab_/metrics.json — LLM-judge accuracy +# evals/snapshots/longmem_mab_/pg_memory.dump — full PG dump +# evals/snapshots/longmem_mab_/results/ — copy of the result dir above +# evals/snapshots/longmem_mab_/{meta,neo4j_graph}.json — snapshot metadata + +set -euo pipefail +cd "$(dirname "$0")/.." + +# ---- pre-flight ------------------------------------------------------------ +if ! lsof -ti:8531 >/dev/null 2>&1; then + echo "ERROR: nothing listening on :8531" >&2 + echo " start the server first: python scripts/start_server.py --port 8531" >&2 + exit 1 +fi + +python - <<'PY' || { echo "ERROR: server /health not 200" >&2; exit 1; } +import sys, urllib.request +try: + r = urllib.request.urlopen("http://localhost:8531/health", timeout=5) + sys.exit(0 if r.status == 200 else 2) +except Exception as e: + print(f" health probe: {e}", file=sys.stderr) + sys.exit(3) +PY + +python - <<'PY' || { echo "ERROR: \`datasets\` not installed in this Python (uv pip install datasets)" >&2; exit 1; } +import importlib, sys +sys.exit(0 if importlib.util.find_spec("datasets") else 1) +PY + +# ---- run ------------------------------------------------------------------- +cd evals +TS=$(date +%Y%m%d_%H%M%S) +OUT="longmem_mab_${TS}" + +# Assemble optional caps from env +EXTRA_ARGS=() +[ -n "${MAX_CHUNKS:-}" ] && EXTRA_ARGS+=( "--max-chunks" "$MAX_CHUNKS" ) +[ -n "${MAX_QS:-}" ] && EXTRA_ARGS+=( "--max-questions" "$MAX_QS" ) +LIMIT_VAL="${LIMIT:-1}" + +echo "[1/3] running longmem_eval.py (MAB-enabled) -> ${OUT}/" +echo " limit=${LIMIT_VAL} max_chunks=${MAX_CHUNKS:-} max_questions=${MAX_QS:-}" +python longmem_eval.py \ + --limit "$LIMIT_VAL" \ + --run-llm \ + --mirix_config_path ./configs/mab.yaml \ + --output_path "$OUT" \ + ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"} + +echo +echo "[2/3] running organize_results.py (LLM judge)" +# longmem_eval writes under results/longmem//; organize_results.py +# defaults its fallback path to results/locomo/, so we pass the full path. +python organize_results.py --mab-judge "results/longmem/${OUT}" + +echo +echo "[3/3] saving snapshot (pg_dump + neo4j export + results copy)" +# Non-fatal: the eval + judge already succeeded by here. If the snapshot +# tool can't reach pg_dump or the neo4j driver is missing we still want +# the run to be considered successful. +if MIRIX_PG_DB="${MIRIX_PG_DB:-mirix}" python memory_snapshot.py save "${OUT}" --agents; then + # Co-locate the results inside the snapshot directory so a single + # archive folder has everything needed to verify later. + cp -R "results/longmem/${OUT}" "snapshots/${OUT}/results" + echo " snapshot + results -> evals/snapshots/${OUT}/" +else + echo " WARN: memory_snapshot.py failed; results still at evals/results/longmem/${OUT}/" >&2 +fi + +# ---- summary --------------------------------------------------------------- +echo +echo "========================================================" +echo " Summary" +echo "========================================================" +python - "results/longmem/${OUT}" <<'PY' +import json, sys +from pathlib import Path + +out_dir = Path(sys.argv[1]) +m = json.loads((out_dir / "metrics.json").read_text()) +mt = m["metrics"] + +print(f" Output: {out_dir}") +print(f" Acc: {mt['accuracy']:.4f} ({mt['total_correct']}/{mt['total_judged']})") +print(f" Total questions: {mt.get('total_questions')}") +print(f" Avg answer lat.: {mt.get('average_answer_latency_seconds', 0):.2f}s") +print(f" Avg mem tokens: {mt.get('average_memory_tokens')}") + +cat_acc = m.get("accuracy_by_category") or {} +if cat_acc: + print() + print(" Per category:") + for k, v in sorted(cat_acc.items()): + n = v.get("total_judged", 0) + a = v.get("accuracy", 0) + print(f" {str(k):20s} n={n:3d} acc={a:.4f}") +PY + +# ---- DB memory counts (handy for cross-run comparison) --------------------- +echo +echo "========================================================" +echo " Memory store counts (DB=\${MIRIX_PG_DB:-mirix})" +echo "========================================================" +PGPASSWORD="${MIRIX_PG_PASSWORD:-mirix}" psql -h "${MIRIX_PG_HOST:-localhost}" \ + -U "${MIRIX_PG_USER:-mirix}" -d "${MIRIX_PG_DB:-mirix}" -tAc \ + "select 'episodic='||count(*) from episodic_memory + union all select 'semantic='||count(*) from semantic_memory + union all select 'procedural='||count(*) from procedural_memory + union all select 'resource='||count(*) from resource_memory + union all select 'knowledge_vault='||count(*) from knowledge_vault" \ + 2>/dev/null || echo " (psql unavailable or DB unreachable; skipping)" diff --git a/evals/run_mab_lru_eval.sh b/evals/run_mab_lru_eval.sh new file mode 100755 index 000000000..fa9c56526 --- /dev/null +++ b/evals/run_mab_lru_eval.sh @@ -0,0 +1,147 @@ +#!/bin/bash +# Run MAB Long_Range_Understanding (LRU) eval end-to-end. +# +# Dataset: +# - infbench_sum_eng_shots2 (100, novel summarisation) ← default +# - detective_qa (10, multiple-choice mystery) +# Switch via SOURCE env var (see Optional overrides below). +# +# Judge (auto-routed by source): +# - infbench_sum_eng_shots2 → mab_summary (gpt-4o-2024-05-13, +# fluency × precision × recall → F1; aligned with official MAB +# llm_based_eval/summarization_evaluate.py) +# - detective_qa → substring (gold answers are short +# option strings like "C. The Brandt couple") +# Override with JUDGE env var. +# +# Prereqs: +# 1. `.env` has OPENAI_API_KEY. +# 2. Server is up: http://localhost:8531/health returns 200. +# (start with `python scripts/start_server.py --port 8531`) +# 3. `datasets` is installed in the venv. +# +# Optional env overrides: +# SOURCE=detective_qa — switch from infbench summarisation +# LIMIT=N — number of samples (default: 1) +# MAX_CHUNKS=N — cap ingest chunks per sample (default: full) +# MAX_QS=N — cap questions per sample (default: all) +# JUDGE=substring — override the source-based judge default +# +# Outputs: +# evals/results/lru/lru_mab_/.json +# evals/results/lru/lru_mab_/_memories.json +# evals/results/lru/lru_mab_/metrics.json +# evals/snapshots/lru_mab_/pg_memory.dump +# evals/snapshots/lru_mab_/results/ + +set -euo pipefail +cd "$(dirname "$0")/.." + +# ---- pre-flight ------------------------------------------------------------ +if ! lsof -ti:8531 >/dev/null 2>&1; then + echo "ERROR: nothing listening on :8531" >&2 + echo " start the server first: python scripts/start_server.py --port 8531" >&2 + exit 1 +fi + +python - <<'PY' || { echo "ERROR: server /health not 200" >&2; exit 1; } +import sys, urllib.request +try: + r = urllib.request.urlopen("http://localhost:8531/health", timeout=5) + sys.exit(0 if r.status == 200 else 2) +except Exception as e: + print(f" health probe: {e}", file=sys.stderr) + sys.exit(3) +PY + +python - <<'PY' || { echo "ERROR: \`datasets\` not installed in this Python (uv pip install datasets)" >&2; exit 1; } +import importlib, sys +sys.exit(0 if importlib.util.find_spec("datasets") else 1) +PY + +# ---- run ------------------------------------------------------------------- +cd evals +TS=$(date +%Y%m%d_%H%M%S) +OUT="lru_mab_${TS}" + +SOURCE_VAL="${SOURCE:-infbench_sum_eng_shots2}" +LIMIT_VAL="${LIMIT:-1}" + +# Source -> default judge mapping (overridable via JUDGE). +case "$SOURCE_VAL" in + infbench_sum_eng_shots2) DEFAULT_JUDGE="mab_summary" ;; + detective_qa) DEFAULT_JUDGE="substring" ;; + *) DEFAULT_JUDGE="mab_summary" ;; +esac +JUDGE_VAL="${JUDGE:-$DEFAULT_JUDGE}" + +EXTRA_ARGS=() +[ -n "${MAX_CHUNKS:-}" ] && EXTRA_ARGS+=( "--max-chunks" "$MAX_CHUNKS" ) +[ -n "${MAX_QS:-}" ] && EXTRA_ARGS+=( "--max-questions" "$MAX_QS" ) + +echo "[1/3] running lru_eval.py -> ${OUT}/" +echo " source=${SOURCE_VAL} limit=${LIMIT_VAL} judge=${JUDGE_VAL} max_chunks=${MAX_CHUNKS:-} max_questions=${MAX_QS:-}" +python lru_eval.py \ + --limit "$LIMIT_VAL" \ + --source "$SOURCE_VAL" \ + --run-llm \ + --mirix_config_path ./configs/mab.yaml \ + --output_path "$OUT" \ + ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"} + +echo +echo "[2/3] running organize_results.py (${JUDGE_VAL} judge)" +python organize_results.py --judge "$JUDGE_VAL" "results/lru/${OUT}" + +echo +echo "[3/3] saving snapshot (pg_dump + neo4j export + results copy)" +if MIRIX_PG_DB="${MIRIX_PG_DB:-mirix}" python memory_snapshot.py save "${OUT}" --agents; then + cp -R "results/lru/${OUT}" "snapshots/${OUT}/results" + echo " snapshot + results -> evals/snapshots/${OUT}/" +else + echo " WARN: memory_snapshot.py failed; results still at evals/results/lru/${OUT}/" >&2 +fi + +# ---- summary --------------------------------------------------------------- +echo +echo "========================================================" +echo " Summary" +echo "========================================================" +python - "results/lru/${OUT}" <<'PY' +import json, sys +from pathlib import Path + +out_dir = Path(sys.argv[1]) +m = json.loads((out_dir / "metrics.json").read_text()) +mt = m["metrics"] + +print(f" Output: {out_dir}") +acc = mt.get('accuracy') +if acc is not None: + print(f" Acc / mean F1: {acc:.4f} ({mt['total_correct']:.2f}/{mt['total_judged']})") +print(f" Total questions: {mt.get('total_questions')}") +lat = mt.get('average_answer_latency_seconds') +if lat is not None: + print(f" Avg answer lat.: {lat:.2f}s") +print(f" Avg mem tokens: {mt.get('average_memory_tokens')}") + +# Per-sub-score breakdown when mab_summary was used. +for k in ("gpt-4-fluency", "gpt-4-recall", "gpt-4-precision", "gpt-4-f1"): + v = mt.get(k) + if v is not None: + print(f" {k:20s} {v:.4f}") +PY + +# ---- DB memory counts ------------------------------------------------------ +echo +echo "========================================================" +echo " Memory store counts (DB=\${MIRIX_PG_DB:-mirix})" +echo "========================================================" +PGPASSWORD="${MIRIX_PG_PASSWORD:-mirix}" psql -h "${MIRIX_PG_HOST:-localhost}" \ + -U "${MIRIX_PG_USER:-mirix}" -d "${MIRIX_PG_DB:-mirix}" -tAc \ + "select 'episodic='||count(*) from episodic_memory + union all select 'semantic='||count(*) from semantic_memory + union all select 'procedural='||count(*) from procedural_memory + union all select 'resource='||count(*) from resource_memory + union all select 'knowledge_vault='||count(*) from knowledge_vault" \ + 2>/dev/null || echo " (psql unavailable or DB unreachable; skipping)" diff --git a/evals/run_mab_ruler_eval.sh b/evals/run_mab_ruler_eval.sh new file mode 100755 index 000000000..7420d5a21 --- /dev/null +++ b/evals/run_mab_ruler_eval.sh @@ -0,0 +1,123 @@ +#!/bin/bash +# Run MAB RULER QA eval (SHDOCQA / MHDOCQA) end-to-end. +# +# Dataset: +# - SHDOCQA = ruler_qa1_197K (single-hop, SQuAD-derived) ← default +# - MHDOCQA = ruler_qa2_421K (multi-hop, HotpotQA-derived) +# Switch via SOURCE env var (see Optional overrides below). +# +# Prereqs: +# 1. `.env` has OPENAI_API_KEY. +# 2. Server is up: http://localhost:8531/health returns 200. +# (start with `python scripts/start_server.py --port 8531`) +# 3. `datasets` is installed in the venv. +# +# Optional env overrides: +# SOURCE=ruler_qa2_421K — switch to MHDOCQA (default is SHDOCQA / qa1_197K) +# LIMIT=N — number of conversations (default: 1) +# MAX_CHUNKS=N — cap ingest chunks per conv (default: full) +# MAX_QS=N — cap QA questions per conv (default: all 100) +# +# Outputs: +# evals/results/ruler/ruler_mab_/ruler_qa_0.json — per-question records +# evals/results/ruler/ruler_mab_/ruler_qa_0_memories.json +# evals/results/ruler/ruler_mab_/metrics.json — substring judge accuracy +# evals/snapshots/ruler_mab_/pg_memory.dump — full PG dump +# evals/snapshots/ruler_mab_/results/ — copy of the result dir + +set -euo pipefail +cd "$(dirname "$0")/.." + +# ---- pre-flight ------------------------------------------------------------ +if ! lsof -ti:8531 >/dev/null 2>&1; then + echo "ERROR: nothing listening on :8531" >&2 + echo " start the server first: python scripts/start_server.py --port 8531" >&2 + exit 1 +fi + +python - <<'PY' || { echo "ERROR: server /health not 200" >&2; exit 1; } +import sys, urllib.request +try: + r = urllib.request.urlopen("http://localhost:8531/health", timeout=5) + sys.exit(0 if r.status == 200 else 2) +except Exception as e: + print(f" health probe: {e}", file=sys.stderr) + sys.exit(3) +PY + +python - <<'PY' || { echo "ERROR: \`datasets\` not installed in this Python (uv pip install datasets)" >&2; exit 1; } +import importlib, sys +sys.exit(0 if importlib.util.find_spec("datasets") else 1) +PY + +# ---- run ------------------------------------------------------------------- +cd evals +TS=$(date +%Y%m%d_%H%M%S) +OUT="ruler_mab_${TS}" + +SOURCE_VAL="${SOURCE:-ruler_qa1_197K}" +LIMIT_VAL="${LIMIT:-1}" + +EXTRA_ARGS=() +[ -n "${MAX_CHUNKS:-}" ] && EXTRA_ARGS+=( "--max-chunks" "$MAX_CHUNKS" ) +[ -n "${MAX_QS:-}" ] && EXTRA_ARGS+=( "--max-questions" "$MAX_QS" ) + +echo "[1/3] running ruler_eval.py -> ${OUT}/" +echo " source=${SOURCE_VAL} limit=${LIMIT_VAL} max_chunks=${MAX_CHUNKS:-} max_questions=${MAX_QS:-}" +python ruler_eval.py \ + --limit "$LIMIT_VAL" \ + --source "$SOURCE_VAL" \ + --run-llm \ + --mirix_config_path ./configs/mab.yaml \ + --output_path "$OUT" \ + ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"} + +echo +echo "[2/3] running organize_results.py (substring judge)" +# RULER's official metric is substring_exact_match — no LLM judge needed. +python organize_results.py --judge substring "results/ruler/${OUT}" + +echo +echo "[3/3] saving snapshot (pg_dump + neo4j export + results copy)" +if MIRIX_PG_DB="${MIRIX_PG_DB:-mirix}" python memory_snapshot.py save "${OUT}" --agents; then + cp -R "results/ruler/${OUT}" "snapshots/${OUT}/results" + echo " snapshot + results -> evals/snapshots/${OUT}/" +else + echo " WARN: memory_snapshot.py failed; results still at evals/results/ruler/${OUT}/" >&2 +fi + +# ---- summary --------------------------------------------------------------- +echo +echo "========================================================" +echo " Summary" +echo "========================================================" +python - "results/ruler/${OUT}" <<'PY' +import json, sys +from pathlib import Path + +out_dir = Path(sys.argv[1]) +m = json.loads((out_dir / "metrics.json").read_text()) +mt = m["metrics"] + +print(f" Output: {out_dir}") +print(f" Acc: {mt['accuracy']:.4f} ({mt['total_correct']}/{mt['total_judged']})") +print(f" Total questions: {mt.get('total_questions')}") +print(f" Avg answer lat.: {mt.get('average_answer_latency_seconds', 0):.2f}s") +print(f" Avg mem tokens: {mt.get('average_memory_tokens')}") + +# RULER has no question_types, so accuracy_by_category is uninformative. +PY + +# ---- DB memory counts ------------------------------------------------------ +echo +echo "========================================================" +echo " Memory store counts (DB=\${MIRIX_PG_DB:-mirix})" +echo "========================================================" +PGPASSWORD="${MIRIX_PG_PASSWORD:-mirix}" psql -h "${MIRIX_PG_HOST:-localhost}" \ + -U "${MIRIX_PG_USER:-mirix}" -d "${MIRIX_PG_DB:-mirix}" -tAc \ + "select 'episodic='||count(*) from episodic_memory + union all select 'semantic='||count(*) from semantic_memory + union all select 'procedural='||count(*) from procedural_memory + union all select 'resource='||count(*) from resource_memory + union all select 'knowledge_vault='||count(*) from knowledge_vault" \ + 2>/dev/null || echo " (psql unavailable or DB unreachable; skipping)" diff --git a/evals/task_agent.py b/evals/task_agent.py index be34a1bde..c16fcbe3e 100644 --- a/evals/task_agent.py +++ b/evals/task_agent.py @@ -32,7 +32,11 @@ def __init__( self.model = model self.user_id = user_id self.max_tool_rounds = max_tool_rounds - self.mirix_client = MirixClient(client_id=client_id, org_id=org_id, base_url="http://127.0.0.1:8531", write_scope="read_write") + # timeout: per-request budget. Question-answer calls are short, + # but this client is shared with the ingest path (see + # MirixMemorySystem) where a single /memory/add_sync on a 4096- + # token chunk takes 3-6 min server-side. Keep 30 min headroom. + self.mirix_client = MirixClient(client_id=client_id, org_id=org_id, base_url="http://127.0.0.1:8531", write_scope="read_write", timeout=1800) self.user_id = user_id if user_id is not None else str(uuid.uuid4()) config_path = Path(mirix_config_path) with config_path.open("r", encoding="utf-8") as handle: @@ -147,6 +151,10 @@ def _search_memory( if not params or not isinstance(params, dict): return {"success": False, "error": "Missing search parameters.", "skipped": True} + # Drop keys the LLM occasionally hallucinates with falsy/blank names — + # `**params` would TypeError on '' or None keys and kill the whole eval. + params = {k: v for k, v in params.items() if isinstance(k, str) and k} + # Set reasonable defaults for better search quality if 'search_method' not in params or params['search_method'] is None: params['search_method'] = "embedding" @@ -155,7 +163,14 @@ def _search_memory( if 'limit' not in params or params['limit'] is None or params['limit'] < 10: params['limit'] = 15 # Get more candidates for better coverage - results = asyncio.run(self.mirix_client.search(user_id=resolved_user_id, **params)) + try: + results = asyncio.run(self.mirix_client.search(user_id=resolved_user_id, **params)) + except TypeError as e: + # MirixClient.search rejected an unknown kwarg (LLM produced + # a key not in the schema). Skip this search rather than + # crashing the entire eval — model will see empty results + # and try another query. + return {"success": False, "error": f"Invalid search args: {e}", "skipped": True} if results['success']: for result in results['results']: From bb274bb1c964056f73e2bfa1ec134ac47f2ad1de Mon Sep 17 00:00:00 2001 From: "Huang, Wei-Chieh" Date: Thu, 4 Jun 2026 14:49:17 +0800 Subject: [PATCH 2/3] MIRIX: Redis modules-missing latch + small robustness fixes - mirix/database/redis_client.py: add an instance _modules_missing flag plus _check_modules_missing(exc) helper. Every set_json / search_* method short-circuits to its empty sentinel ([], False) once a Redis ResponseError ("unknown command 'JSON.SET' / 'FT.SEARCH'") has been seen. On a stock OSS Redis without RedisJSON + RediSearch the modules are absent and every cache write raised on each memory item; a full SHDOCQA ingest emitted thousands of ERROR lines. The first such error now logs a single WARNING and flips the latch; subsequent calls return silently. Functionally equivalent for the PG-fallback path (managers' if results: branch still sees [] and falls through to PostgreSQL), only the log volume changes. - mirix/functions/function_sets/memory_tools.py: in semantic_memory_insert, read item["source"] via item.get("source", "") so an LLM tool call that omits the optional source field doesn't crash the agent with KeyError mid-ingest. Observed on RULER ingest where ~5% of semantic items had a missing source. - mirix/client/remote_client.py: replace the inaccurate comment next to the per-request timeout in _request. The old text claimed AsyncClient(timeout=N) was silently overridden by the wrapped AsyncHTTPTransport; a fake-transport repro showed RetryTransport propagates request.extensions['timeout'] unchanged. The correct framing is that self.timeout is the caller-controlled budget the per-request argument reads from. --- mirix/client/remote_client.py | 7 ++- mirix/database/redis_client.py | 61 +++++++++++++++++++ mirix/functions/function_sets/memory_tools.py | 2 +- 3 files changed, 68 insertions(+), 2 deletions(-) diff --git a/mirix/client/remote_client.py b/mirix/client/remote_client.py index a4b866e33..5c96596fa 100644 --- a/mirix/client/remote_client.py +++ b/mirix/client/remote_client.py @@ -414,8 +414,13 @@ async def _request( if json: logger.debug("[MirixClient] Request body: %s", json) + # Per-request timeout is set from self.timeout so a caller can + # raise it via MirixClient(timeout=...) without rebuilding the + # AsyncClient. RetryTransport propagates request.extensions + # ['timeout'] to the wrapped AsyncHTTPTransport unchanged. response = await self._client.request( - method=method, url=url, json=json, params=params, headers=headers + method=method, url=url, json=json, params=params, headers=headers, + timeout=self.timeout, ) try: response.raise_for_status() diff --git a/mirix/database/redis_client.py b/mirix/database/redis_client.py index 8b47325dc..6fa054cfe 100644 --- a/mirix/database/redis_client.py +++ b/mirix/database/redis_client.py @@ -112,6 +112,18 @@ def __init__( self.client = Redis(connection_pool=self.pool) + # Latched flag: when Redis is reachable but missing the + # RedisJSON + RediSearch modules (a stock OSS Redis), every + # set_json / search_* call raises ResponseError("unknown + # command 'JSON.SET'" / "'FT.SEARCH'"). The system still + # works (PG is authoritative; Redis is a cache layer), but + # without latching we log one error per memory item ingested + # — thousands of lines on a full eval. First failure logs a + # single WARNING and flips this flag; subsequent calls + # short-circuit via _check_modules_missing(). Reset to + # False to re-probe. + self._modules_missing = False + # Log configuration logger.info( "Redis connection pool initialized: %s (max_connections=%d, " "socket_timeout=%ds, keepalive=%s)", @@ -136,6 +148,27 @@ def _mask_uri(self, uri: str) -> str: return f"{protocol}://****@{parts[1]}" return uri + def _check_modules_missing(self, exc: Exception) -> bool: + """Detect ``unknown command 'JSON.SET'`` / ``'FT.SEARCH'`` errors + from a stock Redis missing RedisJSON + RediSearch and latch + ``self._modules_missing`` so future calls short-circuit. + + Returns True when the caller should swallow the exception and + return its empty/False sentinel; False when the caller should + keep its normal error-logging path. + """ + if "unknown command" in str(exc).lower(): + if not self._modules_missing: + logger.warning( + "Redis JSON/Search modules not loaded — skipping " + "Redis cache writes/queries for the rest of this " + "process. PG remains authoritative. Original: %s", + exc, + ) + self._modules_missing = True + return True + return False + async def ping(self) -> bool: """Test Redis connection.""" try: @@ -809,6 +842,8 @@ async def set_json(self, key: str, data: Dict[str, Any], ttl: Optional[int] = No Returns: True if successful """ + if self._modules_missing: + return False try: # Use JSON.SET command await self.client.json().set(key, "$", data) @@ -819,6 +854,8 @@ async def set_json(self, key: str, data: Dict[str, Any], ttl: Optional[int] = No logger.debug("Stored JSON: %s", key) return True except Exception as e: + if self._check_modules_missing(e): + return False logger.error("Failed to set JSON for %s: %s", key, e) return False @@ -916,6 +953,8 @@ async def search_text( end_date=datetime(2025, 11, 19, 23, 59, 59) ) """ + if self._modules_missing: + return [] try: from mirix.database.filter_tags_query import can_redis_handle @@ -1004,6 +1043,8 @@ def escape_redis_query(text: str) -> str: return documents except Exception as e: + if self._check_modules_missing(e): + return [] logger.warning("Redis text search failed for index %s with query '%s': %s", index_name, query[:50], e) return [] @@ -1050,6 +1091,8 @@ async def search_vector( filter_tags={"expert_id": "expert-123"} ) """ + if self._modules_missing: + return [] try: from mirix.database.filter_tags_query import can_redis_handle @@ -1137,6 +1180,8 @@ async def search_vector( return documents except Exception as e: + if self._check_modules_missing(e): + return [] logger.warning( "Redis vector search failed for index %s (vector_field: %s): %s", index_name, vector_field, e ) @@ -1184,6 +1229,8 @@ async def search_recent( end_date=datetime(2025, 11, 19, 23, 59, 59) ) """ + if self._modules_missing: + return [] try: from mirix.database.filter_tags_query import can_redis_handle @@ -1255,6 +1302,8 @@ async def search_recent( return documents except Exception as e: + if self._check_modules_missing(e): + return [] logger.warning("Redis recent search failed for index %s (sort_by: %s): %s", index_name, sort_by, e) return [] @@ -1286,6 +1335,8 @@ async def search_recent_by_org( Returns: List of recent documents sorted by timestamp (descending) """ + if self._modules_missing: + return [] try: from mirix.database.filter_tags_query import can_redis_handle @@ -1345,6 +1396,8 @@ def escape_text_value(value: str) -> str: return [self._doc_to_dict(doc) for doc in results.docs] except Exception as e: + if self._check_modules_missing(e): + return [] logger.warning("Redis org search failed for index %s: %s", index_name, e) return [] @@ -1378,6 +1431,8 @@ async def search_vector_by_org( Returns: List of documents sorted by similarity """ + if self._modules_missing: + return [] try: from mirix.database.filter_tags_query import can_redis_handle @@ -1439,6 +1494,8 @@ def escape_text_value(value: str) -> str: return [self._doc_to_dict(doc) for doc in results.docs] except Exception as e: + if self._check_modules_missing(e): + return [] logger.warning("Redis vector search failed for org search: %s", e) return [] @@ -1472,6 +1529,8 @@ async def search_text_by_org( Returns: List of matching documents """ + if self._modules_missing: + return [] try: from mirix.database.filter_tags_query import can_redis_handle @@ -1527,6 +1586,8 @@ def escape_text_value(value: str) -> str: return [self._doc_to_dict(doc) for doc in results.docs] except Exception as e: + if self._check_modules_missing(e): + return [] logger.warning("Redis text search failed for org search: %s", e) return [] diff --git a/mirix/functions/function_sets/memory_tools.py b/mirix/functions/function_sets/memory_tools.py index f809249bd..6260c67e4 100644 --- a/mirix/functions/function_sets/memory_tools.py +++ b/mirix/functions/function_sets/memory_tools.py @@ -654,7 +654,7 @@ async def semantic_memory_insert(self: "Agent", items: List[SemanticMemoryItemBa name=item["name"], summary=item["summary"], details=item["details"], - source=item["source"], + source=item.get("source", ""), organization_id=self.actor.organization_id, actor=self.actor, filter_tags=filter_tags if filter_tags else None, From 1d79bda42555928619b85d4f5a5c56e74578a6a3 Mon Sep 17 00:00:00 2001 From: "Huang, Wei-Chieh" Date: Thu, 4 Jun 2026 14:49:17 +0800 Subject: [PATCH 3/3] docs: MAB_EVAL_SUITE.md - design + change log 478-line reference covering: - Eval inventory (every runner, helper, judge, runner shell, config with file:line refs) - MIRIX core patches (Redis modules-missing latch, defensive dict access in semantic_memory_insert, per-request timeout plumbing in remote_client) - all additive, no structural change - Benchmark results (SHDOCQA char-4096 82% vs token-4096 69% on fixed retrieval; smoke tests across 4 datasets; memory store granularity numbers) - Bugs and lessons learned (macOS tmp_cleaner /tmp/.venv wipe; users.organization_id NULL killing retrieval silently; Redis user cache hiding the PG fix; chunk-size apples-to-apples confounder; the RetryTransport-timeout claim that was not true; eval crash from empty-string kwarg in LLM tool calls) - Next steps (push branch, optional full runs, write-through cache invalidation, summarisation-workload chunk-size re-measure) --- evals/MAB_EVAL_SUITE.md | 478 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 478 insertions(+) create mode 100644 evals/MAB_EVAL_SUITE.md diff --git a/evals/MAB_EVAL_SUITE.md b/evals/MAB_EVAL_SUITE.md new file mode 100644 index 000000000..62470fd9f --- /dev/null +++ b/evals/MAB_EVAL_SUITE.md @@ -0,0 +1,478 @@ +# MAB eval suite — design + change log + +## Overview + +This change adds a MAB (MemoryAgentBench) eval suite covering four benchmarks — LongMemEval-S, RULER QA1/QA2, and LRU (infbench + detective) — and lands two small robustness fixes plus a Redis modules-missing latch in MIRIX core. The eval suite is commit `ca9bb36` ("Add MAB eval suite: LongMemEval-S, RULER QA1/QA2, LRU infbench/detective"); the MIRIX core patches are commit `b27fe93` ("MIRIX: Redis modules-missing latch + small robustness fixes"). Both commits sit on branch `mab_runner`, currently 2 commits ahead of `origin/main`. A backup branch `mab_runner_pre_squash_260604-1402` preserves the pre-squash 14-commit history as a local-only safety net. + +## The MAB eval suite + +Commit `ca9bb36` ("Add MAB eval suite: LongMemEval-S, RULER QA1/QA2, LRU infbench/detective") adds an end-to-end harness for four MemoryAgentBench (MAB) benchmarks that all run against MIRIX via the remote client. Every runner shares the same shape — `MirixMemorySystem.add_chunk` ingest → `wrap_user_prompt` → `TaskAgent.answer` → judge → snapshot — and writes the same per-sample JSON schema so `organize_results.py` works on every output unchanged. Sample IDs are namespaced per-source so multiple subsets can share one Postgres DB. + +### Runners (one per dataset) + +#### [`evals/longmem_eval.py`](/Users/weichiehhuang/MIRIX_eval/evals/longmem_eval.py) — 433 lines + +- **Dataset / HF source:** `ai-hyz/MemoryAgentBench`, split `Accurate_Retrieval`, filtered to rows whose `metadata.source` equals `--source` (default constant `DEFAULT_LONGMEM_SOURCE = "longmemeval_s*"`, an exact match that mirrors the official MAB judge CLI `--dataset longmemeval_s*`). One HF row = one ~1.6M-char "long context" plus a parallel list of questions / answers / `question_types` / `question_ids` ([longmem_eval.py:40-90](/Users/weichiehhuang/MIRIX_eval/evals/longmem_eval.py#L40-L90)). +- **Per-sample loop ([longmem_eval.py:290-429](/Users/weichiehhuang/MIRIX_eval/evals/longmem_eval.py#L290-L429)):** + 1. `sample_id = f"longmem_s_{idx}"` (line 74). Construct `TaskAgent` and `MirixMemorySystem`, sharing the same `MirixClient` so both halves of the eval hit the same in-memory connection pool. + 2. `parse_sessions(context)` ([longmem_eval.py:96-178](/Users/weichiehhuang/MIRIX_eval/evals/longmem_eval.py#L96-L178)) — `ast.literal_eval`s the HF `context` field (alternating `"Chat Time: 2022/11/17 (Thu) 12:04"` strings and per-session message lists), parses each header with a regex into ISO 8601 (`"2022-11-17T12:04:00"`), prepends the header to the body, and pushes each session through `chunk_text_into_sentences` so each sub-chunk inherits the session's `occurred_at`. Returns `[{occurred_at, text}, ...]`. + 3. Reset server-side token counter via `POST /debug/token_stats/reset`, then ingest each chunk: `memory_system.add_chunk(chunk["text"], raw_input=..., occurred_at=...)`. Per-chunk wall time is recorded into `timings.add_chunk[idx]`, and the chunk's response goes into `responses[idx] = {type, chunk_index, question_index: None, occurred_at, response}`. Idempotent — re-runs skip indices already in `responses` (line 326). + 4. Snapshot ingest token stats into `token_stats.build_raw` / `build_sum`, then call `measure_memory_size(sample_id)` and stash under `memory_stats`. + 5. For each question, call `memory_system.wrap_user_prompt(question)` (records `timings.wrap_user_prompt[qidx]`), then `task_agent.answer(input_messages, user_id=sample_id)` (records `timings.answer[qidx]`). The record (`records[qidx_key]`) stores `sample_id, question_index, question_id, question, expected_answer, evidence:None, category, input_messages, predicted_answer, messages, usage, usage_total`. The `_abs` suffix on `question_id` routes to the MAB abstention prompt later. + 6. After QA, dump full PG memories with `dump_memories(sample_id)` to `_memories.json` and write `token_stats.query_raw` / `query_sum` (post-QA snapshot minus build totals). +- **CLI flags:** `--limit`, `--max-chunks`, `--max-questions`, `--run-llm` (default `True`), `--output_path` (default `Path("longmem_run")`; relative paths resolve under `evals/results/longmem/`; absolute paths warn), `--source` (default `"longmemeval_s*"`), `--mirix_config_path`. +- **Env vars:** `MIRIX_CLIENT_ID` (default `"mirix-eval-client"`), `MIRIX_ORG_ID` (default `"mirix-eval-org"`). +- **Output schema (`.json`):** `{sample_id, timings: {add_chunk, wrap_user_prompt, answer}, responses: {idx → ...}, records: {qidx → ...}, memory_stats, token_stats: {build_raw, build_sum, query_raw, query_sum}}`. Sidecar `_memories.json` is the full PG dump from `dump_memories`. + +#### [`evals/ruler_eval.py`](/Users/weichiehhuang/MIRIX_eval/evals/ruler_eval.py) — 388 lines + +- **Dataset / HF source:** `ai-hyz/MemoryAgentBench`, split `Accurate_Retrieval`, `metadata.source` in `{"ruler_qa1_197K"` (SHDOCQA, single-hop, SQuAD-derived; default), `"ruler_qa2_421K"` (MHDOCQA, multi-hop, HotpotQA-derived)`}` ([ruler_eval.py:43](/Users/weichiehhuang/MIRIX_eval/evals/ruler_eval.py#L43)). One row = one ~1-2M-char needle-in-haystack context of concatenated `Document N:` blocks + 100 short QA pairs whose answers are lists of acceptable substrings (judge must be `substring` — no LLM call needed). +- **Sample-id namespacing ([ruler_eval.py:76-90](/Users/weichiehhuang/MIRIX_eval/evals/ruler_eval.py#L76-L90)):** `ruler_qa1_197K → "shdocqa"`, `ruler_qa2_421K → "mhdocqa"`. `sample_id = f"{source_tag}_ruler_qa_{idx}"` so both subsets can share the same Postgres `user_id` namespace without collisions in `episodic_memory.user_id`. +- **Per-sample loop:** identical to `longmem_eval` ([ruler_eval.py:245-384](/Users/weichiehhuang/MIRIX_eval/evals/ruler_eval.py#L245-L384)) except the chunker is `parse_documents(context)` ([ruler_eval.py:116-133](/Users/weichiehhuang/MIRIX_eval/evals/ruler_eval.py#L116-L133)). That returns `[{occurred_at: None, text}, ...]` — RULER has no timestamps, so `occurred_at` is always `None`. The function no longer pre-splits on `Document N:` markers (the official chunker doesn't either); sentences span document boundaries but match leaderboard chunk shape. +- **CLI flags:** `--limit`, `--max-chunks`, `--max-questions`, `--run-llm`, `--output_path` (default `Path("ruler_run")`; relative under `evals/results/ruler/`), `--source` (default `"ruler_qa1_197K"`), `--mirix_config_path`. +- **`question_types`/`question_ids`:** empty lists ([ruler_eval.py:97-99](/Users/weichiehhuang/MIRIX_eval/evals/ruler_eval.py#L97-L99)) — RULER has no categorization, abstention is moot. +- **Output schema:** same as longmem_eval. + +#### [`evals/lru_eval.py`](/Users/weichiehhuang/MIRIX_eval/evals/lru_eval.py) — 404 lines + +- **Dataset / HF source:** `ai-hyz/MemoryAgentBench`, split `Long_Range_Understanding`, `metadata.source` in `{"infbench_sum_eng_shots2"` (100 rows, novel summarization, default), `"detective_qa"` (10 rows, multiple-choice mystery QA)`}` ([lru_eval.py:52](/Users/weichiehhuang/MIRIX_eval/evals/lru_eval.py#L52)). +- **Sample-id namespacing:** `infbench_sum_eng_shots2 → "infbench"`, `detective_qa → "detective"` ([lru_eval.py:82-95](/Users/weichiehhuang/MIRIX_eval/evals/lru_eval.py#L82-L95)). +- **Per-sample loop ([lru_eval.py:250-400](/Users/weichiehhuang/MIRIX_eval/evals/lru_eval.py#L250-L400)):** same as `ruler_eval`. Chunker is `parse_context` ([lru_eval.py:125-138](/Users/weichiehhuang/MIRIX_eval/evals/lru_eval.py#L125-L138)) — also `chunk_text_into_sentences`, `occurred_at` always `None`. +- **Summarization-judge plumbing ([lru_eval.py:99-108](/Users/weichiehhuang/MIRIX_eval/evals/lru_eval.py#L99-L108) and [357-379](/Users/weichiehhuang/MIRIX_eval/evals/lru_eval.py#L357-L379)):** for `infbench_sum_eng_shots2` each row carries `metadata.keypoints` and `metadata.qa_pair_ids`, which the runner propagates into each record as `keypoints` and `qa_pair_id` so `llm_judge_mab_summary` can score without a re-fetch from HF. For `detective_qa` these are empty (substring judge ignores them). +- **CLI flags:** `--limit`, `--max-chunks`, `--max-questions`, `--run-llm`, `--output_path` (default `Path("lru_run")`; relative under `evals/results/lru/`), `--source` (default `"infbench_sum_eng_shots2"`), `--mirix_config_path`. +- **Output schema:** same as longmem_eval, plus `records[qidx_key].keypoints` and `qa_pair_id` for the summarization judge. + +#### [`evals/main_eval.py`](/Users/weichiehhuang/MIRIX_eval/evals/main_eval.py) — 315 lines (LoCoMo runner, modified in `ca9bb36`) + +- **Dataset:** LoCoMo, loaded from a local JSON path (`--data`, default `data/locomo10.json`) via `load_locomo` ([main_eval.py:24-32](/Users/weichiehhuang/MIRIX_eval/evals/main_eval.py#L24-L32)). Each item has `sample_id`, `conversation` (with `session_N` lists and `session_N_date_time`), and `qa` list. +- **Per-sample loop ([main_eval.py:178-311](/Users/weichiehhuang/MIRIX_eval/evals/main_eval.py#L178-L311)):** + 1. `iter_sessions(conversation)` walks `session_` keys; `format_session_chunk` prepends the LoCoMo `instructions` block and timestamp header, then dumps each turn as `speaker: text`. + 2. `memory_system.add_chunk(chunk, raw_input=chunk)` (no `occurred_at` — LoCoMo embeds the timestamp in the prompt instead). + 3. For each `qa` entry, `wrap_user_prompt(question)` → `task_agent.answer(...)` → record with `sample_id, question_index, question, expected_answer, evidence, category, input_messages, predicted_answer, messages, usage, usage_total`. + 4. `dump_memories(sample_id)` → `_memories.json` ([main_eval.py:300-311](/Users/weichiehhuang/MIRIX_eval/evals/main_eval.py#L300-L311)). +- **CLI flags:** `--data`, `--limit`, `--max-questions`, `--run-llm`, `--output_path` (default `Path("results")`), `--mirix_config_path`. +- **Output schema:** identical to the others, minus `token_stats` / `memory_stats` (those were added only on the new runners). + +### Shared helpers + +#### [`evals/_chunking.py`](/Users/weichiehhuang/MIRIX_eval/evals/_chunking.py) — 98 lines + +Port of `utils/eval_other_utils.chunk_text_into_sentences` from the official MAB repo. Used by all three new runners so RULER, LRU and LongMemEval ingest the same shape the leaderboard runs use. + +- **Atom:** NLTK `sent_tokenize` (auto-downloads `punkt_tab` then `punkt`, both `quiet=True`, via `_ensure_nltk()` at import time — [_chunking.py:37-64](/Users/weichiehhuang/MIRIX_eval/evals/_chunking.py#L37-L64)). +- **Budget:** `DEFAULT_CHUNK_TOKENS = 4096`, measured with `tiktoken.encoding_for_model("gpt-4o-mini")` (`DEFAULT_TOKEN_MODEL`). Encoder is cached at import as `_ENCODING`. +- **Public function:** `chunk_text_into_sentences(text: str, chunk_size: int = DEFAULT_CHUNK_TOKENS) -> List[str]` ([_chunking.py:68-98](/Users/weichiehhuang/MIRIX_eval/evals/_chunking.py#L68-L98)). Greedy: pile sentences into the current buffer until the next one would push past `chunk_size`, flush with `" ".join(buf)`, start a new buffer with the overflowing sentence. No mid-sentence split — a single sentence over the budget still ships in its own chunk (matches official). Empty input returns `[]`. +- **Why this matters:** char-based 4096 emitted ~4× more chunks than the official chunker (~1024 tokens), spreading each semantic unit across multiple memories and tanking retrieval. The module docstring spells this out. + +#### [`evals/_eval_db.py`](/Users/weichiehhuang/MIRIX_eval/evals/_eval_db.py) — 183 lines + +Direct-PG helpers. Both call `psql` via `subprocess.run` with connection params from `MIRIX_PG_{HOST,USER,DB,PASSWORD}` env vars (defaults `localhost / mirix / mirix / mirix`) and `psql` binary from `shutil.which("psql")` falling back to `/usr/local/opt/postgresql@17/bin/psql` ([_eval_db.py:25-32](/Users/weichiehhuang/MIRIX_eval/evals/_eval_db.py#L25-L32)). + +- **`measure_memory_size(sample_id) -> Dict`** ([_eval_db.py:74-139](/Users/weichiehhuang/MIRIX_eval/evals/_eval_db.py#L74-L139)). For each of `episodic_memory` and `semantic_memory`, scalar-queries `count(*)` and `sum(length(coalesce(summary,'')||coalesce(details,'')))` (semantic also folds in `name`), keyed on `user_id`. Returns `{unit: "chars+tokens", flat: {table → {rows, chars}}, flat_total_chars, graph: {...}, graph_total_chars, total_chars, total_tokens}` where `total_tokens = total_chars // 4`. Also queries Neo4j (`MATCH (n) WHERE n.user_id=$uid` then `MATCH (a)-[r]->(b) WHERE a.user_id=$uid`) for node/edge counts and char totals, then reports `total_chars = max(flat_total, graph_total)` so flat-PG and graph backends are on the same yardstick. +- **`dump_memories(sample_id) -> Dict`** ([_eval_db.py:142-183](/Users/weichiehhuang/MIRIX_eval/evals/_eval_db.py#L142-L183)). Streams all rows for `user_id = sample_id` from `episodic_memory` and `semantic_memory` using `_pg_rows` (which uses `-F "\x01" -R "\x02"` so embedded newlines in summaries don't break row parsing). Returns `{user_id, memories: {episodic: {total_count, items: [...]}, semantic: {total_count, items: [...]}}}`. +- **Why bypassing `/memory/components` matters:** that endpoint caps each memory type at 50 by default and 200 even with an explicit `limit`. On conversations with thousands of memory items, the resulting `_memories.json` token count would silently under-report. Going straight to PG sidesteps the cap. + +#### [`evals/llm_judge_substring.py`](/Users/weichiehhuang/MIRIX_eval/evals/llm_judge_substring.py) — 78 lines + +Deterministic, no-LLM judge. Mirrors official `utils/eval_other_utils.substring_exact_match`. + +- **`normalize_answer(text)`** ([llm_judge_substring.py:23-36](/Users/weichiehhuang/MIRIX_eval/evals/llm_judge_substring.py#L23-L36)): lowercase → strip `string.punctuation` → drop `a|an|the` at word boundaries → collapse whitespace. +- **`_accepted_answers(expected)`** ([llm_judge_substring.py:39-53](/Users/weichiehhuang/MIRIX_eval/evals/llm_judge_substring.py#L39-L53)): normalises gold answers from list, semicolon-joined string (`flatten_answer` output), or `None` into a flat `list[str]`. +- **`evaluate_substring_judge(predicted, expected) -> int`** ([llm_judge_substring.py:56-78](/Users/weichiehhuang/MIRIX_eval/evals/llm_judge_substring.py#L56-L78)): returns `1` iff `normalize_answer(predicted)` contains `normalize_answer(any accepted answer)` as a substring; `0` otherwise. Used by RULER QA1/QA2 and `detective_qa`. + +#### [`evals/llm_judge_mab.py`](/Users/weichiehhuang/MIRIX_eval/evals/llm_judge_mab.py) — 144 lines + +Port of `llm_based_eval/longmem_qa_evaluate.py`. + +- **Metric model:** `DEFAULT_METRIC_MODEL = "gpt-4o"` ([llm_judge_mab.py:34](/Users/weichiehhuang/MIRIX_eval/evals/llm_judge_mab.py#L34)) — *not* `gpt-4o-mini` like the generic LoCoMo judge. `temperature=0`, `max_tokens=10`, `n=1`. +- **Five task-category prompts** ([llm_judge_mab.py:37-85](/Users/weichiehhuang/MIRIX_eval/evals/llm_judge_mab.py#L37-L85)): + - `_SINGLE_AND_MULTI` — used for `single-session-user`, `single-session-assistant`, `multi-session`. + - `_TEMPORAL` — for `temporal-reasoning`; verbatim "do not penalize off-by-one errors" clause. + - `_KNOWLEDGE_UPDATE` — for `knowledge-update`; accepts responses that contain previous + updated info. + - `_PREFERENCE` — for `single-session-preference`; "does not need to reflect all the points in the rubric". +- **Plus `_ABSTENTION`** ([llm_judge_mab.py:77-85](/Users/weichiehhuang/MIRIX_eval/evals/llm_judge_mab.py#L77-L85)) — routed to whenever `abstention=True`, regardless of category. Reframes the question as "unanswerable". +- **`SUPPORTED_TASKS`** ([llm_judge_mab.py:88-97](/Users/weichiehhuang/MIRIX_eval/evals/llm_judge_mab.py#L88-L97)) is the frozenset of the five known categories. +- **`get_anscheck_prompt(task, question, answer, response, abstention=False)`** ([llm_judge_mab.py:100-118](/Users/weichiehhuang/MIRIX_eval/evals/llm_judge_mab.py#L100-L118)): if `abstention`, return `_ABSTENTION`; else route by `task`; else `NotImplementedError`. +- **`evaluate_mab_judge(...) -> int`** ([llm_judge_mab.py:121-144](/Users/weichiehhuang/MIRIX_eval/evals/llm_judge_mab.py#L121-L144)): scoring rule is the verbatim official binary rule — `1 if "yes" in completion.lower() else 0`. Not JSON-parsed CORRECT/WRONG. + +#### [`evals/llm_judge_mab_summary.py`](/Users/weichiehhuang/MIRIX_eval/evals/llm_judge_mab_summary.py) — 294 lines + +Port of `llm_based_eval/summarization_evaluate.py`. + +- **Metric model:** `DEFAULT_METRIC_MODEL = "gpt-4o-2024-05-13"` — pinned, *not* the `gpt-4o` alias (alias drifts, diverges from leaderboard). `DEFAULT_TEMPERATURE = 0.1`, `DEFAULT_MAX_TOKENS = 4096` ([llm_judge_mab_summary.py:51-53](/Users/weichiehhuang/MIRIX_eval/evals/llm_judge_mab_summary.py#L51-L53)). +- **Three calls per record** ([llm_judge_mab_summary.py:233-294](/Users/weichiehhuang/MIRIX_eval/evals/llm_judge_mab_summary.py#L233-L294)): + 1. **Fluency** (`_FLUENCY_PROMPT_BOOK`, [llm_judge_mab_summary.py:56-72](/Users/weichiehhuang/MIRIX_eval/evals/llm_judge_mab_summary.py#L56-L72)) — binary 0/1 on coherence/non-repetitiveness. Returns `{"fluency": 0 or 1}`. + 2. **Recall** (`_RECALL_PROMPT_BOOK`, [llm_judge_mab_summary.py:75-159](/Users/weichiehhuang/MIRIX_eval/evals/llm_judge_mab_summary.py#L75-L159)) — counts how many of the expert `keypoints` are supported by the generated summary. Returns `{"supported_key_points": [...], "recall": N}`. + 3. **Precision** (`_PRECISION_PROMPT_BOOK`, [llm_judge_mab_summary.py:162-196](/Users/weichiehhuang/MIRIX_eval/evals/llm_judge_mab_summary.py#L162-L196)) — counts how many sentences in the generated summary are supported by the expert summary. Returns `{"precision": N, "sentence_count": M}`. +- **`_parse_json(text)`** ([llm_judge_mab_summary.py:199-219](/Users/weichiehhuang/MIRIX_eval/evals/llm_judge_mab_summary.py#L199-L219)): greedy brace match with `re.findall(r"\{.*?\}", ..., re.DOTALL)`, fallback to ` ```json ``` ` fenced block; returns the *last* JSON object (model often emits an example object in its reasoning before the final score). +- **Aggregation** ([llm_judge_mab_summary.py:266-294](/Users/weichiehhuang/MIRIX_eval/evals/llm_judge_mab_summary.py#L266-L294)): `recall = recall_found / len(keypoints)`, `precision = precision_found / sentence_count`, then + ``` + F1 = fluency * 2 * recall * precision / (recall + precision) + ``` + (so a 0 on fluency zeros F1 entirely). Any missing-or-unparseable judge call silently collapses that sub-metric to 0, matching the official rule of skipping `None` outputs. Returns `{fluency, recall_total, recall_found, recall, precision_total, precision_found, precision, f1, raw: {fluency_out, recall_out, precision_out}}`. +- **Book-variant prompts only:** `infbench_sum_eng_shots2` is novel summarization, so `*_book` prompts are used; the official script's lawsuit variants (`multi_lexsum`) aren't in the MAB LRU split. + +### Modified files + +#### [`evals/organize_results.py`](/Users/weichiehhuang/MIRIX_eval/evals/organize_results.py) — 563 lines + +The aggregator now supports four judge modes via `--judge {default,mab,substring,mab_summary}` ([organize_results.py:310-326](/Users/weichiehhuang/MIRIX_eval/evals/organize_results.py#L310-L326)). Legacy `--mab-judge` boolean still resolves to `--judge mab` ([organize_results.py:338-339](/Users/weichiehhuang/MIRIX_eval/evals/organize_results.py#L338-L339)). + +- **Judge-task tuple is now 9-wide** ([organize_results.py:155-162](/Users/weichiehhuang/MIRIX_eval/evals/organize_results.py#L155-L162)) — `(idx, sample_id, qidx, category, question, expected, predicted, question_id, keypoints)`. `question_id` was added in `ca9bb36` so `judge_task_mab` can detect the `_abs` abstention suffix; `keypoints` was added so `judge_task_mab_summary` can score per-record without re-fetching HF. The other three judges destructure `_keypoints` and ignore it. +- **`judge_task_mab`** ([organize_results.py:242-267](/Users/weichiehhuang/MIRIX_eval/evals/organize_results.py#L242-L267)): if `category in MAB_SUPPORTED_TASKS` route to `evaluate_mab_judge(... abstention='_abs' in question_id ...)`, label `CORRECT`/`WRONG` plus `-abs` suffix when abstention was triggered; unknown categories fall back to the generic LoCoMo judge tagged `WRONG-unknown-cat` so mixed-source runs don't crash. +- **`judge_task_mab_summary`** ([organize_results.py:270-292](/Users/weichiehhuang/MIRIX_eval/evals/organize_results.py#L270-L292)): runs `evaluate_mab_summary_judge(generated_summary=predicted, expert_summary=expected, keypoints=keypoints)`. Per-record `score = F1` (float in `[0,1]`); the four sub-scores live in `details = {fluency, recall_*, precision_*, f1, raw}`. Label is rendered like `F1=0.831 (fluency=1, recall=18/26, precision=22/30)`. +- **Cache invalidation** ([organize_results.py:343](/Users/weichiehhuang/MIRIX_eval/evals/organize_results.py#L343)): `use_cache = (args.judge == "default")` — non-default judges always re-judge (cached labels were produced by a different judge and aren't comparable). +- **Pool sizing** ([organize_results.py:432-437](/Users/weichiehhuang/MIRIX_eval/evals/organize_results.py#L432-L437)): substring runs with `pool_size=1` (local, free), `mab_summary` with `4` (three sequential gpt-4o calls per task — respect rate limits), all other judges `16`. +- **Floats throughout** ([organize_results.py:474-485](/Users/weichiehhuang/MIRIX_eval/evals/organize_results.py#L474-L485)): `total_correct` is a float so `mab_summary`'s per-record F1 averages correctly; for binary judges this still gives an integer count. +- **`summarization_metrics`** ([organize_results.py:515-528](/Users/weichiehhuang/MIRIX_eval/evals/organize_results.py#L515-L528)): only populated when `--judge mab_summary`. Builds `{"gpt-4-fluency": mean, "gpt-4-recall": mean, "gpt-4-precision": mean, "gpt-4-f1": mean}` from per-record `details` and merges into the output `metrics` dict via `**(summarization_metrics or {})` at [organize_results.py:549](/Users/weichiehhuang/MIRIX_eval/evals/organize_results.py#L549) — matching the official `averaged_metrics` shape. + +#### [`evals/main_eval.py`](/Users/weichiehhuang/MIRIX_eval/evals/main_eval.py) + +Switched the LoCoMo memory snapshot from the capped `/memory/components` endpoint to `dump_memories(sample_id)` ([main_eval.py:11](/Users/weichiehhuang/MIRIX_eval/evals/main_eval.py#L11) and [main_eval.py:300-311](/Users/weichiehhuang/MIRIX_eval/evals/main_eval.py#L300-L311)). LoCoMo's `_memories.json` token counts now reflect the real PG store, not the 50-per-type / 200-cap truncated view. + +#### [`evals/mirix_memory_system.py`](/Users/weichiehhuang/MIRIX_eval/evals/mirix_memory_system.py) — 169 lines + +- **`MirixClient(timeout=1800)`** ([mirix_memory_system.py:54](/Users/weichiehhuang/MIRIX_eval/evals/mirix_memory_system.py#L54)) — bumped from the default. With 4096-token chunks (~16k chars), each `/memory/add_sync` takes 3-6 min server-side; 30 min headroom prevents httpx `ReadTimeout` killing the eval mid-ingest. + +#### [`evals/task_agent.py`](/Users/weichiehhuang/MIRIX_eval/evals/task_agent.py) — 357 lines + +- **`MirixClient(timeout=1800)`** ([task_agent.py:39](/Users/weichiehhuang/MIRIX_eval/evals/task_agent.py#L39)) — same reason; this client is shared with the ingest path via `task_agent.mirix_client`. +- **`_search_memory` hardening** ([task_agent.py:142-190](/Users/weichiehhuang/MIRIX_eval/evals/task_agent.py#L142-L190)): + - Filter out empty-string / non-string keys before `**params` ([task_agent.py:156](/Users/weichiehhuang/MIRIX_eval/evals/task_agent.py#L156)): `params = {k: v for k, v in params.items() if isinstance(k, str) and k}` — the LLM occasionally produces a tool call with a `""` key, which would `TypeError` on `**params` and kill the whole eval. + - Wrap `client.search(**params)` in `try/except TypeError` ([task_agent.py:166-173](/Users/weichiehhuang/MIRIX_eval/evals/task_agent.py#L166-L173)) — if the LLM hallucinates a kwarg not in the schema, return `{"success": False, "error": ..., "skipped": True}` instead of crashing. The model sees empty results and tries another query. + +### Runner shell scripts + +All three follow the same three-stage `[1/3] runner → [2/3] organize_results → [3/3] memory_snapshot` shape, share the same pre-flight (`lsof :8531` + `/health` probe + `datasets` import probe), and end with a per-category summary block and a `psql` count of `episodic / semantic / procedural / resource / knowledge_vault` rows. + +#### [`evals/run_mab_longmem_eval.sh`](/Users/weichiehhuang/MIRIX_eval/evals/run_mab_longmem_eval.sh) — 131 lines + +- **Pre-flight ([run_mab_longmem_eval.sh:29-48](/Users/weichiehhuang/MIRIX_eval/evals/run_mab_longmem_eval.sh#L29-L48)):** `lsof -ti:8531` must return PID; `urllib.request.urlopen("http://localhost:8531/health", timeout=5).status == 200`; `importlib.util.find_spec("datasets")` must succeed. +- **[1/3] runner ([run_mab_longmem_eval.sh:50-68](/Users/weichiehhuang/MIRIX_eval/evals/run_mab_longmem_eval.sh#L50-L68)):** `TS=$(date +%Y%m%d_%H%M%S); OUT="longmem_mab_${TS}"`. `EXTRA_ARGS` is a bash array: each `MAX_CHUNKS` / `MAX_QS` env var adds `--max-chunks N` / `--max-questions N`. `LIMIT_VAL="${LIMIT:-1}"` default. Invokes `python longmem_eval.py --limit $LIMIT_VAL --run-llm --mirix_config_path ./configs/mab.yaml --output_path $OUT "${EXTRA_ARGS[@]}"`. +- **[2/3] organize_results ([run_mab_longmem_eval.sh:71-74](/Users/weichiehhuang/MIRIX_eval/evals/run_mab_longmem_eval.sh#L71-L74)):** `python organize_results.py --mab-judge "results/longmem/${OUT}"` — full path passed because `organize_results` defaults its fallback to `results/locomo/`. +- **[3/3] snapshot ([run_mab_longmem_eval.sh:77-88](/Users/weichiehhuang/MIRIX_eval/evals/run_mab_longmem_eval.sh#L77-L88)):** `MIRIX_PG_DB="${MIRIX_PG_DB:-mirix}" python memory_snapshot.py save "${OUT}" --agents`, then `cp -R "results/longmem/${OUT}" "snapshots/${OUT}/results"`. Failure is non-fatal — eval + judge already succeeded. +- **Summary:** prints `Acc`, `total_correct/total_judged`, `total_questions`, avg answer latency, avg memory tokens, and per-category accuracy (LongMemEval has `single-session-*`, `multi-session`, `temporal-reasoning`, `knowledge-update`, `single-session-preference`). + +#### [`evals/run_mab_ruler_eval.sh`](/Users/weichiehhuang/MIRIX_eval/evals/run_mab_ruler_eval.sh) — 123 lines + +- **Pre-flight:** identical to longmem. +- **[1/3] runner ([run_mab_ruler_eval.sh:53-73](/Users/weichiehhuang/MIRIX_eval/evals/run_mab_ruler_eval.sh#L53-L73)):** `OUT="ruler_mab_${TS}"`. Adds `SOURCE_VAL="${SOURCE:-ruler_qa1_197K}"` env var to pick SHDOCQA (default) vs MHDOCQA (`SOURCE=ruler_qa2_421K`). Invokes `python ruler_eval.py --limit $LIMIT_VAL --source "$SOURCE_VAL" --run-llm --mirix_config_path ./configs/mab.yaml --output_path $OUT "${EXTRA_ARGS[@]}"`. +- **[2/3] organize_results ([run_mab_ruler_eval.sh:78](/Users/weichiehhuang/MIRIX_eval/evals/run_mab_ruler_eval.sh#L78)):** hardcoded to `--judge substring` — RULER's official metric is `substring_exact_match`, no LLM judge needed. +- **[3/3] snapshot:** `python memory_snapshot.py save "${OUT}" --agents` then `cp -R "results/ruler/${OUT}" "snapshots/${OUT}/results"`. +- **Summary** omits per-category accuracy — RULER has no `question_types`, so it would be uninformative. + +#### [`evals/run_mab_lru_eval.sh`](/Users/weichiehhuang/MIRIX_eval/evals/run_mab_lru_eval.sh) — 147 lines + +- **Pre-flight:** identical to longmem. +- **[1/3] runner ([run_mab_lru_eval.sh:62-90](/Users/weichiehhuang/MIRIX_eval/evals/run_mab_lru_eval.sh#L62-L90)):** `OUT="lru_mab_${TS}"`. `SOURCE_VAL="${SOURCE:-infbench_sum_eng_shots2}"` picks `infbench` (default) or `detective_qa`. Same `EXTRA_ARGS` shape. +- **Source-routed judge ([run_mab_lru_eval.sh:71-76](/Users/weichiehhuang/MIRIX_eval/evals/run_mab_lru_eval.sh#L71-L76)):** + ```bash + case "$SOURCE_VAL" in + infbench_sum_eng_shots2) DEFAULT_JUDGE="mab_summary" ;; + detective_qa) DEFAULT_JUDGE="substring" ;; + *) DEFAULT_JUDGE="mab_summary" ;; + esac + JUDGE_VAL="${JUDGE:-$DEFAULT_JUDGE}" + ``` + Overridable via `JUDGE=...`. So infbench → three gpt-4o-2024-05-13 calls per record; detective_qa → free local substring match. +- **[2/3] organize_results ([run_mab_lru_eval.sh:94](/Users/weichiehhuang/MIRIX_eval/evals/run_mab_lru_eval.sh#L94)):** `python organize_results.py --judge "$JUDGE_VAL" "results/lru/${OUT}"`. +- **[3/3] snapshot:** `python memory_snapshot.py save "${OUT}" --agents` then `cp -R "results/lru/${OUT}" "snapshots/${OUT}/results"`. +- **Summary ([run_mab_lru_eval.sh:106-133](/Users/weichiehhuang/MIRIX_eval/evals/run_mab_lru_eval.sh#L106-L133)):** prints `Acc / mean F1`, then iterates `gpt-4-fluency`, `gpt-4-recall`, `gpt-4-precision`, `gpt-4-f1` from the `metrics` block when `mab_summary` was used — surfacing the four official summarization sub-scores. + +### Config + +#### [`evals/configs/mab.yaml`](/Users/weichiehhuang/MIRIX_eval/evals/configs/mab.yaml) — 46 lines + +Standalone MAB profile, independent from the LoCoMo `0201c` profile so tuning can diverge. + +- **`llm_config`** ([mab.yaml:9-14](/Users/weichiehhuang/MIRIX_eval/evals/configs/mab.yaml#L9-L14)): `model: "gpt-4.1-mini"`, `model_endpoint_type: openai`, `model_endpoint: https://api.openai.com/v1`, `context_window: 128000`. This is the answer LLM and the per-agent memory-update LLM. `api_key: your-api-key` is the placeholder substituted at load time by `_resolve_api_keys` in [mirix_memory_system.py](/Users/weichiehhuang/MIRIX_eval/evals/mirix_memory_system.py#L31-L44) using `OPENAI_API_KEY` from `.env`. +- **`topic_extraction_llm_config`** ([mab.yaml:16-21](/Users/weichiehhuang/MIRIX_eval/evals/configs/mab.yaml#L16-L21)): `model: "gpt-4.1-nano"`, same `openai` shape — used for the cheap topic-extraction pass before routing into per-agent memory operations. +- **`embedding_config`** ([mab.yaml:23-28](/Users/weichiehhuang/MIRIX_eval/evals/configs/mab.yaml#L23-L28)): `text-embedding-3-small`, dim `1536`. `build_embeddings_for_memory: true` ([mab.yaml:30](/Users/weichiehhuang/MIRIX_eval/evals/configs/mab.yaml#L30)) — embeddings are built at ingest so `search_method=embedding` in `task_agent._search_memory` ([task_agent.py:159-160](/Users/weichiehhuang/MIRIX_eval/evals/task_agent.py#L159-L160)) has vectors to hit. +- **`meta_agent_config`** ([mab.yaml:32-46](/Users/weichiehhuang/MIRIX_eval/evals/configs/mab.yaml#L32-L46)): `system_prompts_folder: prompts/0201a/`. Agents: `core_memory_agent, resource_memory_agent, semantic_memory_agent, episodic_memory_agent, procedural_memory_agent, knowledge_vault_memory_agent`. Initial core memory ships with `label: "human"` (empty value) and `label: "persona"` (`"I am a helpful assistant."`). +- **`chaining=false` (not in YAML — set per-call):** `MirixMemorySystem.add_chunk` ([mirix_memory_system.py:82](/Users/weichiehhuang/MIRIX_eval/evals/mirix_memory_system.py#L82)) hardcodes `chaining=False` on every `/memory/add_sync` call so the multi-agent chain doesn't run on every session, plus `filter_tags={"scope": "read_write", "kind": "conversation_session"}`. That's the eval-only knob that keeps ingest from blowing past the timeout budget. + +## MIRIX core patches + +Exhaustive reference for commit `b27fe93` ("MIRIX: Redis modules-missing latch + small robustness fixes") at `/Users/weichiehhuang/MIRIX_eval`. Three files changed, all additive — `+68/-2` lines total. + +--- + +### `mirix/database/redis_client.py` — Redis modules-missing latch + +**What's added:** +- New instance attribute `self._modules_missing = False` initialized in `RedisMemoryClient.__init__` (just after `self.client = Redis(connection_pool=self.pool)`). +- New private helper `_check_modules_missing(exc: Exception) -> bool` that returns `True` iff `"unknown command"` appears in `str(exc).lower()`, and on the first hit emits one `logger.warning(...)` and sets `self._modules_missing = True`. +- 7 callers get an entry guard `if self._modules_missing: return ` and their `except Exception as e:` blocks call the helper before falling through to the original `logger.warning/error` line: + - `set_json` (sentinel: `return False`) + - `search_text` (sentinel: `return []`) + - `search_vector` (sentinel: `return []`) + - `search_recent` (sentinel: `return []`) + - `search_recent_by_org` (sentinel: `return []`) + - `search_vector_by_org` (sentinel: `return []`) + - `search_text_by_org` (sentinel: `return []`) + +**Why:** stock OSS Redis without RedisJSON + RediSearch modules raised `ResponseError("unknown command 'JSON.SET'" / "'FT.SEARCH'")` on every cache write/query. A full SHDOCQA ingest emitted thousands of ERROR lines. First failure now logs a single WARNING and flips the latch; every subsequent call returns its empty sentinel silently. + +**Behavior delta:** the PG fallback path is identical. Managers' `if results: ...` branch still sees `[]` and falls through to PostgreSQL exactly as before — only log volume changes. + +**Risk:** the latch can only flip when the exception's `str(...)` contains `"unknown command"`. Real Redis `ResponseError`s for other reasons (auth, connection, OOM, etc.) continue down the original error-log path unchanged. + +**Verification:** full SHDOCQA Q&A re-run on 2026-06-03 emitted exactly 1 WARNING line (`"Redis JSON/Search modules not loaded — skipping Redis cache writes/queries for the rest of this process"`) then silence for the rest of the run. + +**Before vs After (representative — `set_json`; the same shape applies to all 7 callers):** + +Before: +```python +async def set_json(self, key: str, data: dict, ttl: Optional[int] = None) -> bool: + try: + await self.client.json().set(key, "$", data) + ... + return True + except Exception as e: + logger.error("Failed to set JSON for %s: %s", key, e) + return False +``` + +After: +```python +async def set_json(self, key: str, data: dict, ttl: Optional[int] = None) -> bool: + if self._modules_missing: + return False + try: + await self.client.json().set(key, "$", data) + ... + return True + except Exception as e: + if self._check_modules_missing(e): + return False + logger.error("Failed to set JSON for %s: %s", key, e) + return False +``` + +New helper (added once, used by all 7 callers): +```python +def _check_modules_missing(self, exc: Exception) -> bool: + if "unknown command" in str(exc).lower(): + if not self._modules_missing: + logger.warning( + "Redis JSON/Search modules not loaded — skipping " + "Redis cache writes/queries for the rest of this " + "process. PG remains authoritative. Original: %s", + exc, + ) + self._modules_missing = True + return True + return False +``` + +**No class, method signature, or public API changed** — the patch only adds one private attribute, one private helper method, and intra-method guards. All public methods keep their existing names, parameter lists, and return types/sentinels. + +--- + +### `mirix/functions/function_sets/memory_tools.py` — defensive dict access in `semantic_memory_insert` + +**What's added:** a single character-level change at line 657 inside the per-item construction loop of `semantic_memory_insert`. + +**Exact change:** `source=item["source"]` → `source=item.get("source", "")`. + +**Why:** an LLM tool call for `semantic_memory_insert` occasionally omits the optional `source` field; the bracketed access raised `KeyError` and aborted the rest of the items in that batch. Observed on RULER ingest where ~5% of semantic items had a missing `source` during the original SHDOCQA full run. + +**Risk:** zero — `source` was always optional at the storage layer; an empty string is preferable to a hard crash mid-batch. + +**Before vs After:** + +Before: +```python +name=item["name"], +summary=item["summary"], +details=item["details"], +source=item["source"], +organization_id=self.actor.organization_id, +``` + +After: +```python +name=item["name"], +summary=item["summary"], +details=item["details"], +source=item.get("source", ""), +organization_id=self.actor.organization_id, +``` + +**No class, method signature, or public API changed** — only the dict-access expression for one optional field inside the existing function body was softened. + +--- + +### `mirix/client/remote_client.py` — per-request timeout plumbing + corrected comment + +**What's added:** two changes inside `MirixClient._request()`, immediately around the `await self._client.request(...)` call. + +1. **Timeout kwarg:** added `timeout=self.timeout` to the `request(...)` call so the per-request budget is plumbed through `request.extensions['timeout']`. +2. **Comment rewrite:** the old comment claimed `AsyncClient(timeout=N)` was silently overridden by the wrapped `AsyncHTTPTransport`. A fake-transport repro (see the changelog's lessons-learned section) showed `RetryTransport` propagates `request.extensions['timeout']` unchanged. The new comment frames `self.timeout` as the caller-controlled budget the per-request argument reads from. + +**Why the timeout kwarg:** without it the request inherited only the `AsyncClient.timeout` configured at construction. The eval was constructing `MirixClient()` without an explicit timeout, so `self.timeout` stayed at the 60s default and big chunks (>60s server-side) timed out. Passing `timeout=self.timeout` lets callers raise it via `MirixClient(timeout=...)` without rebuilding the underlying `AsyncClient`. + +**Why the comment rewrite:** technical accuracy — `RetryTransport` does not strip the per-request timeout; it forwards `request.extensions['timeout']` to the wrapped `AsyncHTTPTransport` unchanged, so the previous "silently overridden" claim was wrong. + +**Risk:** zero — only adds a kwarg sourced from the existing client's own `self.timeout` attribute. Pre-existing callers that constructed `MirixClient()` without an explicit timeout still get the 60s default. + +**Before vs After:** + +Before: +```python +response = await self._client.request( + method=method, url=url, json=json, params=params, headers=headers +) +``` + +After: +```python +# Per-request timeout is set from self.timeout so a caller can +# raise it via MirixClient(timeout=...) without rebuilding the +# AsyncClient. RetryTransport propagates request.extensions +# ['timeout'] to the wrapped AsyncHTTPTransport unchanged. +response = await self._client.request( + method=method, url=url, json=json, params=params, headers=headers, + timeout=self.timeout, +) +``` + +**No class, method signature, or public API changed** — `MirixClient._request` keeps its existing parameters and return type; only one keyword argument is forwarded to the underlying `httpx` call, and an adjacent comment was corrected. + +--- + +**Summary across all 3 patches:** every change is strictly additive. No class hierarchy, method signature, parameter name, return type, sentinel value, or public/imported symbol was renamed, removed, or repurposed. The patches reduce log noise (Redis), prevent a single-item crash from aborting an ingest batch (semantic memory), and let the caller's timeout budget reach the wire (remote client) — without touching anything callers depend on. + +## Benchmark results + +### SHDOCQA full runs (1 conversation × 100 questions, `substring_exact_match` judge) + +| date | chunking | n_chunks | acc | total_ingest_min | per-chunk_median_s | notes | +|---|---|---|---|---|---|---| +| 2026-06-01 | char-4096 | 273 | 82% | 169 | 32.6 | baseline | +| 2026-06-02 | token-4096 | 50 | 27% | 97 | 111.8 | **invalid — broken retrieval (`user.organization_id` NULL bug). Do not cite.** | +| 2026-06-03 | token-4096 | 50 | 69% | 97 | 111.8 | true apples-to-apples vs row 1 | + +Net: char-4096 wins on accuracy by 13 pp; token-4096 wins on ingest wall-clock by 1.74×. + +### Smoke tests (1 chunk × 1 question per dataset, pipeline validation) + +| dataset | retrieval block size | predicted answer (truncated) | result | +|---|---|---|---| +| SHDOCQA | 2424 chars | "Normandy is located in France." | CORRECT | +| MHDOCQA | 1777 chars | "Scott Derrickson and Ed Wood were both American." | substring miss (semantically right but answer schema wants literal "yes") | +| LongMemEval-S | 1829 chars | "There is no clear information about how many hours you work..." | appropriate abstention (single-session smoke can't contain work-hours) | +| LRU/infbench | 2524 chars | "Miss Jennifer Pete is a young woman of plain but impressive beauty..." | F1=0, fluency=1 — coherent but 1-of-95-chunks ingestion limits coverage | +| LRU/detective | 258 chars | "{answer: D. Miss House}" | substring miss (wrong option) but confident, retrieval working | + +### Memory store characterisation (SHDOCQA, 50 token-4096 chunks → 1040 epi + 1194 sem items) + +- Proper-noun density per semantic item: 18.77 (char) → 15.39 (token); -18% +- Year-token density per semantic item: 1.51 → 1.22; -19% +- Mean summary length: ~163 chars (char) vs ~152 chars (token); -7% +- Mean details length: ~457 chars (char) vs ~393 chars (token); -14% +- Distinct named entities: 6368 (char) vs 5715 (token); 1104 entities only in char store, 451 only in token store, 5264 in both + +### Cost / efficiency (SHDOCQA, same row) + +- Total ingest time: 10131s (char) vs 5832s (token); 1.74× speedup +- Per-chunk median time: 32.6s (char) vs 111.8s (token); 3.4× heavier per call +- Per-correct-answer cost: 124s (char) vs 85s (token); -32% (token gets more accuracy per unit time but lower absolute accuracy) +- Average answer latency at retrieval time: 2.82s (char) vs 2.54s (token); similar + +### System health (all 4 dataset smokes on 2026-06-03 / -04) + +- Server uptime: 1d 14h continuous +- 5 user records (`shdocqa_ruler_qa_0`, `mhdocqa_ruler_qa_0`, `longmem_s_0`, `infbench_0`, `conv-26`) all with `organization_id='mirix-eval-org'` +- Redis user cache consistent with PG +- `retrieve_with_conversation` returns non-empty `episodic_memory` block for every smoke +- Zero `httpx.ReadTimeout`, zero Q&A crashes after the empty-kwarg patch + +## Bugs discovered + lessons learned + +A chronicle of the rabbit holes hit while standing up the MAB eval suite. Read this before debugging anything that smells like one of these — most of them mimic "model got worse" or "framework bug" but are something else entirely. + +### 1. macOS `tmp_cleaner` wiping `/tmp/mirix_isolate/.venv` at midnight + +- **Symptom:** A SHDOCQA full run failed mid-stream with `FileNotFoundError [Errno 2]` thrown from inside `httpx`/`ssl` while calling OpenAI embeddings. The client log showed `/memory/add_sync` returning HTTP 500 from the server. Up until ~00:00 the run was healthy; after midnight every request failed identically. +- **Root cause:** `/usr/libexec/tmp_cleaner` (LaunchDaemon `com.apple.tmp_cleaner`, `StartCalendarInterval` Hour=0) wiped 76 packages under `/tmp/mirix_isolate/.venv/lib/python3.12/site-packages/` at 00:00 on 2026-06-02. `certifi/cacert.pem` was among the casualties, so `ssl.create_default_context(cafile=certifi.where())` raised `FileNotFoundError` on the very next outbound HTTPS call. +- **Diagnostic path:** + 1. Server log pinpointed the throw to `.../mirix/embeddings.py:421`. + 2. `ls -la` inside the venv showed 76 empty package directories, all with `mtime` of `2026-06-02 00:00:0X`. + 3. `cat /System/Library/LaunchDaemons/com.apple.tmp_cleaner.plist` confirmed the midnight schedule and the `/tmp` scope. +- **Fix:** Moved the whole `MIRIX_eval` working tree out of `/tmp` to `~/MIRIX_eval`, recreated the venv there, restarted the server. No more midnight reaper. +- **Forward-looking advice:** Never put a long-lived eval venv under `/tmp` on macOS. If something forces a tmp-ish path, use `/private/var/tmp` or another location outside `tmp_cleaner`'s scope. Add a guard at the top of every runner script: + ```bash + [[ "$(realpath .venv)" != /tmp/* ]] || { echo "venv is under /tmp; tmp_cleaner will eat it"; exit 1; } + ``` + +### 2. `users.organization_id` NULL silently kills retrieval (and the Redis user cache hides the fix) + +- **Symptom:** A new SHDOCQA full run with token-4096 chunking scored 27%, down 55 pp from char-4096's 82%. The model produced abstentions ("no specific information available") for most questions, as if its memory had been wiped. +- **Confounder:** Two unrelated problems were stacked. The bigger one was that retrieval was returning zero items for every question — completely independent of the chunker change. +- **Root cause:** I'd migrated the user row `ruler_qa_0` → `shdocqa_ruler_qa_0` with a raw `INSERT INTO users SELECT ... FROM users WHERE id=...` that did not list `organization_id`. The new row's `organization_id` was NULL. Every `list_*_items` query filters `WHERE organization_id = user.organization_id`, which becomes `WHERE organization_id IS NULL` — no row in the memory tables matches, so retrieval returns empty. +- **Second-level confounder:** `UserManager.get_user_by_id` caches the Pydantic user in Redis via plain `HGETALL` (not RedisJSON, so unaffected by the modules-missing latch). A bare PG `UPDATE` does not invalidate that cache. After `UPDATE users SET organization_id='mirix-eval-org'`, `get_user_by_id` was still returning `organization_id=None` from cache, so retrieval kept returning empty. +- **Diagnostic path:** + 1. `retrieve_with_conversation` reported `total_count=1194` but `items=0` for semantic memory — the smoking gun (data exists, filter excludes it). + 2. Manager-level `list_semantic_items(use_cache=False)` also returned 0, ruling out item-level cache. + 3. Raw ORM `SELECT` against `users` showed `organization_id='mirix-eval-org'`. + 4. `UserManager.get_user_by_id` Pydantic still returned `organization_id=None`. + 5. `redis-cli HGETALL user:shdocqa_ruler_qa_0` showed the `organization_id` field literally empty. +- **Fix:** + ```sql + UPDATE users SET organization_id='mirix-eval-org' WHERE id='shdocqa_ruler_qa_0'; + ``` + ```bash + redis-cli DEL user:shdocqa_ruler_qa_0 + ``` +- **Forward-looking advice:** + - Any raw-SQL user migration MUST copy `organization_id`. The Pydantic model defaults it to `DEFAULT_ORG_ID`, but raw `INSERT ... SELECT` will happily leave it NULL. + - Smoke test after every user mutation: `curl '/memory/components?user_id=X&memory_type=semantic&limit=3'`. If `total_count > 0` but `items == []`, you are in this trap. + - Anywhere DB user rows are mutated programmatically, either invalidate `user:` in Redis or write through. A `DEL user:` after any user UPDATE is cheap insurance. + +### 3. Char-4096 vs token-4096 chunking — apples-to-apples is harder than it looks + +- **Symptom:** SHDOCQA scores read: char-4096 = 82%, token-4096 (broken) = 27%, token-4096 (fixed) = 69%. Taken at face value, the first comparison "proved" the chunker change was catastrophic. It wasn't — the 27% was Bug #2. +- **Root cause (the real, post-fix finding):** Token-4096 chunks pack roughly 3.4× more text per call than char-4096, but the per-chunk extraction budget (~45 items) stayed constant. The extractor compensates by summarising at a higher abstraction level. Specific entities (Conrad of Montferrat, Maciot de Bethencourt, etc.) get folded into thematic items or dropped entirely. +- **Numbers:** Proper-noun density dropped from 18.77 → 15.39 per item (-18%). 1,104 distinct entities present in the char-4096 store had no counterpart in the token-4096 store. +- **Diagnostic path:** Once Bug #2 was fixed and the new score landed at 69%, comparing entity-density and entity-set diffs between the two stores explained the remaining ~13 pp gap. +- **Forward-looking advice:** + - When changing two things at once (chunker + user_id), prove each one independently. Re-run the question phase with the new chunker *and* the old chunker against the same `user_id` before drawing conclusions about chunking. + - For needle-retrieval evals (RULER, LongMemEval-S short-answer), prefer the smaller chunk. For summarisation workloads (LRU, InfBench), the tradeoff may invert — bigger chunks may help by giving the extractor more context to abstract over. + +### 4. `RetryTransport` timeout propagation — the prior comment was wrong + +- **Symptom:** Eval calls were silently dying at the 60s mark even though `MirixClient(timeout=1800)` was nominally being used. A comment in `remote_client.py` blamed the framework: + > "The wrapped `RetryTransport` otherwise falls back to `httpx.AsyncHTTPTransport`'s default 60s for read/write, which silently overrides `AsyncClient(timeout=N)`." +- **Root cause:** That comment is wrong. A minimal fake-transport repro showed that `RetryTransport.handle_async_request` forwards the request unchanged to `self._wrapped.handle_async_request`, and the wrapped transport reads `request.extensions['timeout']` exactly as it would without the wrapper. `AsyncClient(timeout=N)` and per-request `client.request(..., timeout=N)` both propagate correctly through the wrapper. The real cause of the eval's 60s ceiling: the eval was constructing `MirixClient()` *without* passing `timeout`, so `self.timeout` stayed at the default 60s, and the per-request `timeout=self.timeout` then enforced 60s. +- **Diagnostic path:** Built a fake `httpx.AsyncBaseTransport` that recorded the resolved timeout from each incoming request and verified it matched the configured value in every case — including through `RetryTransport`. +- **Fix:** Pass `timeout=1800` when constructing `MirixClient` in both `mirix_memory_system` and `task_agent`. +- **Forward-looking advice:** When a "framework bug" claim lives in a code comment, re-verify with a minimal repro before trusting it. Comments age out faster than code, and the "obvious" framework bug is almost always a misconfiguration one level up. + +### 5. Eval crashed at Q57 from an empty-string kwarg key in an LLM tool call + +- **Symptom:** SHDOCQA Q&A run died with `TypeError: MirixClient.search() got an unexpected keyword argument ''` after answering 56 of 100 questions. The first 56 questions were fine; Q57 happened to elicit a tool call whose JSON args contained an empty-string key. +- **Root cause:** `task_agent.py:_search_memory` does `mirix_client.search(user_id=..., **params)` where `params` is the JSON-decoded args dict from an OpenAI tool call. The LLM occasionally emits a key that is the empty string (`""`). Python rejects an empty kwarg name when it reaches the receiver's signature, and the whole eval falls over. +- **Fix:** Filter falsy/non-string keys before the splat, plus wrap the call in a `try/except TypeError` so the model can retry with a cleaner query: + ```python + params = {k: v for k, v in params.items() if isinstance(k, str) and k} + try: + result = mirix_client.search(user_id=user_id, **params) + except TypeError: + return {"success": False, "skipped": True, "reason": "bad tool args"} + ``` +- **Forward-looking advice:** Any `**dict` splat of LLM-provided args needs (a) a key-shape filter and (b) a `TypeError` safety net. Cost: one `if`-statement. Benefit: the eval doesn't die ~halfway through a 100-question run because the model emitted one weird tool call. + +--- + +**Meta-lesson across all five:** Most of these masqueraded as "model got worse" or "framework is broken." In every case the real cause was lower in the stack — a LaunchDaemon, a missing column, a stale Redis row, a misread comment, an unsanitised splat. Before blaming the model or the framework, prove the eval's own substrate (filesystem, DB rows, cache, client config, tool-args plumbing) is intact. + +## Next steps + +- Push branch to origin: `git push --force-with-lease origin mab_runner`. The backup branch `mab_runner_pre_squash_260604-1402` is a local-only safety net preserving the pre-squash 14-commit history; it stays unpushed. +- Optional full runs queued behind validated smokes: + - MHDOCQA — 100 questions on the multi-hop RULER QA2 subset + - LongMemEval-S — 500 questions across the full single-session-* / multi-session / temporal-reasoning / knowledge-update / single-session-preference categories + - LRU / infbench — 100 questions × ~95 chunks each (summarisation workload, three gpt-4o-2024-05-13 calls per record at judge time) +- Future investigation: the chunk-size tradeoff measured on SHDOCQA (char-4096 +13 pp vs token-4096) is a needle-retrieval finding. Summarisation workloads (LRU/infbench) may invert it because the extractor benefits from more context per call. Re-measure with an infbench full run to see whether token-4096 wins on F1 there. +- Future infra: add write-through invalidation of the Redis user cache (`user:`) whenever a DB user row is mutated by any direct SQL path — closes the foot-gun that hid the `organization_id` NULL fix during Bug #2.