feat: structure-corpus converter + flat-vs-structured findings (#40)

Adds `bm-bench convert structure-corpus`, which rewrites a flat
conversation corpus into Basic Memory's native representation (typed
`- [category]` observations and `- relation [[Entity]]` links) via the
plan-billed `claude -p` extractor (no API spend), so BM can be measured
"at its best" instead of purely as a text search over chat logs.

Two modes: `augment` (keep the transcript and append the structured
block — how BM really works, since a note keeps its prose while its
parsed observations/relations become first-class searchable units) and
`replace` (structured block only; lossy). Doc ids and frontmatter are
preserved, so retrieval ground truth, recall, QA, and the failure
diagnostic stay directly comparable to the flat corpus. Output mirrors
the input layout, so both flat (`docs/`) and grouped (`groups/<g>/docs/`)
corpora work. 8 tests, no live LLM calls.

Findings (benchmarks/results/structured-ingestion-findings.md):
structure is a WASH where retrieval saturates (ConvoMem implicit_connection
augment ties flat 0.449; replace hurts via lossy distillation) and HELPS
retrieval where retrieval has headroom (LoCoMo multi_hop recall@5
0.778->0.833, recall@10 0.841->0.889, MRR 0.679->0.715; QA +1 because 81%
of those failures are answerer-bound). The recall gain is from observations
being better retrieval targets than facts buried in transcript, NOT
query-time graph traversal (which BM does not do today) — which points to
relation-traversal-at-search as a real future retrieval lever.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: groksrc

benchmarks/results/structured-ingestion-findings.md

@@ -0,0 +1,96 @@
+# Structured ingestion: does Basic Memory's native representation help?
+
+**Date:** 2026-06-14 · **Status:** internal finding · **Harness:** `convert structure-corpus`
+
+## Question
+
+The default benchmark ingests each conversation as a flat `## Conversation`
+transcript, so Basic Memory is exercised purely as a text-search engine — none
+of its native representation (typed `- [category]` observations and
+`- relation [[Entity]]` links) is used. Does running BM "at its best", with the
+conversations distilled into its knowledge-graph form, change the numbers?
+
+## Method
+
+`convert structure-corpus` rewrites each flat conversation doc into a BM-native
+note via the plan-billed `claude -p` extractor (no API spend), in two modes:
+
+- **augment** (faithful): keep the transcript **and** append an extracted
+  `## Observations` + `## Relations` block. This mirrors how BM actually
+  works — a note keeps its prose while its parsed observations/relations become
+  first-class searchable units.
+- **replace**: substitute the structured block for the transcript (lossy).
+
+Document ids and frontmatter are preserved, so retrieval ground truth, recall,
+QA, and the failure diagnostic stay directly comparable to the flat corpus.
+Answerer (`claude-haiku-4-5`) and judge (`claude-sonnet-4-6`) are held constant;
+only the corpus representation changes.
+
+## Results
+
+### ConvoMem `implicit_connection` (49 q) — retrieval saturated
+
+| arm | QA acc | correct |
+|---|---|---|
+| flat (transcripts) | **0.449** | 22/49 |
+| structured-replace (lossy) | 0.367 | 18/49 |
+| structured-augment (faithful) | **0.449** | 22/49 |
+
+Augment **ties** flat exactly (it flips 10/49 questions, +5/−5 — a wash);
+replace **hurts** because distillation drops the specific personal details these
+questions hinge on. **Why:** ConvoMem cs10 retrieval is *saturated* (the
+diagnostic shows a retrieval ceiling of 1.000 — the answerer already receives the
+entire 10-doc haystack), so representation can only reshuffle what is already in
+context. **Implication:** flat markdown is a *fair* representation for BM on
+ConvoMem — the benchmark was not secretly hobbling BM there.
+
+### LoCoMo `multi_hop` (63 q, full 272-doc haystack) — retrieval has headroom
+
+| metric | flat | structured-augment | Δ |
+|---|---|---|---|
+| recall@5 | 0.778 | **0.833** | **+0.055** |
+| recall@10 | 0.841 | **0.889** | **+0.048** |
+| MRR | 0.679 | **0.715** | +0.036 |
+| QA accuracy | 0.556 (35/63) | 0.571 (36/63) | +0.015 |
+
+Structure **improves retrieval**: +5 questions newly reach their gold evidence in
+the top-10 (−2 lost, net +3). Example — *"Where did Joanna travel in July 2022?"*:
+flat never retrieves the evidence (→ "I don't know"); structured does (→
+"Woodhaven", correct). QA barely moves (+1) because the diagnostic attributes 81%
+of multi_hop failures to the fixed answerer, so retrieval gains don't fully
+convert end-to-end.
+
+## Interpretation
+
+Structure is a **wash where retrieval saturates** (ConvoMem) and **helps recall
+where retrieval has headroom** (LoCoMo multi_hop) — a coherent two-slice story.
+
+**Mechanism (important for credibility):** Basic Memory does **not** traverse
+relations at query time today, so the recall gain is **not** graph multi-hop. It
+comes from the concise observations being *better retrieval targets* than the same
+facts buried in a long transcript — a better lexical/semantic surface.
+
+**Confound to close before publishing:** augmented docs are longer (transcript +
+structure), giving more match surface. A replace-mode retrieval ablation would
+isolate "structure" from "more text". The LoCoMo effect is also modest (n=63);
+confirm on the full q300 / multi_hop+temporal slice before any external claim.
+
+**Product lever this surfaces:** following `[[Entity]]` relations during search
+(query-time relation traversal) would turn the structured representation into
+genuine multi-hop retrieval — a concrete future basic-memory retrieval
+improvement, distinct from the lexical-surface gain measured here.
+
+## Reproduce
+
+```bash
+# Structure a corpus (flat or grouped layout; augment keeps the transcript)
+bm-bench convert structure-corpus \
+  --input-dir benchmarks/generated/locomo-corrected-v2/docs \
+  --output-dir benchmarks/generated/locomo-corrected-v2-augmented/docs \
+  --mode augment --extractor claude:claude-haiku-4-5
+
+# Run flat vs structured over the same queries; compare recall + QA + diagnose
+bm-bench run retrieval --providers bm-local --corpus-dir <corpus> --queries-path <queries> ...
+bm-bench run qa --run-dir <run> --answerer claude:claude-haiku-4-5 --judge claude:claude-sonnet-4-6
+bm-bench run diagnose --run-dir <run>
+```

src/basic_memory_benchmarks/cli.py

@@ -102,6 +102,63 @@ def convert_locomo(
     console.print(f"Queries: [cyan]{queries_path}[/cyan] ({query_count})")
 
 
+@convert_app.command("structure-corpus")
+def convert_structure_corpus(
+    input_dir: Path = typer.Option(
+        ..., "--input-dir", help="Source corpus root (a flat docs dir or a grouped …/groups dir)"
+    ),
+    output_dir: Path = typer.Option(
+        ..., "--output-dir", help="Destination root; the input layout is mirrored beneath it"
+    ),
+    mode: str = typer.Option(
+        "augment",
+        "--mode",
+        help="augment: keep transcript + append structure (faithful); replace: structure only (lossy)",
+    ),
+    categories: str = typer.Option(
+        "",
+        "--categories",
+        help="Grouped corpora only: comma-separated category labels to restructure (matches group-id prefix). Empty = all docs.",
+    ),
+    extractor: str = typer.Option(
+        "claude:claude-haiku-4-5", "--extractor", help="LLM runner spec for fact extraction"
+    ),
+    max_workers: int = typer.Option(4, "--max-workers"),
+) -> None:
+    """Restructure flat conversation docs into Basic Memory observations/relations.
+
+    Produces a structured twin of a corpus with doc ids/frontmatter preserved, so
+    a flat-vs-structured run isolates the representation and recall stays
+    comparable. Works on both grouped and flat corpora (layout is mirrored).
+    """
+    from basic_memory_benchmarks.converters.structure_corpus import (
+        group_prefix_filter,
+        structure_corpus,
+    )
+    from basic_memory_benchmarks.llm.runners import create_runner
+
+    if mode not in ("augment", "replace"):
+        raise typer.BadParameter("--mode must be 'augment' or 'replace'")
+    cats = {c.strip() for c in categories.split(",") if c.strip()}
+    path_filter = group_prefix_filter(cats) if cats else None
+    runner = create_runner(extractor)
+    output_dir.mkdir(parents=True, exist_ok=True)
+
+    doc_count = structure_corpus(
+        input_root=input_dir,
+        output_root=output_dir,
+        runner=runner,
+        mode=mode,  # type: ignore[arg-type]
+        path_filter=path_filter,
+        max_workers=max_workers,
+    )
+
+    console.print(f"Structured ([green]{mode}[/green]): [cyan]{output_dir}[/cyan] ({doc_count} docs)")
+    console.print(f"Extractor: [green]{extractor}[/green]")
+    if cats:
+        console.print(f"Filtered to categories: {sorted(cats)}")
+
+
 @convert_app.command("longmemeval")
 def convert_longmemeval(
     dataset_path: Path = typer.Option(

src/basic_memory_benchmarks/converters/structure_corpus.py

@@ -0,0 +1,156 @@
+"""Restructure a flat conversation corpus into Basic Memory's native form.
+
+The default benchmark corpus stores each conversation as a flat
+``## Conversation`` transcript, so Basic Memory is exercised purely as a
+text-search engine over chat logs — none of its actual representation (typed
+``- [category]`` observations and ``- relation [[Entity]]`` links) is used. That
+can undersell a system whose strength is a structured knowledge graph,
+especially where the answer requires joining facts the prose states separately.
+
+This converter rewrites each conversation doc into the note shape a Basic Memory
+user/agent would actually write. Two modes:
+
+- ``augment`` (default, the faithful one): keep the original transcript and
+  *append* an LLM-extracted ``## Observations`` + ``## Relations`` block. This is
+  how Basic Memory really works — a note keeps its prose AND its parsed
+  observations/relations become first-class searchable units — so it can only
+  add signal, never lose the source text.
+- ``replace``: swap the transcript for the extracted structure only. Useful to
+  measure the pure distilled representation, but strictly lossy.
+
+Document ids and frontmatter are always preserved, so retrieval ground truth and
+recall stay directly comparable to the flat corpus, and every downstream stage —
+provider ingest, retrieval scoring, QA, the failure diagnostic — works unchanged.
+The extractor runs through the same plan-billed ``claude -p`` runner used
+elsewhere, so there is no API spend. Output mirrors the input directory layout,
+so both grouped (``groups/<g>/docs``) and flat (``docs``) corpora are supported.
+"""
+
+from __future__ import annotations
+
+import re
+from concurrent.futures import ThreadPoolExecutor
+from pathlib import Path
+from typing import Callable, Literal
+
+from basic_memory_benchmarks.llm.runners import LLMRunner
+
+Mode = Literal["augment", "replace"]
+
+EXTRACTION_PROMPT = """\
+You are converting one conversation into a Basic Memory structured note.
+
+Basic Memory represents knowledge as typed observations and relations, NOT prose:
+- An observation is a single fact written as: `- [category] the fact #optional-tag`
+  Categories are short lowercase labels you choose to fit the fact, e.g.
+  [fact], [event], [preference], [decision], [requirement], [risk], [recommendation].
+- A relation links the note's subject to another entity:
+  `- relation_type [[Entity Name]]`
+  Use natural, consistent entity names (e.g. [[Email Communication]],
+  [[Phishing Incident]]) and verb-like relation types (relates_to, caused_by,
+  mitigated_by, prefers, concerns, depends_on).
+
+Extract EVERY concrete, durable fact the conversation establishes — what was
+said, decided, preferred, experienced, or recommended — as observations,
+preserving specific names, dates, numbers, and personal details verbatim. Then
+capture how the entities involved connect to each other as relations. Make
+implicit connections explicit: if the conversation links two things (an incident
+to a vulnerability, a preference to a constraint), write that as a relation or an
+observation. Do not editorialize, summarize the chit-chat, or invent facts that
+are not supported by the text.
+
+Output ONLY markdown in exactly this form, nothing else (no preamble, no fences):
+
+## Observations
+- [category] ...
+- [category] ...
+
+## Relations
+- relation_type [[Entity]]
+- relation_type [[Entity]]
+
+Conversation to convert:
+{conversation}
+"""
+
+_CONVERSATION_RE = re.compile(r"\n## Conversation\b", re.DOTALL)
+
+
+def _split_doc(flat_text: str) -> tuple[str, str]:
+    """Return (header, conversation_body): header is everything up to (excluding)
+    the ``## Conversation`` heading, body is the transcript after it."""
+    match = _CONVERSATION_RE.search(flat_text)
+    if not match:
+        raise ValueError("Doc has no '## Conversation' section to restructure")
+    header = flat_text[: match.start()].rstrip() + "\n"
+    body = flat_text[match.end() :].strip()
+    return header, body
+
+
+def _clean_extraction(text: str) -> str:
+    """Strip stray code fences / preamble and ensure it starts at Observations."""
+    cleaned = text.strip()
+    if cleaned.startswith("```"):
+        cleaned = cleaned.strip("`")
+        cleaned = re.sub(r"^[a-zA-Z]+\n", "", cleaned).strip()
+    idx = cleaned.find("## Observations")
+    if idx > 0:
+        cleaned = cleaned[idx:]
+    return cleaned.strip()
+
+
+def structure_doc(flat_text: str, runner: LLMRunner, *, mode: Mode = "augment") -> str:
+    """Convert one flat conversation doc into a BM-native note (doc id kept).
+
+    ``augment`` retains the transcript and appends the structured block;
+    ``replace`` substitutes the structured block for the transcript.
+    """
+    header, conversation = _split_doc(flat_text)
+    result = runner.complete(EXTRACTION_PROMPT.format(conversation=conversation))
+    structured = _clean_extraction(result.text)
+    if "## Observations" not in structured:
+        # Fail loud so a bad slice can't masquerade as a structured corpus.
+        raise ValueError(f"Extractor returned no observations for a doc:\n{result.text[:400]}")
+    if mode == "replace":
+        return f"{header}\n{structured}\n"
+    return f"{flat_text.rstrip()}\n\n{structured}\n"
+
+
+def structure_corpus(
+    *,
+    input_root: Path,
+    output_root: Path,
+    runner: LLMRunner,
+    mode: Mode = "augment",
+    path_filter: Callable[[Path], bool] | None = None,
+    max_workers: int = 4,
+) -> int:
+    """Restructure every ``*.md`` under ``input_root`` into ``output_root``,
+    mirroring the relative directory layout. ``path_filter`` (given the source
+    path) selects which docs to convert. Returns the doc count."""
+    docs = [
+        path
+        for path in sorted(input_root.rglob("*.md"))
+        if path_filter is None or path_filter(path)
+    ]
+
+    def _work(doc_path: Path) -> None:
+        structured = structure_doc(doc_path.read_text(encoding="utf-8"), runner, mode=mode)
+        dest = output_root / doc_path.relative_to(input_root)
+        dest.parent.mkdir(parents=True, exist_ok=True)
+        dest.write_text(structured, encoding="utf-8")
+
+    with ThreadPoolExecutor(max_workers=max_workers) as pool:
+        list(pool.map(_work, docs))
+
+    return len(docs)
+
+
+def group_prefix_filter(categories: set[str]) -> Callable[[Path], bool]:
+    """Path filter matching grouped corpora: keep docs whose group directory name
+    starts with one of ``categories`` (e.g. ``implicit_connection-cs10-…``)."""
+
+    def _filter(path: Path) -> bool:
+        return any(part.startswith(f"{cat}-") for part in path.parts for cat in categories)
+
+    return _filter