perf(recall): cut recall-bundle tokens ~20%+ with recall held constant

[TJCurnutte]

Goal

Cut the tokens Memory Hive serves per recall by ≥20% while keeping inference/recall the same or better.

Approach

Two changes to bundle rendering only — the FTS index and query selection are untouched, so which chunks are retrieved is identical and recall is preserved by construction:

1. Markdown compaction (compact_snippet): reflow wrapped prose, join bullet continuations, collapse whitespace, trim table cell padding, and drop pure table separator rows. Fenced code is preserved verbatim. Every query-relevant word is kept (unit-tested: words(input) == words(output)), and compaction is non-increasing (never inflates a snippet).
2. Near-duplicate suppression (dedupe): drop a candidate whose token-set Jaccard ≥ 0.85 against an already-kept snippet. Memory Hive's write-back ritual echoes the same fact across logs/memory/learnings and across silos; those redundant copies cost tokens without adding recall.

Both default on in bundle() (new compact / dedupe params, off = previous behavior for A/B). bundle_json, the CLI bundle, and MCP ask_hive all benefit automatically.

Results (benchmark: scripts/bench/recall_token_efficiency.py)

Honest A/B at a budget large enough that neither bundle truncates, so the retrieved set is identical and the delta is pure rendering efficiency:

Corpus	Token cut	Recall
Realistic hive (write-back echoes)	62.7% (7832 → 2923)	0 facts lost; tight-budget recall equal
— compaction only	11.5%	—
— dedupe only	59.8%	—
Low-redundancy hive (compaction in isolation)	12.4%	recall improved 7 → 9 distinct facts
Real CLI-generated hive (add + learn)	56.2% (1800 → 788)	top citation preserved every query

The ≥20% target is met on realistic and real product-generated hives. Even with zero redundancy, compaction alone cuts ~12% and frees budget to fit more distinct facts (recall "or better"). No benchmark query lost a fact.

recall_token_efficiency_benchmark.txt

Review fixes (Bugbot)

• Compaction could inflate a tightly-packed table row (|a|b| → | a | b |). Now renders the shorter of padded/collapsed forms; added test_compaction_never_increases_length. No recall impact (words always preserved); this only tightens the token guarantee.

Testing

• ✅ PYTHONPATH=$(pwd) python3 -m unittest discover -s tests — 45 passed (7 new; existing bundle redaction/budget/JSON tests still green)
• ✅ scripts/bench/recall_token_efficiency.py — PASS (≥20% cut, 0 facts lost, recall ≥ baseline)
• ✅ Adversarial edge probes: empty input, unbalanced code fence, code blocks containing |/---, separator-only tables, distinct short facts (not deduped)

Notes / tradeoffs

• dedupe hides independent corroborating copies of a fact by design; the unique fact is always retained. Threshold 0.85 only collapses true near-duplicates (distinct facts rarely share 85%+ of vocabulary).
• Token estimate uses the engine's existing chars/4 heuristic.

_{To show artifacts inline, enable in settings.}

  

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_4c708116-9b36-4864-ac25-92cf46914186)