[FEATURE] integrate @tobilu/qmd for search and dedup in FileMemoryStore

[lizradway]

Integrate @tobilu/qmd for search and dedup in FileMemoryStore

Problem

FileMemoryStore has two weaknesses:

1. Search quality: keyword token-overlap doesn't weight terms, misses paraphrases, and produces unintuitive rankings
2. Duplicate accumulation: same-slug upsert only catches entries with identical first lines — different phrasing of the same fact creates endless duplicates

Both are similarity problems. qmd solves both with one integration.

Proposed solution

Use @tobilu/qmd (optional peer dependency) for:

• search(): BM25 scoring via searchLex() instead of token-overlap
• add(): before writing, searchLex() the new content against existing entries — if BM25 score exceeds a threshold, overwrite that file instead of creating a new one

Flow: add(content)

1. Ensure qmd store is initialized (lazy)
2. Index any new/changed files: store.update()
3. Search existing entries: store.searchLex(firstLine, { limit: 1 })
4. If top result score > threshold → overwrite that file path
5. Else → write to new slug-derived path
6. Re-index after write

Flow: search(query)

1. Ensure qmd store is initialized (lazy)
2. Index any new/changed files: store.update()
3. Return store.searchLex(query, { limit: maxResults })
4. Map results back to MemoryEntry format

Without qmd installed

Falls back to current behavior:

• search(): keyword token-overlap
• add(): slug-based upsert (same first line = overwrite, different = new file)

Integration details

Lifecycle

• Lazy init: createStore() on first add() or search() call
• DB location: .strands/memory/<store-name>/.qmd.sqlite
• Re-index: store.update() before search/dedup checks (tracks filesystem changes)
• Dispose: expose close() on FileMemoryStore for cleanup

Dedup threshold

• Default: configurable, probably BM25 score > some empirical threshold (needs testing with real memory content)
• Exposed on config: FileMemoryStoreConfig.deduplicationThreshold?: number
• Set to Infinity to disable dedup (always create new files)
• Set to 0 to always overwrite top match (aggressive)

Constraint: requires LocalFileStorage

qmd needs a filesystem path. When the Storage backend isn't file-based, fall back to keyword scoring and slug-based upsert (current behavior). Detection:

if (this._storage instanceof LocalFileStorage) {
  // qmd available — use BM25 for search + dedup
} else {
  // non-filesystem backend — keyword fallback
}

Package changes

"peerDependencies": {
  "@tobilu/qmd": "^2.5.0"
},
"peerDependenciesMeta": {
  "@tobilu/qmd": { "optional": true }
}

File structure

strands-ts/src/vended-memory-stores/file-memory-store/
  file-memory-store.ts    # Main class — delegates to qmd-search when available
  qmd-search.ts           # Lazy qmd lifecycle, search, dedup-before-write
  types.ts                # Add deduplicationThreshold to config
  __tests__/
    qmd-search.test.node.ts

Why this works

qmd's BM25 naturally captures "these two documents are about the same thing" better than token-overlap because it weights rare terms higher than stopwords. A high BM25 score between a new fact and an existing one is a strong signal they're the same topic — no embeddings needed for the common case.

For the long tail (complete paraphrases with zero shared keywords), vector search (store.searchVector()) could be added later as a second-pass check, but BM25 alone handles the majority of real-world dedup cases.

Fallback matrix

qmd installed?	Storage is file-based?	search() uses	add() dedup uses
Yes	Yes	BM25 via searchLex	BM25 score threshold
Yes	No	keyword token-overlap	slug-based upsert
No	Yes	keyword token-overlap	slug-based upsert
No	No	keyword token-overlap	slug-based upsert