Skip to content

feat: cross-tool memory via FTS5 index#21

Merged
NJannasch merged 3 commits into
mainfrom
feat/fts5-cross-tool-memory
May 9, 2026
Merged

feat: cross-tool memory via FTS5 index#21
NJannasch merged 3 commits into
mainfrom
feat/fts5-cross-tool-memory

Conversation

@NJannasch

Copy link
Copy Markdown
Owner

Summary

  • Adds a single SQLite/FTS5 store at ~/.config/vibecockpit/cache/memory.db that indexes every transcript message from every provider, so Claude Code / Cursor / Codex / Gemini all share the same memory substrate.
  • Exposes it through three surfaces backed by one engine:
    • Web — dedicated Memory page with search, ±N context drawer, host badges, Active/Excluded tabs, per-result delete, browser-driven import/export.
    • MCPsearch_memory + get_session_context tools (agents see what the human sees).
    • CLIvibecockpit memory {reindex,search,stats,export,import}.

Notable design choices

  • Per-message FTS5 rows (porter + unicode61, BM25), with read-time dedup to one anchor message per session for the list view.
  • Sessions stamped with os.Hostname() at index time. Orphan cleanup is host-scoped, so imported rows aren't pruned just because their JSONL isn't local.
  • session_tombstones table — user deletes are sticky across reindex AND import. Restorable from the Excluded view.
  • Idempotent re-import: re-uploading the same .db is a no-op (no message duplication; session_meta presence + tombstone filters guard both directions).
  • Lenient query parser: punctuation barewords (example.com, v1.2.3, feat/login) auto-quote on FTS5 syntax-error retry. Power-user expressions like auth NOT okta still parse first try.
  • Web boot kicks off async IndexAll, repeats on every scan-cache refresh. MCP tool calls audit-logged via existing path.
  • Bonus: a11y warnings cleared in Memory, Scheduler, ChatView (overlay div → <button>, dialog role conflict, <fieldset>/<legend> for grouped checkboxes, $state() for bind:this targets).

Test plan

  • make check (CI runs golangci-lint + race tests + frontend lint)
  • Web: open Memory page, search across providers, click result → context drawer, change ±window
  • Web: Export → reload page → Import same file → confirm added: 0 and stats unchanged
  • Web: Delete a session → Excluded tab shows it with snapshotted metadata → re-import → stays gone → Restore → re-import → reappears
  • MCP: search_memory with punctuation tokens (example.com, v1.2.3) returns hits without manual quoting
  • MCP: get_session_context window from 0 to 50 returns the expected message slice with isCenter flagged

NJannasch added 3 commits May 9, 2026 21:58
Single SQLite/FTS5 store under ~/.config/vibecockpit/cache/memory.db
indexes every transcript message from every provider so Claude Code,
Cursor, Codex, etc. can all recall what was said in any prior session.

Surfaces:
- Web: dedicated Memory page with search, ±N context drawer, host badge,
  Active/Excluded tabs, per-result delete, file-based import/export.
- MCP: search_memory + get_session_context tools. Agents share the
  same backend the user does.
- CLI: vibecockpit memory {reindex,search,stats,export,import}.

Index design:
- Per-message FTS5 rows (porter+unicode61, BM25), dedup-by-best-message
  on read so the list view stays clean while context anchors stay
  precise.
- Sessions stamped with os.Hostname() at index time so cross-machine
  imports keep their origin label; orphan cleanup is host-scoped so
  imported rows aren't pruned just because their JSONL isn't local.
- session_tombstones table makes user-deleted sessions sticky across
  reindex AND import; restorable from the Excluded view.
- Idempotent re-import: session_meta presence + WHERE filters mean
  uploading the same .db twice is a no-op (no message duplication).
- Lenient query parser: punctuation barewords (example.com, v1.2.3,
  feat/login) auto-quote on FTS5 syntax-error retry; power-user
  expressions like "auth NOT okta" still parse first try.

Also folded in:
- a11y warnings cleared across Memory, Scheduler, ChatView (drawer
  overlay → button, role/dialog combo, fieldset+legend for grouped
  checkboxes, $state for bind:this targets).
- Web kicks off async IndexAll on boot and on every scan-cache refresh.
- All MCP tool calls audit-logged via existing s.audit.Log path.
- memory_cli: split each subcommand into its own error-returning helper;
  runMemory dispatches and exits centrally. Lets the deferred Close
  actually run (gocritic exitAfterDefer) and silences the unchecked-Close
  errcheck via the standard `defer func(){ _ = idx.Close() }()` pattern.
- index.Search: add a //nolint:gosec G201 suppression with a comment
  explaining that the only formatted SQL fragment is built from a fixed
  literal allowlist; all user values stay parameterized via ?.
The previous nolint directive sat on the closing line of a multi-line
fmt.Sprintf, where golangci-lint's nolint scanner doesn't apply it.
Lift the SQL into a package-level searchSQLTemplate const so the call
itself is one line, and the //nolint:gosec directive sits next to it.

Verified locally with the same golangci-lint v2.1.6 the CI workflow uses
(v2.1.6 reports 0 issues).
@NJannasch NJannasch merged commit 2c30f1d into main May 9, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant