From f75143a287618193dcf1b2e5d8184f293ab1a889 Mon Sep 17 00:00:00 2001 From: gethin Date: Thu, 23 Jul 2026 22:54:08 +0100 Subject: [PATCH 01/17] docs: deferred task-conditioned injection design (PR-D, absorbs PR-C) Co-Authored-By: Claude Fable 5 --- .../2026-07-23-deferred-injection-design.md | 159 ++++++++++++++++++ 1 file changed, 159 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-23-deferred-injection-design.md diff --git a/docs/superpowers/specs/2026-07-23-deferred-injection-design.md b/docs/superpowers/specs/2026-07-23-deferred-injection-design.md new file mode 100644 index 0000000..eed073e --- /dev/null +++ b/docs/superpowers/specs/2026-07-23-deferred-injection-design.md @@ -0,0 +1,159 @@ +# PR-D: Deferred task-conditioned memory injection + +**Date:** 2026-07-23 +**Status:** approved design, pending implementation +**Predecessors:** PR #81 (rating loop), PR #83 (Wilson prior, exploration slot, +embeddings, three-leg fusion). Absorbs PR-C (CLAUDE.md rewrite + drift +sentinel) from the 2026-07-23 retrieval-quality design; PR-B (evidence +ratings) remains separate and follows. + +## Guardrails (from planning memories + standards) + +- **[[da7ff62e]]** (conf 0.9, useful 15): the user-level CLAUDE.md mandates a + broad startup `memory_retrieve` + `knowledge_list`. A near-empty bootstrap + with that mandate intact just re-creates the firehose through the tool + channel — the mandate rewrite ships IN THIS PR. The knowledge-base channel + keeps its startup mandate (bootstrap never carried it). +- **[[be7ad6bf]]** (conf 0.9, useful 13): planning memories + standards + consulted before drafting — done (ralph-runtime.md rules: feature branch at + task start, confidence-scored plan, visualiser handoffs). +- **[[98056ebc]]** (conf 1.0, useful 18): website prose synced in the same PR. + +## Problem + +Bootstrap is the retrieval system's weakest channel — measured 12.9–17% +useful after every ranking fix, vs 21–26% for query-conditioned retrieval — +because at SessionStart there is nothing to condition on: the same top-N +memories inject every session regardless of what the session will do. The +fix is architectural, not a better ranker: **don't inject until there is +task signal**, then let the (already live) contextual channel carry +injection with a scorer worthy of being the primary channel. + +Measured basis: contextual channel hit 25% useful in its best round with a +crude keyword scorer; its instability (11–25% across rounds) is the scorer, +not the timing. + +## Design + +### 1. Injection architecture + +`BETTER_MEMORY_INJECT_MODE` config (env), values `deferred` | `legacy`. +`legacy` = today's behaviour, byte-identical. Initial live default: +`legacy` (flipped to `deferred` after the post-deploy gate passes; legacy +path deleted after a stable week). + +**SessionStart, deferred mode** (`hooks/session_bootstrap.py` + +`services/session_bootstrap.py`): +- Inject in full: ALL general-scope semantic memories (standing user rules). +- Inject one index line: "better-memory knows N reflections + M semantic + memories for this project; relevant ones will surface as you work — or + ask via memory_retrieve with a task query." +- Exposures recorded only for the fully-rendered general semantics. +- Episode open, session marker, failure-isolation: unchanged. + +**Contextual channel** (`hooks/contextual_inject.py`): +- `UserPromptSubmit`: fires every prompt (unchanged trigger). +- `PreToolUse`: matcher widens to ALL tools with a first-fire-per-session + latch persisted in the session's `context_seen` state; after the first + fire, later PreToolUse events no-op immediately. (Today's `Skill|Task| + Write` matcher misses sessions that open with Read/Bash.) +- Per-memory once-per-session dedup: unchanged. + +**CLAUDE.md rewrite (absorbed PR-C):** +- `better_memory/skills/CLAUDE.snippet.md` rewritten to behavioural + instructions only — "pass a task-describing `query` when you begin a + task", "credit with evidence" — never enumerating parameter names/types. + The startup mandate becomes: `knowledge_list` at startup (unchanged); + `memory_retrieve` WITH a task query when a task begins — no broad + no-query dump at session start. +- The user's live `~/.claude/CLAUDE.md` better-memory section gets the same + edit (local change alongside the PR, not a repo artifact). +- **Drift sentinel** in `session_bootstrap`: scans the CLAUDE.md + better-memory section for parameter tokens adjacent to tool names that + are absent from the live registry (imported from `mcp/tools.py`, so it + tracks future schema changes). On drift: one warning line appended to + bootstrap additionalContext. Silent when clean; best-effort; never blocks. + +### 2. Scorer + gate (`services/relevant.py` rewrite) + +Candidate pool unchanged: active project + general reflections (non-neutral +unless `include_neutral`) + semantic memories, minus session-seen. + +Three legs, reusing PR-A machinery: +- **BM25**: prompt text via `sanitize_fts5_query` (stopwords, >2-char + tokens, OR-joined) against `reflection_fts`; semantics via their existing + text-match path. +- **Vec**: prompt embedded once per firing through the SHARED `SyncEmbedder` + (same instance and circuit breaker as the services — wired through the + hook's construction). Cosine against `reflection_embeddings` / + `semantic_embeddings`. Breaker open → leg absent. +- **Prior**: `wilson_lower_bound` from `services/scoring.py`. The ad-hoc + `_activation` formula and the `min_hits` keyword floor are DELETED. + +Fusion: RRF, same shape/constant as retrieve. + +**Gate — inject vs stay silent:** a memory qualifies only with positive +relevance evidence: a BM25 match OR vec cosine ≥ `CONTEXT_VEC_FLOOR` +(config `BETTER_MEMORY_CONTEXT_VEC_FLOOR`, default 0.55, calibrated during +the eval task against labelled prompt/memory pairs from the A/B corpus). +Qualifiers ranked by fused score; top `context_max_items` (3) inject; zero +qualifiers → the hook emits nothing at all. The Wilson prior ranks among +qualifiers but can NEVER qualify a memory alone — popularity cannot force +an irrelevant injection. + +**Latency:** hook is synchronous in the prompt path. Healthy Ollama ≈ +20–40 ms per firing. Ollama down: one ≤5 s stall, then the breaker gives +60 s of instant BM25-only firings. + +### 3. Exploration tagging + metric redefinition + +- Migration (next free number at implementation time; PR-B holds 0015): + `session_memory_exposure` gains `via_exploration INTEGER NOT NULL + DEFAULT 0`. Additive. +- `retrieve_reflections`' bucket-fill threads the reserved-slot flag into + the exposure write. Contextual/bootstrap exposures are always 0. + First-source-wins dedup unchanged — an exploration serve of an + already-exposed memory writes nothing. +- Rating flow unchanged: explorers still rated; their ratings feed Wilson. +- **Standing metric definition (the 30% goal is judged on headline):** + - headline `useful_pct` = useful / exposures where `via_exploration = 0` + - all-in (old definition) reported for continuity with 22.96 / 18.47 + - exploration conversion rate = fraction of exploration serves whose + memory later earns a non-ignored rating + `metric.py` and `analyze.py` report all three. + +### 4. Delivery, rollout, validation + +- One PR, branch `feat/deferred-injection`, then PR-B. +- Merge + deploy with live default `legacy` — zero behaviour change on + merge day. +- **Post-deploy A/B, env-flag arms:** both arms run identical live code; + the runner sets only `BETTER_MEMORY_INJECT_MODE` per arm. This is the + structural fix for the A8 contamination (user-level hooks and runner + hooks execute the same code either way). 24 sessions per arm. +- Gate: deferred arm's headline useful% not statistically below legacy's + (one-sided z, α=0.05), AND deferred injection precision ≥ legacy's + bootstrap+contextual combined. Pass → flip live default to `deferred`; + delete legacy path after a stable week. + +### Error handling + +Every new path inherits existing contracts: SyncEmbedder never raises; +hooks wrap in their existing best-effort shells; gate-with-no-qualifiers +emits nothing (no empty block); sentinel silent-when-clean and never +blocks bootstrap; mode flag misparse coerces to `legacy` (fail-safe to +known behaviour). + +### Testing + +- Unit: gate floor (BM25-only / vec-only / neither / both), first-tool + latch (fires once, any tool, persists across events), mode flag both + ways (deferred renders general-semantics+index; legacy byte-identical to + today), `via_exploration` tagging incl. dedup interaction, metric + arithmetic for all three numbers, sentinel (phantom param detected / + clean silent / malformed CLAUDE.md never raises), vec-floor boundary. +- Integration: hook-level e2e for both modes; contextual e2e for a + Read-first session (latch fires). +- Deleted with `_activation`/`min_hits`: their tests. +- Website prose sync (guardrail): bootstrap/injection descriptions in + website/*.md updated in-PR. From 56e3649885caddfc49a9456f17526e7027dfd788 Mon Sep 17 00:00:00 2001 From: gethin Date: Thu, 23 Jul 2026 23:02:24 +0100 Subject: [PATCH 02/17] docs: deferred-injection implementation plan (PR-D) Co-Authored-By: Claude Fable 5 --- .../plans/2026-07-23-deferred-injection.md | 977 ++++++++++++++++++ 1 file changed, 977 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-23-deferred-injection.md diff --git a/docs/superpowers/plans/2026-07-23-deferred-injection.md b/docs/superpowers/plans/2026-07-23-deferred-injection.md new file mode 100644 index 0000000..7f760c2 --- /dev/null +++ b/docs/superpowers/plans/2026-07-23-deferred-injection.md @@ -0,0 +1,977 @@ +# Deferred Injection (PR-D) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Near-empty SessionStart injection; the contextual channel becomes primary with a three-leg evidence-gated scorer; exploration serves tagged and excluded from the headline metric. + +**Architecture:** `BETTER_MEMORY_INJECT_MODE` flag selects deferred vs legacy bootstrap. `services/relevant.py` is rewritten around BM25 (`reflection_fts`) + vec cosine (unit-norm vectors ⇒ L2 threshold) + Wilson prior, with an evidence gate that keeps popularity from qualifying anything. Hook processes get a file-persisted embed cooldown because the in-process breaker dies with each hook process. CLAUDE.snippet rewrite + drift sentinel ship in the same PR (absorbed PR-C). + +**Tech Stack:** Python 3.12, sqlite + sqlite-vec + FTS5, Ollama nomic-embed-text (768-dim unit-norm), pytest. + +**Spec:** `docs/superpowers/specs/2026-07-23-deferred-injection-design.md` + +## Global Constraints + +- Branch: `feat/deferred-injection` (exists; spec `f75143a`). +- Test command: `./.venv/Scripts/python.exe -m pytest -v`; typecheck `./.venv/Scripts/python.exe -m pyright` stays 0 errors. +- Hooks NEVER raise and never block (existing `except BaseException` shells stay). +- `legacy` mode must be byte-identical to today's bootstrap output; flag misparse coerces to `legacy`. +- Every Ollama touchpoint best-effort; hook-side stall bounded by the file cooldown. +- ASCII only in code/comments/CLI output. Ruff line length 100. +- Website sync guardrail (conf-1.0): prose updated in-PR (Task 9). +- Commits per task, footer `Co-Authored-By: Claude Fable 5 `. + +## Verified-against-source facts (do not re-derive) + +| Fact | Where verified | +|---|---| +| Stored reflection embeddings are unit-norm (5/5 sampled, norm 1.0000) ⇒ cosine c ⇔ L2 dist² ≤ 2(1−c); floor 0.55 ⇒ dist² ≤ 0.9 | live DB probe 2026-07-23 | +| Hook = fresh process per firing ⇒ in-process breaker state dies each firing; needs file persistence | `hooks/contextual_inject.py` structure (main() per invocation) | +| Hook currently builds backend with `embedder=None`; payload carries `session_id`, `cwd`, `prompt`/`tool_name`+`tool_input`; mode gate `_enabled(event, cfg.context_inject_mode)` | `contextual_inject.py:62-116` | +| `SeenStore`: JSON file per session in `/state`, `{"turn": int, "seen": {"kind:id": turn}}`, `bump_turn/filter_unseen/mark_seen`, `prune_stale(7d)`; corrupt ⇒ empty | `services/context_seen.py` | +| `retrieve_relevant(backend, *, query, project, min_hits, max_items, include_neutral, now)` returns `RelevantMemory(kind,id,text,polarity,confidence,useful_count,age_days,hits,score)`; `format_relevant` renders block; `_activation` + `min_hits` are the parts being deleted | `services/relevant.py` (full read) | +| Semantic memories have NO FTS index (LIKE/trigram only) ⇒ no BM25 leg for semantics | schema + `services/semantic.py` | +| kNN template: `SELECT reflection_id, distance FROM reflection_embeddings WHERE embedding MATCH ? AND k = ? ORDER BY distance` (distance selectable) | `search/hybrid.py:274-296`, PR-A `_vec_ranks` | +| Bootstrap render: semantic full + reflection full slices by `bootstrap_top_n`, index lines, `_FOOTER`; exposures recorded only for full-rendered ids | `services/session_bootstrap.py:188-316` | +| Config pattern: dataclass field + `_resolve_*` in `get_config()`; existing keys `context_inject_mode`, `context_min_hits`, `context_max_items`, `context_reinject_turns`, `bootstrap_top_n` | `config.py:225-357` | +| Hook install shapes are golden-value-tested (`tests/cli/test_install_hooks.py`, `tests/e2e/test_install_hooks.py::EXPECTED_ENTRIES`, `tests/e2e/test_setup_sh.py`) — PreToolUse matcher change must update them | prior PR experience (#81) | +| `SyncEmbedder(factory, *, clock, cooldown, timeout)`; `_down_until` in-process; `embed_text/embed_batch` → value or None | `embeddings/sync_embed.py` | +| `wilson_lower_bound(positive, n)` in `services/scoring.py`; counters exposed on retrieve rows and `SemanticMemory` incl. `times_ignored` | PR-A | +| Exposure write is first-source-wins INSERT..WHERE NOT EXISTS in three places; `record_exposures(session_id, items, source)` on backend | #81 dedup + `storage/protocol.py` | +| Migration numbering: 0014 is the latest; THIS PR takes **0015** (PR-B renumbers to 0016) | `db/migrations/` | + +--- + +### Task 1: Config keys — inject mode + vec floor + +**Files:** +- Modify: `better_memory/config.py` (dataclass ~225-235, resolver ~348-357) +- Test: `tests/test_config.py` (append) + +**Interfaces:** +- Produces: `Config.inject_mode: Literal["deferred", "legacy"]` (env `BETTER_MEMORY_INJECT_MODE`, default `"legacy"`, unknown values coerce to `"legacy"`); `Config.context_vec_floor: float` (env `BETTER_MEMORY_CONTEXT_VEC_FLOOR`, default `0.55`, clamped to [0.0, 1.0], malformed ⇒ default). Consumed by Tasks 4, 5, 6. + +- [ ] **Step 1: Write the failing tests** (append to `tests/test_config.py`, mirroring its existing monkeypatch-env style — read the file's newest config test first and copy its fixture usage): + +```python +class TestInjectModeConfig: + def test_default_is_legacy(self, monkeypatch): + monkeypatch.delenv("BETTER_MEMORY_INJECT_MODE", raising=False) + assert get_config().inject_mode == "legacy" + + def test_deferred_selected(self, monkeypatch): + monkeypatch.setenv("BETTER_MEMORY_INJECT_MODE", "deferred") + assert get_config().inject_mode == "deferred" + + def test_unknown_coerces_to_legacy(self, monkeypatch): + monkeypatch.setenv("BETTER_MEMORY_INJECT_MODE", "yolo") + assert get_config().inject_mode == "legacy" + + +class TestVecFloorConfig: + def test_default(self, monkeypatch): + monkeypatch.delenv("BETTER_MEMORY_CONTEXT_VEC_FLOOR", raising=False) + assert get_config().context_vec_floor == 0.55 + + def test_override_and_clamp(self, monkeypatch): + monkeypatch.setenv("BETTER_MEMORY_CONTEXT_VEC_FLOOR", "0.7") + assert get_config().context_vec_floor == 0.7 + monkeypatch.setenv("BETTER_MEMORY_CONTEXT_VEC_FLOOR", "1.7") + assert get_config().context_vec_floor == 1.0 + + def test_malformed_falls_back(self, monkeypatch): + monkeypatch.setenv("BETTER_MEMORY_CONTEXT_VEC_FLOOR", "high") + assert get_config().context_vec_floor == 0.55 +``` + +(If `get_config` caches, mirror how existing tests bust the cache — check the file.) + +- [ ] **Step 2: Run to verify failure** + +Run: `./.venv/Scripts/python.exe -m pytest tests/test_config.py -v -k "InjectMode or VecFloor"` +Expected: FAIL — attributes missing. + +- [ ] **Step 3: Implement** — dataclass fields: + +```python + inject_mode: Literal["deferred", "legacy"] + context_vec_floor: float +``` + +Resolvers (place beside `_resolve_context_inject_mode`, following its shape): + +```python +def _resolve_inject_mode() -> Literal["deferred", "legacy"]: + raw = (os.environ.get("BETTER_MEMORY_INJECT_MODE") or "legacy").strip().lower() + # Fail-safe: anything unrecognised means today's behaviour. + return "deferred" if raw == "deferred" else "legacy" + + +def _resolve_vec_floor() -> float: + raw = os.environ.get("BETTER_MEMORY_CONTEXT_VEC_FLOOR") + if raw is None: + return 0.55 + try: + return min(1.0, max(0.0, float(raw))) + except ValueError: + return 0.55 +``` + +Wire both into the `get_config()` constructor call. + +- [ ] **Step 4: Run** `./.venv/Scripts/python.exe -m pytest tests/test_config.py -q` — all pass. + +- [ ] **Step 5: Commit** + +```bash +git add better_memory/config.py tests/test_config.py +git commit -m "feat(config): inject_mode + context_vec_floor keys + +Co-Authored-By: Claude Fable 5 " +``` + +--- + +### Task 2: Migration 0015 — via_exploration tag + write-path threading + +**Files:** +- Create: `better_memory/db/migrations/0015_via_exploration.sql` +- Modify: `better_memory/services/reflection.py` (bucket-fill from Task PR-A.4; exposure write ~1500s) +- Test: `tests/db/test_schema.py` (append), `tests/services/test_exploration_tagging.py` + +**Interfaces:** +- Produces: `session_memory_exposure.via_exploration INTEGER NOT NULL DEFAULT 0`; `retrieve_reflections` writes 1 for the exposure row of a memory that entered its bucket via the reserved slot. `_bucket_item` gains an `"_exploration": bool` transient key stripped before return (or a parallel id-set — see Step 3). Consumed by the harness metric (Task 10). + +- [ ] **Step 1: Write the failing tests** + +Migration test (append to `tests/db/test_schema.py`, same pattern as 0014's): + +```python +def test_0015_via_exploration_column(tmp_memory_db): + conn = connect(tmp_memory_db) + apply_migrations(conn) + cols = [r[1] for r in conn.execute("PRAGMA table_info(session_memory_exposure)")] + assert "via_exploration" in cols + conn.close() +``` + +Tagging tests: + +```python +# tests/services/test_exploration_tagging.py +"""Exploration-slot serves are tagged so the headline metric can exclude them. + +Exploration is an investment the ranker makes, not a relevance claim; +counting it in useful% punishes the system for learning (measured ~2-4pts +drag in the PR-A A/B). Rating flow is unchanged — explorers still rated. +""" +from __future__ import annotations + +from pathlib import Path + +import pytest + +from better_memory.db.connection import connect +from better_memory.db.schema import apply_migrations +from better_memory.services.reflection import ReflectionSynthesisService + + +@pytest.fixture +def conn(tmp_memory_db: Path): + c = connect(tmp_memory_db) + apply_migrations(c) + try: + yield c + finally: + c.close() + + +def _seed(conn, rid, *, useful=0, ignored=0): + conn.execute( + """INSERT INTO reflections + (id, title, project, phase, polarity, use_cases, hints, + confidence, created_at, updated_at, useful_count, times_ignored) + VALUES (?, ?, 'p', 'general', 'do', 'uc', '[]', 0.5, + '2026-01-01', '2026-01-01', ?, ?)""", + (rid, rid, useful, ignored), + ) + conn.commit() + + +def _flags(conn): + return { + r[0]: r[1] for r in conn.execute( + "SELECT memory_id, via_exploration FROM session_memory_exposure") + } + + +class TestExplorationTagging: + def test_slot_serve_tagged_others_not(self, conn, monkeypatch): + monkeypatch.setenv("CLAUDE_SESSION_ID", "s1") + for i in range(3): + _seed(conn, f"r-proven-{i}", useful=5, ignored=5) + _seed(conn, "r-untested") + svc = ReflectionSynthesisService(conn) + svc.retrieve_reflections(project="p", limit_per_bucket=3) + flags = _flags(conn) + assert flags["r-untested"] == 1 + assert flags["r-proven-0"] == 0 + + def test_dedup_wins_over_tag(self, conn, monkeypatch): + # Memory already exposed normally: a later exploration serve writes + # nothing, so the flag stays 0 (first-source-wins). + monkeypatch.setenv("CLAUDE_SESSION_ID", "s1") + _seed(conn, "r-x") + svc = ReflectionSynthesisService(conn) + svc.retrieve_reflections(project="p", limit_per_bucket=None) # normal serve + for i in range(3): + _seed(conn, f"r-proven-{i}", useful=5, ignored=5) + svc.retrieve_reflections(project="p", limit_per_bucket=3) # r-x now explorer + assert _flags(conn)["r-x"] == 0 + + def test_unlimited_cap_never_tags(self, conn, monkeypatch): + monkeypatch.setenv("CLAUDE_SESSION_ID", "s1") + _seed(conn, "r-untested") + svc = ReflectionSynthesisService(conn) + svc.retrieve_reflections(project="p", limit_per_bucket=None) + assert _flags(conn)["r-untested"] == 0 +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `./.venv/Scripts/python.exe -m pytest tests/db -q -k 0015 && ./.venv/Scripts/python.exe -m pytest tests/services/test_exploration_tagging.py -v` +Expected: FAIL — column missing. + +- [ ] **Step 3: Implement** + +Migration: + +```sql +-- Migration 0015: tag exposures that came from the exploration slot. +-- +-- The reserved per-bucket slot (spec 2026-07-23 retrieval-quality, section 2) +-- serves under-rated memories to earn them ratings. Those serves are an +-- investment the ranker makes, not a relevance claim, so the headline +-- usefulness metric excludes them. Ratings still apply to them normally. + +ALTER TABLE session_memory_exposure + ADD COLUMN via_exploration INTEGER NOT NULL DEFAULT 0; +``` + +In `reflection.py`'s two-pass bucket fill: collect the ids that took the +reserved slot into a local set (`exploration_ids: set[str]`) — the index +chosen from `untested_idx[0]` per bucket. Then in the exposure write for the +retrieve path, thread the flag (the write is the INSERT..WHERE NOT EXISTS +from #81): + +```python + self._conn.executemany( + "INSERT INTO session_memory_exposure " + "(session_id, memory_kind, memory_id, exposed_at, " + " source, via_exploration) " + "SELECT ?, 'reflection', ?, ?, 'retrieve', ? " + "WHERE NOT EXISTS (" + " SELECT 1 FROM session_memory_exposure " + " WHERE session_id = ? AND memory_kind = 'reflection' " + " AND memory_id = ?)", + [(sid, rid, now, 1 if rid in exploration_ids else 0, + sid, rid) for rid in all_ids], + ) +``` + +(`all_ids` already exists in that block. `exploration_ids` is empty when +`reserve` is False — the unlimited-cap case, pinning test 3.) + +- [ ] **Step 4: Run** `./.venv/Scripts/python.exe -m pytest tests/db tests/services/test_exploration_tagging.py tests/services/test_exposure_dedup.py tests/services/test_exploration_slot.py -q` — all pass. + +- [ ] **Step 5: Commit** + +```bash +git add -A +git commit -m "feat(db): tag exploration-slot exposures (migration 0015) + +Co-Authored-By: Claude Fable 5 " +``` + +--- + +### Task 3: File-persisted embed cooldown for hook processes + +**Files:** +- Modify: `better_memory/embeddings/sync_embed.py` +- Test: `tests/embeddings/test_sync_embed.py` (append) + +**Interfaces:** +- Produces: `SyncEmbedder(factory, *, clock=time.monotonic, cooldown=60.0, timeout=15.0, down_state_file: Path | None = None)`. When `down_state_file` is set: before any embed, if the file exists and holds a wall-clock epoch (`time.time()`) in the future ⇒ instant None; on failure, write `time.time() + cooldown` to it (best-effort). In-process `_down_until` behaviour unchanged. Consumed by Task 5 (hooks). + +**Why:** hooks run a fresh process per firing; the in-process breaker dies with the process, so an Ollama outage would stall EVERY prompt ~5s instead of once per cooldown window. Wall-clock (not monotonic) because the timestamp crosses processes. + +- [ ] **Step 1: Write the failing tests** (append): + +```python +class TestFilePersistedCooldown: + def test_failure_writes_down_file_and_second_instance_skips(self, tmp_path): + down = tmp_path / "embed_down_until" + fake = FakeEmbedder(fail=True) + s1 = SyncEmbedder(lambda: fake, down_state_file=down) + assert s1.embed_text("x") is None + assert down.exists() + # Fresh instance = fresh hook process. Must skip without touching + # the embedder. + fake2 = FakeEmbedder() + s2 = SyncEmbedder(lambda: fake2, down_state_file=down) + assert s2.embed_text("y") is None + assert fake2.calls == [] + + def test_expired_down_file_allows_embedding(self, tmp_path): + down = tmp_path / "embed_down_until" + down.write_text("1.0", encoding="utf-8") # epoch 1970 — expired + fake = FakeEmbedder() + s = SyncEmbedder(lambda: fake, down_state_file=down) + assert s.embed_text("x") is not None + + def test_corrupt_down_file_is_ignored(self, tmp_path): + down = tmp_path / "embed_down_until" + down.write_text("not-a-number", encoding="utf-8") + fake = FakeEmbedder() + s = SyncEmbedder(lambda: fake, down_state_file=down) + assert s.embed_text("x") is not None + + def test_no_file_param_keeps_old_behaviour(self): + fake = FakeEmbedder() + s = SyncEmbedder(lambda: fake) + assert s.embed_text("x") is not None +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `./.venv/Scripts/python.exe -m pytest tests/embeddings/test_sync_embed.py -v -k FilePersisted` +Expected: FAIL — unexpected keyword `down_state_file`. + +- [ ] **Step 3: Implement** — ctor stores `self._down_file = down_state_file`; in `_run`, after the in-process check: + +```python + if self._down_file is not None and self._file_down(): + return None +``` + +and in the exception handler, alongside setting `_down_until`: + +```python + self._write_down_file() +``` + +Helpers: + +```python + def _file_down(self) -> bool: + try: + until = float(self._down_file.read_text(encoding="utf-8").strip()) + except Exception: + return False + return time.time() < until + + def _write_down_file(self) -> None: + if self._down_file is None: + return + try: + self._down_file.parent.mkdir(parents=True, exist_ok=True) + self._down_file.write_text( + str(time.time() + self._cooldown), encoding="utf-8") + except Exception: + pass +``` + +(Wall-clock `time.time()` here deliberately, documented in the docstring; +the injected `clock` stays monotonic for the in-process window.) + +- [ ] **Step 4: Run** `./.venv/Scripts/python.exe -m pytest tests/embeddings -q` — all pass. + +- [ ] **Step 5: Commit** + +```bash +git add better_memory/embeddings/sync_embed.py tests/embeddings/test_sync_embed.py +git commit -m "feat(embeddings): file-persisted embed cooldown for per-process hooks + +Co-Authored-By: Claude Fable 5 " +``` + +--- + +### Task 4: relevant.py rewrite — three-leg scorer + evidence gate + +**Files:** +- Modify: `better_memory/services/relevant.py` (full rewrite of scoring; `RelevantMemory` + `format_relevant` keep their shapes) +- Delete tests: the `_activation`/min-hits-specific cases in `tests/services/test_relevant.py` (replace file content; `tests/services/test_relevant_format.py` stays untouched — format unchanged) +- Test: `tests/services/test_relevant.py` (rewritten) + +**Interfaces:** +- Consumes: `wilson_lower_bound` (scoring.py), `sanitize_fts5_query` (search/query.py), `sqlite_vec.serialize_float32`, `count_keyword_hits`/`extract_keywords` (keywords.py — kept for the fallback path), `SyncEmbedder` instance. +- Produces: + +```python +def retrieve_relevant( + backend: Any, + *, + query: str, + project: str, + conn: sqlite3.Connection | None = None, # sqlite FTS/vec access; None = fallback path + sync_embedder: Any = None, # SyncEmbedder | None + vec_floor: float = 0.55, + max_items: int = 3, + include_neutral: bool = False, + now: Callable[[], datetime] | None = None, +) -> list[RelevantMemory] +``` + +`RelevantMemory` unchanged except `hits` now holds BM25-or-fallback hit count (0 allowed when vec-qualified). `min_hits` parameter and `_activation` deleted. Consumed by Task 5. + +- [ ] **Step 1: Write the failing tests** (replace `tests/services/test_relevant.py`; read the old file first and keep any fixture helpers that seed reflections through the backend — the new tests construct a real sqlite backend the same way the old ones did; copy that setup): + +```python +# tests/services/test_relevant.py (rewritten for the three-leg gate) +"""Contextual relevance: BM25 + vec + Wilson prior behind an evidence gate. + +The gate is the point: a memory injects only with positive relevance +evidence (BM25 match on the query, or vec cosine >= floor). The Wilson +prior RANKS qualifiers but can never qualify a memory alone — popularity +must not force irrelevant injections (that failure mode measured 13% useful +as bootstrap). Vectors are unit-norm, so cosine >= c is L2 dist^2 <= 2(1-c). +""" +``` + +Test cases (write full bodies mirroring the old file's backend-construction +helper; each seeds via SQL like PR-A tests): + +1. `test_bm25_match_qualifies` — reflection titled "Retention archives by + confidence"; query "how does retention archive things"; no embedder; + returned. +2. `test_no_evidence_no_injection` — popular reflection (useful=50) with + unrelated title; query shares no tokens; no embedder ⇒ `[]` (Wilson + cannot qualify alone). +3. `test_vec_match_qualifies_without_token_overlap` — DirectedEmbedder + (import from `tests/services/_embedding_fakes.py`; extend that file with + the PR-A `DirectedEmbedder` if not already shared) maps query and target + title to the same unit vector; embedding stored for the reflection + (INSERT via `sqlite_vec.serialize_float32`); query wording shares no + tokens ⇒ returned. +4. `test_vec_below_floor_does_not_qualify` — embedder maps query and memory + to orthogonal unit vectors (cos 0) ⇒ `[]`. +5. `test_wilson_ranks_among_qualifiers` — two BM25-qualifying reflections, + one 5/6 useful-rated, one 0/6 ⇒ better hit-rate first. +6. `test_semantic_qualifies_via_vec` — semantic memory + matching directed + embedding in `semantic_embeddings` ⇒ returned with kind "semantic". +7. `test_semantic_fallback_keyword_when_no_embedder` — no embedder; + semantic content shares >=2 distinct keywords with query ⇒ returned + (keyword fallback keeps semantics reachable without Ollama). +8. `test_max_items_cap` — 5 qualifiers ⇒ 3 returned. +9. `test_no_conn_falls_back_to_keywords` — `conn=None` (agentcore shape): + reflections qualify via >=2 keyword hits, ranked by Wilson; no FTS/vec + used (no crash). +10. `test_embedder_failure_degrades_to_bm25` — FakeEmbedder(fail=True); + BM25-qualifying memory still returned. + +- [ ] **Step 2: Run to verify failure** + +Run: `./.venv/Scripts/python.exe -m pytest tests/services/test_relevant.py -v` +Expected: FAIL — new signature/behaviour absent. + +- [ ] **Step 3: Implement** + +Rewrite the scoring half of `relevant.py` (keep `RelevantMemory`, `_age_days`, +`format_relevant`, header/footer constants): + +```python +_FALLBACK_MIN_HITS = 2 # keyword evidence floor when FTS/vec are unavailable + + +def _bm25_qualifiers(conn, query: str) -> dict[str, int]: + """reflection_id -> BM25 rank (0 best) for reflections matching query.""" + sanitized = sanitize_fts5_query(query) + tokens = [t for t in sanitized.split() if len(t) > 2] + if not tokens or conn is None: + return {} + try: + rows = conn.execute( + "SELECT r.id, bm25(reflection_fts) AS bm " + "FROM reflection_fts JOIN reflections r ON r.rowid = reflection_fts.rowid " + "WHERE reflection_fts MATCH ? ORDER BY bm ASC", + (" OR ".join(tokens),), + ).fetchall() + except sqlite3.OperationalError: + return {} + return {row[0]: i for i, row in enumerate(rows)} + + +def _vec_qualifiers(conn, table: str, id_col: str, query_vector, vec_floor: float, + ) -> dict[str, int]: + """id -> vec rank for rows within the cosine floor (unit-norm vectors: + cosine >= c <=> L2 distance^2 <= 2*(1-c)).""" + if conn is None or query_vector is None: + return {} + max_dist_sq = 2.0 * (1.0 - vec_floor) + try: + rows = conn.execute( + f"SELECT {id_col}, distance FROM {table} " + f"WHERE embedding MATCH ? AND k = ? ORDER BY distance", + (sqlite_vec.serialize_float32(query_vector), 50), + ).fetchall() + except sqlite3.OperationalError: + return {} + out: dict[str, int] = {} + for row in rows: + if float(row[1]) ** 2 <= max_dist_sq or float(row[1]) <= max_dist_sq: + # sqlite-vec reports L2 distance; accept either dist or dist^2 + # convention defensively — see Step 3a calibration note. + out[row[0]] = len(out) + return out +``` + +**Step 3a (MANDATORY, do before finalising `_vec_qualifiers`):** probe which +convention sqlite-vec returns (distance vs squared distance) with a 3-line +script against two known unit vectors (`[1,0,...]` vs `[0,1,...]` ⇒ L2 +distance sqrt(2)≈1.414, squared 2.0). Keep ONLY the correct comparison and +delete the defensive double-check; add the probe result as a comment. + +Main function: fetch buckets + semantics through the backend exactly as the +old code did (`track_exposure=False`); compute: + +```python + bm = _bm25_qualifiers(conn, query) + qvec = sync_embedder.embed_text(query) if sync_embedder is not None else None + vec_r = _vec_qualifiers(conn, "reflection_embeddings", "reflection_id", + qvec, vec_floor) + vec_s = _vec_qualifiers(conn, "semantic_embeddings", "memory_id", + qvec, vec_floor) + keywords = extract_keywords(query) # fallback evidence only +``` + +Qualification per reflection row `r`: + +```python + fts_unavailable = conn is None + qualifies = (r_id in bm) or (r_id in vec_r) or ( + fts_unavailable and count_keyword_hits(text, keywords) >= _FALLBACK_MIN_HITS + ) +``` + +Semantic rows: `s_id in vec_s or (qvec is None and keyword hits >= _FALLBACK_MIN_HITS)` +(semantics have no FTS — the keyword fallback applies whenever the vec leg +is absent, not only when conn is None). + +Score for ranking (RRF over available legs + Wilson prior rank): + +```python + # Build rank lists among qualifiers, then RRF-fuse: + # prior_rank: order by wilson_lower_bound desc (reflections use + # useful+overlooked / +ignored counters; semantics likewise) + # bm rank, vec rank: from the dicts above (absent -> no term) + score = sum(1.0 / (60 + rank) for rank in present_ranks) +``` + +Sort `(-score, id)`, cap `max_items`. `hits` field = BM25 presence ? keyword +hit count for display : fallback hits (0 when vec-only). Delete +`_activation`; `min_hits` parameter gone. + +- [ ] **Step 4: Run** `./.venv/Scripts/python.exe -m pytest tests/services/test_relevant.py tests/services/test_relevant_format.py -q` — all pass. + +- [ ] **Step 5: Commit** + +```bash +git add -A +git commit -m "feat(contextual): three-leg evidence-gated relevance scorer + +Co-Authored-By: Claude Fable 5 " +``` + +--- + +### Task 5: Hook wiring — embedder, latch, matcher + +**Files:** +- Modify: `better_memory/hooks/contextual_inject.py` +- Modify: `better_memory/services/context_seen.py` (latch flag) +- Modify: `better_memory/cli/install_hooks.py` (PreToolUse matcher `Skill|Task|Write` → None/all-tools) +- Modify: `tests/cli/test_install_hooks.py`, `tests/e2e/test_install_hooks.py` (EXPECTED_ENTRIES), `tests/e2e/test_setup_sh.py` (golden shapes) +- Test: `tests/hooks/test_contextual_inject.py` (extend existing file — read it first, follow its payload-driven style), `tests/services/test_context_seen.py` (append latch tests) + +**Interfaces:** +- Consumes: Task 3's `down_state_file`, Task 4's signature, Task 1's config. +- Produces: `SeenStore.pretool_fired() -> bool` and `SeenStore.mark_pretool_fired() -> None` (persisted as `{"pretool_fired": true}` in the same JSON). Hook behaviour: PreToolUse events after the first per session exit immediately (before any DB/embedder work); embedder built as `SyncEmbedder(lambda: OllamaEmbedder(timeout=5.0, max_retries=1), down_state_file=cfg.home / "state" / "embed_down_until")` when `cfg.embeddings_backend == "ollama"` else None. + +- [ ] **Step 1: Write the failing tests** + +`test_context_seen.py` additions: + +```python +class TestPretoolLatch: + def test_defaults_false_then_persists(self, tmp_path): + s = SeenStore(tmp_path, "sess") + assert s.pretool_fired() is False + s.mark_pretool_fired() + assert SeenStore(tmp_path, "sess").pretool_fired() is True + + def test_corrupt_state_means_not_fired(self, tmp_path): + (tmp_path / "context_seen_sess.json").write_text("{", encoding="utf-8") + assert SeenStore(tmp_path, "sess").pretool_fired() is False +``` + +`test_contextual_inject.py` additions (mirror the file's existing +subprocess/payload harness — read first): +- PreToolUse fires once: two PreToolUse payloads same session ⇒ second emits + empty additionalContext and does not bump `contextual_fired_pretool` twice. +- UserPromptSubmit unaffected by the latch. + +- [ ] **Step 2: Run to verify failure** — latch methods missing. + +- [ ] **Step 3: Implement** + +`context_seen.py`: + +```python + def pretool_fired(self) -> bool: + return bool(self._data.get("pretool_fired")) + + def mark_pretool_fired(self) -> None: + self._data["pretool_fired"] = True + self._save() +``` + +`contextual_inject.py` in `main()`, after `seen = SeenStore(...)`: + +```python + if event == "PreToolUse": + if seen.pretool_fired(): + raise _SkipInjection() # module-local sentinel; caught + seen.mark_pretool_fired() # below, renders empty output +``` + +(Implement with a small module-level `class _SkipInjection(Exception)` caught +in the existing try to leave `rendered = ""` — do NOT use BaseException +paths for control flow; add an explicit `except _SkipInjection: rendered=""` +BEFORE the broad handler.) + +Embedder + new call: + +```python + sync_embedder = None + if cfg.embeddings_backend == "ollama": + from better_memory.embeddings.ollama import OllamaEmbedder + from better_memory.embeddings.sync_embed import SyncEmbedder + sync_embedder = SyncEmbedder( + lambda: OllamaEmbedder(timeout=5.0, max_retries=1), + down_state_file=cfg.home / "state" / "embed_down_until", + ) + items = retrieve_relevant( + backend, query=query, project=project, + conn=conn, + sync_embedder=sync_embedder, + vec_floor=cfg.context_vec_floor, + max_items=cfg.context_max_items, + ) +``` + +(imports move to module top per house style; `conn` is None in agentcore +mode already — matches Task 4's fallback contract. `cfg.context_min_hits` +config key stays for back-compat but is no longer read here; note it as +deprecated in config.py's comment.) + +`install_hooks.py`: PreToolUse HookSpec matcher `"Skill|Task|Write"` → `None` +(unscoped = all tools), with a comment: first-fire latch in the hook makes +an unscoped matcher cheap — one real firing per session, later events +short-circuit on the state file. Update the three golden-shape test files +accordingly (matcher None ⇒ no `matcher` key in the group, mirroring how +SessionStart/Stop entries assert). + +- [ ] **Step 4: Run** + +Run: `./.venv/Scripts/python.exe -m pytest tests/hooks tests/services/test_context_seen.py tests/cli/test_install_hooks.py tests/e2e/test_install_hooks.py tests/e2e/test_setup_sh.py -q` +Expected: all pass. + +- [ ] **Step 5: Commit** + +```bash +git add -A +git commit -m "feat(hooks): contextual inject gains vec leg, per-session PreToolUse latch, all-tools matcher + +Co-Authored-By: Claude Fable 5 " +``` + +--- + +### Task 6: Deferred bootstrap mode + +**Files:** +- Modify: `better_memory/services/session_bootstrap.py` (render path, ~188-316) +- Test: `tests/services/test_session_bootstrap.py` (append class) + +**Interfaces:** +- Consumes: `get_config().inject_mode` (Task 1). +- Produces: in `deferred` mode `bootstrap()` renders: header (unchanged), ALL general-scope semantic memories in full (existing `_render_semantic_full`), one index line `"better-memory knows {n_refl} reflections + {n_sem} semantic memories for this project; relevant ones will surface as you work - or ask via memory_retrieve with a task query."`, footer (unchanged). Exposures recorded ONLY for the general semantics rendered. Project-scoped semantics and ALL reflections: neither rendered nor exposed (counts still computed for the index line). `legacy` mode: byte-identical output to today. + +- [ ] **Step 1: Write the failing tests** (append; reuse the file's existing fixtures/seed helpers — read it first): + +```python +class TestDeferredBootstrap: + def test_deferred_renders_general_semantics_and_index_only(self, ..., monkeypatch): + monkeypatch.setenv("BETTER_MEMORY_INJECT_MODE", "deferred") + # seed: 2 general semantics, 3 project semantics, 4 reflections + out = + assert "" in out.context + assert "" not in out.context + assert "" not in out.context + assert "knows 4 reflections + 5 semantic memories" in out.context + + def test_deferred_exposes_only_general_semantics(self, ..., monkeypatch): + # session_memory_exposure rows: exactly the general-semantic ids, + # source='bootstrap' + + def test_legacy_mode_byte_identical(self, ..., monkeypatch): + # render once with flag unset, once with BETTER_MEMORY_INJECT_MODE=legacy, + # assert outputs equal; then with =deferred assert different +``` + +(Write full bodies against the file's real fixture names during +implementation — the assertions above are the contract; the seeding +boilerplate is copied from neighbouring tests in the same file.) + +- [ ] **Step 2: Run to verify failure.** + +- [ ] **Step 3: Implement** — in `bootstrap()`, after fetching semantics + +reflections (fetches stay — the index line needs counts), branch: + +```python + if get_config().inject_mode == "deferred": + general = [s for s in semantics if s.scope == "general"] + n_refl = sum(len(v) for v in refl_buckets.values()) + n_sem = len(semantics) + parts = [_HEADER..., *(_render_semantic_full(s) for s in general), + f"better-memory knows {n_refl} reflections + {n_sem} " + "semantic memories for this project; relevant ones will " + "surface as you work - or ask via memory_retrieve with " + "a task query.", + _FOOTER] + self._record_exposure(session_id=session_id, + reflection_ids=[], + semantic_ids=[s.id for s in general]) + return +``` + +(Adapt names to the function's real locals — the implementer reads the +function; the contract is fixed by the tests. The legacy path is NOT +touched — same lines, same order.) + +- [ ] **Step 4: Run** `./.venv/Scripts/python.exe -m pytest tests/services/test_session_bootstrap.py tests/hooks -q` — all pass. + +- [ ] **Step 5: Commit** + +```bash +git add -A +git commit -m "feat(bootstrap): deferred mode - general semantics + index line only + +Co-Authored-By: Claude Fable 5 " +``` + +--- + +### Task 7: CLAUDE.snippet rewrite + drift sentinel + +**Files:** +- Modify: `better_memory/skills/CLAUDE.snippet.md` (behavioural rewrite) +- Create: `better_memory/hooks/_claude_md_sentinel.py` +- Modify: `better_memory/hooks/session_bootstrap.py` (append sentinel warning to additionalContext, best-effort) +- Test: `tests/hooks/test_claude_md_sentinel.py` + +**Interfaces:** +- Produces: `check_claude_md(text: str, schemas: dict[str, set[str]]) -> list[str]` — pure function; `schemas` maps MCP-rendered tool name (e.g. `memory_retrieve`) to its valid property names; returns warning strings. `build_schemas() -> dict[str, set[str]]` derives it from `better_memory.mcp.tools.tool_definitions()` (wire name dots → underscores). Hook glue reads `~/.claude/CLAUDE.md` (path via `Path.home()`), appends at most one warning line. + +- [ ] **Step 1: Write the failing tests** + +```python +# tests/hooks/test_claude_md_sentinel.py +"""CLAUDE.md drift sentinel: prose that enumerates tool params rots. + +The 2026-07 incident: the user-level CLAUDE.md documented component/ +scope_path/window on memory_retrieve for weeks after they ceased to exist, +training every session to make silently-degraded calls. The rewrite removes +enumerations; the sentinel catches regressions. +""" +from __future__ import annotations + +from better_memory.hooks._claude_md_sentinel import build_schemas, check_claude_md + + +def test_phantom_param_detected(): + schemas = {"memory_retrieve": {"query", "project", "tech"}} + text = "call memory_retrieve with query and scope_path=src/" + warnings = check_claude_md(text, schemas) + assert warnings and "scope_path" in warnings[0] + + +def test_valid_params_silent(): + schemas = {"memory_retrieve": {"query", "project", "tech"}} + text = "call memory_retrieve with query='task' and project=x" + assert check_claude_md(text, schemas) == [] + + +def test_lines_without_tool_names_ignored(): + schemas = {"memory_retrieve": {"query"}} + assert check_claude_md("window=30d is a fine phrase alone", schemas) == [] + + +def test_common_words_not_flagged(): + schemas = {"memory_retrieve": {"query"}} + # 'e.g.' / 'i.e.' style tokens and words without = or : suffix don't count + assert check_claude_md("memory_retrieve is documented here", schemas) == [] + + +def test_build_schemas_covers_retrieve(): + schemas = build_schemas() + assert "query" in schemas["memory_retrieve"] + + +def test_malformed_input_never_raises(): + assert check_claude_md("", {}) == [] +``` + +- [ ] **Step 2: Run to verify failure** — module missing. + +- [ ] **Step 3: Implement** + +```python +# better_memory/hooks/_claude_md_sentinel.py +"""Detect parameter-enumeration drift in the user's CLAUDE.md. + +Pure functions; the session_bootstrap hook wires them in best-effort. +Only lines that mention a better-memory tool name are scanned, and only +tokens shaped like parameter usage (word= or word:) are checked against +the live schema, so prose can't false-positive. +""" + +from __future__ import annotations + +import re + +_PARAM_RE = re.compile(r"\b([a-z_][a-z0-9_]{2,})\s*[=:]") +_IGNORE = {"http", "https", "note", "example", "warning", "default"} + + +def build_schemas() -> dict[str, set[str]]: + from better_memory.mcp.tools import tool_definitions + out: dict[str, set[str]] = {} + for tool in tool_definitions(): + rendered = tool.name.replace(".", "_") + props = set((tool.inputSchema or {}).get("properties", {}).keys()) + out[rendered] = props + return out + + +def check_claude_md(text: str, schemas: dict[str, set[str]]) -> list[str]: + warnings: list[str] = [] + try: + for line in (text or "").splitlines(): + hit_tools = [name for name in schemas if name in line] + if not hit_tools: + continue + for token in _PARAM_RE.findall(line): + if token in _IGNORE or any(token in schemas[t] for t in hit_tools): + continue + if token in schemas: # a tool name followed by ':' etc. + continue + warnings.append( + f"CLAUDE.md documents parameter '{token}' near " + f"{'/'.join(hit_tools)} but the live tool schema has no " + "such parameter - fix or drop the enumeration.") + except Exception: + return [] + return warnings[:1] # at most one line of noise per session +``` + +Hook glue in `session_bootstrap.py` (inside the existing try, after the +context is built): read `Path.home() / ".claude" / "CLAUDE.md"` (missing → +skip), run `check_claude_md(text, build_schemas())`, append the single +warning to the additionalContext string. + +Snippet rewrite (`CLAUDE.snippet.md`): behavioural only. Core lines: +retrieval — "When you begin a task, call `memory_retrieve` with a `query` +describing it. Do not do broad no-query retrieval at session start; memories +surface contextually as you work."; knowledge — startup `knowledge_list` +mandate unchanged; recording — unchanged sections carried over minus every +parameter table/enumeration; credit — "credit with a one-line evidence +statement when a memory shapes your work" (forward-compatible with PR-B). + +- [ ] **Step 4: Run** `./.venv/Scripts/python.exe -m pytest tests/hooks -q` — all pass. + +- [ ] **Step 5: Commit** + +```bash +git add -A +git commit -m "feat(docs): behavioural CLAUDE.snippet + parameter-drift sentinel + +Co-Authored-By: Claude Fable 5 " +``` + +--- + +### Task 8: Vec-floor calibration + +**Files:** +- Create: `C:/Users/gethi/source/autoresearch/memuse-260721-run/calibrate_floor.py` (harness-side, not committed to repo) +- Possibly modify: `better_memory/config.py` default (only if calibration contradicts 0.55) + +**Interfaces:** consumes the A/B corpus (192 sandbox session DBs + their prompts in `tasks.json`) and live embeddings. + +- [ ] **Step 1:** Write `calibrate_floor.py`: for each (task prompt, memory rated cited/shaped in that task's sessions) pair → positive example; for each (prompt, memory rated ignored ≥3 times for that task) → negative. Embed prompts + memory texts via `OllamaEmbedder` (direct asyncio.run, batches). Report cosine distributions and the floor maximising F1, plus precision/recall at 0.45/0.50/0.55/0.60/0.65. +- [ ] **Step 2:** Run it. If best-F1 floor ∈ [0.50, 0.60], keep default 0.55 (within noise); else change the `config.py` default in a dedicated commit citing the numbers. +- [ ] **Step 3:** Paste the output table into the task report; note the chosen floor in the plan ledger. + +--- + +### Task 9: Website sync, pyright, full suite + +**Files:** `website/architecture.md`, `website/configuration.md`, `website/mcp-tools.md`, `website/index.md` (only stale paragraphs) + +- [ ] **Step 1:** `grep -rn "bootstrap\|inject\|BOOTSTRAP_TOP_N\|min_hits\|context_min" website/ | head -30`; update: bootstrap section describes both modes + flag (legacy default until gate); contextual section describes three-leg gate + floor + latch; configuration.md documents `BETTER_MEMORY_INJECT_MODE`, `BETTER_MEMORY_CONTEXT_VEC_FLOOR`, marks `BETTER_MEMORY_CONTEXT_MIN_HITS` deprecated. Semantic synonyms in the grep (lesson: "usefulness" escaped the last sweep — also grep "relevant\|keyword"). +- [ ] **Step 2:** `./.venv/Scripts/python.exe -m pyright` → 0 errors. +- [ ] **Step 3:** `./.venv/Scripts/python.exe -m pytest tests -q` → zero failures. +- [ ] **Step 4:** Commit `docs(website): deferred-injection prose + config keys`. + +--- + +### Task 10: Merge (legacy default), deploy, env-flag A/B gate, flip + +**Interfaces:** gate = deferred arm's headline useful% (via_exploration=0 denominator) not statistically below legacy arm's (one-sided z, α=0.05, 24 sessions/arm) AND deferred injection precision ≥ legacy bootstrap+contextual combined. + +- [ ] **Step 1:** Push, `gh pr create` (body: spec link, mode-flag rollout, sentinel, calibration numbers; footer `🤖 Generated with [Claude Code](https://claude.com/claude-code)`), babysit to squash-merge (checks green + threads resolved). +- [ ] **Step 2:** Deploy: pull main; restart note (migration 0015 applies on next server start). Live default stays `legacy` — no env var set anywhere yet. Apply the live `~/.claude/CLAUDE.md` edit now (mirror the snippet rewrite). +- [ ] **Step 3:** Harness updates (`memuse-260721-run/`): `metric.py` + `analyze.py` gain the three numbers (headline excl. exploration, all-in, exploration conversion rate — conversion = exploration-tagged exposures whose memory has a later non-ignored rating in any session). Arms `D-legacy` / `D-deferred` added to `runner.py`: identical `spec` (live MAIN_REPO code, sync stop, contextual on) differing ONLY in `spec["env"]["BETTER_MEMORY_INJECT_MODE"]`. Refresh sandbox base. 24 sessions per arm. +- [ ] **Step 4:** Gate: + +```bash +python analyze.py --arms D-legacy,D-deferred +python - <<'EOF' +import math +# fill from analyze headline columns: +u_l, n_l = USEFUL_LEGACY, EXPOSED_LEGACY_EXCL_EXPLORATION +u_d, n_d = USEFUL_DEFERRED, EXPOSED_DEFERRED_EXCL_EXPLORATION +p_l, p_d = u_l/n_l, u_d/n_d +p = (u_l+u_d)/(n_l+n_d) +z = (p_d-p_l)/math.sqrt(p*(1-p)*(1/n_l+1/n_d)) +print(f"legacy={p_l:.4f} deferred={p_d:.4f} z={z:.3f}") +print("GATE:", "PASS" if z > -1.645 else "FAIL (one re-run allowed)") +EOF +``` + +Plus the precision condition from analyze's per-source table (deferred contextual useful% vs legacy bootstrap+contextual pooled). + +- [ ] **Step 5:** PASS → flip live: add `"BETTER_MEMORY_INJECT_MODE": "deferred"` to the `env` of the better-memory server entry in `~/.claude.json` AND to the `env` block of `~/.claude/settings.json` (hooks read process env via the session env block — the runner arms prove this channel works). Restart note. FAIL twice → leave legacy live, report. +- [ ] **Step 6:** Ledger the outcome; schedule the legacy-path deletion PR after a stable week (note only). + +--- + +## Self-review notes + +- Spec §1-§4 all covered: flag+floor (T1), tagging+metric (T2, T10), breaker persistence gap discovered in spike (T3 — spec's latency promise required it), scorer+gate (T4), cadence+latch+matcher (T5), deferred bootstrap (T6), absorbed PR-C (T7), calibration (T8 — the spec's R1 mitigation), website guardrail (T9), rollout+gate+flip (T10). +- Two implementer-judgment points are contract-pinned rather than line-pinned (T6 render locals, T5 payload-harness style) with explicit read-first instructions — the files' internals shift too easily for line-level edits to survive. +- One in-plan probe (T4 Step 3a: sqlite-vec distance convention) deliberately left to implementation with an exact 3-line experiment — cheaper than pinning a possibly-wrong convention now. +- Migration number 0015 claimed here; PR-B's plan must renumber to 0016. From 60d420ea64aba1b56433e6074667c19af410e72d Mon Sep 17 00:00:00 2001 From: gethin Date: Thu, 23 Jul 2026 23:07:24 +0100 Subject: [PATCH 03/17] feat(config): inject_mode + context_vec_floor keys Co-Authored-By: Claude Fable 5 --- better_memory/config.py | 20 ++++++++++++++++++++ tests/test_config.py | 30 ++++++++++++++++++++++++++++++ 2 files changed, 50 insertions(+) diff --git a/better_memory/config.py b/better_memory/config.py index d069c6c..7d0b339 100644 --- a/better_memory/config.py +++ b/better_memory/config.py @@ -232,6 +232,8 @@ class Config: context_min_hits: int context_max_items: int context_reinject_turns: int + inject_mode: Literal["deferred", "legacy"] + context_vec_floor: float _DEFAULT_CONTEXT_INJECT_MODE = "both" @@ -260,6 +262,22 @@ def _resolve_embeddings_backend() -> Literal["ollama", "sqlite"]: return raw # type: ignore[return-value] +def _resolve_inject_mode() -> Literal["deferred", "legacy"]: + raw = (os.environ.get("BETTER_MEMORY_INJECT_MODE") or "legacy").strip().lower() + # Fail-safe: anything unrecognised means today's behaviour. + return "deferred" if raw == "deferred" else "legacy" + + +def _resolve_vec_floor() -> float: + raw = os.environ.get("BETTER_MEMORY_CONTEXT_VEC_FLOOR") + if raw is None: + return 0.55 + try: + return min(1.0, max(0.0, float(raw))) + except ValueError: + return 0.55 + + def _read_settings_storage_backend(home: Path) -> str | None: """Read ``storage_backend`` from ``/settings.json``. @@ -355,4 +373,6 @@ def get_config() -> Config: context_min_hits=_resolve_nonneg_int("BETTER_MEMORY_CONTEXT_MIN_HITS", 2), context_max_items=_resolve_nonneg_int("BETTER_MEMORY_CONTEXT_MAX_ITEMS", 3), context_reinject_turns=_resolve_nonneg_int("BETTER_MEMORY_CONTEXT_REINJECT_TURNS", 0), + inject_mode=_resolve_inject_mode(), + context_vec_floor=_resolve_vec_floor(), ) diff --git a/tests/test_config.py b/tests/test_config.py index 14e4b22..430ac1c 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -603,3 +603,33 @@ def test_injection_knobs_invalid_raises(monkeypatch): monkeypatch.setenv("BETTER_MEMORY_CONTEXT_MIN_HITS", "-1") with pytest.raises(ValueError): get_config() + + +class TestInjectModeConfig: + def test_default_is_legacy(self, monkeypatch): + monkeypatch.delenv("BETTER_MEMORY_INJECT_MODE", raising=False) + assert get_config().inject_mode == "legacy" + + def test_deferred_selected(self, monkeypatch): + monkeypatch.setenv("BETTER_MEMORY_INJECT_MODE", "deferred") + assert get_config().inject_mode == "deferred" + + def test_unknown_coerces_to_legacy(self, monkeypatch): + monkeypatch.setenv("BETTER_MEMORY_INJECT_MODE", "yolo") + assert get_config().inject_mode == "legacy" + + +class TestVecFloorConfig: + def test_default(self, monkeypatch): + monkeypatch.delenv("BETTER_MEMORY_CONTEXT_VEC_FLOOR", raising=False) + assert get_config().context_vec_floor == 0.55 + + def test_override_and_clamp(self, monkeypatch): + monkeypatch.setenv("BETTER_MEMORY_CONTEXT_VEC_FLOOR", "0.7") + assert get_config().context_vec_floor == 0.7 + monkeypatch.setenv("BETTER_MEMORY_CONTEXT_VEC_FLOOR", "1.7") + assert get_config().context_vec_floor == 1.0 + + def test_malformed_falls_back(self, monkeypatch): + monkeypatch.setenv("BETTER_MEMORY_CONTEXT_VEC_FLOOR", "high") + assert get_config().context_vec_floor == 0.55 From 6174345e0d7a5b7e9f92f4ea682f02e8c714b5dd Mon Sep 17 00:00:00 2001 From: gethin Date: Thu, 23 Jul 2026 23:17:56 +0100 Subject: [PATCH 04/17] feat(db): tag exploration-slot exposures (migration 0015) Co-Authored-By: Claude Fable 5 --- .../db/migrations/0015_via_exploration.sql | 9 +++ better_memory/services/reflection.py | 10 ++- tests/db/test_migration_0009.py | 1 + tests/db/test_migration_0010.py | 1 + ...test_migration_0012_contextual_exposure.py | 1 + tests/db/test_schema.py | 8 ++ tests/services/test_exploration_tagging.py | 76 +++++++++++++++++++ 7 files changed, 103 insertions(+), 3 deletions(-) create mode 100644 better_memory/db/migrations/0015_via_exploration.sql create mode 100644 tests/services/test_exploration_tagging.py diff --git a/better_memory/db/migrations/0015_via_exploration.sql b/better_memory/db/migrations/0015_via_exploration.sql new file mode 100644 index 0000000..de8f320 --- /dev/null +++ b/better_memory/db/migrations/0015_via_exploration.sql @@ -0,0 +1,9 @@ +-- Migration 0015: tag exposures that came from the exploration slot. +-- +-- The reserved per-bucket slot (spec 2026-07-23 retrieval-quality, section 2) +-- serves under-rated memories to earn them ratings. Those serves are an +-- investment the ranker makes, not a relevance claim, so the headline +-- usefulness metric excludes them. Ratings still apply to them normally. + +ALTER TABLE session_memory_exposure + ADD COLUMN via_exploration INTEGER NOT NULL DEFAULT 0; diff --git a/better_memory/services/reflection.py b/better_memory/services/reflection.py index 26c7392..05a98f5 100644 --- a/better_memory/services/reflection.py +++ b/better_memory/services/reflection.py @@ -1524,6 +1524,7 @@ def retrieve_reflections( reserve = limit_per_bucket is not None and cap >= 2 buckets: dict[str, list[dict]] = {"do": [], "dont": [], "neutral": []} by_polarity: dict[str, list] = {"do": [], "dont": [], "neutral": []} + exploration_ids: set[str] = set() for r in rows: by_polarity[r["polarity"]].append(r) for polarity, group in by_polarity.items(): @@ -1537,6 +1538,7 @@ def retrieve_reflections( chosen = tested_idx[: cap - 1] if untested_idx: chosen.append(untested_idx[0]) + exploration_ids.add(group[untested_idx[0]]["id"]) if len(chosen) < cap: # top up from the remainder taken = set(chosen) for i in range(len(group)): @@ -1586,13 +1588,15 @@ def retrieve_reflections( # re-serves must not add rows. self._conn.executemany( "INSERT INTO session_memory_exposure " - "(session_id, memory_kind, memory_id, exposed_at, source) " - "SELECT ?, 'reflection', ?, ?, 'retrieve' " + "(session_id, memory_kind, memory_id, exposed_at, " + " source, via_exploration) " + "SELECT ?, 'reflection', ?, ?, 'retrieve', ? " "WHERE NOT EXISTS (" " SELECT 1 FROM session_memory_exposure " " WHERE session_id = ? AND memory_kind = 'reflection' " " AND memory_id = ?)", - [(sid, rid, now, sid, rid) for rid in all_ids], + [(sid, rid, now, 1 if rid in exploration_ids else 0, + sid, rid) for rid in all_ids], ) _diag.step(fn, "exposure_commit") self._conn.commit() diff --git a/tests/db/test_migration_0009.py b/tests/db/test_migration_0009.py index 9901ba3..1dcec5e 100644 --- a/tests/db/test_migration_0009.py +++ b/tests/db/test_migration_0009.py @@ -29,6 +29,7 @@ def test_table_created_with_expected_columns(self, conn): assert cols == { "session_id", "memory_kind", "memory_id", "exposed_at", "source", "rated_at", "classification", + "via_exploration", } def test_primary_key_includes_exposed_at(self, conn): diff --git a/tests/db/test_migration_0010.py b/tests/db/test_migration_0010.py index eb7717f..5d3f4a1 100644 --- a/tests/db/test_migration_0010.py +++ b/tests/db/test_migration_0010.py @@ -59,6 +59,7 @@ def test_exposure_table_columns_preserved(self, conn): assert cols == { "session_id", "memory_kind", "memory_id", "exposed_at", "source", "rated_at", "classification", + "via_exploration", } def test_primary_key_preserved(self, conn): diff --git a/tests/db/test_migration_0012_contextual_exposure.py b/tests/db/test_migration_0012_contextual_exposure.py index 76b0718..7c1d153 100644 --- a/tests/db/test_migration_0012_contextual_exposure.py +++ b/tests/db/test_migration_0012_contextual_exposure.py @@ -63,6 +63,7 @@ def test_exposure_table_columns_preserved(self, conn): assert cols == { "session_id", "memory_kind", "memory_id", "exposed_at", "source", "rated_at", "classification", + "via_exploration", } def test_primary_key_preserved(self, conn): diff --git a/tests/db/test_schema.py b/tests/db/test_schema.py index e9c1ad5..bd6abb3 100644 --- a/tests/db/test_schema.py +++ b/tests/db/test_schema.py @@ -799,3 +799,11 @@ def test_0014_semantic_embeddings_table(tmp_memory_db: Path) -> None: assert row is not None finally: conn.close() + + +def test_0015_via_exploration_column(tmp_memory_db): + conn = connect(tmp_memory_db) + apply_migrations(conn) + cols = [r[1] for r in conn.execute("PRAGMA table_info(session_memory_exposure)")] + assert "via_exploration" in cols + conn.close() diff --git a/tests/services/test_exploration_tagging.py b/tests/services/test_exploration_tagging.py new file mode 100644 index 0000000..a9ad742 --- /dev/null +++ b/tests/services/test_exploration_tagging.py @@ -0,0 +1,76 @@ +"""Exploration-slot serves are tagged so the headline metric can exclude them. + +Exploration is an investment the ranker makes, not a relevance claim; +counting it in useful% punishes the system for learning (measured ~2-4pts +drag in the PR-A A/B). Rating flow is unchanged — explorers still rated. +""" +from __future__ import annotations + +from pathlib import Path + +import pytest + +from better_memory.db.connection import connect +from better_memory.db.schema import apply_migrations +from better_memory.services.reflection import ReflectionSynthesisService + + +@pytest.fixture +def conn(tmp_memory_db: Path): + c = connect(tmp_memory_db) + apply_migrations(c) + try: + yield c + finally: + c.close() + + +def _seed(conn, rid, *, useful=0, ignored=0): + conn.execute( + """INSERT INTO reflections + (id, title, project, phase, polarity, use_cases, hints, + confidence, created_at, updated_at, useful_count, times_ignored) + VALUES (?, ?, 'p', 'general', 'do', 'uc', '[]', 0.5, + '2026-01-01', '2026-01-01', ?, ?)""", + (rid, rid, useful, ignored), + ) + conn.commit() + + +def _flags(conn): + return { + r[0]: r[1] for r in conn.execute( + "SELECT memory_id, via_exploration FROM session_memory_exposure") + } + + +class TestExplorationTagging: + def test_slot_serve_tagged_others_not(self, conn, monkeypatch): + monkeypatch.setenv("CLAUDE_SESSION_ID", "s1") + for i in range(3): + _seed(conn, f"r-proven-{i}", useful=5, ignored=5) + _seed(conn, "r-untested") + svc = ReflectionSynthesisService(conn) + svc.retrieve_reflections(project="p", limit_per_bucket=3) + flags = _flags(conn) + assert flags["r-untested"] == 1 + assert flags["r-proven-0"] == 0 + + def test_dedup_wins_over_tag(self, conn, monkeypatch): + # Memory already exposed normally: a later exploration serve writes + # nothing, so the flag stays 0 (first-source-wins). + monkeypatch.setenv("CLAUDE_SESSION_ID", "s1") + _seed(conn, "r-x") + svc = ReflectionSynthesisService(conn) + svc.retrieve_reflections(project="p", limit_per_bucket=None) # normal serve + for i in range(3): + _seed(conn, f"r-proven-{i}", useful=5, ignored=5) + svc.retrieve_reflections(project="p", limit_per_bucket=3) # r-x now explorer + assert _flags(conn)["r-x"] == 0 + + def test_unlimited_cap_never_tags(self, conn, monkeypatch): + monkeypatch.setenv("CLAUDE_SESSION_ID", "s1") + _seed(conn, "r-untested") + svc = ReflectionSynthesisService(conn) + svc.retrieve_reflections(project="p", limit_per_bucket=None) + assert _flags(conn)["r-untested"] == 0 From f932dff2838e42de3c1ab33e27fa4f3074fcf26a Mon Sep 17 00:00:00 2001 From: gethin Date: Thu, 23 Jul 2026 23:22:39 +0100 Subject: [PATCH 05/17] feat(embeddings): file-persisted embed cooldown for per-process hooks Co-Authored-By: Claude Fable 5 --- better_memory/embeddings/sync_embed.py | 23 +++++++++++++++++ tests/embeddings/test_sync_embed.py | 34 ++++++++++++++++++++++++++ 2 files changed, 57 insertions(+) diff --git a/better_memory/embeddings/sync_embed.py b/better_memory/embeddings/sync_embed.py index f259536..052cbc5 100644 --- a/better_memory/embeddings/sync_embed.py +++ b/better_memory/embeddings/sync_embed.py @@ -19,6 +19,7 @@ import time from collections.abc import Callable +from pathlib import Path from typing import Any from better_memory.async_bridge import run_async_in_worker @@ -38,12 +39,14 @@ def __init__( clock: Callable[[], float] = time.monotonic, cooldown: float = _DEFAULT_COOLDOWN, timeout: float = _WORKER_TIMEOUT, + down_state_file: Path | None = None, ) -> None: self._factory = factory self._clock = clock self._cooldown = cooldown self._timeout = timeout self._down_until = 0.0 + self._down_file = down_state_file def embed_text(self, text: str) -> list[float] | None: return self._run(lambda emb: emb.embed(text)) @@ -56,6 +59,8 @@ def _run(self, op: Callable[[Any], Any]): return None if self._clock() < self._down_until: return None + if self._down_file is not None and self._file_down(): + return None factory = self._factory @@ -72,4 +77,22 @@ async def _go(): return run_async_in_worker(_go, timeout=self._timeout) except Exception: self._down_until = self._clock() + self._cooldown + self._write_down_file() return None + + def _file_down(self) -> bool: + try: + until = float(self._down_file.read_text(encoding="utf-8").strip()) + except Exception: + return False + return time.time() < until + + def _write_down_file(self) -> None: + if self._down_file is None: + return + try: + self._down_file.parent.mkdir(parents=True, exist_ok=True) + self._down_file.write_text( + str(time.time() + self._cooldown), encoding="utf-8") + except Exception: + pass diff --git a/tests/embeddings/test_sync_embed.py b/tests/embeddings/test_sync_embed.py index 20ae985..f3a89f7 100644 --- a/tests/embeddings/test_sync_embed.py +++ b/tests/embeddings/test_sync_embed.py @@ -72,3 +72,37 @@ async def embed(self, text): s = SyncEmbedder(Bare) assert s.embed_text("x") is not None + + +class TestFilePersistedCooldown: + def test_failure_writes_down_file_and_second_instance_skips(self, tmp_path): + down = tmp_path / "embed_down_until" + fake = FakeEmbedder(fail=True) + s1 = SyncEmbedder(lambda: fake, down_state_file=down) + assert s1.embed_text("x") is None + assert down.exists() + # Fresh instance = fresh hook process. Must skip without touching + # the embedder. + fake2 = FakeEmbedder() + s2 = SyncEmbedder(lambda: fake2, down_state_file=down) + assert s2.embed_text("y") is None + assert fake2.calls == [] + + def test_expired_down_file_allows_embedding(self, tmp_path): + down = tmp_path / "embed_down_until" + down.write_text("1.0", encoding="utf-8") # epoch 1970 — expired + fake = FakeEmbedder() + s = SyncEmbedder(lambda: fake, down_state_file=down) + assert s.embed_text("x") is not None + + def test_corrupt_down_file_is_ignored(self, tmp_path): + down = tmp_path / "embed_down_until" + down.write_text("not-a-number", encoding="utf-8") + fake = FakeEmbedder() + s = SyncEmbedder(lambda: fake, down_state_file=down) + assert s.embed_text("x") is not None + + def test_no_file_param_keeps_old_behaviour(self): + fake = FakeEmbedder() + s = SyncEmbedder(lambda: fake) + assert s.embed_text("x") is not None From 05a2ee00e6abeafc2a311093105dd7f3f07487ac Mon Sep 17 00:00:00 2001 From: gethin Date: Thu, 23 Jul 2026 23:24:59 +0100 Subject: [PATCH 06/17] fix(embeddings): narrow Optional down_state_file for pyright Co-Authored-By: Claude Fable 5 --- better_memory/embeddings/sync_embed.py | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/better_memory/embeddings/sync_embed.py b/better_memory/embeddings/sync_embed.py index 052cbc5..d0eb736 100644 --- a/better_memory/embeddings/sync_embed.py +++ b/better_memory/embeddings/sync_embed.py @@ -81,18 +81,22 @@ async def _go(): return None def _file_down(self) -> bool: + down = self._down_file + if down is None: + return False try: - until = float(self._down_file.read_text(encoding="utf-8").strip()) + until = float(down.read_text(encoding="utf-8").strip()) except Exception: return False return time.time() < until def _write_down_file(self) -> None: - if self._down_file is None: + down = self._down_file + if down is None: return try: - self._down_file.parent.mkdir(parents=True, exist_ok=True) - self._down_file.write_text( + down.parent.mkdir(parents=True, exist_ok=True) + down.write_text( str(time.time() + self._cooldown), encoding="utf-8") except Exception: pass From c1459b71c21b99ca89b5e968a6f30acd8cd05943 Mon Sep 17 00:00:00 2001 From: gethin Date: Thu, 23 Jul 2026 23:34:46 +0100 Subject: [PATCH 07/17] feat(contextual): three-leg evidence-gated relevance scorer Co-Authored-By: Claude Fable 5 --- better_memory/services/relevant.py | 265 +++++++++++++++++------ tests/services/_embedding_fakes.py | 27 +++ tests/services/test_relevant.py | 325 +++++++++++++++++++---------- 3 files changed, 446 insertions(+), 171 deletions(-) diff --git a/better_memory/services/relevant.py b/better_memory/services/relevant.py index c392152..453f0e9 100644 --- a/better_memory/services/relevant.py +++ b/better_memory/services/relevant.py @@ -1,22 +1,49 @@ """Relevance filter over the curated memory set (semantic + reflections). -Fetches the small, already-ranked sets through the StorageBackend abstraction -(works on sqlite AND agentcore) and scores them against a query using -hits x activation: distinct whole-word keyword hits (title hits count double) -multiplied by an activation factor built from useful_count and confidence, -halved when a memory has misled more often than it has helped. Items below -a min-hits floor are dropped; the remainder is ranked by score and capped -to max_items. Pure-Python; no embeddings, no new schema. +Three-leg evidence-gated scorer, replacing the old pure-keyword +hits-x-activation model: a memory injects only when it has positive +relevance EVIDENCE -- + +- BM25 match against ``reflection_fts`` (title / use_cases / hints), or +- vector cosine similarity >= ``vec_floor`` against its embedding, or +- (only when the vec/FTS legs are structurally unavailable -- no sqlite + ``conn``, or no embedder for semantics) a keyword-hit floor as a + degraded fallback. + +The Wilson lower-bound prior (see ``services.scoring``) never qualifies a +memory by itself -- it only RANKS among qualifiers via reciprocal rank +fusion (RRF), alongside the BM25 and vector ranks. Popularity forcing +irrelevant injections was the old failure mode (13% useful as bootstrap); +the gate exists specifically to close it. + +Fetches the small, already-ranked sets through the StorageBackend +abstraction (works on sqlite AND agentcore); the BM25/vec legs additionally +require a raw sqlite ``conn`` (agentcore passes ``conn=None`` and degrades +to the keyword fallback for reflections). """ from __future__ import annotations import math +import sqlite3 from collections.abc import Callable from dataclasses import dataclass from datetime import UTC, datetime from typing import Any +import sqlite_vec + +from better_memory.search.query import sanitize_fts5_query from better_memory.services.keywords import count_keyword_hits, extract_keywords +from better_memory.services.scoring import wilson_lower_bound + +#: Keyword-hit floor used only when the FTS/vec legs are structurally +#: unavailable (no sqlite conn for reflections; no query vector for +#: semantics, which have no FTS substrate at all). +_FALLBACK_MIN_HITS = 2 + +#: Reciprocal rank fusion constant, matching search/hybrid.py and +#: ReflectionSynthesisService._fuse_by_relevance. +_RRF_K = 60 @dataclass @@ -44,13 +71,88 @@ def _age_days(iso_ts: str | None, now: datetime) -> int | None: return max(0, (now - ts).days) -def _activation(*, useful_count: int, times_misled: int, confidence: float | None) -> float: - act = (1.0 + 0.2 * math.log1p(max(0, useful_count))) - if confidence is not None: - act *= max(0.1, float(confidence)) - if times_misled > useful_count: - act *= 0.5 - return act +def _bm25_qualifiers(conn: sqlite3.Connection | None, query: str) -> dict[str, int]: + """reflection_id -> BM25 rank (0 best) for reflections matching query.""" + sanitized = sanitize_fts5_query(query) + tokens = [t for t in sanitized.split() if len(t) > 2] + if not tokens or conn is None: + return {} + try: + rows = conn.execute( + "SELECT r.id, bm25(reflection_fts) AS bm " + "FROM reflection_fts JOIN reflections r ON r.rowid = reflection_fts.rowid " + "WHERE reflection_fts MATCH ? ORDER BY bm ASC", + (" OR ".join(tokens),), + ).fetchall() + except sqlite3.OperationalError: + return {} + return {row[0]: i for i, row in enumerate(rows)} + + +def _vec_qualifiers( + conn: sqlite3.Connection | None, + table: str, + id_col: str, + query_vector: list[float] | None, + vec_floor: float, +) -> dict[str, int]: + """id -> vec rank for rows within the cosine floor. + + Vectors are unit-norm, so cosine >= c <=> L2 distance <= sqrt(2*(1-c)). + + Step 3a probe (2026-07-23, sqlite-vec as vendored in this repo's + .venv): a vec0 kNN query for [1,0,0,0] against a stored [0,1,0,0] row + returned ``distance=1.4142135381698608`` -- sqrt(2) at float32 + precision, not 2.0. Confirms sqlite-vec's ``distance`` column is plain + L2 distance, NOT squared L2. The floor comparison below is against the + plain (unsquared) distance only; the squared-distance branch that a + defensive dual-check would need is dead code and has been omitted. + """ + if conn is None or query_vector is None: + return {} + max_dist = math.sqrt(2.0 * (1.0 - vec_floor)) + try: + rows = conn.execute( + f"SELECT {id_col}, distance FROM {table} " + f"WHERE embedding MATCH ? AND k = ? ORDER BY distance", + (sqlite_vec.serialize_float32(query_vector), 50), + ).fetchall() + except sqlite3.OperationalError: + return {} + out: dict[str, int] = {} + for row in rows: + if float(row[1]) <= max_dist: + out[row[0]] = len(out) + return out + + +def _wilson_for(useful: int, overlooked: int, ignored: int) -> float: + positive = useful + overlooked + n = useful + overlooked + ignored + return wilson_lower_bound(positive, n) + + +def _rrf_score(candidates: list[dict]) -> list[tuple[float, dict]]: + """RRF-fuse the Wilson prior with the BM25/vec ranks already stashed + on each candidate dict (``bm_rank`` / ``vec_rank``, ``None`` if absent). + + The prior rank is computed fresh here (desc by Wilson score) rather + than carried in, since it only makes sense relative to the other + qualifiers in this same candidate set. + """ + order_by_wilson = sorted(range(len(candidates)), key=lambda i: -candidates[i]["wilson"]) + prior_rank = {candidates[i]["id"]: rank for rank, i in enumerate(order_by_wilson)} + + scored: list[tuple[float, dict]] = [] + for c in candidates: + present_ranks = [prior_rank[c["id"]]] + if c["bm_rank"] is not None: + present_ranks.append(c["bm_rank"]) + if c["vec_rank"] is not None: + present_ranks.append(c["vec_rank"]) + score = sum(1.0 / (_RRF_K + rank) for rank in present_ranks) + scored.append((score, c)) + return scored def retrieve_relevant( @@ -58,76 +160,119 @@ def retrieve_relevant( *, query: str, project: str, - min_hits: int = 2, + conn: sqlite3.Connection | None = None, + sync_embedder: Any = None, + vec_floor: float = 0.55, max_items: int = 3, include_neutral: bool = False, now: Callable[[], datetime] | None = None, ) -> list[RelevantMemory]: - """Score curated memories against the query; return top max_items whose - distinct-keyword hits >= min_hits, ordered by score desc. Never raises.""" - keywords = extract_keywords(query) - if not keywords: - return [] - _now = (now or (lambda: datetime.now(UTC)))() + """Gate + rank curated memories (semantic + reflections) for ``query``. - out: list[RelevantMemory] = [] + A memory is returned only if it clears the evidence gate: a BM25 match, + a vector cosine >= ``vec_floor``, or (only when that leg is structurally + unavailable) a keyword-hit fallback. Among qualifiers, ranking is RRF + over the Wilson prior plus whichever of BM25/vec ranks are present. + Never raises -- any backend/leg failure degrades that leg to "absent" + rather than propagating. + """ + _now = (now or (lambda: datetime.now(UTC)))() try: buckets = backend.retrieve(project=project, track_exposure=False) except Exception: # noqa: BLE001 - degrade to no reflections buckets = {} + try: + semantic = backend.semantic_list(project=project, track_exposure=False) + except Exception: # noqa: BLE001 - degrade to no semantic + semantic = [] + + bm = _bm25_qualifiers(conn, query) + qvec = sync_embedder.embed_text(query) if sync_embedder is not None else None + vec_r = _vec_qualifiers(conn, "reflection_embeddings", "reflection_id", qvec, vec_floor) + vec_s = _vec_qualifiers(conn, "semantic_embeddings", "memory_id", qvec, vec_floor) + keywords = extract_keywords(query) # fallback evidence only + + fts_unavailable = conn is None + + refl_candidates: list[dict] = [] order = ["do", "dont"] + (["neutral"] if include_neutral else []) for bucket in order: for r in buckets.get(bucket, []) or []: + r_id = str(r.get("id")) title = str(r.get("title") or "") body = " ".join( [str(r.get("use_cases") or "")] + [str(h) for h in (r.get("hints") or [])] ) - title_hits = count_keyword_hits(title, keywords) - total_hits = count_keyword_hits(f"{title} {body}", keywords) - if total_hits < min_hits: + text = f"{title} {body}" + kw_hits = count_keyword_hits(text, keywords) + + in_bm = r_id in bm + in_vec = r_id in vec_r + fallback_ok = fts_unavailable and kw_hits >= _FALLBACK_MIN_HITS + if not (in_bm or in_vec or fallback_ok): continue - act = _activation( - useful_count=int(r.get("useful_count") or 0), - times_misled=int(r.get("times_misled") or 0), - confidence=r.get("confidence"), - ) - score = (total_hits + title_hits) * act # title hits count double - out.append(RelevantMemory( - kind="reflection", id=str(r.get("id")), - text=f"{title}: {body}".strip(": "), - polarity=bucket if bucket in ("do", "dont") else None, - confidence=r.get("confidence"), - useful_count=int(r.get("useful_count") or 0), - age_days=_age_days(r.get("updated_at"), _now), - hits=total_hits, score=score, - )) - try: - semantic = backend.semantic_list(project=project, track_exposure=False) - except Exception: # noqa: BLE001 - degrade to no semantic - semantic = [] + refl_candidates.append({ + "id": r_id, "kind": "reflection", + "polarity": bucket if bucket in ("do", "dont") else None, + "text": f"{title}: {body}".strip(": "), + "confidence": r.get("confidence"), + "useful_count": int(r.get("useful_count") or 0), + "age_days": _age_days(r.get("updated_at"), _now), + "hits": kw_hits if (in_bm or fallback_ok) else 0, + "bm_rank": bm.get(r_id), + "vec_rank": vec_r.get(r_id), + "wilson": _wilson_for( + int(r.get("useful_count") or 0), + int(r.get("times_overlooked") or 0), + int(r.get("times_ignored") or 0), + ), + }) + + sem_candidates: list[dict] = [] for s in semantic or []: + s_id = str(getattr(s, "id", "")) content = getattr(s, "content", "") or "" - hits = count_keyword_hits(content, keywords) - if hits < min_hits: + kw_hits = count_keyword_hits(content, keywords) + + in_vec = s_id in vec_s + # Semantics have no FTS/BM25 leg at all, so the keyword fallback + # applies whenever the vec leg itself is absent (no embedder / + # embed failure), not only when conn is None. + fallback_ok = qvec is None and kw_hits >= _FALLBACK_MIN_HITS + if not (in_vec or fallback_ok): continue - act = _activation( - useful_count=int(getattr(s, "useful_count", 0) or 0), - times_misled=int(getattr(s, "times_misled", 0) or 0), - confidence=None, + + sem_candidates.append({ + "id": s_id, "kind": "semantic", "polarity": None, + "text": content, + "confidence": None, + "useful_count": int(getattr(s, "useful_count", 0) or 0), + "age_days": _age_days(getattr(s, "updated_at", None), _now), + "hits": kw_hits if fallback_ok else 0, + "bm_rank": None, + "vec_rank": vec_s.get(s_id), + "wilson": _wilson_for( + int(getattr(s, "useful_count", 0) or 0), + int(getattr(s, "times_overlooked", 0) or 0), + int(getattr(s, "times_ignored", 0) or 0), + ), + }) + + all_scored = _rrf_score(refl_candidates) + _rrf_score(sem_candidates) + all_scored.sort(key=lambda t: (-t[0], t[1]["id"])) + + out = [ + RelevantMemory( + kind=c["kind"], id=c["id"], text=c["text"], polarity=c["polarity"], + confidence=c["confidence"], useful_count=c["useful_count"], + age_days=c["age_days"], hits=c["hits"], score=score, ) - out.append(RelevantMemory( - kind="semantic", id=str(getattr(s, "id", "")), - text=content, polarity=None, confidence=None, - useful_count=int(getattr(s, "useful_count", 0) or 0), - age_days=_age_days(getattr(s, "updated_at", None), _now), - hits=hits, score=hits * act, - )) - - out.sort(key=lambda m: (-m.score, m.id)) - return out[:max_items] + for score, c in all_scored[:max_items] + ] + return out _TEXT_MAX_CHARS = 400 diff --git a/tests/services/_embedding_fakes.py b/tests/services/_embedding_fakes.py index d9258b2..227220e 100644 --- a/tests/services/_embedding_fakes.py +++ b/tests/services/_embedding_fakes.py @@ -22,3 +22,30 @@ async def embed_batch(self, texts: list[str]) -> list[list[float]]: async def aclose(self) -> None: self.closed += 1 + + +class DirectedEmbedder(FakeEmbedder): + """Maps texts containing any trigger phrase to one vector; noise else. + + Lets a test make the query and one reflection/semantic memory + 'semantically identical' while sharing zero tokens -- isolating the + vec leg from BM25/keyword matching. Shared here (was previously + private to test_vec_fusion.py) so any embedding-path test can use it. + """ + + def __init__(self, *triggers: str): + super().__init__() + self.triggers = triggers + + def _vec(self, text: str) -> list[float]: + if any(t in text for t in self.triggers): + return [1.0] + [0.0] * 767 + return [0.0, 1.0] + [0.0] * 766 + + async def embed(self, text): + self.calls.append(text) + return self._vec(text) + + async def embed_batch(self, texts): + self.calls.append(list(texts)) + return [self._vec(t) for t in texts] diff --git a/tests/services/test_relevant.py b/tests/services/test_relevant.py index 36766cb..36fa96d 100644 --- a/tests/services/test_relevant.py +++ b/tests/services/test_relevant.py @@ -1,124 +1,227 @@ -"""Tests for retrieve_relevant scoring (hits x activation, min-hits floor).""" +"""Contextual relevance: BM25 + vec + Wilson prior behind an evidence gate. + +The gate is the point: a memory injects only with positive relevance +evidence (BM25 match on the query, or vec cosine >= floor). The Wilson +prior RANKS qualifiers but can never qualify a memory alone — popularity +must not force irrelevant injections (that failure mode measured 13% useful +as bootstrap). Vectors are unit-norm, so cosine >= c is L2 dist^2 <= 2(1-c). +""" from __future__ import annotations from datetime import UTC, datetime +from pathlib import Path import pytest +import sqlite_vec +from better_memory.db.connection import connect +from better_memory.db.schema import apply_migrations +from better_memory.embeddings.sync_embed import SyncEmbedder from better_memory.services.relevant import RelevantMemory, retrieve_relevant +from better_memory.storage.sqlite import SqliteBackend +from tests.services._embedding_fakes import DirectedEmbedder, FakeEmbedder FIXED_NOW = datetime(2026, 7, 11, 12, 0, 0, tzinfo=UTC) -class _FakeBackend: - """Minimal stand-in for a StorageBackend: only the two methods - retrieve_relevant calls (retrieve, semantic_list) are implemented.""" - - def __init__(self): - self.reflections: dict = {"do": [], "dont": [], "neutral": []} - self.semantic: list = [] - self.retrieve_error: Exception | None = None - self.semantic_error: Exception | None = None - - def retrieve(self, *, project, track_exposure=False): - if self.retrieve_error is not None: - raise self.retrieve_error - return self.reflections - - def semantic_list(self, *, project, track_exposure=False): - if self.semantic_error is not None: - raise self.semantic_error - return self.semantic - - @pytest.fixture -def fake_backend(): - return _FakeBackend() - - -def _reflection(id="r1", title="pytest windows redirect", hints=None, - useful_count=0, times_misled=0, confidence=0.8, - updated_at="2026-07-01T00:00:00+00:00", polarity="do"): - return { - "id": id, "title": title, "phase": "general", "use_cases": "", - "hints": hints or [], "confidence": confidence, "tech": None, - "evidence_count": 1, "useful_count": useful_count, - "times_misled": times_misled, "updated_at": updated_at, - "_polarity": polarity, # test helper only; buckets carry polarity - } - - -def test_backend_error_degrades_to_empty(fake_backend): - fake_backend.retrieve_error = RuntimeError("boom") - out = retrieve_relevant(fake_backend, query="alpha beta", project="p", - now=lambda: FIXED_NOW) - assert out == [] - - -def test_empty_keyword_query_returns_empty(fake_backend): - fake_backend.reflections = {"do": [_reflection(title="alpha beta")], "dont": [], "neutral": []} - out = retrieve_relevant(fake_backend, query=" ", project="p", now=lambda: FIXED_NOW) - assert out == [] - - -def test_floor_rejects_single_hit_by_default(fake_backend): - fake_backend.reflections = {"do": [_reflection(title="pytest only-one")], - "dont": [], "neutral": []} - out = retrieve_relevant(fake_backend, query="pytest something unrelated", - project="p", now=lambda: FIXED_NOW) - assert out == [] # 1 distinct hit < min_hits=2 - - -def test_floor_admits_two_hits(fake_backend): - fake_backend.reflections = {"do": [_reflection(title="pytest windows redirect")], - "dont": [], "neutral": []} - out = retrieve_relevant(fake_backend, query="run pytest on windows", - project="p", now=lambda: FIXED_NOW) - assert [m.id for m in out] == ["r1"] - assert out[0].hits == 2 - assert out[0].age_days == 10 - assert out[0].kind == "reflection" - - -def test_title_hits_count_double_in_score(fake_backend): - title_match = _reflection(id="rt", title="alpha beta", updated_at="2026-07-01T00:00:00+00:00") - hint_match = _reflection(id="rh", title="zzz yyy", hints=["alpha beta"], - updated_at="2026-07-01T00:00:00+00:00") - fake_backend.reflections = {"do": [hint_match, title_match], "dont": [], "neutral": []} - out = retrieve_relevant(fake_backend, query="alpha beta", project="p", now=lambda: FIXED_NOW) - assert out[0].id == "rt" # same hits, but title-weighted score wins - - -def test_misled_penalty_halves_score(fake_backend): - clean = _reflection(id="rc", title="alpha beta", useful_count=0, times_misled=0) - burned = _reflection(id="rb", title="alpha beta", useful_count=0, times_misled=3) - fake_backend.reflections = {"do": [burned, clean], "dont": [], "neutral": []} - out = retrieve_relevant(fake_backend, query="alpha beta", project="p", now=lambda: FIXED_NOW) - assert out[0].id == "rc" - assert out[1].score < out[0].score - - -def test_max_items_cap(fake_backend): - fake_backend.reflections = {"do": [ - _reflection(id=f"r{i}", title="alpha beta") for i in range(6) - ], "dont": [], "neutral": []} - out = retrieve_relevant(fake_backend, query="alpha beta", project="p", - max_items=3, now=lambda: FIXED_NOW) - assert len(out) == 3 - - -def test_missing_metadata_is_neutral(fake_backend): - r = _reflection(id="rm", title="alpha beta") - del r["times_misled"] # older backend shape - del r["updated_at"] - fake_backend.reflections = {"do": [r], "dont": [], "neutral": []} - out = retrieve_relevant(fake_backend, query="alpha beta", project="p", now=lambda: FIXED_NOW) - assert out[0].age_days is None - assert out[0].score > 0 - - -def test_returns_relevantmemory(fake_backend): - fake_backend.reflections = {"do": [_reflection(title="alpha beta")], "dont": [], "neutral": []} - out = retrieve_relevant(fake_backend, query="alpha beta", project="p", now=lambda: FIXED_NOW) +def conn(tmp_memory_db: Path): + c = connect(tmp_memory_db) + apply_migrations(c) + try: + yield c + finally: + c.close() + + +def _backend(conn, *, sync_embedder=None, project="p"): + return SqliteBackend( + memory_conn=conn, sync_embedder=sync_embedder, + session_id=None, project=project, + ) + + +def _seed_reflection( + conn, rid, *, title, use_cases="uc", hints="[]", + useful=0, overlooked=0, ignored=0, polarity="do", + updated_at="2026-01-01T00:00:00+00:00", +): + conn.execute( + """INSERT INTO reflections + (id, title, project, phase, polarity, use_cases, hints, + confidence, created_at, updated_at, + useful_count, times_overlooked, times_ignored) + VALUES (?, ?, 'p', 'general', ?, ?, ?, 0.5, + '2026-01-01T00:00:00+00:00', ?, ?, ?, ?)""", + (rid, title, polarity, use_cases, hints, updated_at, + useful, overlooked, ignored), + ) + conn.commit() + + +def _seed_semantic( + conn, sid, *, content, useful=0, overlooked=0, ignored=0, + updated_at="2026-01-01T00:00:00+00:00", +): + conn.execute( + """INSERT INTO semantic_memories + (id, content, project, scope, created_at, updated_at, + useful_count, times_overlooked, times_ignored) + VALUES (?, ?, 'p', 'project', '2026-01-01T00:00:00+00:00', ?, + ?, ?, ?)""", + (sid, content, updated_at, useful, overlooked, ignored), + ) + conn.commit() + + +def _embed_reflection(conn, rid, vector): + conn.execute( + "INSERT INTO reflection_embeddings (reflection_id, embedding) VALUES (?, ?)", + (rid, sqlite_vec.serialize_float32(vector)), + ) + conn.commit() + + +def _embed_semantic(conn, sid, vector): + conn.execute( + "INSERT INTO semantic_embeddings (memory_id, embedding) VALUES (?, ?)", + (sid, sqlite_vec.serialize_float32(vector)), + ) + conn.commit() + + +class TestBM25Gate: + def test_bm25_match_qualifies(self, conn): + _seed_reflection(conn, "r1", title="Retention archives by confidence") + backend = _backend(conn) + out = retrieve_relevant( + backend, query="how does retention archive things", project="p", + conn=conn, now=lambda: FIXED_NOW, + ) + assert [m.id for m in out] == ["r1"] + + def test_no_evidence_no_injection(self, conn): + _seed_reflection(conn, "r1", title="Zebra flamingo unrelated topic", useful=50) + backend = _backend(conn) + out = retrieve_relevant( + backend, query="how does retention archive things", project="p", + conn=conn, now=lambda: FIXED_NOW, + ) + assert out == [] + + +class TestVecGate: + def test_vec_match_qualifies_without_token_overlap(self, conn): + # DirectedEmbedder maps EITHER trigger phrase to the same unit + # vector, so the title (matching via "Stdout handling") and the + # query (matching via "console output") land at cosine 1 despite + # sharing zero tokens -- isolating the vec leg from BM25. + title = "Stdout handling on win32 interpreters" + emb = DirectedEmbedder("Stdout handling", "console output") + _seed_reflection(conn, "r1", title=title) + _embed_reflection(conn, "r1", emb._vec(title)) + sync_embedder = SyncEmbedder(lambda: emb) + backend = _backend(conn, sync_embedder=sync_embedder) + out = retrieve_relevant( + backend, query="console output disappears on windows", project="p", + conn=conn, sync_embedder=sync_embedder, now=lambda: FIXED_NOW, + ) + assert [m.id for m in out] == ["r1"] + + def test_vec_below_floor_does_not_qualify(self, conn): + # Orthogonal unit vectors: the title's trigger is absent from the + # query, so DirectedEmbedder's else-branch gives the query the + # noise vector [0,1,...] while the stored embedding is [1,0,...]. + title = "Zebra flamingo unrelated topic" + emb = DirectedEmbedder("only-in-the-title-xyz") + _seed_reflection(conn, "r1", title=title) + _embed_reflection(conn, "r1", emb._vec("only-in-the-title-xyz")) + sync_embedder = SyncEmbedder(lambda: emb) + backend = _backend(conn, sync_embedder=sync_embedder) + out = retrieve_relevant( + backend, query="totally different wording", project="p", + conn=conn, sync_embedder=sync_embedder, now=lambda: FIXED_NOW, + ) + assert out == [] + + +class TestWilsonRanking: + def test_wilson_ranks_among_qualifiers(self, conn): + _seed_reflection(conn, "r-hi", title="Retention thresholds alpha", + useful=5, ignored=1) + _seed_reflection(conn, "r-lo", title="Retention thresholds beta", + useful=0, ignored=6) + backend = _backend(conn) + out = retrieve_relevant( + backend, query="retention thresholds", project="p", + conn=conn, now=lambda: FIXED_NOW, + ) + assert [m.id for m in out] == ["r-hi", "r-lo"] + + +class TestSemantics: + def test_semantic_qualifies_via_vec(self, conn): + content = "Stdout handling on win32 interpreters" + emb = DirectedEmbedder("Stdout handling", "console output") + _seed_semantic(conn, "s1", content=content) + _embed_semantic(conn, "s1", emb._vec(content)) + sync_embedder = SyncEmbedder(lambda: emb) + backend = _backend(conn, sync_embedder=sync_embedder) + out = retrieve_relevant( + backend, query="console output disappears on windows", project="p", + conn=conn, sync_embedder=sync_embedder, now=lambda: FIXED_NOW, + ) + assert [(m.kind, m.id) for m in out] == [("semantic", "s1")] + + def test_semantic_fallback_keyword_when_no_embedder(self, conn): + _seed_semantic(conn, "s1", content="repo uses uv run pytest on windows") + backend = _backend(conn) + out = retrieve_relevant( + backend, query="uv run pytest windows setup", project="p", + conn=conn, now=lambda: FIXED_NOW, + ) + assert [(m.kind, m.id) for m in out] == [("semantic", "s1")] + + +class TestCapsAndDegradation: + def test_max_items_cap(self, conn): + for i in range(5): + _seed_reflection(conn, f"r{i}", title=f"Retention thresholds variant {i}") + backend = _backend(conn) + out = retrieve_relevant( + backend, query="retention thresholds", project="p", + conn=conn, max_items=3, now=lambda: FIXED_NOW, + ) + assert len(out) == 3 + + def test_no_conn_falls_back_to_keywords(self, conn): + _seed_reflection(conn, "r-hi", title="Retention thresholds alpha", + useful=5, ignored=1) + _seed_reflection(conn, "r-lo", title="Retention thresholds beta", + useful=0, ignored=6) + backend = _backend(conn) + out = retrieve_relevant( + backend, query="retention thresholds", project="p", + conn=None, now=lambda: FIXED_NOW, + ) + assert [m.id for m in out] == ["r-hi", "r-lo"] + + def test_embedder_failure_degrades_to_bm25(self, conn): + _seed_reflection(conn, "r1", title="Retention archives by confidence") + sync_embedder = SyncEmbedder(lambda: FakeEmbedder(fail=True)) + backend = _backend(conn, sync_embedder=sync_embedder) + out = retrieve_relevant( + backend, query="how does retention archive things", project="p", + conn=conn, sync_embedder=sync_embedder, now=lambda: FIXED_NOW, + ) + assert [m.id for m in out] == ["r1"] + + +def test_returns_relevantmemory(conn): + _seed_reflection(conn, "r1", title="Retention archives by confidence") + backend = _backend(conn) + out = retrieve_relevant( + backend, query="how does retention archive things", project="p", + conn=conn, now=lambda: FIXED_NOW, + ) assert out and all(isinstance(m, RelevantMemory) for m in out) - assert all(m.kind in ("reflection", "semantic") for m in out) From 71526ae32e8643a43ab1b5474b7b772b035ea21d Mon Sep 17 00:00:00 2001 From: gethin Date: Fri, 24 Jul 2026 00:03:18 +0100 Subject: [PATCH 08/17] feat(hooks): contextual inject gains vec leg, per-session PreToolUse latch, all-tools matcher Wires the three-leg evidence-gated scorer (BM25/vector/keyword-fallback) into contextual_inject.py: builds a SyncEmbedder(down_state_file=...) when embeddings_backend=ollama and calls retrieve_relevant with the new conn/sync_embedder/vec_floor signature (no more min_hits). Adds SeenStore.pretool_fired()/mark_pretool_fired() so PreToolUse does real retrieval work only once per session; later PreToolUse events short-circuit via a module-local _SkipInjection sentinel before touching the DB. Widens install_hooks.py's PreToolUse matcher from "Skill|Task|Write" to None (unscoped) since the latch makes an all-tools matcher cheap. Also restores the blank-query short-circuit in retrieve_relevant that the Task 4 rewrite dropped (a contentless query has no evidence by definition), and fixes a SeenStore._load() bug found while testing the latch: it silently dropped the pretool_fired key on every reload because it only carried forward "turn"/"seen". Updates the 5 tests in test_contextual_inject.py that assumed the old keyword min-hits gate (BM25 now matches on token overlap, so test_below_floor_injects_nothing needed genuinely zero-evidence content), adds latch coverage, and fixes the golden hook-shape assertions in test_install_hooks.py (both) and test_setup_sh.py for the matcher change. Also updates test_agentcore_semantic_parity.py's stale min_hits= call, collateral from the Task 4 signature change but outside its declared scope. Co-Authored-By: Claude Fable 5 --- better_memory/cli/install_hooks.py | 9 ++-- better_memory/config.py | 5 ++ better_memory/hooks/contextual_inject.py | 48 +++++++++++++---- better_memory/services/context_seen.py | 17 +++++- better_memory/services/relevant.py | 3 ++ tests/cli/test_install_hooks.py | 2 +- tests/e2e/test_install_hooks.py | 2 +- tests/e2e/test_setup_sh.py | 8 ++- tests/hooks/test_contextual_inject.py | 52 ++++++++++++++++++- tests/services/test_context_seen.py | 12 +++++ tests/services/test_relevant.py | 19 +++++++ .../storage/test_agentcore_semantic_parity.py | 5 +- 12 files changed, 158 insertions(+), 24 deletions(-) diff --git a/better_memory/cli/install_hooks.py b/better_memory/cli/install_hooks.py index e669731..bf88284 100644 --- a/better_memory/cli/install_hooks.py +++ b/better_memory/cli/install_hooks.py @@ -67,9 +67,12 @@ class HookSpec: # stdout). Gated at runtime by BETTER_MEMORY_CONTEXT_INJECT_MODE; both # events install and the mode no-ops whichever isn't selected. HookSpec("better_memory.hooks.contextual_inject", "UserPromptSubmit", None, False, True), - HookSpec( - "better_memory.hooks.contextual_inject", "PreToolUse", "Skill|Task|Write", False, True - ), + # Matcher is None (unscoped = all tools) rather than a tool-name + # alternation: the hook's per-session PreToolUse latch (SeenStore + # .pretool_fired/.mark_pretool_fired) makes an unscoped matcher cheap — + # only the first PreToolUse event per session does real work; every + # later one short-circuits on the state file before touching the DB. + HookSpec("better_memory.hooks.contextual_inject", "PreToolUse", None, False, True), ) # Module paths that are no longer registered but may be present in users' diff --git a/better_memory/config.py b/better_memory/config.py index 7d0b339..6cd1a27 100644 --- a/better_memory/config.py +++ b/better_memory/config.py @@ -234,6 +234,11 @@ class Config: context_reinject_turns: int inject_mode: Literal["deferred", "legacy"] context_vec_floor: float + # NOTE: context_min_hits is DEPRECATED. The three-leg evidence-gated + # scorer in services/relevant.py (BM25 / vector cosine / keyword-hit + # fallback) replaced the old pure keyword-hits floor; contextual_inject.py + # no longer reads this field. Kept for back-compat with any external + # BETTER_MEMORY_CONTEXT_MIN_HITS overrides that still resolve here. _DEFAULT_CONTEXT_INJECT_MODE = "both" diff --git a/better_memory/hooks/contextual_inject.py b/better_memory/hooks/contextual_inject.py index f158be2..05be6c2 100644 --- a/better_memory/hooks/contextual_inject.py +++ b/better_memory/hooks/contextual_inject.py @@ -2,18 +2,20 @@ current prompt or tool-input. Gated by BETTER_MEMORY_CONTEXT_INJECT_MODE (userprompt | pretool | both | off). Never raises; always exits 0. -Candidates are scored via retrieve_relevant and must clear a min-hits floor -(cfg.context_min_hits) and fit within a max-items cap (cfg.context_max_items). -A per-session SeenStore dedups injected memories across turns within a run -(cfg.context_reinject_turns controls re-injection after N turns). Survivors -are recorded as 'contextual' exposures (best-effort; a write failure never -blocks injection) and counted in rating_diagnostics for observability -(contextual_fired_userprompt/pretool, contextual_injected, +Candidates are scored via retrieve_relevant's three-leg evidence gate (BM25 / +vector cosine / keyword-hit fallback — see services/relevant.py) and capped +at cfg.context_max_items. A per-session SeenStore dedups injected memories +across turns within a run (cfg.context_reinject_turns controls re-injection +after N turns). Survivors are recorded as 'contextual' exposures (best-effort; +a write failure never blocks injection) and counted in rating_diagnostics for +observability (contextual_fired_userprompt/pretool, contextual_injected, contextual_suppressed_floor, contextual_suppressed_dedup). -NOTE: whether PreToolUse fires for the built-in Skill/Task tools is -environment-dependent (see the plan's Task 0 probe); UserPromptSubmit is the -reliable trigger. +PreToolUse is latched to one real firing per session (SeenStore.pretool_fired +/ mark_pretool_fired): the installed matcher is unscoped (all tools), so +without the latch every tool call would re-run the full retrieval path. +Later PreToolUse events in the same session short-circuit on the state file +before any DB/embedder work. UserPromptSubmit is unaffected by the latch. """ from __future__ import annotations @@ -26,6 +28,8 @@ from better_memory.config import get_config, project_name from better_memory.db.connection import connect +from better_memory.embeddings.ollama import OllamaEmbedder +from better_memory.embeddings.sync_embed import SyncEmbedder from better_memory.hooks._error_log import record_hook_error from better_memory.services.context_seen import SeenStore, prune_stale from better_memory.services.relevant import format_relevant, retrieve_relevant @@ -34,6 +38,14 @@ _MAX_STDIN_BYTES = 1_000_000 +class _SkipInjection(Exception): + """Module-local sentinel: PreToolUse latch already fired this session. + + Caught explicitly (never via the outer BaseException guard) to leave + ``rendered = ""`` without treating the skip as an error. + """ + + def _bump_diagnostic(conn, cfg, metric: str) -> None: """Best-effort observability counter. Sqlite mode only; never raises.""" if cfg.storage_backend != "sqlite" or conn is None: @@ -95,6 +107,10 @@ def main() -> None: state_dir = cfg.home / "state" prune_stale(state_dir, now=datetime.now(UTC)) seen = SeenStore(state_dir, session_id) + if event == "PreToolUse": + if seen.pretool_fired(): + raise _SkipInjection() # module-local sentinel; caught below + seen.mark_pretool_fired() seen.bump_turn() conn_ctx = ( closing(connect(cfg.memory_db)) @@ -114,9 +130,17 @@ def main() -> None: session_id=session_id or None, project=project, ) + sync_embedder = None + if cfg.embeddings_backend == "ollama": + sync_embedder = SyncEmbedder( + lambda: OllamaEmbedder(timeout=5.0, max_retries=1), + down_state_file=cfg.home / "state" / "embed_down_until", + ) items = retrieve_relevant( backend, query=query, project=project, - min_hits=cfg.context_min_hits, + conn=conn, + sync_embedder=sync_embedder, + vec_floor=cfg.context_vec_floor, max_items=cfg.context_max_items, ) had_candidates = bool(items) @@ -145,6 +169,8 @@ def main() -> None: _bump_diagnostic(conn, cfg, "contextual_suppressed_dedup") else: _bump_diagnostic(conn, cfg, "contextual_suppressed_floor") + except _SkipInjection: + rendered = "" except BaseException as exc: # noqa: BLE001 try: record_hook_error(hook_name="contextual_inject", exc=exc) diff --git a/better_memory/services/context_seen.py b/better_memory/services/context_seen.py index 11ae420..f3e611f 100644 --- a/better_memory/services/context_seen.py +++ b/better_memory/services/context_seen.py @@ -6,7 +6,9 @@ degrades to "nothing seen". File format: ``context_seen_.json`` -> -``{"turn": int, "seen": {":": last_injected_turn}}``. +``{"turn": int, "seen": {":": last_injected_turn}, +"pretool_fired": bool}``. ``pretool_fired`` latches PreToolUse to one real +firing per session (see :meth:`SeenStore.pretool_fired`). """ from __future__ import annotations @@ -34,7 +36,11 @@ def _load(self) -> dict: try: raw = json.loads(self._path.read_text(encoding="utf-8")) if isinstance(raw, dict) and isinstance(raw.get("seen"), dict): - return {"turn": int(raw.get("turn") or 0), "seen": raw["seen"]} + return { + "turn": int(raw.get("turn") or 0), + "seen": raw["seen"], + "pretool_fired": bool(raw.get("pretool_fired")), + } except BaseException: # noqa: BLE001 - corrupt/missing -> empty pass return {"turn": 0, "seen": {}} @@ -70,6 +76,13 @@ def mark_seen(self, ids: list[tuple[str, str]]) -> None: self._data["seen"][_key(kind, id_)] = turn self._save() + def pretool_fired(self) -> bool: + return bool(self._data.get("pretool_fired")) + + def mark_pretool_fired(self) -> None: + self._data["pretool_fired"] = True + self._save() + def prune_stale(state_dir: Path, *, now: datetime, max_age_days: int = 7) -> None: """Delete context_seen files older than max_age_days. Never raises.""" diff --git a/better_memory/services/relevant.py b/better_memory/services/relevant.py index 453f0e9..30e2921 100644 --- a/better_memory/services/relevant.py +++ b/better_memory/services/relevant.py @@ -176,6 +176,9 @@ def retrieve_relevant( Never raises -- any backend/leg failure degrades that leg to "absent" rather than propagating. """ + if not (query or "").strip(): + return [] + _now = (now or (lambda: datetime.now(UTC)))() try: diff --git a/tests/cli/test_install_hooks.py b/tests/cli/test_install_hooks.py index 2028889..2cfb967 100644 --- a/tests/cli/test_install_hooks.py +++ b/tests/cli/test_install_hooks.py @@ -94,7 +94,7 @@ def test_contextual_inject_registered_for_both_events(self) -> None: ups = next(s for s in ci if s.event == "UserPromptSubmit") assert ups.matcher is None and ups.is_async is False and ups.needs_stdout is True ptu = next(s for s in ci if s.event == "PreToolUse") - assert ptu.matcher == "Skill|Task|Write" and ptu.needs_stdout is True + assert ptu.matcher is None and ptu.needs_stdout is True def test_session_bootstrap_is_session_start_event_no_matcher(self) -> None: sb = next(s for s in _OUR_HOOKS if s.module.endswith("session_bootstrap")) diff --git a/tests/e2e/test_install_hooks.py b/tests/e2e/test_install_hooks.py index 880e805..d2ed1af 100644 --- a/tests/e2e/test_install_hooks.py +++ b/tests/e2e/test_install_hooks.py @@ -65,7 +65,7 @@ # honoured for. Async/pythonw => nothing ever gets rated. "Stop": ("better_memory.hooks.session_close", VENV_PY, False, None), "UserPromptSubmit": ("better_memory.hooks.contextual_inject", VENV_PY, False, None), - "PreToolUse": ("better_memory.hooks.contextual_inject", VENV_PY, False, "Skill|Task|Write"), + "PreToolUse": ("better_memory.hooks.contextual_inject", VENV_PY, False, None), } diff --git a/tests/e2e/test_setup_sh.py b/tests/e2e/test_setup_sh.py index 60896d0..1cd624e 100644 --- a/tests/e2e/test_setup_sh.py +++ b/tests/e2e/test_setup_sh.py @@ -397,9 +397,13 @@ def test_settings_hooks_py_pyw_split(self, decline_run: SetupRun) -> None: for key, interp in entries.items(): assert Path(interp) == expected_interp[key], (key, interp) assert matchers[("PostToolUse", "better_memory.hooks.observer")] == "Write|Edit|Bash" + # Unscoped (matcher None -> no `matcher` key in the group), mirroring + # how SessionStart/Stop assert: the PreToolUse latch in the hook + # (SeenStore.pretool_fired/mark_pretool_fired) makes an all-tools + # matcher cheap -- only the first PreToolUse event per session does + # real work. assert ( - matchers[("PreToolUse", "better_memory.hooks.contextual_inject")] - == "Skill|Task|Write" + matchers[("PreToolUse", "better_memory.hooks.contextual_inject")] is None ) def test_all_writes_landed_under_tmp(self, decline_run: SetupRun) -> None: diff --git a/tests/hooks/test_contextual_inject.py b/tests/hooks/test_contextual_inject.py index a20300e..7d93dc2 100644 --- a/tests/hooks/test_contextual_inject.py +++ b/tests/hooks/test_contextual_inject.py @@ -159,7 +159,11 @@ def test_second_run_suppressed_by_seen_store(bm_home, monkeypatch, capsys): def test_below_floor_injects_nothing(bm_home, monkeypatch, capsys): - _seed_reflection(bm_home, "refl-widget-only-4", title="widget playbook") + # No BM25 overlap (zero shared tokens with the prompt), no vec leg + # (embeddings_backend=sqlite by default -> sync_embedder is None), and + # conn is present so the keyword fallback never kicks in either: no + # evidence on any leg -> no injection. + _seed_reflection(bm_home, "refl-zebra-flamingo-4", title="zebra flamingo unrelated topic") res = _run( {"hook_event_name": "UserPromptSubmit", "prompt": "deploy the widget service now", "cwd": ".", "session_id": "sess-4"}, @@ -256,3 +260,49 @@ def _raise(*args, **kwargs): ) ctx = res["hookSpecificOutput"]["additionalContext"] assert "refl-widget-deploy-6" in ctx + + +def test_pretool_fires_once_per_session(bm_home, monkeypatch, capsys): + """Second PreToolUse in the same session hits the latch: empty + additionalContext, and contextual_fired_pretool is bumped only once.""" + _seed_reflection(bm_home, "refl-widget-deploy-7", title="widget deploy playbook") + payload = { + "hook_event_name": "PreToolUse", "tool_name": "Skill", + "tool_input": {"skill": "deploy the widget service now"}, + "cwd": ".", "session_id": "sess-7", + } + first = _run(payload, monkeypatch, capsys) + assert first["hookSpecificOutput"]["additionalContext"] != "" + assert _diag_value(bm_home, "contextual_fired_pretool") == 1 + + second = _run(payload, monkeypatch, capsys) + assert second["hookSpecificOutput"]["additionalContext"] == "" + assert _diag_value(bm_home, "contextual_fired_pretool") == 1 + + +def test_userprompt_unaffected_by_pretool_latch(bm_home, monkeypatch, capsys): + """The PreToolUse latch must not suppress UserPromptSubmit injections, + even after a PreToolUse event has already fired in the same session. + + The PreToolUse payload deliberately shares no evidence with the seeded + reflection (no BM25/vec/keyword match) so it latches without injecting or + marking anything seen -- isolating the latch from the unrelated + SeenStore dedup mechanism, which would otherwise also explain a second + empty result. + """ + _seed_reflection(bm_home, "refl-widget-deploy-8", title="widget deploy playbook") + session_id = "sess-8" + pretool_payload = { + "hook_event_name": "PreToolUse", "tool_name": "Bash", + "tool_input": {"command": "zebra flamingo unrelated topic"}, + "cwd": ".", "session_id": session_id, + } + first = _run(pretool_payload, monkeypatch, capsys) # fires and latches PreToolUse + assert first["hookSpecificOutput"]["additionalContext"] == "" + + prompt_payload = { + "hook_event_name": "UserPromptSubmit", "prompt": "deploy the widget service now", + "cwd": ".", "session_id": session_id, + } + res = _run(prompt_payload, monkeypatch, capsys) + assert res["hookSpecificOutput"]["additionalContext"] != "" diff --git a/tests/services/test_context_seen.py b/tests/services/test_context_seen.py index 42823ed..37bf231 100644 --- a/tests/services/test_context_seen.py +++ b/tests/services/test_context_seen.py @@ -64,3 +64,15 @@ def test_missing_state_dir_never_raises(tmp_path): s = SeenStore(tmp_path / "does-not-exist-yet", "sess-1") assert s.bump_turn() == 1 # creates the dir s.mark_seen([("reflection", "r1")]) + + +class TestPretoolLatch: + def test_defaults_false_then_persists(self, tmp_path): + s = SeenStore(tmp_path, "sess") + assert s.pretool_fired() is False + s.mark_pretool_fired() + assert SeenStore(tmp_path, "sess").pretool_fired() is True + + def test_corrupt_state_means_not_fired(self, tmp_path): + (tmp_path / "context_seen_sess.json").write_text("{", encoding="utf-8") + assert SeenStore(tmp_path, "sess").pretool_fired() is False diff --git a/tests/services/test_relevant.py b/tests/services/test_relevant.py index 36fa96d..2187327 100644 --- a/tests/services/test_relevant.py +++ b/tests/services/test_relevant.py @@ -217,6 +217,25 @@ def test_embedder_failure_degrades_to_bm25(self, conn): assert [m.id for m in out] == ["r1"] +@pytest.mark.parametrize("blank_query", ["", " ", "\t\n"]) +def test_blank_query_returns_empty(conn, blank_query): + # DirectedEmbedder with no trigger args maps EVERY text to the same + # unit vector, so it would match anything if the vec leg were reached -- + # the point is that a contentless query short-circuits before that. + title = "Retention archives by confidence" + emb = DirectedEmbedder() + _seed_reflection(conn, "r1", title=title) + _embed_reflection(conn, "r1", emb._vec(title)) + sync_embedder = SyncEmbedder(lambda: emb) + backend = _backend(conn, sync_embedder=sync_embedder) + out = retrieve_relevant( + backend, query=blank_query, project="p", + conn=conn, sync_embedder=sync_embedder, now=lambda: FIXED_NOW, + ) + assert out == [] + assert emb.calls == [] + + def test_returns_relevantmemory(conn): _seed_reflection(conn, "r1", title="Retention archives by confidence") backend = _backend(conn) diff --git a/tests/storage/test_agentcore_semantic_parity.py b/tests/storage/test_agentcore_semantic_parity.py index 7fba97b..ec0fa8d 100644 --- a/tests/storage/test_agentcore_semantic_parity.py +++ b/tests/storage/test_agentcore_semantic_parity.py @@ -195,8 +195,8 @@ def test_relevant_reads_agentcore_semantic_counters_and_content( """End-to-end §6.3 proof: retrieve_relevant over the agentcore backend surfaces the real content + non-zero useful_count. Before the fix, semantic_list returned dicts and getattr(s,'content') / getattr(s, - 'useful_count') defaulted to '' / 0, so this memory would never score - above the min-hits floor and semantic injection was dead.""" + 'useful_count') defaulted to '' / 0, so this memory would never clear + the keyword-fallback evidence gate and semantic injection was dead.""" # No reflections; semantic namespace has one strongly-matching memory. mock_data_client.list_memory_records.return_value = { "memoryRecordSummaries": [ @@ -210,7 +210,6 @@ def test_relevant_reads_agentcore_semantic_counters_and_content( backend, query="uv pip python", project="testproj", - min_hits=2, now=lambda: datetime(2026, 7, 13, tzinfo=UTC), ) assert len(out) == 1 From 87dca9be50f05fa221073315e2b97265e0d5e7d4 Mon Sep 17 00:00:00 2001 From: gethin Date: Fri, 24 Jul 2026 00:10:14 +0100 Subject: [PATCH 09/17] =?UTF-8?q?test(hooks):=20pin=20sqlite=20embeddings?= =?UTF-8?q?=20backend=20=E2=80=94=20no=20live-Ollama=20dependency?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit tests/hooks/test_contextual_inject.py's bm_home fixture never pinned BETTER_MEMORY_EMBEDDINGS_BACKEND, so every non-blank-query test relied on config.py's actual default (ollama): the hook built a real SyncEmbedder and issued an HTTP call to localhost:11434 on every run. This worked only because no test seeds embeddings, so a connection-refused/failed embed degrades silently to the qvec=None fallback path with no observable difference — a live Ollama on that port would have gone unnoticed too. Pins BETTER_MEMORY_EMBEDDINGS_BACKEND=sqlite in the fixture, mirroring tests/ui/conftest.py's pin, and corrects the stale comment on test_below_floor_injects_nothing that claimed sqlite was the config default rather than the fixture's pin. Co-Authored-By: Claude Fable 5 --- tests/hooks/test_contextual_inject.py | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/tests/hooks/test_contextual_inject.py b/tests/hooks/test_contextual_inject.py index 7d93dc2..da16026 100644 --- a/tests/hooks/test_contextual_inject.py +++ b/tests/hooks/test_contextual_inject.py @@ -34,9 +34,18 @@ def bm_home(tmp_path: Path, monkeypatch) -> Path: (e.g. test_userprompt_emits_envelope, test_mode_off_is_noop) -- otherwise hook.main() falls back to the developer's real ~/.better-memory and writes memory.db, diagnostics, and state/ files there. + + Pins ``BETTER_MEMORY_EMBEDDINGS_BACKEND=sqlite`` (mirrors + tests/ui/conftest.py's pin) so the hook never builds a real SyncEmbedder + / OllamaEmbedder and never issues an HTTP call to localhost:11434 -- + config.py's actual default is "ollama", not sqlite. Without this pin + every non-blank-query test here would silently depend on there being no + live Ollama daemon to answer (works today only because no test seeds + embeddings, so the qvec=None fallback path never observably differs). """ monkeypatch.setenv("BETTER_MEMORY_HOME", str(tmp_path)) monkeypatch.setenv("BETTER_MEMORY_PROJECT", _PROJECT) + monkeypatch.setenv("BETTER_MEMORY_EMBEDDINGS_BACKEND", "sqlite") conn = connect(tmp_path / "memory.db") try: apply_migrations(conn) @@ -160,9 +169,9 @@ def test_second_run_suppressed_by_seen_store(bm_home, monkeypatch, capsys): def test_below_floor_injects_nothing(bm_home, monkeypatch, capsys): # No BM25 overlap (zero shared tokens with the prompt), no vec leg - # (embeddings_backend=sqlite by default -> sync_embedder is None), and - # conn is present so the keyword fallback never kicks in either: no - # evidence on any leg -> no injection. + # (the bm_home fixture pins embeddings_backend=sqlite -> sync_embedder is + # None), and conn is present so the keyword fallback never kicks in + # either: no evidence on any leg -> no injection. _seed_reflection(bm_home, "refl-zebra-flamingo-4", title="zebra flamingo unrelated topic") res = _run( {"hook_event_name": "UserPromptSubmit", "prompt": "deploy the widget service now", From 7133c16d6fe98a137a3a2e4e540c42817ec93594 Mon Sep 17 00:00:00 2001 From: gethin Date: Fri, 24 Jul 2026 00:15:56 +0100 Subject: [PATCH 10/17] feat(bootstrap): deferred mode - general semantics + index line only Co-Authored-By: Claude Fable 5 --- better_memory/services/session_bootstrap.py | 45 +++++++ tests/services/test_session_bootstrap.py | 137 ++++++++++++++++++++ 2 files changed, 182 insertions(+) diff --git a/better_memory/services/session_bootstrap.py b/better_memory/services/session_bootstrap.py index 0add45d..8e3263b 100644 --- a/better_memory/services/session_bootstrap.py +++ b/better_memory/services/session_bootstrap.py @@ -257,6 +257,51 @@ def bootstrap( "neutral": len(buckets["neutral"]), } + from better_memory.config import get_config as _get_config_deferred + + if _get_config_deferred().inject_mode == "deferred": + general_only = [m for m in semantic if m.scope == "general"] + deferred_now = self._clock() + n_refl = sum(reflections_counts.values()) + n_sem = semantic_count + index_line = ( + f"better-memory knows {n_refl} reflections + {n_sem} semantic " + "memories for this project; relevant ones will surface as you " + "work - or ask via memory_retrieve with a task query." + ) + deferred_sections: list[str] = [ + _render_header( + project=project, + source=coerced_source, + action=action, + episode_id=episode_id, + ), + ] + deferred_sem_section, deferred_semantic_ids = _render_semantic_full( + general_only, deferred_now, + ) + if deferred_sem_section: + deferred_sections.append(deferred_sem_section) + deferred_sections.append(index_line) + deferred_sections.append("---") + deferred_sections.append(_FOOTER) + + self._record_exposure( + session_id=session_id, + reflection_ids=[], + semantic_ids=deferred_semantic_ids, + ) + + return BootstrapResult( + additional_context="\n\n".join(deferred_sections), + project=project, + source=coerced_source, + episode_id=episode_id, + episode_action=action, + semantic_count=semantic_count, + reflections_counts=reflections_counts, + ) + if self._top_n is not None: top_n = self._top_n else: diff --git a/tests/services/test_session_bootstrap.py b/tests/services/test_session_bootstrap.py index 6b93b03..c4752fc 100644 --- a/tests/services/test_session_bootstrap.py +++ b/tests/services/test_session_bootstrap.py @@ -521,3 +521,140 @@ def test_list_session_exposures_empty_session_id_returns_none_envelope(conn) -> svc = SessionBootstrapService(conn) result = svc.list_session_exposures(session_id="") assert result == {"session_id": None, "exposures": []} + + +# --------------------------------------------------------------------------- +# Task 6: deferred bootstrap mode. +# --------------------------------------------------------------------------- + + +class TestDeferredBootstrap: + def test_deferred_renders_general_semantics_and_index_only( + self, conn, git_repo: Path, monkeypatch + ) -> None: + monkeypatch.setenv("BETTER_MEMORY_INJECT_MODE", "deferred") + proj = git_repo.name + + _seed_semantic(conn, content="general-fact-one", project="anyproj", scope="general") + _seed_semantic(conn, content="general-fact-two", project="otherproj", scope="general") + _seed_semantic(conn, content="project-fact-one", project=proj, scope="project") + _seed_semantic(conn, content="project-fact-two", project=proj, scope="project") + _seed_semantic(conn, content="project-fact-three", project=proj, scope="project") + for i in range(4): + _seed_reflection( + conn, project=proj, polarity="do", scope="project", title=f"refl-title-{i}", + ) + + svc = SessionBootstrapService(conn) + text = svc.bootstrap( + source="startup", session_id="sess-deferred-1", cwd=git_repo, + ).additional_context + + assert "general-fact-one" in text + assert "general-fact-two" in text + assert "project-fact-one" not in text + assert "project-fact-two" not in text + assert "project-fact-three" not in text + assert "refl-title-0" not in text + assert "knows 4 reflections + 5 semantic memories" in text + + def test_deferred_exposes_only_general_semantics( + self, conn, git_repo: Path, monkeypatch + ) -> None: + monkeypatch.setenv("BETTER_MEMORY_INJECT_MODE", "deferred") + proj = git_repo.name + + gen_ids = [ + _seed_semantic(conn, content="general-a", project="anyproj", scope="general"), + _seed_semantic(conn, content="general-b", project="otherproj", scope="general"), + ] + _seed_semantic(conn, content="proj-a", project=proj, scope="project") + _seed_semantic(conn, content="proj-b", project=proj, scope="project") + _seed_reflection(conn, project=proj, polarity="do", scope="project") + + svc = SessionBootstrapService(conn) + svc.bootstrap(source="startup", session_id="sess-deferred-2", cwd=git_repo) + + rows = conn.execute( + "SELECT memory_kind, memory_id, source FROM session_memory_exposure " + "WHERE session_id = ?", + ("sess-deferred-2",), + ).fetchall() + exposed = {(r["memory_kind"], r["memory_id"]) for r in rows} + assert exposed == {("semantic", gid) for gid in gen_ids} + assert all(r["source"] == "bootstrap" for r in rows) + + def test_legacy_mode_byte_identical( + self, tmp_path: Path, git_repo: Path, monkeypatch + ) -> None: + from uuid import UUID + + proj = git_repo.name + fixed_now = datetime(2026, 1, 1, tzinfo=UTC) + fixed_uuid = UUID("a" * 32) + + monkeypatch.setattr( + "better_memory.services.episode.uuid4", lambda: fixed_uuid, + ) + + def make_conn(name: str): + db = tmp_path / name + c = connect(db) + apply_migrations(c, migrations_dir=_MIGRATIONS) + return c + + # Explicit ids + a fixed timestamp (rather than the _seed_* helpers, + # which mint a random uuid per call) so the two conns render + # byte-identical text — only the inject_mode env var differs. + fixed_ts = "2026-01-01T00:00:00+00:00" + + def seed(c) -> None: + c.execute( + "INSERT INTO semantic_memories " + "(id, content, project, scope, created_at, updated_at) " + "VALUES (?, ?, ?, ?, ?, ?)", + ("1" * 32, "proj-fact", proj, "project", fixed_ts, fixed_ts), + ) + c.execute( + "INSERT INTO semantic_memories " + "(id, content, project, scope, created_at, updated_at) " + "VALUES (?, ?, ?, ?, ?, ?)", + ("2" * 32, "gen-fact", "anyproj", "general", fixed_ts, fixed_ts), + ) + c.execute( + "INSERT INTO reflections " + "(id, title, project, tech, phase, polarity, use_cases, hints, " + " confidence, status, evidence_count, created_at, updated_at, scope) " + "VALUES (?, 'refl-a', ?, NULL, 'implementation', 'do', 'uc', ?, 0.9, " + " 'confirmed', 1, ?, ?, 'project')", + ("3" * 32, proj, json.dumps(["h"]), fixed_ts, fixed_ts), + ) + c.commit() + + monkeypatch.delenv("BETTER_MEMORY_INJECT_MODE", raising=False) + conn_unset = make_conn("unset.db") + seed(conn_unset) + svc_unset = SessionBootstrapService(conn_unset, clock=lambda: fixed_now) + text_unset = svc_unset.bootstrap( + source="startup", session_id="sess-same", cwd=git_repo, + ).additional_context + + monkeypatch.setenv("BETTER_MEMORY_INJECT_MODE", "legacy") + conn_legacy = make_conn("legacy.db") + seed(conn_legacy) + svc_legacy = SessionBootstrapService(conn_legacy, clock=lambda: fixed_now) + text_legacy = svc_legacy.bootstrap( + source="startup", session_id="sess-same", cwd=git_repo, + ).additional_context + + assert text_unset == text_legacy + + monkeypatch.setenv("BETTER_MEMORY_INJECT_MODE", "deferred") + conn_deferred = make_conn("deferred.db") + seed(conn_deferred) + svc_deferred = SessionBootstrapService(conn_deferred, clock=lambda: fixed_now) + text_deferred = svc_deferred.bootstrap( + source="startup", session_id="sess-same", cwd=git_repo, + ).additional_context + + assert text_deferred != text_unset From ab5321b67ac52349cde8b244fdfcc06626e588c5 Mon Sep 17 00:00:00 2001 From: gethin Date: Fri, 24 Jul 2026 00:23:27 +0100 Subject: [PATCH 11/17] feat(docs): behavioural CLAUDE.snippet + parameter-drift sentinel Co-Authored-By: Claude Fable 5 --- better_memory/hooks/_claude_md_sentinel.py | 45 ++++++++++ better_memory/hooks/session_bootstrap.py | 17 ++++ better_memory/skills/CLAUDE.snippet.md | 100 +++++++++++---------- tests/hooks/test_claude_md_sentinel.py | 43 +++++++++ 4 files changed, 156 insertions(+), 49 deletions(-) create mode 100644 better_memory/hooks/_claude_md_sentinel.py create mode 100644 tests/hooks/test_claude_md_sentinel.py diff --git a/better_memory/hooks/_claude_md_sentinel.py b/better_memory/hooks/_claude_md_sentinel.py new file mode 100644 index 0000000..c9a2351 --- /dev/null +++ b/better_memory/hooks/_claude_md_sentinel.py @@ -0,0 +1,45 @@ +"""Detect parameter-enumeration drift in the user's CLAUDE.md. + +Pure functions; the session_bootstrap hook wires them in best-effort. +Only lines that mention a better-memory tool name are scanned, and only +tokens shaped like parameter usage (word= or word:) are checked against +the live schema, so prose can't false-positive. +""" + +from __future__ import annotations + +import re + +_PARAM_RE = re.compile(r"\b([a-z_][a-z0-9_]{2,})\s*[=:]") +_IGNORE = {"http", "https", "note", "example", "warning", "default"} + + +def build_schemas() -> dict[str, set[str]]: + from better_memory.mcp.tools import tool_definitions + out: dict[str, set[str]] = {} + for tool in tool_definitions(): + rendered = tool.name.replace(".", "_") + props = set((tool.inputSchema or {}).get("properties", {}).keys()) + out[rendered] = props + return out + + +def check_claude_md(text: str, schemas: dict[str, set[str]]) -> list[str]: + warnings: list[str] = [] + try: + for line in (text or "").splitlines(): + hit_tools = [name for name in schemas if name in line] + if not hit_tools: + continue + for token in _PARAM_RE.findall(line): + if token in _IGNORE or any(token in schemas[t] for t in hit_tools): + continue + if token in schemas: # a tool name followed by ':' etc. + continue + warnings.append( + f"CLAUDE.md documents parameter '{token}' near " + f"{'/'.join(hit_tools)} but the live tool schema has no " + "such parameter - fix or drop the enumeration.") + except Exception: + return [] + return warnings[:1] # at most one line of noise per session diff --git a/better_memory/hooks/session_bootstrap.py b/better_memory/hooks/session_bootstrap.py index 345e2cc..4c81942 100644 --- a/better_memory/hooks/session_bootstrap.py +++ b/better_memory/hooks/session_bootstrap.py @@ -114,6 +114,23 @@ def main() -> None: source=source, session_id=session_id, cwd=Path(cwd_str), ) rendered = result.additional_context + + try: + claude_md_path = Path.home() / ".claude" / "CLAUDE.md" + if claude_md_path.is_file(): + from better_memory.hooks._claude_md_sentinel import ( + build_schemas, + check_claude_md, + ) + + claude_md_text = claude_md_path.read_text( + encoding="utf-8", errors="ignore", + ) + sentinel_warnings = check_claude_md(claude_md_text, build_schemas()) + if sentinel_warnings: + rendered = rendered + "\n\n" + sentinel_warnings[0] + except BaseException: # noqa: BLE001 — sentinel is best-effort + pass except BaseException as exc: # noqa: BLE001 try: record_hook_error(hook_name="session_bootstrap", exc=exc) diff --git a/better_memory/skills/CLAUDE.snippet.md b/better_memory/skills/CLAUDE.snippet.md index fd90137..d31bf3c 100644 --- a/better_memory/skills/CLAUDE.snippet.md +++ b/better_memory/skills/CLAUDE.snippet.md @@ -9,52 +9,58 @@ This project uses better-memory for persistent AI knowledge. - Validation arrives (evidence in hand) → read `better_memory/skills/memory-feedback.md` - Session ending → read `better_memory/skills/session-close.md` -### Memory outcomes — the evidence-in-hand rule +### Retrieval + +When you begin a task, call `memory_retrieve` with a `query` describing +it. Do not do broad no-query retrieval at session start; memories +surface contextually as you work. + +`memory_retrieve` returns distilled **reflections** — generalised +lessons from prior observations — bucketed by polarity: `do`, `dont`, +and `neutral`. Treat `dont` as a hard constraint list: do not repeat +approaches that live there. + +For raw observation lookup — e.g. citing a specific past event or +hunting for an exact prior decision — use `memory_retrieve_observations` +instead. With a free-text query, results are ranked by hybrid search; +without one, they're ordered newest-first. + +### Knowledge + +At the start of a session, call `knowledge_list` to see what curated +knowledge is indexed for this project. Call `knowledge_search` when the +current task may be covered by curated documentation rather than raw +memory. + +### Recording — the evidence-in-hand rule + +At a decision point, record with a concise factual summary, an explicit +outcome, and component/theme tags where they apply. Every observation has an `outcome`: `success`, `failure`, or `neutral`. -- **Default to `neutral`** at observe time. Only claim `success` or `failure` when the evidence exists RIGHT NOW (tests ran, approach reverted, user confirmed). -- For decisions whose outcome is not yet provable, write `neutral`, keep the returned id, and close the loop later with `memory.record_use(id, outcome=...)` once validation arrives. -- Record failures at the same cadence as successes — the `dont` bucket depends on it. - -### Retrieval buckets - -`memory.retrieve` returns `{do, dont, neutral}` — distilled **reflections** -(generalised lessons from prior observations), bucketed by polarity. Each -bucket lists items with `{id, title, phase, use_cases, hints, confidence, -tech, evidence_count}`, ordered by `confidence DESC, updated_at DESC`, -capped at 20 per bucket. - -Treat `dont` as a hard constraint list: do not repeat approaches that -live there. - -For raw observation lookup (e.g. citing specific past events), use -`memory.retrieve_observations(project?, episode_id?, component?, theme?, -outcome?, query?, limit?)`. With `query`, results are ranked by hybrid -search; without, ordered newest-first. - -### MCP tools - -- `memory.observe(content, component?, theme?, trigger_type?, outcome?, tech?)` -- `memory.retrieve(project?, tech?, phase?, polarity?, limit_per_bucket?)` — reflections, bucketed by polarity -- `memory.retrieve_observations(project?, episode_id?, component?, theme?, outcome?, query?, limit?)` — raw observations -- `memory.record_use(id, outcome?)` -- `memory.credit(kind, id, class)` — **opportunistic per-tool-use rating.** When you actively use a memory retrieved this session (quote it, follow its guidance, or it misled you), call this immediately. Class is `cited`/`shaped`/`misled` (NOT `ignored`). The session-end sweep catches anything you don't credit, defaulting to `ignored`. Credit-as-you-go survives compaction. -- `memory.list_session_exposures()` / `memory.apply_session_ratings(ratings)` — end-of-session sweep tooling used by the `rate-session-memories` skill. -- `memory.start_episode(goal, tech?)` — declares a goal; returns `{episode_id, reflections}` -- `memory.close_episode(outcome, summary?, close_reason?)` — closes the active episode -- `memory.reconcile_episodes()` — list unclosed episodes from prior sessions -- `memory.list_episodes(project?, outcome?, only_open?)` — episodes for UI/inspection -- `memory.run_retention(retention_days?, prune?, prune_age_days?, dry_run?)` — apply spec §9 retention: flip stale observations to `status='archived'` per the three rules (linked-only-to-retired-reflection, consumed_without_reflection, no_outcome episode), optionally prune archived rows older than `prune_age_days`. Reflections are never auto-deleted. User-invoked; no auto-scheduling. -- `knowledge.search(query, project?)` -- `knowledge.list(project?)` +- **Default to `neutral`** at observe time. Only claim `success` or + `failure` when the evidence exists RIGHT NOW (tests ran, approach + reverted, user confirmed). +- For decisions whose outcome is not yet provable, write `neutral`, + keep the returned id, and close the loop later with `record_use` + once validation arrives. +- Record failures at the same cadence as successes — the `dont` bucket + depends on it. + +### Crediting memories you use + +When a retrieved memory shapes your work — you quote it, follow its +guidance, or it misleads you — credit it with a one-line evidence +statement when a memory shapes your work. Do this as you go rather +than batching it at session end; anything left uncredited is caught by +the end-of-session sweep, but fresh-context credit is more reliable. ## Session-start reconciliation -After the mandatory better-memory retrieve at session start, call -`memory.reconcile_episodes()` to check for episodes left open by prior -sessions. The tool returns a list of unclosed episodes, each with -`episode_id`, `project`, `tech`, `goal`, and `started_at`. +After the mandatory better-memory retrieve at session start, check for +episodes left open by prior sessions. Each unclosed episode carries a +goal, project, tech, and start time. **For each returned episode**, prompt the user in chat: @@ -65,8 +71,8 @@ sessions. The tool returns a list of unclosed episodes, each with > > How did it end? (success / abandoned / partial / no_outcome / continuing) -Apply the user's answer via `memory.close_episode(outcome=..., summary=...)` -— EXCEPT for `continuing`, which is a no-op at the service layer (the +Apply the user's answer by closing the episode with that outcome — +EXCEPT for `continuing`, which is a no-op at the service layer (the episode stays open and subsequent observations bind to it). If the user ignores the prompt or proceeds without answering, default to `abandoned` — it still feeds synthesis as a negative signal, so nothing is lost. @@ -119,16 +125,12 @@ including absence, is a no-op. ### Plan-complete close When the `superpowers:executing-plans` workflow (or any equivalent -multi-step plan run) finishes, close the active episode directly: - -``` -memory.close_episode(outcome="success", close_reason="plan_complete") -``` +multi-step plan run) finishes, close the active episode directly with +outcome `success` and close reason `plan_complete`. Do this INSTEAD of the commit trailer if the final commit of the plan doesn't itself map 1:1 to the plan's goal (e.g. the plan comprises several logically-separate commits and the final one isn't the "completion" commit). If the last commit of the plan already carries `Closes-Episode: true`, the plan-complete call is a no-op (the episode -is already closed) — still safe to call; the drain layer swallows the -no-active-episode ValueError. +is already closed) — still safe to call. diff --git a/tests/hooks/test_claude_md_sentinel.py b/tests/hooks/test_claude_md_sentinel.py new file mode 100644 index 0000000..5c951b5 --- /dev/null +++ b/tests/hooks/test_claude_md_sentinel.py @@ -0,0 +1,43 @@ +"""CLAUDE.md drift sentinel: prose that enumerates tool params rots. + +The 2026-07 incident: the user-level CLAUDE.md documented component/ +scope_path/window on memory_retrieve for weeks after they ceased to exist, +training every session to make silently-degraded calls. The rewrite removes +enumerations; the sentinel catches regressions. +""" +from __future__ import annotations + +from better_memory.hooks._claude_md_sentinel import build_schemas, check_claude_md + + +def test_phantom_param_detected(): + schemas = {"memory_retrieve": {"query", "project", "tech"}} + text = "call memory_retrieve with query and scope_path=src/" + warnings = check_claude_md(text, schemas) + assert warnings and "scope_path" in warnings[0] + + +def test_valid_params_silent(): + schemas = {"memory_retrieve": {"query", "project", "tech"}} + text = "call memory_retrieve with query='task' and project=x" + assert check_claude_md(text, schemas) == [] + + +def test_lines_without_tool_names_ignored(): + schemas = {"memory_retrieve": {"query"}} + assert check_claude_md("window=30d is a fine phrase alone", schemas) == [] + + +def test_common_words_not_flagged(): + schemas = {"memory_retrieve": {"query"}} + # 'e.g.' / 'i.e.' style tokens and words without = or : suffix don't count + assert check_claude_md("memory_retrieve is documented here", schemas) == [] + + +def test_build_schemas_covers_retrieve(): + schemas = build_schemas() + assert "query" in schemas["memory_retrieve"] + + +def test_malformed_input_never_raises(): + assert check_claude_md("", {}) == [] From fbaba3360643965f981470bde2379dc05602902e Mon Sep 17 00:00:00 2001 From: gethin Date: Fri, 24 Jul 2026 00:33:44 +0100 Subject: [PATCH 12/17] fix(sentinel): catch prose-style param enumeration; snippet consistency fixes Co-Authored-By: Claude Fable 5 --- better_memory/hooks/_claude_md_sentinel.py | 26 ++++++++++++++++----- better_memory/skills/CLAUDE.snippet.md | 17 +++++++------- tests/hooks/test_claude_md_sentinel.py | 27 ++++++++++++++++++++++ tests/hooks/test_session_bootstrap.py | 8 +++++++ 4 files changed, 64 insertions(+), 14 deletions(-) diff --git a/better_memory/hooks/_claude_md_sentinel.py b/better_memory/hooks/_claude_md_sentinel.py index c9a2351..24ab46b 100644 --- a/better_memory/hooks/_claude_md_sentinel.py +++ b/better_memory/hooks/_claude_md_sentinel.py @@ -1,9 +1,14 @@ """Detect parameter-enumeration drift in the user's CLAUDE.md. Pure functions; the session_bootstrap hook wires them in best-effort. -Only lines that mention a better-memory tool name are scanned, and only -tokens shaped like parameter usage (word= or word:) are checked against -the live schema, so prose can't false-positive. +Only lines that mention a better-memory tool name are scanned. Two token +shapes are checked against the live schema, so prose can't false-positive +on unrelated words: + +- `word=` / `word:` — explicit parameter-usage syntax. +- `` `word` `` — a backtick-quoted snake_case identifier, the shape real + CLAUDE.md prose uses when enumerating params conversationally (e.g. + "optional `component` ... and `scope_path`"). """ from __future__ import annotations @@ -11,6 +16,7 @@ import re _PARAM_RE = re.compile(r"\b([a-z_][a-z0-9_]{2,})\s*[=:]") +_BACKTICK_RE = re.compile(r"`([a-z_]{4,})`") _IGNORE = {"http", "https", "note", "example", "warning", "default"} @@ -31,10 +37,18 @@ def check_claude_md(text: str, schemas: dict[str, set[str]]) -> list[str]: hit_tools = [name for name in schemas if name in line] if not hit_tools: continue - for token in _PARAM_RE.findall(line): - if token in _IGNORE or any(token in schemas[t] for t in hit_tools): + valid: set[str] = set() + for t in hit_tools: + valid |= schemas[t] + candidates = _PARAM_RE.findall(line) + _BACKTICK_RE.findall(line) + seen: set[str] = set() + for token in candidates: + if token in seen: continue - if token in schemas: # a tool name followed by ':' etc. + seen.add(token) + if token in _IGNORE or token in valid: + continue + if token in schemas: # a tool name itself, not a param continue warnings.append( f"CLAUDE.md documents parameter '{token}' near " diff --git a/better_memory/skills/CLAUDE.snippet.md b/better_memory/skills/CLAUDE.snippet.md index d31bf3c..1833223 100644 --- a/better_memory/skills/CLAUDE.snippet.md +++ b/better_memory/skills/CLAUDE.snippet.md @@ -43,8 +43,8 @@ Every observation has an `outcome`: `success`, `failure`, or `neutral`. `failure` when the evidence exists RIGHT NOW (tests ran, approach reverted, user confirmed). - For decisions whose outcome is not yet provable, write `neutral`, - keep the returned id, and close the loop later with `record_use` - once validation arrives. + keep the returned id, and close the loop later with + `memory_record_use` once validation arrives. - Record failures at the same cadence as successes — the `dont` bucket depends on it. @@ -52,15 +52,16 @@ Every observation has an `outcome`: `success`, `failure`, or `neutral`. When a retrieved memory shapes your work — you quote it, follow its guidance, or it misleads you — credit it with a one-line evidence -statement when a memory shapes your work. Do this as you go rather -than batching it at session end; anything left uncredited is caught by -the end-of-session sweep, but fresh-context credit is more reliable. +statement. Do this as you go rather than batching it at session end; +anything left uncredited is caught by the end-of-session sweep, but +fresh-context credit is more reliable. ## Session-start reconciliation -After the mandatory better-memory retrieve at session start, check for -episodes left open by prior sessions. Each unclosed episode carries a -goal, project, tech, and start time. +At session start, after the mandatory `knowledge_list` call and before +task-specific `memory_retrieve` calls begin, check for episodes left +open by prior sessions. Each unclosed episode carries a goal, project, +tech, and start time. **For each returned episode**, prompt the user in chat: diff --git a/tests/hooks/test_claude_md_sentinel.py b/tests/hooks/test_claude_md_sentinel.py index 5c951b5..d1680e8 100644 --- a/tests/hooks/test_claude_md_sentinel.py +++ b/tests/hooks/test_claude_md_sentinel.py @@ -41,3 +41,30 @@ def test_build_schemas_covers_retrieve(): def test_malformed_input_never_raises(): assert check_claude_md("", {}) == [] + + +# --- Real 2026-07 incident lines (verbatim), prose-style backtick enumeration --- + + +def test_prose_style_phantom_param_detected(): + schemas = {"memory_retrieve": {"query", "project", "tech"}} + text = ( + "- `mcp__better-memory__memory_retrieve` with a broad `query` " + "describing the project or task area. ... optional `component` " + "(subsystem/module) and `scope_path` (file or directory)" + ) + warnings = check_claude_md(text, schemas) + assert warnings + assert "component" in warnings[0] or "scope_path" in warnings[0] + + +def test_prose_style_no_tool_name_not_flagged(): + schemas = {"memory_retrieve": {"query"}} + text = "Tune `window` (default `30d`)" + assert check_claude_md(text, schemas) == [] + + +def test_prose_style_valid_param_not_flagged(): + schemas = {"memory_retrieve": {"query"}} + text = "call `memory_retrieve` with a `query` describing it" + assert check_claude_md(text, schemas) == [] diff --git a/tests/hooks/test_session_bootstrap.py b/tests/hooks/test_session_bootstrap.py index b681d0b..9ad806c 100644 --- a/tests/hooks/test_session_bootstrap.py +++ b/tests/hooks/test_session_bootstrap.py @@ -336,6 +336,14 @@ def test_agentcore_mode_does_not_open_sqlite_connection( proj.mkdir() monkeypatch.setenv("BETTER_MEMORY_HOME", str(home)) monkeypatch.setenv("BETTER_MEMORY_STORAGE_BACKEND", "agentcore") + # Hermetic against the CLAUDE.md drift sentinel: point Path.home() at an + # empty dir so no real ~/.claude/CLAUDE.md is picked up (the sentinel + # skips silently when the file is missing) and the exact-equality + # assertion below on the remote backend's passthrough context holds + # regardless of the machine's real CLAUDE.md contents. + no_home = tmp_path / "no-home" + monkeypatch.setenv("USERPROFILE", str(no_home)) + monkeypatch.setenv("HOME", str(no_home)) connect_calls: list = [] monkeypatch.setattr( From ba641c335900c9702acb3f1ffed6438862f8498b Mon Sep 17 00:00:00 2001 From: gethin Date: Fri, 24 Jul 2026 00:40:07 +0100 Subject: [PATCH 13/17] fix(sentinel): schema-enum exemption + param-signal precision for backtick detection Co-Authored-By: Claude Fable 5 --- better_memory/hooks/_claude_md_sentinel.py | 41 +++++++++++++++++++--- tests/hooks/test_claude_md_sentinel.py | 37 +++++++++++++++++++ 2 files changed, 73 insertions(+), 5 deletions(-) diff --git a/better_memory/hooks/_claude_md_sentinel.py b/better_memory/hooks/_claude_md_sentinel.py index 24ab46b..2405cf8 100644 --- a/better_memory/hooks/_claude_md_sentinel.py +++ b/better_memory/hooks/_claude_md_sentinel.py @@ -8,7 +8,16 @@ - `word=` / `word:` — explicit parameter-usage syntax. - `` `word` `` — a backtick-quoted snake_case identifier, the shape real CLAUDE.md prose uses when enumerating params conversationally (e.g. - "optional `component` ... and `scope_path`"). + "optional `component` ... and `scope_path`"). To avoid flagging plain + identifier mentions (e.g. "called from `session_bootstrap` module") this + branch only fires on lines that also carry a parameter-signal word (see + `_SIGNAL_WORDS`). + +The accepted-token set per tool is schema-derived, not just property +names: enum *values* (e.g. `memory_retrieve`'s `polarity` enum +`do`/`dont`/`neutral`) are valid tokens too — those are documented return +shapes, not phantom parameters, and real CLAUDE.md prose legitimately +backtick-quotes them right next to the tool name. """ from __future__ import annotations @@ -17,7 +26,15 @@ _PARAM_RE = re.compile(r"\b([a-z_][a-z0-9_]{2,})\s*[=:]") _BACKTICK_RE = re.compile(r"`([a-z_]{4,})`") -_IGNORE = {"http", "https", "note", "example", "warning", "default"} +_IGNORE = {"http", "https", "note", "example", "warning", "default", "docs"} + +# Words/phrases whose presence on a line signals "this backtick token is +# being documented as a parameter", as opposed to a plain identifier or +# module mention. Checked case-insensitively as substrings. +_SIGNAL_WORDS = ( + "optional", "parameter", "param", "pass", "passing", "tune", + "filter", "argument", "arg", "field", "defaults", "set ", +) def build_schemas() -> dict[str, set[str]]: @@ -25,11 +42,23 @@ def build_schemas() -> dict[str, set[str]]: out: dict[str, set[str]] = {} for tool in tool_definitions(): rendered = tool.name.replace(".", "_") - props = set((tool.inputSchema or {}).get("properties", {}).keys()) - out[rendered] = props + properties = (tool.inputSchema or {}).get("properties", {}) or {} + tokens: set[str] = set() + for prop_name, spec in properties.items(): + tokens.add(prop_name) + if isinstance(spec, dict): + enum_values = spec.get("enum") + if isinstance(enum_values, list): + tokens.update(v for v in enum_values if isinstance(v, str)) + out[rendered] = tokens return out +def _has_signal_word(line: str) -> bool: + lowered = line.lower() + return any(word in lowered for word in _SIGNAL_WORDS) + + def check_claude_md(text: str, schemas: dict[str, set[str]]) -> list[str]: warnings: list[str] = [] try: @@ -40,7 +69,9 @@ def check_claude_md(text: str, schemas: dict[str, set[str]]) -> list[str]: valid: set[str] = set() for t in hit_tools: valid |= schemas[t] - candidates = _PARAM_RE.findall(line) + _BACKTICK_RE.findall(line) + candidates = list(_PARAM_RE.findall(line)) + if _has_signal_word(line): + candidates += _BACKTICK_RE.findall(line) seen: set[str] = set() for token in candidates: if token in seen: diff --git a/tests/hooks/test_claude_md_sentinel.py b/tests/hooks/test_claude_md_sentinel.py index d1680e8..528153d 100644 --- a/tests/hooks/test_claude_md_sentinel.py +++ b/tests/hooks/test_claude_md_sentinel.py @@ -68,3 +68,40 @@ def test_prose_style_valid_param_not_flagged(): schemas = {"memory_retrieve": {"query"}} text = "call `memory_retrieve` with a `query` describing it" assert check_claude_md(text, schemas) == [] + + +# --- Round 3: schema-enum exemption + param-signal precision --- + + +def test_bucket_line_with_enum_values_not_flagged(): + # Real line: `do`/`dont`/`neutral` are memory_retrieve's polarity ENUM + # VALUES (documented return shape), not phantom parameters. No signal + # word on this line either, so the backtick branch wouldn't even fire — + # but the schema also carries the enum values as valid tokens as a + # second line of defense. + schemas = {"memory_retrieve": {"query", "do", "dont", "neutral"}} + text = ( + "- `mcp__better-memory__memory_retrieve` with a broad `query` " + "describing the project or task area. Returns observations " + "bucketed by outcome (`do`, `dont`, `neutral`)." + ) + assert check_claude_md(text, schemas) == [] + + +def test_identifier_mention_not_flagged(): + schemas = {"memory_retrieve": {"query"}} + text = "memory_retrieve is called from `session_bootstrap` module" + assert check_claude_md(text, schemas) == [] + + +def test_docs_colon_not_flagged(): + schemas = {"memory_retrieve": {"query"}} + text = "memory_retrieve docs: see https://x" + assert check_claude_md(text, schemas) == [] + + +def test_build_schemas_includes_enum_values(): + schemas = build_schemas() + # memory.retrieve's `polarity` property has enum ["do", "dont", "neutral"]. + for value in ("do", "dont", "neutral"): + assert value in schemas["memory_retrieve"] From 7f130c95e3a1841171b70e5f9a49564214300da3 Mon Sep 17 00:00:00 2001 From: gethin Date: Fri, 24 Jul 2026 00:45:19 +0100 Subject: [PATCH 14/17] fix(sentinel): word-boundary signal matching, trim signal list Co-Authored-By: Claude Fable 5 --- better_memory/hooks/_claude_md_sentinel.py | 20 +++++++++++--------- tests/hooks/test_claude_md_sentinel.py | 19 +++++++++++++++++++ 2 files changed, 30 insertions(+), 9 deletions(-) diff --git a/better_memory/hooks/_claude_md_sentinel.py b/better_memory/hooks/_claude_md_sentinel.py index 2405cf8..601cedd 100644 --- a/better_memory/hooks/_claude_md_sentinel.py +++ b/better_memory/hooks/_claude_md_sentinel.py @@ -11,7 +11,7 @@ "optional `component` ... and `scope_path`"). To avoid flagging plain identifier mentions (e.g. "called from `session_bootstrap` module") this branch only fires on lines that also carry a parameter-signal word (see - `_SIGNAL_WORDS`). + `_SIGNAL_RE`). The accepted-token set per tool is schema-derived, not just property names: enum *values* (e.g. `memory_retrieve`'s `polarity` enum @@ -28,12 +28,15 @@ _BACKTICK_RE = re.compile(r"`([a-z_]{4,})`") _IGNORE = {"http", "https", "note", "example", "warning", "default", "docs"} -# Words/phrases whose presence on a line signals "this backtick token is -# being documented as a parameter", as opposed to a plain identifier or -# module mention. Checked case-insensitively as substrings. -_SIGNAL_WORDS = ( - "optional", "parameter", "param", "pass", "passing", "tune", - "filter", "argument", "arg", "field", "defaults", "set ", +# Words whose presence on a line signal "this backtick token is being +# documented as a parameter", as opposed to a plain identifier or module +# mention. Matched case-insensitively on word boundaries — a naive +# substring check would match "set" inside "reset"/"preset"/"subset". +# "field" and "defaults" were dropped: both false-positived on unrelated +# backticked identifiers and no incident fixture depends on either. +_SIGNAL_RE = re.compile( + r"\b(optional|parameter|param|pass|passing|tune|filter|argument|arg|set)\b", + re.IGNORECASE, ) @@ -55,8 +58,7 @@ def build_schemas() -> dict[str, set[str]]: def _has_signal_word(line: str) -> bool: - lowered = line.lower() - return any(word in lowered for word in _SIGNAL_WORDS) + return _SIGNAL_RE.search(line) is not None def check_claude_md(text: str, schemas: dict[str, set[str]]) -> list[str]: diff --git a/tests/hooks/test_claude_md_sentinel.py b/tests/hooks/test_claude_md_sentinel.py index 528153d..6b9fc86 100644 --- a/tests/hooks/test_claude_md_sentinel.py +++ b/tests/hooks/test_claude_md_sentinel.py @@ -105,3 +105,22 @@ def test_build_schemas_includes_enum_values(): # memory.retrieve's `polarity` property has enum ["do", "dont", "neutral"]. for value in ("do", "dont", "neutral"): assert value in schemas["memory_retrieve"] + + +# --- Round 4: word-boundary signal matching, trimmed signal list --- + + +def test_set_inside_reset_not_a_signal_word(): + # Substring matching would have matched "set" inside "reset"; word + # boundaries must not. + schemas = {"memory_retrieve": {"query"}} + text = "reset the `cache` before calling `memory_retrieve` again" + assert check_claude_md(text, schemas) == [] + + +def test_field_no_longer_a_signal_word(): + # "field" was dropped from the signal list — a backticked identifier + # near a "field" mention must not be flagged. + schemas = {"memory_retrieve": {"query"}} + text = "after calling `memory_retrieve`, edit the `hints` field in the UI drawer" + assert check_claude_md(text, schemas) == [] From 8df5eb667b00a0b447d05fa33342ab6b525cb1d4 Mon Sep 17 00:00:00 2001 From: gethin Date: Fri, 24 Jul 2026 00:59:57 +0100 Subject: [PATCH 15/17] docs(website): deferred-injection prose + config keys Sync website/architecture.md, configuration.md, mcp-tools.md with the feat/deferred-injection branch: BETTER_MEMORY_INJECT_MODE (legacy default | deferred bootstrap), the three-leg evidence-gated contextual scorer (BM25 / vector cosine against BETTER_MEMORY_CONTEXT_VEC_FLOOR / keyword fallback), deprecate BETTER_MEMORY_CONTEXT_MIN_HITS, document the PreToolUse-widened-to-all-tools once-per-session latch, the file-persisted Ollama outage cooldown shared across hook processes, and via_exploration exposure tagging excluded from the headline usefulness metric. --- website/architecture.md | 34 ++++++++++++++++++++++++---------- website/configuration.md | 14 ++++++++------ website/mcp-tools.md | 5 ++++- 3 files changed, 36 insertions(+), 17 deletions(-) diff --git a/website/architecture.md b/website/architecture.md index c53a21c..44816f3 100644 --- a/website/architecture.md +++ b/website/architecture.md @@ -58,18 +58,22 @@ See [Configuration](configuration.md) for env vars and [AgentCore setup](agentco ## Injection strategies -better-memory gets memory in front of Claude two ways: a slimmed-down dump at session start, and targeted mid-session injection keyed to what Claude is actually doing. +better-memory gets memory in front of Claude two ways: a dump at session start, and targeted mid-session injection keyed to what Claude is actually doing. -**Bootstrap (SessionStart).** `SessionBootstrapService.bootstrap` (`better_memory/services/session_bootstrap.py`) renders project-scoped semantic memories and reflections in full only up to `BETTER_MEMORY_BOOTSTRAP_TOP_N` per set (default 5; general-scope semantic memories are always shown in full, uncapped). The remainder collapses into a one-line `### Index (not expanded - retrieve on demand)` section plus a footer affordance pointing at `memory.retrieve` / `memory.retrieve_observations`. Semantic memory ids in the rendered output are the full ids (not truncated), each stamped with an age suffix (`(Nd old)`). Setting `BETTER_MEMORY_BOOTSTRAP_TOP_N=0` disables slimming and renders everything in full (legacy behavior). +**Bootstrap (SessionStart).** Governed by `BETTER_MEMORY_INJECT_MODE` (`legacy` | `deferred`; default `legacy`): -**Contextual injection (UserPromptSubmit / PreToolUse).** The `contextual_inject` hook (`better_memory/hooks/contextual_inject.py`) scores the same curated memory set against the current prompt (UserPromptSubmit) or tool name + input (PreToolUse, matcher `Skill|Task|Write`) via `retrieve_relevant` (`better_memory/services/relevant.py`): +- **`legacy`** (default, byte-identical to pre-deferred-injection behavior). `SessionBootstrapService.bootstrap` (`better_memory/services/session_bootstrap.py`) renders project-scoped semantic memories and reflections in full only up to `BETTER_MEMORY_BOOTSTRAP_TOP_N` per set (default 5; general-scope semantic memories are always shown in full, uncapped). The remainder collapses into a one-line `### Index (not expanded - retrieve on demand)` section plus a footer affordance pointing at `memory.retrieve` / `memory.retrieve_observations`. Semantic memory ids in the rendered output are the full ids (not truncated), each stamped with an age suffix (`(Nd old)`). Setting `BETTER_MEMORY_BOOTSTRAP_TOP_N=0` disables slimming and renders everything in full. +- **`deferred`**. SessionStart renders only general-scope semantic memories in full, plus a single index line ("better-memory knows N reflections + M semantic memories for this project; relevant ones will surface as you work - or ask via memory_retrieve with a task query"). Project-scoped semantic memories and all reflections are not dumped at session start; they surface exclusively through the contextual channel below, or on demand via `memory.retrieve` / `memory.retrieve_observations`. -- Keywords are extracted from the query (lowercased, tokenised, stopwords and <3-char tokens dropped). -- Each candidate's score is `(distinct keyword hits + title hits) × activation`, where activation grows with `useful_count` and confidence and is halved when a memory has misled more often than it has helped. -- Candidates below `BETTER_MEMORY_CONTEXT_MIN_HITS` distinct hits are dropped; survivors are ranked by score and capped to `BETTER_MEMORY_CONTEXT_MAX_ITEMS`. -- A per-session `SeenStore` (`better_memory/services/context_seen.py`, JSON file at `/state/context_seen_.json`) deduplicates memories already injected this session. `BETTER_MEMORY_CONTEXT_REINJECT_TURNS=0` (default) means a memory is injected at most once per session; a positive value allows re-injection after that many turns have passed since it was last shown. A "turn" here is one firing of the `contextual_inject` hook, not one user prompt-response cycle: each user prompt is a turn, and in mode `both` each matched tool call (`Skill`, `Task`, `Write`) is a separate turn too. +**Contextual injection (UserPromptSubmit / PreToolUse).** The `contextual_inject` hook (`better_memory/hooks/contextual_inject.py`) scores the curated memory set (semantic + reflections) against the current prompt (UserPromptSubmit, fires on every prompt) or tool name + input (PreToolUse, matcher now unscoped -- every tool call, not a fixed allowlist) via `retrieve_relevant` (`better_memory/services/relevant.py`), a three-leg evidence-gated scorer: + +- A memory injects only when it clears an evidence gate: a BM25 match against `reflection_fts` (title / use_cases / hints), or a vector cosine similarity >= `BETTER_MEMORY_CONTEXT_VEC_FLOOR` (default `0.55`) against its embedding. No evidence, no injection -- a memory with neither leg present is silently dropped, however popular it is. +- The Wilson lower-bound prior on rated exposures (see [Self-rating loop](#self-rating-loop)) never qualifies a memory by itself; among qualifiers it only ranks, via reciprocal rank fusion together with whichever of the BM25/vector ranks are present. Popularity forcing irrelevant injections into context was the old failure mode the gate exists to close. +- A keyword-hit fallback (>= 2 distinct hits) applies only where the BM25/vector legs are structurally unavailable: for reflections, when there is no raw sqlite `conn` (agentcore mode); for semantic memories -- which have no FTS substrate at all -- whenever no query vector could be produced (no embedder configured, or the Ollama embed call is in cooldown). `BETTER_MEMORY_CONTEXT_MIN_HITS` is **deprecated**: `contextual_inject` no longer reads it, superseded by the evidence gate above. +- Because PreToolUse now matches every tool, a per-session latch (`SeenStore.pretool_fired` / `mark_pretool_fired`, `better_memory/services/context_seen.py`) runs the full retrieval path at most once per session for PreToolUse; every later PreToolUse event in the session short-circuits on the latch state before touching the DB or the embedder. UserPromptSubmit is unaffected by the latch and fires on every prompt. +- Survivors are capped to `BETTER_MEMORY_CONTEXT_MAX_ITEMS`, then filtered through a per-session `SeenStore` (`better_memory/services/context_seen.py`, JSON file at `/state/context_seen_.json`) that deduplicates memories already injected this session. `BETTER_MEMORY_CONTEXT_REINJECT_TURNS=0` (default) means a memory is injected at most once per session; a positive value allows re-injection after that many turns have passed since it was last shown. A "turn" here is one firing of the `contextual_inject` hook, not one user prompt-response cycle: each user prompt is a turn, and each PreToolUse latch-firing is a turn too (subsequent latched-out PreToolUse events do not bump the turn counter). - Survivors render as a `` XML block in `additionalContext`, one entry per item with its kind, id, confidence, `useful_count`, and age, a `dont`-polarity item prefixed `Known pitfall -- do this instead:`, and a footer inviting `memory_credit(kind, id, 'cited'|'shaped'|'misled')` when an entry actually helped or misled. -- Survivors are logged to `session_memory_exposure` with `source='contextual'` (best-effort; a write failure never blocks injection) and counted in `rating_diagnostics` (`contextual_fired_userprompt`, `contextual_fired_pretool`, `contextual_injected`, `contextual_suppressed_floor`, `contextual_suppressed_dedup`). These counters are per-firing, not per-item: a firing that injects one or several memories still increments `contextual_injected` by exactly 1. +- Survivors are logged to `session_memory_exposure` with `source='contextual'` (best-effort; a write failure never blocks injection) and counted in `rating_diagnostics` (`contextual_fired_userprompt`, `contextual_fired_pretool`, `contextual_injected`, `contextual_suppressed_floor`, `contextual_suppressed_dedup`). These counters are per-firing, not per-item: a firing that injects one or several memories still increments `contextual_injected` by exactly 1. `contextual_suppressed_floor` now means "no candidate cleared the evidence gate", not "below the old keyword-hits floor". Gated by `BETTER_MEMORY_CONTEXT_INJECT_MODE` (`userprompt` / `pretool` / `both` / `off`). The hook never raises and always exits 0 — failures are swallowed and logged to `hook_errors`, with no `additionalContext` emitted on that turn. @@ -112,7 +116,12 @@ through `SyncEmbedder` (`better_memory/embeddings/sync_embed.py`), a synchronous facade over the Ollama embedder that is best-effort by design: any failure opens a 60-second circuit breaker (`_DEFAULT_COOLDOWN`) during which every embed call returns `None` immediately instead of -retrying against a dead Ollama. A missing vector is never treated as an +retrying against a dead Ollama. In `contextual_inject`, that cooldown is +additionally persisted to a state file (`/state/embed_down_until`) +so an Ollama outage is paid for once, not once per hook process -- every +`contextual_inject` invocation started while the file's deadline is still +in the future skips the embed call outright instead of re-discovering the +outage itself. A missing vector is never treated as an error - callers just drop the vector leg from ranking and fall back to the Wilson + BM25 order above. @@ -180,7 +189,12 @@ captures whether memories actually shaped Claude's work: outright to proven rows it competes for a reserved exploration slot: the last slot of each polarity bucket (when the bucket cap allows at least 2 entries) is set aside for the highest-ranked - under-rated memory, if one exists. When `memory.retrieve` is called + under-rated memory, if one exists. That serve is tagged + `via_exploration=1` on its `session_memory_exposure` row (migration + `0015_via_exploration.sql`) -- it's an investment the ranker makes to + earn the memory a rating, not a relevance claim, so it is excluded + from the headline usefulness metric while still being rated normally + through the same self-rating loop. When `memory.retrieve` is called with a `query`, this Wilson-ranked list is re-fused with a BM25 relevance ranking (title / use_cases / hints) and a vector-kNN ranking via the same Reciprocal Rank Fusion used for observations -- diff --git a/website/configuration.md b/website/configuration.md index 7f01a73..ac99b10 100644 --- a/website/configuration.md +++ b/website/configuration.md @@ -17,10 +17,12 @@ One environment variable roots the runtime filesystem layout. Everything else ha | `BETTER_MEMORY_TEST_AGENTCORE` | unset | `1` enables integration tests against real AWS. Default off; never set in CI. | | `BETTER_MEMORY_TEST_AGENTCORE_REGION` | inherits `eu-west-2` | Override region used by integration tests. | | `BETTER_MEMORY_CONTEXT_INJECT_MODE` | `both` | Contextual memory-injection hook trigger: `userprompt` (on prompt only), `pretool` (on tool calls only), `both` (default), or `off`. The `contextual_inject` hook surfaces curated memories (semantic + reflections) relevant to the current prompt / tool-input. | -| `BETTER_MEMORY_BOOTSTRAP_TOP_N` | `5` | Number of project-scoped semantic memories and reflections the SessionStart bootstrap renders in full (beyond that, a one-line index plus a retrieve affordance). `0` disables slimming and dumps everything in full (legacy behavior). | -| `BETTER_MEMORY_CONTEXT_MIN_HITS` | `2` | Minimum number of distinct keyword hits a memory must clear against the current prompt / tool-input before `contextual_inject` will surface it. | -| `BETTER_MEMORY_CONTEXT_MAX_ITEMS` | `3` | Max number of memories `contextual_inject` injects per firing, after the min-hits floor and ranking. | -| `BETTER_MEMORY_CONTEXT_REINJECT_TURNS` | `0` | Turns to wait before `contextual_inject` will re-inject a memory already seen this session. `0` means never re-inject (each memory surfaces at most once per session). A "turn" here means one firing of the `contextual_inject` hook, not one user prompt-response cycle: each user prompt counts as a turn, and in mode `both` each matched tool call (`Skill`, `Task`, `Write`) counts as a separate turn too. | +| `BETTER_MEMORY_INJECT_MODE` | `legacy` | SessionStart bootstrap strategy: `legacy` (default, byte-identical to pre-deferred-injection behavior -- full bootstrap dump gated only by `BETTER_MEMORY_BOOTSTRAP_TOP_N`) or `deferred` (bootstrap renders only general-scope semantic memories plus a one-line index; project-scoped semantic memories and all reflections surface only through the contextual channel or on-demand `memory.retrieve`). Any value other than `deferred` resolves to `legacy`. See [Architecture](architecture.md#injection-strategies). | +| `BETTER_MEMORY_BOOTSTRAP_TOP_N` | `5` | Number of project-scoped semantic memories and reflections the SessionStart bootstrap renders in full (beyond that, a one-line index plus a retrieve affordance). `0` disables slimming and dumps everything in full. Only applies in `BETTER_MEMORY_INJECT_MODE=legacy`. | +| `BETTER_MEMORY_CONTEXT_MIN_HITS` | `2` | **Deprecated, unused.** `contextual_inject` no longer reads this. It was the old minimum-keyword-hits floor, superseded by the three-leg evidence gate (BM25 / vector cosine / keyword fallback) described in [Architecture](architecture.md#injection-strategies). Kept only for back-compat resolution of stray env overrides. | +| `BETTER_MEMORY_CONTEXT_VEC_FLOOR` | `0.55` | Minimum vector cosine similarity a memory's embedding must clear against the query embedding to qualify as vector-leg evidence in `contextual_inject`'s evidence gate. Clamped to `[0.0, 1.0]`; invalid values fall back to the default. See [Architecture](architecture.md#injection-strategies). | +| `BETTER_MEMORY_CONTEXT_MAX_ITEMS` | `3` | Max number of memories `contextual_inject` injects per firing, after the evidence gate and ranking. | +| `BETTER_MEMORY_CONTEXT_REINJECT_TURNS` | `0` | Turns to wait before `contextual_inject` will re-inject a memory already seen this session. `0` means never re-inject (each memory surfaces at most once per session). A "turn" here means one firing of the `contextual_inject` hook, not one user prompt-response cycle: each user prompt counts as a turn, and PreToolUse counts as at most one more turn per session (it's latched to a single real firing -- see [Architecture](architecture.md#injection-strategies)). | !!! note "Removed: `BETTER_MEMORY_AGENTCORE_REGION`" The region env var is gone. The AWS region is single-sourced from `agentcore.json` (written by `better-memory agentcore init --region `); a differing env value could only produce a split-brain where events were written to a region the memories don't live in. If you had it exported, remove it — it is ignored. To change region, re-run `init` (see [AgentCore troubleshooting](troubleshooting/agentcore.md)). @@ -72,9 +74,9 @@ The two SQLite files are never shared between processes — the MCP server owns Four Claude Code hooks ship with better-memory and read or write the filesystem layout above. They are installed automatically by `./scripts/setup.sh` (which calls `python -m better_memory.cli.install_hooks` to merge them idempotently into `~/.claude/settings.json`). The list below is reference material: -- **`better_memory.hooks.session_bootstrap`** (SessionStart) — opens or reuses a background episode for the session and injects the project's curated context (project-scoped and general-scope semantic memories plus distilled reflections in `do` / `dont` / `neutral` buckets, retrieved up to 20 per bucket and ranked by a Wilson-score lower bound on positive-rated exposures, then confidence, then recency, with the bucket's last slot reserved for an under-rated memory - fewer than 3 rated exposures - when one exists; see [Architecture](architecture.md#self-rating-loop)) as `additionalContext` for Claude's first turn. Only the top `BETTER_MEMORY_BOOTSTRAP_TOP_N` project-scoped items (default 5; general-scope semantic memories are always shown in full) render in full; the rest collapse into a one-line index plus a `memory.retrieve` / `memory.retrieve_observations` affordance. Setting `BETTER_MEMORY_BOOTSTRAP_TOP_N=0` restores the legacy full-dump behavior. Runs in-process against `memory.db` (in agentcore mode it routes through the storage backend instead and never opens the local database); failure-isolated: if bootstrap breaks, a fallback directive is injected and the failure is recorded in the `hook_errors` table. +- **`better_memory.hooks.session_bootstrap`** (SessionStart) — opens or reuses a background episode for the session and injects the project's curated context as `additionalContext` for Claude's first turn. Shape depends on `BETTER_MEMORY_INJECT_MODE`: in `legacy` (default) it's project-scoped and general-scope semantic memories plus distilled reflections in `do` / `dont` / `neutral` buckets (retrieved up to 20 per bucket and ranked by a Wilson-score lower bound on positive-rated exposures, then confidence, then recency, with the bucket's last slot reserved for an under-rated memory - fewer than 3 rated exposures - when one exists; see [Architecture](architecture.md#self-rating-loop)), with only the top `BETTER_MEMORY_BOOTSTRAP_TOP_N` project-scoped items (default 5; general-scope semantic memories are always shown in full) rendered in full and the rest collapsed into a one-line index plus a `memory.retrieve` / `memory.retrieve_observations` affordance (`BETTER_MEMORY_BOOTSTRAP_TOP_N=0` restores the full dump). In `deferred` it's general-scope semantic memories in full plus a single one-line index -- project-scoped semantic memories and reflections surface only through `contextual_inject` or on-demand `memory.retrieve`. Runs in-process against `memory.db` (in agentcore mode it routes through the storage backend instead and never opens the local database); failure-isolated: if bootstrap breaks, a fallback directive is injected and the failure is recorded in the `hook_errors` table. - **`better_memory.hooks.observer`** (PostToolUse) — captures tool-call snapshots into `spool/` for later observation creation. - **`better_memory.hooks.session_close`** (Stop) — writes a session-close marker into `spool/`; also emits the rating directive described in [Self-rating loop](architecture.md#self-rating-loop). In agentcore mode (resolved via the env var, else `settings.json`) it additionally fires one `CreateEvent(role=OTHER)` closure event against the current AgentCore session so the episodic strategy extracts within minutes; failure is logged to `hook_errors` and never blocks the marker write. -- **`better_memory.hooks.contextual_inject`** (UserPromptSubmit + PreToolUse) — scores the curated memory set (semantic + reflections) against the current prompt or tool-input via keyword-hit x activation, drops candidates below `BETTER_MEMORY_CONTEXT_MIN_HITS`, caps survivors to `BETTER_MEMORY_CONTEXT_MAX_ITEMS`, and injects them as a `` block in `additionalContext`. A per-session seen-file dedups repeats (`BETTER_MEMORY_CONTEXT_REINJECT_TURNS` controls re-injection). Gated by `BETTER_MEMORY_CONTEXT_INJECT_MODE`. See [Architecture](architecture.md#injection-strategies) for detail. +- **`better_memory.hooks.contextual_inject`** (UserPromptSubmit, every prompt; PreToolUse, now every tool call, latched to one real firing per session) — scores the curated memory set (semantic + reflections) against the current prompt or tool-input through a three-leg evidence gate (BM25 match, vector cosine >= `BETTER_MEMORY_CONTEXT_VEC_FLOOR`, or a keyword-hit fallback when those legs are structurally unavailable; `BETTER_MEMORY_CONTEXT_MIN_HITS` is deprecated and no longer consulted), caps survivors to `BETTER_MEMORY_CONTEXT_MAX_ITEMS`, and injects them as a `` block in `additionalContext`. A per-session seen-file dedups repeats (`BETTER_MEMORY_CONTEXT_REINJECT_TURNS` controls re-injection). Gated by `BETTER_MEMORY_CONTEXT_INJECT_MODE`. See [Architecture](architecture.md#injection-strategies) for detail. See [`README.md`](https://github.com/emp3thy/better-memory/blob/main/README.md#manual-setup) for the exact `~/.claude/settings.json` registration JSON. diff --git a/website/mcp-tools.md b/website/mcp-tools.md index 1ec1311..10751c9 100644 --- a/website/mcp-tools.md +++ b/website/mcp-tools.md @@ -120,7 +120,10 @@ Spawn or reuse the management UI. Returns `{"url": "...", "reused": bool}`. ### `memory.session_bootstrap` -Open or reuse a session episode and inject project + general semantic memories and reflections as `additionalContext` markdown. Reflections are retrieved up to 20 per polarity bucket (`do` / `dont` / `neutral`), ranked by Wilson-prior lower bound on hit-rate, ties by confidence. Only the top `BETTER_MEMORY_BOOTSTRAP_TOP_N` project-scoped items (default 5; general-scope semantic memories always render in full) are shown in full — the rest collapse into a one-line index plus a retrieve affordance (`BETTER_MEMORY_BOOTSTRAP_TOP_N=0` restores the legacy full dump). Mirrors what the SessionStart hook does; callable manually for recovery, testing, or post-`/clear` re-injection. +Open or reuse a session episode and inject curated context as `additionalContext` markdown. Mirrors what the SessionStart hook does; callable manually for recovery, testing, or post-`/clear` re-injection. Shape depends on `BETTER_MEMORY_INJECT_MODE` (see [Architecture](architecture.md#injection-strategies)): + +- **`legacy`** (default) -- project + general semantic memories and reflections. Reflections are retrieved up to 20 per polarity bucket (`do` / `dont` / `neutral`), ranked by Wilson-prior lower bound on hit-rate, ties by confidence. Only the top `BETTER_MEMORY_BOOTSTRAP_TOP_N` project-scoped items (default 5; general-scope semantic memories always render in full) are shown in full -- the rest collapse into a one-line index plus a retrieve affordance (`BETTER_MEMORY_BOOTSTRAP_TOP_N=0` renders everything in full). +- **`deferred`** -- general-scope semantic memories in full plus a single index line; project-scoped semantic memories and reflections are not rendered and surface only via the contextual channel or `memory.retrieve` / `memory.retrieve_observations`. | Parameter | Type | Required | Notes | |---|---|---|---| From 6b251dac91de38215d3f2906f378e52be5687710 Mon Sep 17 00:00:00 2001 From: gethin Date: Fri, 24 Jul 2026 08:58:41 +0100 Subject: [PATCH 16/17] fix(review): agentcore semantic fallback, sentinel docs, cooldown docstring, import hoist Co-Authored-By: Claude Fable 5 --- better_memory/embeddings/sync_embed.py | 7 +++++++ better_memory/services/relevant.py | 17 +++++++++++------ better_memory/services/session_bootstrap.py | 7 +++---- tests/services/test_relevant.py | 15 +++++++++++++++ website/architecture.md | 2 ++ 5 files changed, 38 insertions(+), 10 deletions(-) diff --git a/better_memory/embeddings/sync_embed.py b/better_memory/embeddings/sync_embed.py index d0eb736..972b3d7 100644 --- a/better_memory/embeddings/sync_embed.py +++ b/better_memory/embeddings/sync_embed.py @@ -13,6 +13,13 @@ Returns ``None`` on every failure path — callers treat a missing vector as "no vec leg", never as an error. + +The in-memory ``_down_until`` breaker is per-process, so it does nothing +for hook invocations, which are short-lived subprocesses that exit and +restart on every event. ``down_state_file`` persists the cooldown deadline +to disk (wall-clock ``time.time()``, not the monotonic ``clock``) so a +fresh hook process can see that Ollama was recently marked down and skip +straight to ``None`` instead of re-probing a dead endpoint on every event. """ from __future__ import annotations diff --git a/better_memory/services/relevant.py b/better_memory/services/relevant.py index 30e2921..bb2ea33 100644 --- a/better_memory/services/relevant.py +++ b/better_memory/services/relevant.py @@ -37,8 +37,9 @@ from better_memory.services.scoring import wilson_lower_bound #: Keyword-hit floor used only when the FTS/vec legs are structurally -#: unavailable (no sqlite conn for reflections; no query vector for -#: semantics, which have no FTS substrate at all). +#: unavailable (no sqlite conn for reflections; no query vector, or no +#: sqlite conn, for semantics -- which have no FTS substrate at all and +#: whose vec leg needs a conn even when the embedder itself is healthy). _FALLBACK_MIN_HITS = 2 #: Reciprocal rank fusion constant, matching search/hybrid.py and @@ -171,7 +172,8 @@ def retrieve_relevant( A memory is returned only if it clears the evidence gate: a BM25 match, a vector cosine >= ``vec_floor``, or (only when that leg is structurally - unavailable) a keyword-hit fallback. Among qualifiers, ranking is RRF + unavailable, e.g. ``conn=None`` as in agentcore mode, or no query + vector) a keyword-hit fallback. Among qualifiers, ranking is RRF over the Wilson prior plus whichever of BM25/vec ranks are present. Never raises -- any backend/leg failure degrades that leg to "absent" rather than propagating. @@ -242,9 +244,12 @@ def retrieve_relevant( in_vec = s_id in vec_s # Semantics have no FTS/BM25 leg at all, so the keyword fallback - # applies whenever the vec leg itself is absent (no embedder / - # embed failure), not only when conn is None. - fallback_ok = qvec is None and kw_hits >= _FALLBACK_MIN_HITS + # applies whenever the vec leg is structurally unavailable: no + # query vector (no embedder / embed failure) OR no sqlite conn + # to query against (agentcore mode passes conn=None even when + # the embedder itself is healthy and qvec is real -- the vec + # candidate set still comes back empty in that case). + fallback_ok = (qvec is None or conn is None) and kw_hits >= _FALLBACK_MIN_HITS if not (in_vec or fallback_ok): continue diff --git a/better_memory/services/session_bootstrap.py b/better_memory/services/session_bootstrap.py index 8e3263b..4bbce6b 100644 --- a/better_memory/services/session_bootstrap.py +++ b/better_memory/services/session_bootstrap.py @@ -208,6 +208,8 @@ def bootstrap( cwd: Path | None = None, project: str | None = None, ) -> BootstrapResult: + from better_memory.config import get_config + coerced_source = source if source in _VALID_SOURCES else "startup" if project is None: if cwd is None: @@ -257,9 +259,7 @@ def bootstrap( "neutral": len(buckets["neutral"]), } - from better_memory.config import get_config as _get_config_deferred - - if _get_config_deferred().inject_mode == "deferred": + if get_config().inject_mode == "deferred": general_only = [m for m in semantic if m.scope == "general"] deferred_now = self._clock() n_refl = sum(reflections_counts.values()) @@ -305,7 +305,6 @@ def bootstrap( if self._top_n is not None: top_n = self._top_n else: - from better_memory.config import get_config top_n = get_config().bootstrap_top_n now = self._clock() diff --git a/tests/services/test_relevant.py b/tests/services/test_relevant.py index 2187327..111d266 100644 --- a/tests/services/test_relevant.py +++ b/tests/services/test_relevant.py @@ -182,6 +182,21 @@ def test_semantic_fallback_keyword_when_no_embedder(self, conn): ) assert [(m.kind, m.id) for m in out] == [("semantic", "s1")] + def test_semantic_fallback_keyword_when_conn_none_agentcore(self, conn): + """agentcore mode: conn=None but the embedder is healthy and qvec is + real. The vec candidate set is still structurally empty because vec + queries need the sqlite conn -- so the keyword fallback must fire + here too, not only when qvec itself is None.""" + _seed_semantic(conn, "s1", content="repo uses uv run pytest on windows") + emb = DirectedEmbedder("unrelated trigger phrase") + sync_embedder = SyncEmbedder(lambda: emb) + backend = _backend(conn, sync_embedder=sync_embedder) + out = retrieve_relevant( + backend, query="uv run pytest windows setup", project="p", + conn=None, sync_embedder=sync_embedder, now=lambda: FIXED_NOW, + ) + assert [(m.kind, m.id) for m in out] == [("semantic", "s1")] + class TestCapsAndDegradation: def test_max_items_cap(self, conn): diff --git a/website/architecture.md b/website/architecture.md index 44816f3..3ee4e09 100644 --- a/website/architecture.md +++ b/website/architecture.md @@ -65,6 +65,8 @@ better-memory gets memory in front of Claude two ways: a dump at session start, - **`legacy`** (default, byte-identical to pre-deferred-injection behavior). `SessionBootstrapService.bootstrap` (`better_memory/services/session_bootstrap.py`) renders project-scoped semantic memories and reflections in full only up to `BETTER_MEMORY_BOOTSTRAP_TOP_N` per set (default 5; general-scope semantic memories are always shown in full, uncapped). The remainder collapses into a one-line `### Index (not expanded - retrieve on demand)` section plus a footer affordance pointing at `memory.retrieve` / `memory.retrieve_observations`. Semantic memory ids in the rendered output are the full ids (not truncated), each stamped with an age suffix (`(Nd old)`). Setting `BETTER_MEMORY_BOOTSTRAP_TOP_N=0` disables slimming and renders everything in full. - **`deferred`**. SessionStart renders only general-scope semantic memories in full, plus a single index line ("better-memory knows N reflections + M semantic memories for this project; relevant ones will surface as you work - or ask via memory_retrieve with a task query"). Project-scoped semantic memories and all reflections are not dumped at session start; they surface exclusively through the contextual channel below, or on demand via `memory.retrieve` / `memory.retrieve_observations`. +**CLAUDE.md drift sentinel (SessionStart only).** After bootstrap renders, `hooks/session_bootstrap.py` best-effort-scans the user's `~/.claude/CLAUDE.md` for parameter tokens documented next to a better-memory tool name and flags any the live MCP tool schema does not recognize (`better_memory/hooks/_claude_md_sentinel.py`; schema built from `mcp/tools.py`, so it tracks future tool changes automatically). The accepted-token set per tool is schema-derived, not just its property names: enum *values* (e.g. `memory_retrieve`'s `polarity` values `do` / `dont` / `neutral`) count as valid tokens too, since real CLAUDE.md prose legitimately backtick-quotes them next to the tool name -- this avoids false-positiving on documented return shapes. At most one warning line is appended to bootstrap's `additionalContext` per session; the check is silent when clean, and any exception is swallowed, so the sentinel can never block or degrade injection. It runs only on the SessionStart bootstrap path, not on the contextual channel. Alongside the sentinel, `better_memory/skills/CLAUDE.snippet.md` was rewritten to behavioural instructions only ("pass a task-describing query when you begin a task", "credit with evidence") and never enumerates parameter names or types, so the canonical snippet itself can no longer drift into the failure mode the sentinel exists to catch. + **Contextual injection (UserPromptSubmit / PreToolUse).** The `contextual_inject` hook (`better_memory/hooks/contextual_inject.py`) scores the curated memory set (semantic + reflections) against the current prompt (UserPromptSubmit, fires on every prompt) or tool name + input (PreToolUse, matcher now unscoped -- every tool call, not a fixed allowlist) via `retrieve_relevant` (`better_memory/services/relevant.py`), a three-leg evidence-gated scorer: - A memory injects only when it clears an evidence gate: a BM25 match against `reflection_fts` (title / use_cases / hints), or a vector cosine similarity >= `BETTER_MEMORY_CONTEXT_VEC_FLOOR` (default `0.55`) against its embedding. No evidence, no injection -- a memory with neither leg present is silently dropped, however popular it is. From 25e34673f2b029d8603c3b843c3f499477edc274 Mon Sep 17 00:00:00 2001 From: gethin Date: Fri, 24 Jul 2026 09:18:07 +0100 Subject: [PATCH 17/17] fix(agentcore): expose times_overlooked/times_ignored on reflection dicts for the Wilson prior Addresses PR #84 review: missing keys made the contextual prior compute a falsely-perfect hit rate on the AgentCore backend. Co-Authored-By: Claude Fable 5 --- better_memory/services/relevant.py | 5 +++++ better_memory/storage/agentcore.py | 29 +++++++++++++++++++++++----- better_memory/storage/protocol.py | 7 ++++++- tests/storage/test_agentcore_unit.py | 13 +++++++++++++ 4 files changed, 48 insertions(+), 6 deletions(-) diff --git a/better_memory/services/relevant.py b/better_memory/services/relevant.py index bb2ea33..60dfd01 100644 --- a/better_memory/services/relevant.py +++ b/better_memory/services/relevant.py @@ -229,6 +229,11 @@ def retrieve_relevant( "hits": kw_hits if (in_bm or fallback_ok) else 0, "bm_rank": bm.get(r_id), "vec_rank": vec_r.get(r_id), + # storage.protocol.retrieve guarantees times_overlooked/ + # times_ignored on both backends (sqlite columns; agentcore + # copies its internal overlooked counter and hardcodes + # ignored=0 — see storage/agentcore.py::_parse_reflection_record). + # The .get(...) defaults below are defensive, not load-bearing. "wilson": _wilson_for( int(r.get("useful_count") or 0), int(r.get("times_overlooked") or 0), diff --git a/better_memory/storage/agentcore.py b/better_memory/storage/agentcore.py index e67cf7c..fe38bc9 100644 --- a/better_memory/storage/agentcore.py +++ b/better_memory/storage/agentcore.py @@ -492,10 +492,13 @@ def _count_body_first(body_key: str, meta_key: str) -> int: else: evidence_count = int(_num("useful_count")) + int(_num("missed_count")) + overlooked_count = _count_body_first("times_overlooked", "overlooked_count") + return { # Public shape — must match ReflectionSynthesisService.retrieve_reflections # return: {id, title, phase, use_cases, hints (list), confidence (float), - # tech, evidence_count, useful_count, times_misled, updated_at} + # tech, evidence_count, useful_count, times_overlooked, + # times_ignored, times_misled, updated_at} "id": rec["memoryRecordId"], "title": body.get("title", "") if isinstance(body, dict) else "", "phase": phase_value, @@ -505,13 +508,23 @@ def _count_body_first(body_key: str, meta_key: str) -> int: "tech": tech_value, "evidence_count": evidence_count, "useful_count": _count_body_first("useful_count", "useful_count"), + # Copied from the internal ranking counter below so the Wilson + # prior in services/relevant.py sees the real overlooked count + # instead of silently defaulting to 0 (PR #84 review). + "times_overlooked": overlooked_count, + # AgentCore has no exposure/rating sweep, so "ignored" (shown + # but never rated) is never tracked here — 0 is the true + # recorded signal, not a stand-in for a missing feature. The + # Wilson prior therefore degrades to a monotone function of + # (useful_count + times_overlooked) on this backend, which is + # equivalent to the pre-existing popularity ordering below, not + # a corruption of it. + "times_ignored": 0, "times_misled": _count_body_first("times_misled", "times_misled"), "updated_at": updated_at_value, # Internal ranking/bucketing helpers — stripped by # _fetch_reflection_buckets before the payload leaves the backend. - "_overlooked_count": _count_body_first( - "times_overlooked", "overlooked_count" - ), + "_overlooked_count": overlooked_count, "_updated_at_ts": updated_at_ts, "_polarity": polarity_value, "_status": status_value, @@ -933,7 +946,13 @@ def _semantic_summary_to_model( (useful_count / times_misled / overlooked_count → times_overlooked); the collapsed rating timestamp comes from last_credited_at stringValue. Absent metadata (AWS-extracted or freshly-created records) → zeroed - counters, never None.""" + counters, never None. + + times_ignored is deliberately not populated here: agentcore has no + exposure/rating sweep to derive it from, so it falls through to the + SemanticMemory dataclass default of 0 — the true recorded signal, + not a corruption of it (mirrors the reflection-dict handling in + _parse_reflection_record above).""" # Local import: the SemanticMemory read model lives in the services # layer; importing at module scope would invert the storage→services # layering. This is a lightweight frozen dataclass, imported lazily. diff --git a/better_memory/storage/protocol.py b/better_memory/storage/protocol.py index dc805ee..3830131 100644 --- a/better_memory/storage/protocol.py +++ b/better_memory/storage/protocol.py @@ -96,7 +96,12 @@ def retrieve( Each bucket is a list of reflection dicts: ``{id, title, phase, use_cases, hints (list[str]), confidence (float), tech, - evidence_count, useful_count, times_misled, updated_at}``. Sync — + evidence_count, useful_count, times_overlooked, times_ignored, + times_misled, updated_at}``. ``times_overlooked`` and + ``times_ignored`` feed the Wilson-lower-bound prior in + ``services/relevant.py``; both backends guarantee the keys are + present, but agentcore's ``times_ignored`` is always ``0`` (it has + no exposure/rating sweep to derive it from). Sync — no embedder call (reflections are pre-extracted in both backends; sqlite mode ranks by Wilson lower bound on (useful+overlooked)/rated, computed in Python; agentcore mode applies the legacy linear formula diff --git a/tests/storage/test_agentcore_unit.py b/tests/storage/test_agentcore_unit.py index 95b362b..45addbb 100644 --- a/tests/storage/test_agentcore_unit.py +++ b/tests/storage/test_agentcore_unit.py @@ -1507,6 +1507,11 @@ def test_parse_reflection_extracted_record_unchanged(backend) -> None: # evidence_count computed from metadata useful(4)+missed(2) "evidence_count": 6, "useful_count": 4, + # times_overlooked mirrors _overlooked_count (metadata overlooked_count=3). + "times_overlooked": 3, + # AgentCore has no exposure/rating sweep, so times_ignored is always 0 + # (PR #84 review: this is the true recorded signal, not a corruption). + "times_ignored": 0, "times_misled": 1, # updated_at derived from createdAt (no body updated_at, no sys key) "updated_at": created.isoformat(), @@ -1553,6 +1558,10 @@ def test_parse_reflection_body_state_record_uses_body(backend) -> None: assert parsed["useful_count"] == 7 assert parsed["times_misled"] == 2 assert parsed["_overlooked_count"] == 5 + # Public times_overlooked mirrors the internal counter; times_ignored + # is always 0 on agentcore (no exposure/rating sweep to derive it from). + assert parsed["times_overlooked"] == 5 + assert parsed["times_ignored"] == 0 # evidence_count taken from the body (NOT recomputed as useful+missed=7). assert parsed["evidence_count"] == 12 # updated_at is the body ISO string, and it drives the ranking ts. @@ -1602,6 +1611,10 @@ def stub(**kwargs): assert refl["useful_count"] == 7 assert refl["evidence_count"] == 12 assert refl["updated_at"] == "2026-06-01T09:30:00+00:00" + # times_overlooked/times_ignored must survive to the public payload — + # the Wilson prior in services/relevant.py reads both keys directly. + assert refl["times_overlooked"] == 5 + assert refl["times_ignored"] == 0 # Internal helpers stripped from the public payload. assert not any(k.startswith("_") for k in refl)