From 41bdeb6ac5c35419a369bab712ce23c1319bbb05 Mon Sep 17 00:00:00 2001 From: Wuesteon Date: Thu, 16 Jul 2026 19:14:14 +0800 Subject: [PATCH 01/14] docs(wp10a): sleep-time maintenance spec rev 3 + implementation plan; claim packet MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Spec carries a two-round adversarial verification record (46 findings vs rev 1; independent 15-finding round vs rev 2 — the transitive duplicate-chain resurrection blocker is fixed in rev 3). Registers WP10a/ WP10b in the roadmap, amends the consolidation anti-goal to hosted-only, and claims WP10a (merge stays gated on WP1 launch). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_0198CpP3tf1SbtDGaRX2NWvJ --- .../2026-07-16-sleep-time-maintenance.md | 229 ++++ ...026-07-16-sleep-time-maintenance-design.md | 1024 +++++++++++++++++ docs/superpowers/workpackets.md | 89 +- 3 files changed, 1339 insertions(+), 3 deletions(-) create mode 100644 docs/superpowers/plans/2026-07-16-sleep-time-maintenance.md create mode 100644 docs/superpowers/specs/2026-07-16-sleep-time-maintenance-design.md diff --git a/docs/superpowers/plans/2026-07-16-sleep-time-maintenance.md b/docs/superpowers/plans/2026-07-16-sleep-time-maintenance.md new file mode 100644 index 0000000..45f8e10 --- /dev/null +++ b/docs/superpowers/plans/2026-07-16-sleep-time-maintenance.md @@ -0,0 +1,229 @@ +# Sleep-Time Maintenance (WP10a) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Implement the approved sleep-time maintenance design +(`docs/superpowers/specs/2026-07-16-sleep-time-maintenance-design.md`, rev 3): +an offline job that dedupes, summarizes-old, and evicts-low-value memory +between sessions while preserving the ADD-only spine and as-of semantics per +the spec's §3.1 theorem (three verbs + ingest commutation), with a staged +human review queue exposed over MCP (console UI follows as WP10b). Every +design decision is already made and verified — **do not re-derive**; when a +step below conflicts with the spec, the spec wins. + +**Architecture:** All mutations flow through four sanctioned store verbs +(`supersede_fact` + duplicate-cascade, `retire_duplicate`, `set_tier`, append +via `add_fact`) inside short `batch()` transactions. Two tiny ingest hooks +(duplicate-cascade closure, summary-staleness cascade) restore ingest +commutation and are provable no-ops until maintenance has ever run. Proposals +live in the namespace `.db` (schema v2, user_version-gated migration); the +lease is a partial-unique-index INSERT. Transforms: DEDUP-EXACT (auto), +EVICT auto-band (auto), DEDUP-NEAR / SUMMARIZE / EVICT (proposals). Promotion +is explicit-only, permanently. No physical deletion anywhere. + +**Tech Stack:** Python ≥3.10, sqlite-vec 0.1.9 (vec0 TEXT-metadata UPDATE + +KNN tier filter empirically verified — spec §14), FTS5, pytest offline with +FakeEmbedder/StubTyper, FastMCP (mcp 1.28.0, `@mcp.prompt()` verified), +Ollama behind `[llm]` only. + +## Global Constraints + +- Offline suite green at every commit: `.venv/bin/python -m pytest tests/ -q`. + Console suite too when console files change: + `console/.venv/bin/python -m pytest console/tests -q` — the console has its + OWN venv; the root `.venv` cannot import `lean_memory_console`. +- ADD-only discipline: no DELETE of fact/vec0/FTS rows anywhere in this plan. +- Offline-by-default: extractive stub summarizer is the default; `[llm]` is + opt-in; `LM_FORCE_STUBS` honored; no test touches the network. +- The two ingest hooks must be byte-identical no-ops on any DB where + maintenance never ran (spec §10.10 pins this) — the launch first-run path + is sacred. +- stdout hygiene: nothing in the MCP server process (or any child it spawns) + may write to fd 1 except the JSON-RPC stream. +- Work on branch `wp10a-sleep-maintenance` off `main` (claim the packet in + `workpackets.md`). Commit per task. +- Existing pins must stay green untouched: `tests/test_spine.py`, + `tests/test_asof_sparse.py`, `tests/test_functional_slot_supersession.py`, + `tests/test_search_now.py`. +- Task order matters: 1→2→3 are strictly sequential (plumbing → schema → + hooks); 4–6 build on them; 7–8 are the surfaces; 9 closes out. + +--- + +### Task 1: Store plumbing — busy_timeout, batch(), new verbs + +Spec §4.0, §7.1. The foundation everything else stands on. + +**Files:** +- Modify: `src/lean_memory/store/base.py`, `src/lean_memory/store/sqlite_store.py` +- New: `tests/test_store_maintenance_verbs.py` + +**Interfaces:** +- `SqliteStore.__init__(..., busy_timeout_ms: int = 1500)`; `_connect` issues + `PRAGMA busy_timeout`. Maintenance opens stores with `busy_timeout_ms=5000`. +- `batch()` context manager: `BEGIN IMMEDIATE`, suppresses per-call commits + (a store-level flag checked by every `commit()` call site), single COMMIT + at exit, ROLLBACK on exception. Uses `execute()` only — never + `executescript()` (implicit-commit trap, verified). +- `retire_duplicate(loser_id, survivor_id)`: `is_latest=0` + + `superseded_by=survivor` on fact + `fact_vec.is_latest=0`, `valid_to` + untouched. Chain invariant (spec §4.0, rev-3 blocker fix), maintained two + ways: (i) resolve the survivor arg to its live canonical at call time + (depth 1); (ii) **re-point existing losers of the loser** + (`UPDATE fact SET superseded_by=:survivor WHERE superseded_by=:loser AND + valid_to IS NULL`) so every open retired duplicate always points directly + at an `is_latest=1` row — without (ii), a B→A→D chain resurrects B when D + is later superseded. +- `set_tier(fact_id, tier)`: `fact.tier` + `fact_vec.tier` in one txn. +- `get_embedding(fact_id) -> np.ndarray | None` (read back, never re-embed). +- `iter_latest_facts(after_id=None)`, `iter_slots_touched_since(cursor_id)` + (new-facts-since-cursor → DISTINCT slots; slot transforms then read the + full slot via `find_latest_in_slot`). + +- [ ] **Step 1:** busy_timeout param + PRAGMA in `_connect`; default 1500 ms. Assert existing tests still green (the console `retry_busy` stacking budget is documented in the spec — no console change needed here). +- [ ] **Step 2:** `batch()` with commit-suppression; convert store mutators to route their commit through one helper so suppression is a one-place change. +- [ ] **Step 3:** the four new verbs + the two iterators, each following `supersede_fact`'s two-surface discipline. +- [ ] **Step 4:** tests — two-surface sync for `retire_duplicate`/`set_tier`; batch atomicity (raise mid-batch → nothing committed); `get_embedding` round-trip; `iter_slots_touched_since` finds a duplicate landing on a long-quiet slot (the verified cursor gap); chain invariant: retire B→A then A→D re-points B to D (zero open duplicates pointing at a non-latest row). + +### Task 2: Schema v2 — versioned migration, ledger, proposals + +Spec §5. First real persisted-format change; carries the WP6-flagged +migration obligation. + +**Files:** +- Modify: `src/lean_memory/store/schema.py`, `src/lean_memory/store/base.py`, `src/lean_memory/store/sqlite_store.py` (`_init_schema` + ledger/proposal CRUD), `src/lean_memory/types.py` (`Fact.record_kind` default `'fact'`), `console/src/lean_memory_console/inspect_sql.py` (fingerprint) +- New: `tests/fixtures/v1_format.db` (checked-in, tiny, built by a script committed beside it), `tests/test_schema_migration.py` + +**Interfaces:** +- `_init_schema` restructured: always-run `executescript(SCHEMA_SQL)` (all + IF-NOT-EXISTS) **plus** a `if user_version < 2:` branch running the + non-idempotent DDL (`ALTER TABLE fact ADD COLUMN record_kind ...`) exactly + once, then stamping 2 — never lowering a newer stamp (verified trap: + ALTER raises `duplicate column name` on reopen if left in the blob). +- New tables per spec §5 verbatim: `fact_derivation` (+ + `ix_derivation_source`), `maintenance_run` (+ partial unique index + `ux_run_live ON maintenance_run(namespace) WHERE status='running'`), + `maintenance_proposal` (incl. `expiry_reason`, `evidence_backend`). + +- [ ] **Step 1:** DDL + versioned `_init_schema`; `record_kind` threaded through `Fact`, `add_fact`, `_row_to_fact`. +- [ ] **Step 2:** build + check in the v1-format fixture DB; migration test: opens, upgrades once, **reopens cleanly** (the ALTER-idempotence trap), round-trips a search. +- [ ] **Step 3:** update the console `EXPECTED_SCHEMA_FINGERPRINT` (the new CREATE TABLE lines trip it — verified); console suite green. This lands HERE, in WP10a, with the migration — spec §5; deferring it to WP10b merges this packet with a red console suite. +- [ ] **Step 4:** ledger/proposal CRUD on the store (spec §4.0 "+ ledger/proposal CRUD" — create-half needed by Task 5's runner, so it cannot wait for Task 6): `create_run` / `heartbeat_run` / `finish_run` / `get_live_run`, `stage_proposal` / `get_proposal` / `list_proposals(status=...)`. Pure row CRUD — no decide/apply logic (that's Task 6). Unit tests: round-trip + `ux_run_live` uniqueness (second live INSERT hits the constraint). + +### Task 3: The two ingest hooks (commutation) + +Spec §4.0 (duplicate-cascade), §4.3 (staleness cascade), §3.1 condition 3. +These fix the two empirically-demonstrated rev-1 wrong answers. + +**Files:** +- Modify: `src/lean_memory/store/sqlite_store.py` (`supersede_fact`), `src/lean_memory/memory.py` (`_apply_supersession`) +- New: `tests/test_ingest_commutation.py` + +- [ ] **Step 1:** duplicate-cascade in `supersede_fact`: after closing `old`, `UPDATE fact SET valid_to=? WHERE superseded_by=old.id AND valid_to IS NULL` (same world-time V). `supersede_fact` now RETURNS the full closed-id list — `[old.id]` + cascade-closed ids (spec §4.0, rev 3). +- [ ] **Step 2:** staleness cascade in `_apply_supersession`: feed the closed-id sets **returned by `supersede_fact`** (explicit targets PLUS cascade-closed duplicates — not just the loop's own targets) into a `fact_derivation` lookup via `ix_derivation_source`; any derived summary still `is_latest=1` gets `is_latest=0`, `valid_to=new.valid_at`, `invalidated_by=new.id` (+vec mirror). +- [ ] **Step 3:** the regression tests from spec §10.2/§10.3 — **resurrection** (dedup, then supersede the survivor; `as_of` after the supersession returns only the new fact) **plus the transitive variant** (retire B→A, then A→D, then supersede D: assert B was re-pointed and BOTH A and B closed at V), and **stale summary**. No summarize implementation exists until Tasks 4/6, so the stale-summary test **hand-inserts its fixture**: an `add_fact` summary row (`record_kind='summary'`, `predicate='summary'`, `is_inference=1`) plus a direct `fact_derivation` INSERT (schema exists since Task 2); then contradict a source via ordinary ingest and assert the summary actually flipped (`is_latest=0`, `valid_to=new.valid_at`, `invalidated_by` set). Assert `fact_derivation` is non-empty FIRST, so the test can never pass vacuously. +- [ ] **Step 4:** the no-op pin (spec §10.10): full ingest+search byte-equivalence on a DB where maintenance never ran. + +### Task 4: MaintenanceConfig, scoring, transforms + +Spec §3.6, §4.1–§4.4. The heart of the job — pure functions over the Store +ABC. + +**Files:** +- New: `src/lean_memory/maintain/__init__.py`, `maintain/config.py`, `maintain/score.py`, `maintain/summarize.py`, `maintain/transforms.py` +- New: `tests/test_maintenance_transforms.py`, `tests/test_maintenance_asof_grid.py` + +**Interfaces:** +- `MaintenanceConfig` frozen dataclass with the spec §3.6 defaults + + `config_hash()` (canonical JSON → sha256). +- `score.value(fact, now)` — spec §4.4 formula; recency anchor is VERBATIM + the retriever's: `(last_access or valid_at)` (`retriever.py:97`) — not + `created_at`, and not `max(last_access, valid_at)` (the `max` form diverges + on future-dated facts accessed before their `valid_at`). +- `Summarizer` protocol; `ExtractiveStubSummarizer` (deterministic, + top-salience fact_texts); `OllamaSummarizer` behind `[llm]` import guard. +- Transforms as pure functions `(store, config, embedder, summarizer) → + actions/proposals`: `dedup_exact` (auto; value-preserving normalization = + NFC + casefold + whitespace collapse ONLY; survivor argmin(valid_at), + tiebreak min id; last_access merge = max over cluster of + coalesce(last_access, valid_at); access_count summed), `dedup_near` + (propose; τ≥0.95 on stored embeddings via `get_embedding`; multivalued + flag in payload), `summarize` (propose; per spec §4.3 steps, embedding + computed BEFORE the batch window), `evict` (auto strict band + propose; + guards per §4.4; intra-run ordering: stage all proposals over the + pre-transform snapshot, then auto-apply excluding staged-proposal + targets). + +- [ ] **Step 1:** config + hash + score (unit tests incl. backfill anchor case). +- [ ] **Step 2:** summarizer seam + stub (deterministic output pinned by test); Ollama variant import-guarded. +- [ ] **Step 3:** the four transforms with the guards and ordering above. +- [ ] **Step 4:** the as-of grid test at the **store predicate** (spec §10.1), for what is executable at this task — the AUTO transforms (dedup_exact, evict auto-band) plus staging itself (which must produce a ZERO spine delta): ids satisfying the visibility predicate over a T grid, `is_latest_only=False`, identical for all T < t_m; intended deltas at T ≥ t_m. The propose-transforms' spine effects exist only after the Task-6 apply path — Task 6 Step 4 re-runs this grid post-apply (spec §10.1 rev-3 note). Plus: no inverted intervals; multivalued never auto-merged; ranking-delta pin after DEDUP-EXACT (spec §10.8). + +### Task 5: Runner — lease, cursor, thresholds, ledger + +Spec §6 (thresholds, cursor semantics), §7.2–§7.4. + +**Files:** +- New: `src/lean_memory/maintain/runner.py` +- New: `tests/test_maintenance_runner.py` + +- [ ] **Step 1:** lease claim: BEGIN IMMEDIATE → check live-heartbeat row → INSERT run row (loser hits `ux_run_live` constraint) → COMMIT; heartbeat at every batch commit and ≥ every 30 s; stale threshold `max(5 min, 10× longest observed batch)`; stale takeover marks `'aborted'`. +- [ ] **Step 2:** work thresholds (≥200 facts since cursor OR cumulative new-fact salience ≥300 OR ≥7 days) — below them the run is a clean no-op (spec §10.9). +- [ ] **Step 3:** cursor: advance before writing outputs; candidate scans exclude `record_kind='summary'` and maintenance episodes; slot transforms driven by `iter_slots_touched_since`. +- [ ] **Step 4:** crash/resume test (kill between batches → consistent DB, lease takeover, convergent re-run, no double-summary); two-process lease race test (second runner cleanly skips). + +### Task 6: Proposal lifecycle + Memory façade + retrieval changes + +Spec §5 (CAS + apply re-validation), §8. + +**Files:** +- Modify: `src/lean_memory/memory.py`, `src/lean_memory/store/sqlite_store.py` (tier filters), `src/lean_memory/retrieve/retriever.py` (thread `include_cold`) +- New: `tests/test_proposal_lifecycle.py`, `tests/test_tier_retrieval.py` + +- [ ] **Step 1:** CAS decide + apply, on top of the Task-2 CRUD, exactly per spec §5; apply = one `batch()`: CAS → **re-validate targets** (all still `is_latest=1`, dedup pairs still co-valid; stale ⇒ `status='expired', expiry_reason='stale_target'`, spine untouched) → verbs → `applied_at`. Re-apply retry returns "already applied". +- [ ] **Step 2:** `Memory.maintain(config=...)`, `review_queue()`, `decide()`, `promote()` (explicit-only — no auto-promotion anywhere), `search(..., include_cold=False)`. `maintain()`/apply open a **dedicated maintenance `SqliteStore`** on the namespace file with `busy_timeout_ms=5000` for the run, closed at run end — the serving store stays at 1500 (spec §7.1; this is how the 5000 budget reaches the in-process MCP path). +- [ ] **Step 3:** tier filter: dense `AND tier='hot'` (vec0 metadata, verified), sparse in the per-row recheck; applied only in default latest-mode; `as_of` NEVER filters tier. Byte-identical default when nothing is cold (regression pin). +- [ ] **Step 4:** tests — lifecycle incl. stale-target expiry + full-DB-hash reject invariance (spec §10.7); `as_of × include_cold × tier` matrix + arm agreement (spec §10.6); edited-approve records human provenance and re-scores with `source='user'`; **apply-path as-of grid re-run** (spec §10.1 — the headline claim): approve one summarize + one dedup-near + one evict proposal, then re-assert predicate invariance for all T < t_a and the intended deltas at T ≥ t_a — the only place the propose-transforms' spine effects can be grid-pinned. + +### Task 7: CLI + memory_clear lease-refusal + +Spec §6.1, §7.3. + +**Files:** +- New: `src/lean_memory/maintain/cli.py` +- Modify: `pyproject.toml` (`lean-memory-maintain = "lean_memory.maintain.cli:main"`), `src/lean_memory/mcp_server.py` (`memory_clear`) +- New: `tests/test_maintenance_cli.py` + +- [ ] **Step 1:** `lean-memory-maintain --root [--namespace] [--apply] [--auto-only] [--json]`; **dry-run default**; per-namespace report to stdout (own process — stdout free); `--json` for machines. +- [ ] **Step 2:** `memory_clear` refuses with an explanatory message while a live-heartbeat lease exists; maintenance skips cleared namespaces at the next batch boundary; the residual sliver documented in the tool docstring (spec §7.3 honest statement). +- [ ] **Step 3:** cross-process test: CLI (subprocess) vs a live store writer interleave without unhandled `database is locked` (busy_timeout + lease). + +### Task 8: MCP surfaces — tools ×3, prompt, plugin command, auto-spawn + +Spec §6.3–§6.5, §1.3.10. The v0.1.3 lesson lives here; be exact. + +**Files:** +- Modify: `src/lean_memory/mcp_server.py`, `console/src/lean_memory_console/observe_mcp.py`, `console/src/lean_memory_console/routes/mcp.py`, `plugin/.mcp.json` + `server.json` (reconcile), `tests/test_stdout_hygiene.py` +- New: `plugin/commands/review-memory.md`, `tests/test_mcp_maintenance_tools.py` + +- [ ] **Step 1:** the four tools — `memory_maintenance_run(namespace, apply=False)` (**dry-run default**, symmetric with CLI), `memory_maintenance_status` (must not trigger the lazy model build — reads the DB only; pin with a test), `memory_review_queue(namespace, kind=None, limit=20)` (grouped by entity, evidence inline), `memory_review_decide(namespace, proposal_id, decision, edited_text=None)` (approve|reject|edit|promote) — registered on **all three** surfaces: core stdio, console `observe_mcp.py` (what the plugin ships), console HTTP mount. In the same change, update the exact-set pin `console/tests/test_mcp_parity.py::test_wrapper_exposes_exactly_add_and_search` to the new six-tool set — it asserts set equality and goes red the moment the wrapper grows. +- [ ] **Step 2:** `@mcp.prompt() review-memory-maintenance` on the console stdio server + the plugin command file `plugin/commands/review-memory.md` (the client-portable path). Prompt text: fetch queue → present batched by entity/kind with evidence → collect explicit user verdicts → decide per item → summarize; **forbids the client agent deciding without an explicit user verdict**; batch verbs only on explicit user statements. +- [ ] **Step 3:** auto-spawn (`LM_MAINT_AUTO=1`, default OFF), inside `_mem()` on first tool call: indexed staleness read → `Popen([...,'--apply','--auto-only'], stdin=DEVNULL, stdout=DEVNULL, stderr=, start_new_session=True, close_fds=True)`. fd 1 must never be inherited. +- [ ] **Step 4:** extend the stdout-hygiene test to cover tool calls with `LM_MAINT_AUTO=1` (parent JSON-RPC stream byte-clean while the child runs); manifest reconciliation checked by the existing server-manifest test (extend `tests/test_server_manifest.py` if it doesn't already cover the tool list). + +### Task 9: Docs + close-out + +**Files:** +- Modify: `README.md` (a "Sleep-time maintenance & review" section: the CLI, the cron recipe, the Claude Code review workflow, the safety story in one paragraph), `CHANGELOG.md`, `ARCHITECTURE.md` (status table row), `docs/superpowers/workpackets.md` (WP10a → merged status when done) + +- [ ] **Step 1:** README section — lead with the user story ("overnight cleanup, next-morning click-through in Claude Code"), then the invariants (nothing deleted; history queryable as-of any past time; unreviewed proposals expire). +- [ ] **Step 2:** CHANGELOG entry; ARCHITECTURE row; full offline suite + console suite green; whole-branch review per house flow; update the packet status table. + +--- + +## Explicitly out of scope (do not drift into these) + +- WP10b console Review page (own packet, starts after this merges). +- Physical space reclamation, episode compaction, deletion of any kind (WP5). +- Automatic promote-on-access (decided against permanently, spec §4.4/§12). +- Any change to `bench/` or the frozen calibration constants. diff --git a/docs/superpowers/specs/2026-07-16-sleep-time-maintenance-design.md b/docs/superpowers/specs/2026-07-16-sleep-time-maintenance-design.md new file mode 100644 index 0000000..b912d21 --- /dev/null +++ b/docs/superpowers/specs/2026-07-16-sleep-time-maintenance-design.md @@ -0,0 +1,1024 @@ +# Sleep-Time Maintenance — Design + +Date: 2026-07-16. Status: **approved design, rev 3** (rev 1 was revised +against a six-dimension adversarial verification pass; rev 3 folds in a +second, independent six-dimension round that confirmed one blocker — the +transitive duplicate-chain resurrection, fixed in §4.0/§10.2 — plus +packet-boundary and consistency fixes; see §14. The §12 decisions +were resolved by the user 2026-07-16 and WP10a/WP10b are registered in +`docs/superpowers/workpackets.md`; implementation plan: +`docs/superpowers/plans/2026-07-16-sleep-time-maintenance.md`). All `file:line` references verified against the working tree +at v0.1.3; empirical claims were tested with scratch scripts against the +project venv (sqlite-vec 0.1.9 / SQLite 3.53.2 / Python 3.13). + +Companion docs: `docs/superpowers/specs/2026-07-08-strategic-direction-design.md` +(launch strategy — this work is **post-launch**, see §9), +`docs/superpowers/workpackets.md` (WP4 read-surface, WP5 deletion, WP6 TTL, +and the anti-goal this design partially amends — see §9.1). + +## 0. Decision summary + +An **offline "sleep-time" maintenance job** that runs between sessions and +cleans up stored memory — **deduplicating** entries, **summarizing** older +records into compact form, and **evicting** low-value ones — while preserving +the ADD-only spine and as-of query semantics, plus a **human review queue**: +the job auto-applies only provably-safe transforms overnight and stages +everything judgmental as *proposals* the user clicks through the next day, in +the web console **or conversationally through Claude Code via MCP tools**. + +| Question | Decision | Why | +|---|---|---| +| Where does maintenance write? | Only through sanctioned verbs: append rows, `supersede_fact` (now with duplicate-cascade), `retire_duplicate` (is_latest flip + re-point of existing losers, §4.0), `set_tier` (hot↔cold) | Every verb is as-of-safe at the predicate level (§3.1, §4) | +| Physical deletion? | **None in v1.** No `DELETE` of fact, vec0, or FTS rows | Deleting index rows breaks as-of retrieval (§4.5); space reclaim is a v2 design | +| Ingest-path changes? | Two tiny hooks: duplicate-cascade closure + summary-staleness cascade | Verification proved offline-only transforms go temporally incoherent when later ingest contradicts them (§4.1, §4.3); both hooks are exact no-ops until maintenance has ever run | +| Who approves risky transforms? | Human, via a proposal queue (`pending → approved/rejected/edited/expired`) with apply-time target re-validation | LLM summaries and near-dup merges are judgment calls; no shipping memory product stages changes for approval — verified differentiator (§2.4, §14) | +| Where do proposals live? | New tables in the namespace `.db` (schema v2, user_version-gated migration) | Both MCP surfaces and the console must reach them; decide+apply must be one transaction | +| Unreviewed proposals? | **Expire** after N days (default 30). Never auto-apply | A memory write permanently changes agent behavior; silence ≠ consent | +| Which MCP server gets the tools? | The console stdio server (`observe_mcp.py`) + HTTP mount — what the plugin actually ships — plus core `mcp_server.py` for parity | There are **three** MCP surfaces in the tree; registering only in core would reach no plugin user (the v0.1.3 manifest-gap class, §6.3) | +| Config surface? | Frozen `MaintenanceConfig` dataclass; hash recorded per run in `maintenance_run.config_hash` | The engine has no config mechanism today; a frozen dataclass matches the repo's frozen-config discipline | +| Trigger? | CLI (`lean-memory-maintain`, dry-run by default) + cron recipe + MCP tools (also dry-run by default) + opt-in auto-spawn | The MCP stdio server has no idle/shutdown hook; external triggers are the honest ones (§6) | +| Launch impact? | Zero. Post-launch packet, default-off, no change to the first-run path | The quality gate just closed; nothing here touches it | + +## 1. Map of the current implementation (what maintenance must respect) + +### 1.1 The spine and its two sanctioned mutations + +- **ADD-only means**: no row is ever deleted; the *only* retirement is + `SqliteStore.supersede_fact` (`store/sqlite_store.py:168-176`), which + updates exactly `superseded_by`, `valid_to`, `is_latest` on the old row and + mirrors `is_latest=0` into `fact_vec`. Scoring metadata (`last_access`, + `access_count`) is additionally mutable via `touch()` + (`store/sqlite_store.py:309-314`). +- **As-of visibility predicate** (the invariant): a fact is visible at time + `T` iff `valid_at <= T AND (valid_to IS NULL OR valid_to > T)` — applied + identically in the dense arm (`_apply_as_of`, `store/sqlite_store.py:241-254`) + and the sparse arm (`store/sqlite_store.py:290-293`). It reads **only** + `valid_at`/`valid_to`. It never reads `is_latest`, `tier`, `expired_at`, + `invalidated_by`, or `is_inference`. Note: `Memory.search` defaults + `is_latest_only=True` *even when `as_of` is set* (`memory.py:213-230`) — + the default-flag as-of query is a hybrid surface that ANDs both filters; + the pure point-in-time surface is `is_latest_only=False`. +- **Ingest-time supersession**: `_apply_supersession` (`memory.py:154-173`) + closes old facts at `valid_to = new.valid_at` (world-time), targeting only + `find_latest_in_slot` rows (`is_latest=1`) — a fact that maintenance has + already flipped to `is_latest=0` is **invisible to future ingest closure**. + This is the root of the two interaction bugs verification found (§4.1, + §4.3) and why the two ingest hooks exist. + +### 1.2 Dormant seams the schema already reserved + +- `fact.tier` `'hot'|'cold'` (`types.py:89`, `store/schema.py:57`) **and** a + `tier` metadata column on the vec0 table (`store/schema.py:72`) — written at + insert, never updated, never filtered. **Empirically verified** on the + pinned sqlite-vec 0.1.9: vec0 TEXT-metadata UPDATE works, and KNN `MATCH` + with `AND tier='hot' AND is_latest=1` filters correctly (§14). +- `expired_at` / `invalidated_by` audit columns (`store/schema.py:49-50`) — + declared, round-tripped, read by no query path. Provenance seam. +- Scoring signals: `salience` [0,10] cached at write (`extract/salience.py:73`), + `confidence`, `access_count`/`last_access` bumped on every top-k retrieval + hit (`retrieve/retriever.py:113-114`), `is_inference`. +- Schema-version anchor: `PRAGMA user_version` stamped 1 at + `store/sqlite_store.py:67-73`. + +### 1.3 Operational constraints (verified, load-bearing) + +1. **No `busy_timeout` anywhere in the engine.** `_connect` + (`store/sqlite_store.py:51-61`) sets WAL + foreign_keys only. A second + writer gets an *immediate* `database is locked`. The console compensates + with its own `retry_busy` wrapper (3 attempts, exponential backoff, + `console/.../engine.py:108-127`) — any engine-side timeout **stacks** with + it (§7.1). +2. **Per-statement commits.** Every store mutation commits individually + (`add_fact:166`, `supersede_fact:176`, `touch:314`). No unit-of-work + exists; `batch()` (§4.0) is mandatory new plumbing. Caveat (verified): + Python's `executescript()` implicitly commits — `batch()` must use + `execute()` only. +3. **Three-surface sync.** `fact`, `fact_vec`, `fact_fts` must agree; + `supersede_fact` is the pattern. +4. **Two independent clocks.** Ids sort by *ingestion* time (`new_id()`, + `types.py:21-27`); `valid_at` is *world* time (episode `t_ref`). Backfills + invert their order — no rule may assume id-order == valid_at-order. +5. **`touch()` top-k bias.** Only top-k hits bump access stats; + `access_count=0` can mean "never cracked top-k", not "worthless". Also + verified: `touch()` fires on **as_of searches too** — historical reads + already mutate access stats today (§4.4). +6. **MCP stdio hygiene.** stdout is the JSON-RPC channel + (`mcp_server.py:47-48`); the v0.1.3 banner bug is the cautionary tale. +7. **One SQLite file per namespace** (BET 4) — `find_latest_in_slot` / + `supersede_fact` have no namespace in their WHERE clauses; correct only + because of this. One store per file; never pool facts across stores. +8. **Console architecture** (`console/`, `ui/`): reads via read-only SQL + (`inspect_sql.py`, schema-fingerprint tripwire at `inspect_sql.py:313`), + writes **only** through `EngineGateway` (per-namespace asyncio lock + + `retry_busy` + single worker thread, `engine.py:130-277`, which today + exposes only `add`/`search`/`close`). +9. **No existing maintenance code.** The only deletion in the package is the + whole-namespace file unlink in `memory_clear` (`mcp_server.py:137-147`). +10. **Three MCP surfaces exist** (verification finding, blocker-class): + core `src/lean_memory/mcp_server.py` (stdio, `lean-memory-mcp`); the + console's own stdio server `console/.../observe_mcp.py` (what + `plugin/.mcp.json` actually ships via `lean-memory-console mcp`); and the + console's HTTP mount `console/.../routes/mcp.py` (Docker mode). Tools + registered only in core reach **no plugin user**. + +## 2. Survey — how current systems maintain agent memory + +Full citations in §13. Key mechanisms, grouped by the three transforms: + +### 2.1 Consolidation / summarization + +- **Letta/MemGPT sleep-time compute** (Lin et al. 2025, arXiv:2504.13171; + Letta sleep-time agents): a *background agent* rewrites the primary agent's + learned context between turns (`rethink_memory`), triggered every N=5 + primary steps on the message delta, tracked via a + `last_processed_message_id` cursor. Amortization result (verified against + the paper): ~5× less test-time compute at equal accuracy when many queries + share a context. The canonical "offline pass" shape we adopt — but + expressed as append-only derived facts, not in-place block rewrite. +- **Generative-agents reflection** (Park et al. 2023): trigger = cumulative + importance of recent events > 150 (verified exact); transform = LLM + synthesizes higher-level insights *appended to the same memory stream with + provenance pointers to sources*. The closest published precedent for our + summary facts + derivation links. +- **MemGPT recursive summarization** (Packer et al. 2023): two-threshold + pressure (warn ~70%, flush ~100%); evicted messages *remain permanently in + recall storage* — eviction from the working set is never deletion. +- **RAPTOR** (Sarthi et al. 2024): +20 abs. pts on QuALITY with GPT-4; + builds summaries as *additional* tree nodes over intact leaves and + retrieves across all layers including leaves — architecturally a + summarize-and-retain design, not summarize-and-delete. (The paper does not + ablate leaves-retained vs leaves-deleted; the architectural point stands, + the causal attribution is ours.) +- **Event sourcing**: snapshots/"closing the books" bound replay cost while + the log stays the immutable source of truth (Kurrent; Microsoft patterns; + Kleppmann DDIA ch. 11-12: *derived data may be rebuilt; the source of truth + is never rewritten*). + +### 2.2 Deduplication / contradiction + +- **Mem0** (Chhikara et al. 2025): inline LLM chooses ADD/UPDATE/DELETE/NOOP + against top-10 similar memories — destructive by design; loses history. We + reuse the *compare-against-neighbors-then-classify* step but resolve it as + supersession-append. +- **Zep/Graphiti** (Rasmussen et al. 2025): bi-temporal edge invalidation — + contradicted edges get `t_invalid` set, never deleted. The closest + production kin of lean-memory's spine; LongMemEval +18.5% over + full-context. Validates retirement-by-timestamp. +- **Letta archival memory is ADD-only with NO dedup** — cosine near-dup + consolidation is an *open feature request* (letta-ai/letta #3116, Dec 2025; + verified accurate): "merge duplicates preserving temporal metadata, as a + sleep-time extension / scheduled background task." The leading production + system has not solved exactly the thing this design specifies. +- **Kafka log compaction / LSM**: keep-latest-per-key with physical drops + gated on "no live reader can still need it" (RocksDB drops tombstones only + at the bottommost level with no live snapshot). v1 never physically drops; + v2 reclaim must adopt this gating. + +### 2.3 Eviction / forgetting + +- **MemoryOS** (Kang et al. 2025): heat = α·visits + β·length + γ·exp(−Δt/μ), + promotion threshold τ (verified exact); promote above τ, evict lowest-heat + on overflow. We reuse the formula to *demote*, never delete. +- **MemoryBank** (Zhong et al. 2023): Ebbinghaus retention `e^(−Δt/S)`, + strength grows on recall — decay modulates *ranking*, no deletion + threshold. "Rank, don't delete." +- **FadeMem** (2026): importance-modulated decay with promote/demote + *hysteresis* (θ=0.7/0.3); 45% storage reduction at comparable LoCoMo F1. +- **LUFY** (2025): aggressive forgetting (~90% of utterances) *improved* + retrieval precision >17% — eviction is a precision feature, not a + disk-space feature. +- **Rate-distortion view of memory compaction** (arXiv:2607.08032, 2026; + verified real): frames compaction as a rate-distortion retain-vs-discard + decision and shows query-agnostic keep-signals systematically discard + query-relevant information — supporting a reversible, retrieval-backed + regime. (Attribution note: the abstract does not state the specific + "super-linear vs flat error" dichotomy; we cite the paper for the + reversibility framing only.) Our maintenance stays in the reversible + regime by construction: originals retained, demotions reversible, + summaries re-derivable. + +### 2.4 Human-in-the-loop review (the gap we fill) + +- **No shipping memory product stages agent-proposed memory changes for + approval** — Letta ADE, ChatGPT Manage Memory, Mem0 dashboard, Zep, LangMem + are all *apply-then-curate*. This claim survived an active refutation + attempt during verification (§14). The only approve-before-durable designs + found: a 2026 hermes-agent proposal and generic LangGraph HITL interrupts. +- **Proven interaction patterns from adjacent domains**: Wikipedia pending + changes (invisible-until-accepted, trust-gated auto-approval, + batch-accept), GitHub suggested changes (propose-as-diff, single or batched + commit), alert-fatigue research (confidence-based sampling, batch by + entity, keep queues small or review quality collapses). +- **Security angle** (source pinned per verification): across recent + evaluations, >90% of trials on frontier models (GPT-5-mini, Claude Sonnet + 4.5, Gemini 2.5 Flash) were vulnerable to memory poisoning through normal + query-only interactions, even under strict safety constraints (MCFA + arXiv:2603.15125; MINJA arXiv:2601.05504; Dash et al. arXiv:2606.04329). + CHI 2025 user research shows demand for transparency/control over agent + memory. A human gate on consolidation is a defense layer, not just UX. + +*Human-memory analogy (analogy only): sleep consolidation distills episodic +traces into compact semantic memory without erasing the episodes — waking +writes stay fast and append-only; reorganization happens offline.* + +## 3. Design principles + +### 3.1 The visibility theorem (cornerstone, with verified scope) + +At maintenance time `t_m`, if every action is one of: + +- (a) append a fact with `valid_at >= t_m`, +- (b) close a fact with `valid_to = t_m` (or a later world-time closure + event), +- (c) flip columns the as-of predicate never reads (`is_latest`, `tier`, + `superseded_by`, `expired_at`, `invalidated_by`), + +then for every `T < t_m` the **store-level as-of visibility predicate** is +bit-for-bit unchanged: closed facts still satisfy `valid_to > T`; appended +facts fail `valid_at <= T`; flipped columns are invisible to the predicate. + +**Scope (per verification):** + +1. *Predicate level, not retrieval top-k.* Both retrieval arms fetch a + bounded candidate pool BEFORE as-of filtering (`dense_search` coarse KNN + `LIMIT k*8`, `sparse_search` `k*2` FTS rows); an appended fact can + displace pool candidates and thus perturb a past-T *top-k* result. This + was empirically shown to be **pre-existing engine behavior under ordinary + ingest** — maintenance verb (a) is identical in kind. The invariance + guarantee, and the §10.1 test, are stated at the store predicate. +2. *Surface honesty.* Verb (c) flips ARE visible on the `is_latest` surfaces + — default latest search AND the default-flag as-of query + (`is_latest_only` defaults True even with `as_of`). This is consistent + with how ordinary `supersede_fact` already behaves on those surfaces. The + pure point-in-time surface (`is_latest_only=False`) is bit-identical. +3. *Ingest commutation (the fourth condition, added after verification).* + Transforms must also **commute with future ingest**: any interval that + ordinary ingest would have closed without maintenance must still get + closed with it. Offline-only transforms fail this — retired duplicates + and orphaned summaries are invisible to `find_latest_in_slot` and never + get closed, producing *resurrection* and *stale-summary* wrong answers + (empirically demonstrated, §14). The two ingest hooks (§4.1, §4.3) exist + to restore commutation; both are exact no-ops on databases where + maintenance has never run. + +**Anti-rule** (from the first adversarial round): never close a row at +`valid_to = survivor.valid_at`. Ids sort by ingestion, `valid_at` by world +time; backfills make their order arbitrary → inverted intervals and coverage +gaps. Maintenance closure uses maintenance/world event times only. + +### 3.2 Source of truth vs derived data + +The `fact/episode/entity` spine is never rewritten; vec0 + FTS5 are derived +and *in v1 never deleted either* — as-of retrieval runs KNN/BM25 over them, +so deleting index rows silently removes past facts from as-of retrieval even +when the spine row survives. Space reclamation is a v2 design with LSM-style +reader gating (§9.2). + +### 3.3 Two-tier autonomy + +Auto-apply only transforms that are (i) as-of-safe per §3.1 *including +commutation* and (ii) information-preserving and reversible. Everything +judgmental becomes a **proposal**: + +| Transform | Autonomy | +|---|---| +| Exact-duplicate retirement (identical normalized `fact_text`, same slot) | auto | +| Tier demotion of already-superseded old facts | auto (bookkeeping) | +| Near-duplicate merge (semantic band) | **propose** | +| Summarization (extractive or LLM) | **propose** | +| Eviction (demotion) of still-latest facts | **propose** (auto only below a strict config band) | + +### 3.4 Proposals are invisible until approved + +Staging writes zero spine changes. Approve = re-validate targets, then replay +the verbs at apply-time `t_a` (theorem holds with `t_a`). Reject = zero trace +on the spine. Expire = reject by timeout. The decision trail is itself +append-only data. + +### 3.5 Offline-by-default discipline + +Default summarizer = deterministic extractive stub (top-salience fact_texts; +honest, no model, labeled as such in the proposal). `[llm]` upgrades to +Ollama abstractive. The entire test suite stays offline; `LM_FORCE_STUBS` is +honored. Every proposal records the backend (`stub` / `ollama:` / +embedder id) that produced its evidence, and the apply-time embedder id is +recorded alongside. + +### 3.6 Configuration + +`MaintenanceConfig` — a frozen dataclass (defaults: `tau_near=0.95`, +`age_floor_days=90`, `min_cluster=5`, `evict_threshold`, auto-band +`salience<2 AND access_count=0 AND age>180d`, `proposal_expiry_days=30`, +`proposal_budget_per_run=50`, work thresholds §6) — passed to +`Memory.maintain()` / the CLI. Its canonical-JSON hash is recorded per run in +`maintenance_run.config_hash`, matching the repo's frozen-config discipline. +No env-var soup, no config table in v1. (Verification found every "(config)" +in rev 1 was unbacked — the engine has no config mechanism; this is it.) + +## 4. The transforms, precisely + +### 4.0 New store verbs (the only new mutation surface) + +``` +Store ABC additions (store/base.py): + iter_slots_touched_since(cursor_id) -> Iterator[(subject_id, predicate)] + # slots that GAINED a member since the cursor: join new-facts-since-cursor + # → DISTINCT slots; slot transforms then read the FULL slot via + # find_latest_in_slot. (A bare id-cursor over facts misses duplicates + # landing on long-quiet slots — verified gap.) + iter_latest_facts(after_id=None) -> Iterator[Fact] # id high-water scan (evict/summarize candidates) + get_embedding(fact_id) -> np.ndarray | None # read stored vector back (no re-embed; verified exact + # float32 round-trip via np.frombuffer) + retire_duplicate(loser_id, survivor_id) # is_latest=0 + superseded_by=survivor, valid_to UNTOUCHED, + # + fact_vec.is_latest=0 (two-surface, one txn). + # CHAIN INVARIANT: every OPEN retired duplicate + # (superseded_by set, valid_to NULL) points DIRECTLY at + # an is_latest=1 canonical survivor. Maintained two ways: + # (i) resolve the survivor arg to its live canonical at + # call time (depth 1); (ii) RE-POINT existing losers of + # the loser: UPDATE fact SET superseded_by=:survivor + # WHERE superseded_by=:loser AND valid_to IS NULL. + # (ii) is load-bearing (rev-3 blocker, empirically + # shown): without it, a chain B→A→D leaves B invisible + # to the duplicate-cascade when D is later superseded — + # B resurrects as a permanently-open interval on the + # pure as-of surface. §10.2 pins the transitive case. + # Re-pointing at maintenance time was chosen over a + # recursive cascade (WITH RECURSIVE in supersede_fact) + # to keep the hot ingest path single-level. + set_tier(fact_id, tier) # fact.tier + fact_vec.tier, one txn (vec0 TEXT-metadata + # UPDATE verified working on sqlite-vec 0.1.9) + batch() -> context manager # unit-of-work: BEGIN IMMEDIATE, suspends per-call commits, + # one COMMIT at exit. MUST use execute() — executescript() + # implicitly commits (verified). Model/embedding work is + # FORBIDDEN inside the batch window (§7.1). + + ledger/proposal CRUD (§5) + +Modified verb: + supersede_fact(old_id, new_id, valid_to) -> list[closed_id] + # + DUPLICATE-CASCADE (ingest hook 1): additionally + # UPDATE fact SET valid_to=? WHERE superseded_by=old_id + # AND valid_to IS NULL + # — when a fact closes, its retired duplicates close at + # the same world-time. A single level suffices BECAUSE + # retire_duplicate's chain invariant guarantees every + # open duplicate points directly at old_id. No-op until + # retire_duplicate has ever produced such rows. + # RETURNS the full closed set — [old_id] + cascade-closed + # ids — so the summary-staleness cascade (§4.3) keys on + # every closed row, not just the explicit target + # (rev-3 seam fix). +``` + +### 4.1 DEDUP-EXACT (auto-apply) + +*Target:* co-valid `is_latest=1` facts in one `(subject_id, predicate)` slot +with identical **normalized** `fact_text` — normalization is +**value-preserving only**: Unicode NFC, case-fold, whitespace collapse; never +stemming/synonyms (a lossy normalization could merge distinct multivalued +values — verified risk). These accumulate because an identical restatement is +classified ASSERTS and persisted again (`extract/contradiction.py:203`). + +*Action:* survivor = `argmin(valid_at)` (tiebreak: min id) — preserves "since +when". For each loser: `retire_duplicate(loser, survivor)`; merge usage +stats: `survivor.access_count += loser.access_count`, +`survivor.last_access = max over cluster of coalesce(last_access, valid_at)`. +The last_access rule is deliberate (verified finding): the retriever's +recency anchor is `last_access or valid_at` (`retriever.py:97`) — keeping the +oldest-valid_at survivor without carrying the newest restatement's recency +would silently de-rank the deduped fact on the latest surface. §10 pins the +ranking delta. + +*As-of argument:* `retire_duplicate` is verb (c) — the +`is_latest_only=False` as-of surface is bit-identical for all T (empirically +confirmed over a T grid). The `is_latest` surfaces — default search AND +default-flag as-of — see the collapse, exactly as they do for ordinary +supersession. `valid_to` stays NULL *at dedup time*; **ingest commutation** +is restored by the duplicate-cascade in `supersede_fact` (§4.0): when the +survivor is later closed at world-time V, its duplicates close at V too — all +of them, because the §4.0 chain invariant keeps every open duplicate pointing +directly at the live survivor — identical to the no-dedup counterfactual, +where functional-slot supersession +would have closed every co-valid restatement at V (`memory.py:168-173`). +Without the cascade, a retired duplicate resurrects as a permanently-open +interval after the survivor is superseded — an empirically demonstrated +wrong answer (§14). + +*Notes:* the read-time `fact_text` dedup in `memory_search` +(`mcp_server.py:118-134`) is orthogonal — it collapses cross-slot repeats per +query and must stay. WP4 coordination: `history()` must distinguish +retirement-by-duplication (`superseded_by` set, `valid_to` NULL-until-cascade, +pointer aims at an *older* fact) from world-time supersession, e.g. by edge +kind, or its oldest→newest chain walk will mis-order merges. + +### 4.2 DEDUP-NEAR (propose) + +*Target:* same-slot co-valid pairs with stored-embedding cosine ≥ +`tau_near` (0.95) that are *not* textually identical. Never auto-applied: the +multivalued co-valid band ("likes jazz"/"likes blues" sit at cosine 0.6-0.95 +by design, `extract/contradiction.py:237`) and near-identical-but-distinct +literals ("salary 100k"/"110k") make this a judgment call. + +*Proposal payload:* both fact_texts, cosine, slot, multivalued flag, proposed +survivor, evidence backend. *On approve:* apply-time re-validation (§5), then +DEDUP-EXACT mechanics. Multivalued slots require explicit reviewer +confirmation of co-reference ("same thing said twice, or two different +things?"). + +### 4.3 SUMMARIZE (propose) + +*Target:* per subject entity, latest facts older than `age_floor` in slots +with ≥ `min_cluster` facts, ranked by cluster heat. + +*Action on approve* — embedding and any model call computed **before** the +batch window (§7.1) — then in ONE `batch()` transaction: +1. Re-validate sources (§5): all still `is_latest=1`; else the proposal + expires as stale. +2. Insert a **maintenance episode** (`source='maintenance'`, raw = run/report + JSON) — satisfies the `fact.episode_id` NOT NULL FK + (`store/schema.py:58`). +3. Insert the summary fact: **`predicate='summary'`** (its own slot per + subject — never the source slot, so the functional-slot single-current- + value invariant, `memory.py:154-165`, is untouched; verified), + `record_kind='summary'`, `is_inference=1`, **`valid_at = t_a`** (never + backdated → appears in no past window), `valid_to = NULL`, `tier='hot'`. +4. Insert `fact_derivation(summary_id, source_id, run_id)` lineage rows. +5. `set_tier(source, 'cold')` for each source — sources stay `is_latest=1` + and fully as-of visible; they leave the default hot surface, where the + summary now represents them. +6. If a previous summary exists for the subject: + `supersede_fact(old_summary, new_summary, valid_to=t_a)`. + +**Ingest hook 2 — summary-staleness cascade** (required; without it the +design ships a live contradiction, empirically demonstrated): when ingest +supersedes any fact, `_apply_supersession` additionally looks up +`fact_derivation WHERE source_id IN (closed ids)` (new index, §5) — *closed +ids* being the full set **returned by `supersede_fact`** (explicit target plus +duplicate-cascade-closed rows, §4.0), not merely the loop's own targets — and, +for +each derived summary still `is_latest=1`: sets `is_latest=0`, +`valid_to = new.valid_at`, `invalidated_by = new.id` (+vec mirror). The stale +summary leaves the default surface the moment its content stops being true; +the next maintenance run stages a fresh SUMMARIZE proposal for the subject. +As-of windows `[t_a, closure)` still show the summary — accurate: it was the +believed consolidated state during that period. No-op until +`fact_derivation` has rows. + +*Summarizer seam:* `Summarizer` protocol; `ExtractiveStubSummarizer` default, +`OllamaSummarizer` behind `[llm]`. The reviewer can **edit** the text before +approving — recorded with human-curated provenance and re-scored with +`source='user'`, which the existing salience stub already favors — non-user +sources take a flat penalty (`extract/salience.py:119-120`) — so +human-curated memories naturally outrank machine summaries with zero new +ranking code. + +*Apply ownership* (verified gap): the process executing apply owns the +embedding; it must embed with an embedder matching the namespace's baked vec0 +dims (reuse the `_check_existing_dims` guard, `store/sqlite_store.py:75-104`) +or refuse to apply. Console applies run on `EngineGateway`'s single worker +thread; MCP applies use the server's Memory; CLI applies use its own. + +### 4.4 EVICT (demotion; propose above a floor, auto below a strict band) + +*Value score* (standing, query-free): + +``` +value = 0.5·(salience/10) + + 0.3·exp(−DECAY_LAMBDA · (now − (last_access or valid_at))) # VERBATIM the retriever's anchor (retriever.py:97) + + 0.2·min(1, log1p(access_count)/log1p(10)) +``` + +The recency anchor is deliberately the retriever's own (`last_access or +valid_at`, `retriever.py:97`) — rev 1 used `created_at`, which diverges on +backfills: a fact the retriever de-ranks as stale (old `valid_at`) would have +scored fresh under EVICT (recent ingest). "Stale" now means the same thing in +both places. (Rev 2 briefly wrote the anchor as `max(last_access or 0, +valid_at)`, which still diverged — on future-dated facts accessed before +their `valid_at`; rev 3 uses the retriever's expression verbatim.) + +*Guards:* never propose facts with `salience ≥ 6`, age < `age_floor`, +`record_kind='summary'`, or facts referenced by any staged proposal. +`access_count=0` is never sufficient alone. *Auto band* (config, strict): +demote without review. Below `evict_threshold` otherwise → proposal. + +*Intra-run ordering* (verified gap): stage ALL proposals over the +pre-transform snapshot first, THEN run auto-band transforms, excluding any +fact referenced by a staged proposal — so reviewer evidence never drifts +mid-run. + +*Action:* `set_tier(fact, 'cold')`. *As-of:* verb (c), predicate-invisible. + +*Reversibility (anti-ratchet), revised:* cold facts remain reachable via +`include_cold=True` and as-of search. **Promotion is explicit-only, +permanently** (decision 2026-07-16): review UI / `memory_review_decide` +promote verb / `Memory.promote()`. Rev 1's automatic promote-on-touch is +rejected, not deferred — verification showed `touch()` fires on as-of +searches too, so auto-promotion would make a read-only historical audit +query durably mutate the default hot surface. Reads never durably change +surfaces. + +### 4.5 What deliberately does NOT happen in v1 + +- No `DELETE FROM fact_vec` / `fact_fts` — deleting index rows removes facts + from as-of retrieval candidate pools even when the spine row survives. +- No episode compaction (provenance layer untouched); no `bench/` changes. +- No deletion of any kind — "delete" in the review UI routes to WP5's + designed semantics when that lands. +- No ingest-path changes **except** the two cascades (§4.0, §4.3), each an + exact no-op until maintenance has ever produced the rows they key on — + the first-run path stays byte-identical. + +## 5. Schema v2 (user_version-gated migration, 1→2) + +```sql +-- Inside an `if user_version < 2:` migration branch (NOT the always-run +-- executescript(SCHEMA_SQL) blob): ALTER TABLE ADD COLUMN is not idempotent +-- and raises 'duplicate column name' on reopen (empirically verified). +ALTER TABLE fact ADD COLUMN record_kind TEXT NOT NULL DEFAULT 'fact'; -- 'fact'|'summary' + +CREATE TABLE IF NOT EXISTS fact_derivation ( + summary_id TEXT NOT NULL REFERENCES fact(id), + source_id TEXT NOT NULL REFERENCES fact(id), + run_id TEXT NOT NULL, + created_at INTEGER NOT NULL, + PRIMARY KEY (summary_id, source_id) +); +CREATE INDEX IF NOT EXISTS ix_derivation_source ON fact_derivation(source_id); + -- the staleness cascade's lookup path (§4.3) + +CREATE TABLE IF NOT EXISTS maintenance_run ( + id TEXT PRIMARY KEY, namespace TEXT NOT NULL, + started_at INTEGER NOT NULL, finished_at INTEGER, + heartbeat_at INTEGER, + trigger TEXT NOT NULL, -- 'cli'|'mcp'|'auto'|'console' + cursor_id TEXT, + config_hash TEXT, stats_json TEXT, + status TEXT NOT NULL DEFAULT 'running' -- 'running'|'ok'|'aborted' +); +CREATE UNIQUE INDEX IF NOT EXISTS ux_run_live + ON maintenance_run(namespace) WHERE status='running'; + -- the INSERT is the atomic lease claim: a second runner gets a constraint + -- error, not a silent second row (verified race gap in rev 1) + +CREATE TABLE IF NOT EXISTS maintenance_proposal ( + id TEXT PRIMARY KEY, run_id TEXT NOT NULL REFERENCES maintenance_run(id), + namespace TEXT NOT NULL, + kind TEXT NOT NULL, -- 'dedup_near'|'summarize'|'evict' + payload_json TEXT NOT NULL, + status TEXT NOT NULL DEFAULT 'pending', -- 'pending'|'approved'|'rejected'|'edited'|'expired' + expiry_reason TEXT, -- 'timeout'|'stale_target' + created_at INTEGER NOT NULL, expires_at INTEGER NOT NULL, + decided_at INTEGER, decided_by TEXT, -- 'console'|'mcp'|'expiry' + applied_at INTEGER, edited_text TEXT, + evidence_backend TEXT -- 'stub'|'ollama:'|embedder id +); +``` + +Migration notes (revised per verification): `_init_schema` +(`store/sqlite_store.py:63-73`) must be restructured into a versioned +branch — v2 DDL runs only when `user_version < 2`, then stamps 2 (never +lowering a newer stamp). Ship the WP6 obligation: a checked-in v1-format +fixture DB + regression test (open, search, round-trip after upgrade). The +console fingerprint tripwire (`inspect_sql.py:313`) trips on the new CREATE +TABLE statements (verified: digest keys on lines containing `create` — the +bare ALTER alone would not trip it) and must be updated in the same change. + +**Decide (CAS)** — verified race-safe under two concurrent writers: + +```sql +UPDATE maintenance_proposal SET status=?, decided_at=?, decided_by=?, edited_text=? + WHERE id=? AND status='pending'; -- 0 rows ⇒ already decided; report, don't re-apply. + -- A retry after a committed apply returns + -- 'already applied' (status/applied_at), not an error. +``` + +**Approve-and-apply**, one `batch()` transaction: CAS decide → **re-validate +targets** (every referenced fact still `is_latest=1` and, for dedup, still +co-valid in-slot; namespace file still present) → apply verbs → stamp +`applied_at`. Any stale target ⇒ the whole proposal flips to +`status='expired', expiry_reason='stale_target'` and the spine is untouched. +CAS on the proposal row alone is *not* sufficient after up to 30 days of live +ingest (verified gap). + +## 6. Triggers + +### 6.1 CLI (primary) + +`lean-memory-maintain --root $LM_DATA_ROOT [--namespace NS] [--apply] +[--auto-only] [--json]` — one name, consistent everywhere (rev 1 mixed a +subcommand form; verified inconsistency). Lives in the **core** package +(`[project.scripts]`, beside `lean-memory-mcp`): cross-process safety comes +from the lease + busy_timeout + `batch()` (§7), not from the console's +in-process gateway, so no core→console dependency inversion. **Dry-run is +the default**; `--apply` executes the auto band and stages proposals. + +### 6.2 Scheduler recipe (docs) + +cron/launchd/systemd-timer invoking the CLI off-hours. + +### 6.3 MCP tools + +Registered on the surfaces users actually get (§1.3.10): the console stdio +server `observe_mcp.py` (what the plugin ships) and the HTTP mount +`routes/mcp.py`, with core `mcp_server.py` gaining the same four for +non-plugin stdio users. `server.json` and `plugin/.mcp.json` reconciled in +the same change (the v0.1.3 manifest lesson). + +- `memory_maintenance_run(namespace, apply=False)` — **dry-run by + default, symmetric with the CLI** (rev 1's asymmetry let an agent + mutate state where the CLI would not); returns stats + would-stage / + staged counts. +- `memory_maintenance_status(namespace)` — ledger only; guaranteed not to + force the lazy model build (reads the DB, never constructs backends). +- `memory_review_queue(namespace, kind=None, limit=20)` — pending + proposals with evidence, grouped by entity. +- `memory_review_decide(namespace, proposal_id, decision, edited_text=None)` + — approve | reject | edit | promote. + +### 6.4 Review workflow entry point + +An MCP prompt `review-memory-maintenance` (`@mcp.prompt()` — verified +supported by the shipped mcp 1.28.0) on the console stdio server, **plus** a +plugin command file (`plugin/commands/review-memory.md`) — the mechanism the +plugin already uses — since MCP-prompt surfacing is a client capability we +cannot verify from this repo. The workflow: fetch queue → present batched by +entity/kind with evidence → collect verdicts → decide per item → summarize. +The prompt text **forbids the client agent from deciding without an explicit +user verdict** — agent-mediated review must remain human review. Batch verbs +map only to explicit user statements ("approve all exact dedups"). + +### 6.5 Opt-in auto-spawn + +(`LM_MAINT_AUTO=1`, default OFF): on **first tool call** (inside `_mem()` — +the lazy build means there is no "server start" hook; rev 1 misstated this), +a single indexed `maintenance_run` read decides staleness; if stale, spawn +`lean-memory-maintain --apply --auto-only` with exactly: +`Popen(stdin=DEVNULL, stdout=DEVNULL, stderr=, +start_new_session=True, close_fds=True)` — fd 1 must never be inherited +(the v0.1.3 stdout bug class; §10 extends the stdout-hygiene test to +cover this). The child outlives or dies with the server safely +(own session; no zombies — the parent never waits, `start_new_session` +reparents on parent exit). + +### 6.6 Work thresholds & cursor semantics + +*Work thresholds* (from `MaintenanceConfig`; below them the run is a no-op): +facts since cursor ≥ 200, OR cumulative salience of new facts ≥ 300 +(generative-agents reflection trigger, rescaled), OR ≥ 7 days since last run. + +*Cursor semantics* (split per verification): `iter_latest_facts` uses the id +high-water mark; **slot-level transforms re-scan every slot that gained a +member since the cursor** (`iter_slots_touched_since`) — a duplicate landing +on a long-quiet slot is otherwise invisible. The cursor advances before any +transform output is written, and maintenance-created rows +(`record_kind='summary'`, maintenance episodes) are excluded from candidate +scans, so the job's own outputs never re-enter its candidate set. + +## 7. Concurrency & crash safety + +### 7.1 Lock budget (revised — rev 1's numbers didn't survive verification) + +- Engine-wide `PRAGMA busy_timeout` = **1500 ms** on every `_connect`. + Rationale: the console's shipped `retry_busy` wrapper retries 3× *around* + engine calls — an engine timeout of 5000 ms would stack to ~15 s worst-case + per gateway `add()`; 1500 ms bounds the stack at ≈4.6 s and a single MCP + `memory_add` stall at 1.5 s. Maintenance-opened connections use 5000 ms. + In-process triggers (`Memory.maintain()`, the MCP tools) reach 5000 by + opening a **dedicated maintenance `SqliteStore`** on the namespace file for + the run's duration and closing it after — never by re-tuning the serving + store's connection, whose budget stays untouched (rev-3 wiring fix; the + CLI's own process does the same). +- **No model work inside a batch window.** Embeddings and summaries are + computed before `BEGIN IMMEDIATE`; the lock-hold window contains only row + writes. Empirically verified: a writer holding the lock past another + connection's busy_timeout surfaces `database is locked`, not a clean + retry — long batches would fail the live server's writes. + +### 7.2 Lease (revised — atomic claim) + +The partial unique index `ux_run_live` (§5) makes the `maintenance_run` +INSERT the atomic lease claim: BEGIN IMMEDIATE → check live-heartbeat row → +INSERT (loser hits the constraint) → COMMIT. Heartbeat: `heartbeat_at` +updated at every batch commit and at least every 30 s; stale threshold = +`max(5 min, 10× longest observed single-batch duration)`. A run whose +heartbeat is stale may be marked `'aborted'` and its namespace taken over; +takeover never rolls anything back (idempotent transforms + consistent +per-batch commits make partial runs safe). + +### 7.3 `memory_clear` vs a live maintenance run (honest statement) + +POSIX unlink defeats existence-checking: the maintenance job's open handle +keeps committing to the unlinked inode, and those commits are silently lost +(rev 1's per-batch existence check only reduces wasted work — verified +BROKEN as a safety claim). v1 fix: `memory_clear` refuses (returns an +explanatory error) while a live-heartbeat maintenance lease exists for the +namespace; the maintenance job likewise skips namespaces cleared mid-run at +its next batch boundary. Residual sliver (clear lands between lease-check +and unlink) is documented as a known limitation; full cross-process file +locking is deliberately out of scope for v1. + +### 7.4 Crash safety + +Short `batch()` transactions co-commit every mutation with its +derivation/ledger rows. A crash leaves a consistent DB and a +`status='running'` row with a stale heartbeat; the next run marks it +`'aborted'` and proceeds. Idempotence = resumability: transforms are pure +functions of DB state (already-deduped slots yield nothing; covered sources +are excluded via `fact_derivation`; a re-tried apply hits CAS 0-rows and +reports "already applied"). + +## 8. Retrieval changes (small, gated) + +- `dense_search`/`sparse_search` gain a tier filter — dense via + `AND tier='hot'` on the vec0 metadata column (**empirically verified** on + sqlite-vec 0.1.9), sparse in the existing per-row recheck (single source of + truth: both arms read the flag that `set_tier` writes to both surfaces in + one txn; §10 pins arm agreement). Applied **only** in default latest-mode + searches: **`as_of` queries never filter tier**, and `include_cold=True` + opts out explicitly. +- Every existing row has `tier='hot'` ⇒ byte-identical behavior for anyone + who never runs maintenance. First-run path untouched. +- `Memory` gains `maintain()`, `review_queue()`, `decide()`, `promote()`, + and `search(..., include_cold=False)`. +- **`EngineGateway` gains four public methods** (`maintain`, `review_queue`, + `decide`, `promote`) — real plumbing, not a façade freebie (verified: the + gateway exposes only `add/search/close` today). Each wraps `retry_busy` + + the per-namespace asyncio lock + the single worker thread. Documented + consequence: a console-invoked `maintain()` holds that namespace's gateway + lock, so console-origin maintenance runs in short chunks (per-batch lock + release) to avoid starving live add/search. + +### 8.1 Console review surface + +New `Review` page (`ui/src/pages/Review.tsx` + nav entry): pending proposals +grouped by entity, suggested-changes-style before/after view, verbs +approve / keep (reject) / edit-then-approve / promote; batch approve per +group. Reads beside `routes/views.py` via `inspect_sql` additions (paginated +envelope pattern); decisions POST through the new `EngineGateway` methods. +Review-fatigue levers: batch-by-entity, an evidence-confidence sort +(highest-confidence first, so batch-approve stays safe to reach for), and the +`proposal_budget_per_run` cap (default 50). No lever auto-applies anything — +the spine changes only on an explicit verdict (§3.3, §3.4, §12); rev 3 +removed a stray "auto-approve confidence band" phrase here that contradicted +that invariant. + +## 9. Rollout, packaging, effort + +### 9.1 Roadmap fit — the anti-goal must be amended, not ignored + +`workpackets.md` listed "Consolidation/summarization passes" as a decided +anti-goal — rationale: "*managed-service concerns; the embedded positioning +sidesteps them*". A local, embedded, default-off sleep-time job does not +contradict that rationale; a hosted consolidation service still would. +**Decided 2026-07-16 (user):** the anti-goal is amended to "hosted/managed +consolidation services" and **WP10a/WP10b are registered** in +`workpackets.md` with this spec as their design doc. Sequencing: strictly +**post-launch** (blocked by WP1), a conscious strategy addition recorded in +the packet table — not gated on the six-week read. + +### 9.2 Phasing + +- **v1 (WP10a, ~1.5 weeks — revised up for the ingest hooks + migration + restructuring):** store verbs + `batch()` + busy_timeout + duplicate- + cascade + staleness cascade; versioned migration + fixture test + the + console `EXPECTED_SCHEMA_FINGERPRINT` update (§5 — the v2 DDL trips it, + so it must land with the migration); + DEDUP-EXACT auto (decision 2026-07-16: auto from day one); EVICT auto-band + + proposals; DEDUP-NEAR proposals; SUMMARIZE proposals (extractive stub); + CLI + ledger/lease; `MaintenanceConfig`; MCP tools on all three surfaces + + prompt + plugin command; as-of grid test at the store predicate. Review + via MCP is fully usable without UI. +- **v1.1 (WP10b, ~3-4 days — decision 2026-07-16: starts right after WP10a + merges, not gated on the demand read):** console Review page + + `EngineGateway` methods + `inspect_sql` proposal reads. (The fingerprint + update is **v1** work, per §5 — deferring it would merge WP10a with a red + console suite; rev 3 fixed this doc's earlier triple-assignment.) +- **v2 (design-first):** space reclamation (drop full-dim vectors for cold + facts, keep coarse 256-dim; LSM-style reader gating; explicit + `--reclaim`), Ollama summarizer default-on with `[llm]`, WP5-integrated + deletion verbs in the review UI. (Automatic promote-on-access is decided + against permanently, §4.4 — not a v2 item.) + +### 9.3 Files touched (v1) + +| File | Change | +|---|---| +| `store/base.py`, `store/sqlite_store.py` | new verbs, `batch()`, busy_timeout, duplicate-cascade in `supersede_fact`, tier filters, ledger/proposal CRUD, versioned `_init_schema` | +| `store/schema.py` | schema v2 DDL (§5) | +| `memory.py` | staleness cascade in `_apply_supersession`; `maintain/review_queue/decide/promote` façade; `include_cold` | +| `maintain/{__init__,transforms,score,summarize,runner,config,cli}.py` | NEW package | +| `mcp_server.py` | 4 tools + opt-in auto-spawn (Popen primitives per §6.5) | +| `console/.../observe_mcp.py`, `console/.../routes/mcp.py` | same 4 tools + prompt (the surfaces the plugin ships) | +| `console/.../inspect_sql.py` | `EXPECTED_SCHEMA_FINGERPRINT` update — the §5 DDL trips it; same change as the migration | +| `console/.../engine.py` (v1.1) | 4 new `EngineGateway` methods | +| `plugin/` (`.mcp.json`, `commands/review-memory.md`), `server.json` | manifest reconciliation + command file | +| `pyproject.toml` | `lean-memory-maintain` script | +| `tests/test_maintenance_*.py` | §10 | +| `console/`, `ui/` (v1.1) | review router + `inspect_sql` proposal reads + Review page | + +## 10. Test plan (the invariance argument, executable) + +1. **As-of grid invariance, at the store predicate** (headline): corpus with + backfills, functional + multivalued slots, supersessions; snapshot the + ids satisfying the visibility predicate (direct store query, + `is_latest_only=False`) over a T grid; run every transform; assert + identical sets for all `T < t_m`, intended deltas for `T ≥ t_m`. + Explicitly documented as a predicate-level guarantee — retrieval top-k + has bounded-pool perturbation identical in kind to ordinary ingest + (verified pre-existing). The propose-transforms' legs of this grid + necessarily run against the proposal APPLY path — they have no spine + effect before approval. +2. **Resurrection** (the rev-1 killer): dedup a slot, then supersede the + survivor via ordinary ingest; assert the cascade closed the loser at the + same world-time and `as_of` after the supersession returns only the new + fact. **Transitive variant (the rev-3 killer):** retire B→A, then retire + A→D (the DEDUP-NEAR apply path), then supersede D via ordinary ingest; + assert the second retirement re-pointed B to D and the cascade closed + BOTH A and B at the same world-time. Plus a standing invariant check run + after every maintenance/apply test: zero rows with `superseded_by` set, + `valid_to` NULL, whose `superseded_by` target is not `is_latest=1`. +3. **Stale summary** (the other rev-1 killer): summarize a slot, then + contradict a source via ordinary ingest; assert the summary left the + default surface (`is_latest=0`, `valid_to=new.valid_at`, + `invalidated_by` set) and the next run stages a fresh proposal. +4. Survivor rule argmin(valid_at); no inverted intervals + (`valid_to > valid_at` asserted post-run); normalization is + value-preserving (case/whitespace only). +5. Multivalued guard: "likes jazz"+"likes blues" never auto-merged; near-dup + only proposes. +6. Tier: two-surface sync; dense/sparse arm agreement; `as_of` ignores tier; + byte-identical default when nothing is cold; `as_of × include_cold × + tier` matrix. +7. Proposal lifecycle: CAS double-decide across two frontends; re-apply after + commit returns "already applied", not an error; timeout expiry; + **stale-target expiry** (target superseded between stage and approve → + apply rejects, spine byte-identical by full-DB hash); edited-approve + records human provenance and re-scores with `source='user'`; reject + leaves the spine byte-identical. +8. Ranking honesty: latest-mode top-k delta after DEDUP-EXACT is pinned + (last_access merge rule keeps the deduped fact's recency anchor). +9. Crash/resume: kill between batches; DB consistent; lease takeover; re-run + converges; no double-summary; no orphaned summary. Zero-eligible-work run + is a clean no-op (threshold gate). +10. Ingest hooks are no-ops pre-maintenance: full ingest+search byte- + equivalence on a DB where maintenance never ran (first-run pin). +11. Offline: whole suite on FakeEmbedder/StubTyper; stdout-hygiene test + extended to the maintenance tools and the auto-spawn child (fd 1 never + inherited). +12. Migration: v1-format fixture DB opens, upgrades once, reopens cleanly + (the ALTER-idempotence trap), round-trips. +13. Existing pins stay green: `test_spine.py`, `test_asof_sparse.py`, + `test_functional_slot_supersession.py`, `test_search_now.py`. + +## 11. Comparison to current science — adopted / rejected / novel + +**Adopted:** offline pass decoupled from the write path (Letta sleep-time, +LightMem, LangMem background); reflection-style threshold trigger and +append-with-provenance summaries (generative agents); demotion-not-deletion +paging (MemGPT recall storage; MemoryOS heat as a demotion signal); +summaries-over-intact-leaves (RAPTOR); retirement-by-timestamp (Zep/Graphiti +bi-temporal invalidation); compare-against-neighbors-then-classify (Mem0's +step, our verbs); derived-vs-source split and reader-gated future reclaim +(Kleppmann; Kafka/LSM; Datomic/XTDB logical-vs-physical). + +**Rejected, deliberately:** inline destructive UPDATE/DELETE (Mem0, LangMem); +in-place neighbor rewriting (A-MEM evolution); hard forgetting thresholds +(MemoryOS eviction, LUFY deletion) — we take the precision upside as tier +demotion; auto-apply-on-expiry (Wikipedia timed flagged revisions) — silence +must mean expire, not consent. + +**Novel (not found shipping anywhere):** (1) maintenance with an as-of +preservation argument reduced to three verbs **plus an ingest-commutation +condition**, pinned by executable predicate-level tests; (2) a staged +human-review queue for memory maintenance with dual frontends (web console + +conversational MCP review in Claude Code) — every surveyed product is +apply-then-curate; (3) the reversibility stance made concrete: all compaction +reversible, originals retained, summaries re-derivable and +staleness-invalidated by live ingest. + +## 12. Decisions (resolved with the user, 2026-07-16) + +1. **Roadmap:** anti-goal amended to hosted-only; WP10a/WP10b registered in + `workpackets.md`, blocked by WP1, not gated on the six-week read. (§9.1) +2. **Proposal expiry (default taken, not user-asked):** `expires_at` = 30 + days. Expired *summarize* proposals re-stage naturally on a later run — + the candidate scan re-discovers any still-qualifying cluster that has no + live summary; expired *dedup/evict* proposals likewise re-stage if their + evidence still holds. Nothing stays dead by fiat; nothing auto-applies. +3. **DEDUP-EXACT: auto from day one.** It is the one transform verified + as-of-safe in isolation AND under later ingest (with the cascade), and + review attention belongs on the judgment calls. Both CLI and MCP paths + stay dry-run by default regardless. +4. **Promotion: explicit-only, permanently.** Reads never durably change + surfaces. (§4.4) +5. **WP10b starts right after WP10a merges** — the dual-frontend review + story is core to the feature, not demand-gated. + +## 13. Sources (primary) + +- MemGPT: Packer et al., arXiv:2310.08560 · Sleep-time Compute: Lin et al., + arXiv:2504.13171 · Letta sleep-time agents docs · Letta issue #3116 +- Generative Agents: Park et al., arXiv:2304.03442 · Reflexion: + arXiv:2303.11366 · ExpeL: arXiv:2308.10144 · Voyager: arXiv:2305.16291 +- Mem0: arXiv:2504.19413 · Zep/Graphiti: arXiv:2501.13956 · LangMem docs · + A-MEM: arXiv:2502.12110 · MemoryOS: arXiv:2506.06326 · Memobase · + LightMem: arXiv:2510.18866 +- MemoryBank: arXiv:2305.10250 · LUFY: arXiv:2409.12524 · FadeMem: + arXiv:2601.18642 · RAPTOR: arXiv:2401.18059 · Rate-distortion compaction: + arXiv:2607.08032 · HippoRAG: arXiv:2405.14831 / 2502.14802 · SleepGate: + arXiv:2603.14517 · SCM: arXiv:2604.20943 · Larimar: arXiv:2403.11901 +- Memory poisoning: MCFA arXiv:2603.15125 · MINJA arXiv:2601.05504 · + Dash et al. arXiv:2606.04329 +- Surveys: arXiv:2404.13501, 2504.15965, 2505.00675, 2512.13564 +- Temporal-DB prior art: Datomic excision/noHistory · XTDB DELETE/ERASE · + Kafka log compaction (Confluent) · RocksDB compaction · Kurrent snapshots · + Kleppmann, DDIA ch. 11-12 +- HITL: Letta ADE docs · OpenAI memory controls · Mem0 dashboard · + hermes-agent #44963 · LangGraph HITL · Wikipedia pending/timed flagged + revisions · GitHub suggested changes · CHI EA '25 agent-memory user study + +## 14. Verification record (2026-07-16) + +Six adversarial verifiers ran against rev 1, each attacking one dimension +with code reading, empirical scratch scripts against the project venv, and +web fact-checking. **46 findings: 17 HOLDS, 5 BROKEN, 24 NEEDS_AMENDMENT, +0 unverifiable.** All BROKEN and NEEDS_AMENDMENT findings are folded into +this rev 2. + +Confirmed empirically (HOLDS): vec0 TEXT-metadata UPDATE + KNN tier filter on +sqlite-vec 0.1.9; `batch()` via BEGIN IMMEDIATE on the current connection +config; float32 embedding round-trip; CAS decide under a live two-writer +race; DEDUP-EXACT as-of invariance in isolation over a T grid; the +functional-slot machinery after both dedup and summarize; the console +fingerprint tripwire; packaging of the new package; every §1 file:line +citation. Citations: Letta #3116, sleep-time-compute amortization, the +generative-agents threshold of 150, the MemoryOS heat formula, and the +2607.08032 paper's existence are exact; the "no product stages memory +changes for approval" claim survived an active refutation attempt. + +Refuted in rev 1 and fixed here (BROKEN): duplicate resurrection under later +ingest (→ duplicate-cascade, §4.0); stale summaries under later ingest +(→ staleness cascade, §4.3); MCP tools registered on a server the plugin +doesn't ship (→ §1.3.10, §6.3); unbacked config thresholds +(→ `MaintenanceConfig`, §3.6); `memory_clear` ghost-inode race presented as +mitigated (→ honest lease-based refusal, §7.3). + +Amended: theorem scoped to the store predicate with the ingest-commutation +condition (§3.1); default-flag as-of is an is_latest surface (§1.1, §4.1); +versioned migration (§5); lease atomicity + heartbeat cadence (§7.2); lock +budget vs console retry stacking (§7.1); model work outside batch windows +(§7.1); auto-spawn Popen primitives and first-tool-call timing (§6.5); +apply-time target re-validation (§5); apply-time embedder ownership (§4.3); +recency-anchor alignment and last_access merge (§4.1, §4.4); explicit-only +promotion in v1 (§4.4); intra-run ordering (§4.4); slot-aware cursor (§6); +CLI naming/placement (§6.1); EngineGateway plumbing (§8); RAPTOR / +rate-distortion / poisoning citation precision (§2); test plan +7 scenarios +(§10). + +### Second round (2026-07-16, rev 3) + +A second, fully independent six-dimension adversarial pass ran against rev 2 +(decision fidelity, code grounding, spec→plan coverage, workpackets +integrity, design attack, plan executability — every non-nit finding +re-verified by a dedicated skeptic; 15 verified: 10 CONFIRMED, 4 PARTIAL, +1 REFUTED). Grounding, decision fidelity, and coverage held clean; the +riskiest rev-2 empirical claims (executescript implicit commit, ALTER +duplicate-column on reopen, vec0 metadata UPDATE + KNN tier filter) +reproduced exactly. Confirmed and fixed in this rev 3: + +- **BLOCKER — transitive duplicate-chain resurrection** (found independently + by two reviewers; empirically reproduced three ways): a chain B→A→D left B + permanently open after D's supersession, because the duplicate-cascade is + single-level and rev 2's "depth 1" chain rule never re-pointed *existing* + losers of a fact that itself becomes a loser — a wrong answer on the exact + pure as-of surface the §3.1 theorem guarantees, which the single-level + §10.2 test would have missed. Fix: `retire_duplicate` now re-points + existing losers (§4.0); §10.2 gains the transitive case + a standing + chain-invariant check. +- **Packet-boundary contradictions:** the console fingerprint update was + triple-assigned (WP10a per §5/plan; WP10b per the old §9.2/§9.3/ + workpackets — the WP10b reading merges WP10a with a red console suite); + now unambiguously v1/WP10a. The `EngineGateway` methods row in §9.3 is + now marked (v1.1), matching §9.2/§8/plan/workpackets. +- **Seam fix:** `supersede_fact` now returns the full closed-id set so the + §4.3 staleness cascade provably sees duplicate-cascade-closed sources. +- **§8.1** dropped a stray "auto-approve confidence band" lever that + contradicted the nothing-auto-applies invariant (§3.3/§12). +- **Consistency:** EVICT's recency anchor is now verbatim the retriever's + `last_access or valid_at` (rev 2's `max()` form diverged on future-dated + facts); §6's triggers are numbered subsections so §6.x references resolve; + citation nits (retriever.py:113-114; the salience mechanism is a non-user + penalty, not a user reward). + +Refuted (design already correct as written): the suspected fresh-create +migration trap — the `record_kind` ALTER lives only in the `<2` branch, never +in `SCHEMA_SQL`, and both create paths run clean. Downgraded on verification: +the staleness-cascade placement concern (the spec's "closed ids" wording was +already correct — the return-value contract above makes it unmissable) and +the WP10a lane-footprint understatement (WP10b is hard-serialized behind +WP10a, so no live race; the packet header now discloses the full footprint). +The companion plan was patched in the same rev for five executability gaps: +proposal-CRUD moved ahead of the runner (Task 2), hand-inserted fixtures for +the Task-3 stale-summary test, the console parity-pin update in Task 8, an +apply-path as-of grid re-run in Task 6, and the console suite runner command +(`console/.venv`, not the root venv). diff --git a/docs/superpowers/workpackets.md b/docs/superpowers/workpackets.md index 1c127d8..017d2b3 100644 --- a/docs/superpowers/workpackets.md +++ b/docs/superpowers/workpackets.md @@ -40,7 +40,9 @@ on the same files). | WP7 Async API | `wp7-async` | A | WP4–WP6 | six-week read | open | | WP8 Integrations & distribution wave | `wp8-*` (per sub-packet) | C | WP0 | six-week read (spec §5) | open | | WP9 NLI middle tier (contingent) | `wp9-nli-resolver` | A | WP0 | **contingent** — see trigger | open | -| Memory UI | `worktree-memory-ui` | D | WP0 (rebase gate) | spec §3 rebase gate | **approved, parked** (2026-07-10: spec v2 + 17-task plan committed on the branch; BLOCKED until WP0 Task 11 merges to main, then rebase + subagent-driven execution) | +| WP10a Sleep-time maintenance (engine + MCP review) | `wp10a-sleep-maintenance` | A | WP1 | — (conscious post-launch addition, recorded 2026-07-16) | **claimed** (2026-07-16: implementation started on `wp10a-sleep-maintenance` by user direction; **merge remains gated on WP1 launch**) | +| WP10b Maintenance review UI | `wp10b-review-ui` | D | WP10a | — | open | +| Memory UI | `worktree-memory-ui` | D | — | — | **MERGED** (2026-07-14: PR #2 → main 9d840b6; console 125 + core 141 green on merged main; lane D released) | Lanes: **A** = engine/API surface (`src/lean_memory/` hot zone — strictly sequential within the lane). **B** = benchmarks (`bench/` — parallel-safe with @@ -66,6 +68,7 @@ A and C). **C** = launch/distribution (docs, packaging, listings). **D** = UI. WP0 launch-gate ──► WP1 launch ──► (six-week read, spec §4) ─┬─► WP3 benchmarks ├─► WP4 ─► WP5(impl) ─► WP6 ─► WP7 (lane A, sequential) └─► WP8 integrations +WP1 launch ──► WP10a sleep-maintenance (lane A — serialize with WP4+) ─► WP10b review UI (lane D) WP2 update-integrity ────────────── independent, anytime (lane B) WP5 design doc ──────────────────── independent, anytime (docs only) WP9 NLI tier ────────────────────── contingent on WP0 Task-6 / WP3 telemetry @@ -391,15 +394,95 @@ latency per `add` < 150ms on CPU with the real model; zero new mandatory deps. --- +## WP10a — Sleep-time maintenance: engine + MCP review + +**Branch:** `wp10a-sleep-maintenance` · **Blocked by:** WP1 (strictly +post-launch; a conscious strategy addition recorded 2026-07-16, not gated on +the six-week read) · **Effort:** L (~1.5 weeks) · **Lane A — claims +`memory.py` + `store/*` + `types.py` + `retrieve/retriever.py` + +`mcp_server.py`; serialize with WP4–WP7 within the lane. Also touches, for +the MCP/packaging surfaces only: `console/.../observe_mcp.py`, +`console/.../routes/mcp.py`, `console/.../inspect_sql.py` (fingerprint +constant only — the schema-v2 DDL trips it, spec §5), `plugin/`, +`server.json`. No UI files; WP10b is hard-serialized behind this packet, so +the console-file overlap cannot race.** + +**Design (approved, verified):** +`docs/superpowers/specs/2026-07-16-sleep-time-maintenance-design.md` (rev 3) +— read it in full before starting; it carries a two-round verification +record (46 findings vs rev 1; an independent second round vs rev 2 whose one +blocker — transitive duplicate-chain resurrection — is fixed in rev 3) and +the resolved §12 decisions. +**Plan:** `docs/superpowers/plans/2026-07-16-sleep-time-maintenance.md`. + +**Goal:** Offline "sleep-time" maintenance between sessions — dedupe, +summarize-old, evict-low-value — preserving the ADD-only spine and as-of +semantics (spec §3.1 theorem incl. ingest commutation), with a staged human +review queue served two ways: console UI (WP10b) and conversational review +through Claude Code via MCP tools + a shipped review prompt. + +**Scope (in):** store verbs + `batch()` + engine busy_timeout; schema v2 +(user_version-gated migration + v1 fixture); the two ingest hooks +(duplicate-cascade, summary-staleness cascade); DEDUP-EXACT auto; +EVICT auto-band + proposals; DEDUP-NEAR + SUMMARIZE proposals (extractive +stub default, `[llm]` Ollama opt-in); tier filters + `include_cold`; +`MaintenanceConfig`; lease/ledger/runner + CLI `lean-memory-maintain`; +4 MCP tools on all three MCP surfaces + review prompt + plugin command file; +opt-in auto-spawn. +**Scope (out):** console UI (WP10b); physical space reclamation (v2, +design-first); deletion of any kind (WP5's problem). + +**Interactions:** shares the schema-migration anchor obligation with WP6 +(whoever ships first restructures `_init_schema` into versioned branches and +checks in the v1 fixture DB — coordinate); WP4's `history()` must distinguish +retirement-by-duplication edges from world-time supersession (spec §4.1). + +**Acceptance criteria:** spec §10 test plan green (as-of grid at the store +predicate, incl. the post-apply re-run; resurrection — incl. the transitive +chain variant — + stale-summary regression tests; ingest hooks +byte-identical no-ops pre-maintenance; proposal lifecycle incl. stale-target +expiry; migration fixture; stdout hygiene incl. spawned child); offline suite +green; existing spine/as-of pins untouched. + +--- + +## WP10b — Maintenance review UI (console) + +**Branch:** `wp10b-review-ui` · **Blocked by:** WP10a (decided 2026-07-16: +starts right after WP10a merges, not gated on the demand read) · **Effort:** +M (~3-4 days) · **Lane D + console files.** + +**Goal:** The next-morning click-through: a `Review` page listing pending +proposals grouped by entity with before/after evidence; verbs approve / +keep / edit-then-approve / promote; batch approve per group. + +**Files:** `console/src/lean_memory_console/engine.py` (4 new +`EngineGateway` methods), `console/.../inspect_sql.py` (proposal reads — +the schema fingerprint lands in WP10a with the migration, spec §5), +new review router, `ui/src/pages/Review.tsx`, `ui/src/App.tsx`, +`ui/src/components/Layout.tsx`, `ui/src/api.ts`, console + UI tests. + +**Acceptance criteria:** decisions round-trip through `EngineGateway` (never +raw SQL writes); CAS "already decided elsewhere" surfaced in the UI; console +suite green; spec §8.1 fatigue levers present (entity grouping, batch +approve, budget cap). + +--- + ## Anti-goals (decided, with evidence — do not open packets for these) - **Graph memory / graph-traversal retrieval channel.** Mem0's own published ablation: graph variant +1.6 points overall, ~3× slower search, ~2× token cost — then removed from their OSS entirely. Revisit only if WP3 shows systematic multi-hop failures. -- **Consolidation/summarization passes, webhooks, dashboards, SOC2, hosted +- **Hosted/managed consolidation services, webhooks, dashboards, SOC2, hosted anything.** Managed-service concerns; the embedded positioning sidesteps - them (per-spec out-of-scope). + them (per-spec out-of-scope). *Amended 2026-07-16: the original bullet + banned all "consolidation/summarization passes"; its stated rationale + (managed-service concerns) does not apply to a local, embedded, default-off + sleep-time maintenance job, which is now in scope as WP10a/WP10b (verified design: + `docs/superpowers/specs/2026-07-16-sleep-time-maintenance-design.md`). + Hosted consolidation stays out.* - **CRDT multi-agent sync.** sqlite-memory's differentiated lane; real demand signal required before entering it. - **int8 vectors / LanceStore.** Blocked upstream / deferred (ARCHITECTURE.md, From b6526165d333425b9da130a80e1d5a949b2ac56a Mon Sep 17 00:00:00 2001 From: Wuesteon Date: Thu, 16 Jul 2026 19:23:16 +0800 Subject: [PATCH 02/14] =?UTF-8?q?feat(store):=20add=20maintenance=20plumbi?= =?UTF-8?q?ng=20=E2=80=94=20busy=5Ftimeout,=20batch(),=20new=20verbs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Foundation for the sleep-time maintenance job (design spec §4.0, §7.1). - busy_timeout: SqliteStore gains busy_timeout_ms=1500 (kw-only); _connect issues PRAGMA busy_timeout. Maintenance opens stores with 5000 ms. - batch(): BEGIN IMMEDIATE unit-of-work with commit suppression routed through a single _commit() helper (all existing mutators converted), one COMMIT at exit, ROLLBACK on exception, non-re-entrant. execute() only. - retire_duplicate(loser, survivor): two-surface is_latest flip + superseded_by, valid_to untouched. Chain invariant maintained two ways — resolve survivor to live canonical (depth 1) and re-point existing open losers of the loser — so a transitive B->A->D retirement can't resurrect B. Rejects survivor==loser. - set_tier / get_embedding (float32 round-trip, no re-embed) / iter_latest_facts (id high-water) / iter_slots_touched_since (new-facts- since-cursor -> DISTINCT slots; catches dups on quiet slots). Store ABC updated with the six new abstract methods. Tests: tests/test_store_maintenance_verbs.py (15, offline) — two-surface sync, batch atomicity, get_embedding round-trip, cursor gap, transitive chain invariant. Full suite 155 passed, 1 skipped; four pinned files untouched. Co-Authored-By: Claude Fable 5 --- src/lean_memory/store/base.py | 38 +++- src/lean_memory/store/sqlite_store.py | 139 +++++++++++++- tests/test_store_maintenance_verbs.py | 257 ++++++++++++++++++++++++++ 3 files changed, 426 insertions(+), 8 deletions(-) create mode 100644 tests/test_store_maintenance_verbs.py diff --git a/src/lean_memory/store/base.py b/src/lean_memory/store/base.py index 4e2762f..22516b1 100644 --- a/src/lean_memory/store/base.py +++ b/src/lean_memory/store/base.py @@ -8,7 +8,8 @@ from __future__ import annotations from abc import ABC, abstractmethod -from typing import Optional, Sequence +from contextlib import contextmanager +from typing import Iterator, Optional, Sequence import numpy as np @@ -80,6 +81,41 @@ def hydrate(self, fact_ids: Sequence[str]) -> dict[str, Fact]: def touch(self, fact_id: str, when_ms: int) -> None: """Record an access (recency/decay bookkeeping).""" + # ── maintenance mutation surface (sleep-time job; design spec §4.0) ── + @abstractmethod + @contextmanager + def batch(self) -> Iterator[None]: + """Unit-of-work: BEGIN IMMEDIATE, suspend per-call commits, one COMMIT at + exit, ROLLBACK on exception. Model/embedding work is forbidden inside the + window (§7.1) — the lock-hold span must contain only row writes.""" + ... + + @abstractmethod + def retire_duplicate(self, loser_id: str, survivor_id: str) -> None: + """Retire an exact duplicate: flip loser is_latest=0 + superseded_by=survivor + on fact and fact_vec; valid_to UNTOUCHED (verb (c), as-of-safe). Maintains the + chain invariant — every open retired duplicate points DIRECTLY at an is_latest=1 + survivor — by (i) resolving `survivor_id` to its live canonical and (ii) + re-pointing existing open losers of `loser_id` at that survivor.""" + + @abstractmethod + def set_tier(self, fact_id: str, tier: str) -> None: + """Move a fact between the hot/cold tiers — fact.tier + fact_vec.tier, one txn.""" + + @abstractmethod + def get_embedding(self, fact_id: str) -> Optional[np.ndarray]: + """Read a fact's stored full-dim vector back (no re-embed). None if absent.""" + + @abstractmethod + def iter_latest_facts(self, after_id: Optional[str] = None) -> Iterator[Fact]: + """Id high-water scan over is_latest=1 rows (evict/summarize candidates).""" + + @abstractmethod + def iter_slots_touched_since(self, cursor_id: str) -> Iterator[tuple[str, str]]: + """DISTINCT (subject_id, predicate) slots that GAINED a member (a fact with + id > cursor) since the cursor — including duplicates landing on long-quiet + slots (the verified cursor gap).""" + # ── lifecycle ── @abstractmethod def close(self) -> None: ... diff --git a/src/lean_memory/store/sqlite_store.py b/src/lean_memory/store/sqlite_store.py index 230105b..e920f39 100644 --- a/src/lean_memory/store/sqlite_store.py +++ b/src/lean_memory/store/sqlite_store.py @@ -16,8 +16,9 @@ import re import sqlite3 +from contextlib import contextmanager from pathlib import Path -from typing import Optional, Sequence +from typing import Iterator, Optional, Sequence import numpy as np from sqlite_vec import serialize_float32 @@ -40,10 +41,21 @@ def _serialize(vec: np.ndarray) -> bytes: class SqliteStore(Store): - def __init__(self, path: str | Path, *, dim: int = 768, coarse_dim: int = 256) -> None: + def __init__( + self, + path: str | Path, + *, + dim: int = 768, + coarse_dim: int = 256, + busy_timeout_ms: int = 1500, + ) -> None: self.path = str(path) self.dim = dim self.coarse_dim = coarse_dim + # Lock budget (§7.1): 1500 ms serving default; maintenance opens with 5000. + self.busy_timeout_ms = busy_timeout_ms + # When True, per-call commits are suppressed — set only inside batch(). + self._in_batch = False self._db = self._connect() self._init_schema() @@ -58,8 +70,35 @@ def _connect(self) -> sqlite3.Connection: db.enable_load_extension(False) db.execute("PRAGMA journal_mode=WAL") # better single-writer concurrency db.execute("PRAGMA foreign_keys=ON") + db.execute(f"PRAGMA busy_timeout={int(self.busy_timeout_ms)}") return db + # ── commit routing (single choke point for batch suppression) ── + def _commit(self) -> None: + """Every mutator commits through here so batch() can suppress it in ONE place. + Inside a batch window the commit is deferred to the single COMMIT at exit.""" + if not self._in_batch: + self._db.commit() + + @contextmanager + def batch(self) -> Iterator[None]: + """Atomic unit-of-work (§4.0): BEGIN IMMEDIATE, suppress per-call commits, one + COMMIT at exit, ROLLBACK on exception. execute() only — executescript() would + implicitly commit and break atomicity. Not re-entrant (a nested batch is a bug).""" + if self._in_batch: + raise RuntimeError("batch() is not re-entrant") + self._db.execute("BEGIN IMMEDIATE") + self._in_batch = True + try: + yield + except BaseException: + self._in_batch = False + self._db.rollback() + raise + else: + self._in_batch = False + self._db.commit() + def _init_schema(self) -> None: self._check_existing_dims() sql = SCHEMA_SQL.format(dim=self.dim, coarse_dim=self.coarse_dim) @@ -111,7 +150,7 @@ def add_episode(self, episode: Episode) -> None: (episode.id, episode.namespace, episode.raw, episode.source, episode.t_ref, episode.created_at), ) - self._db.commit() + self._commit() # ── entities ── def upsert_entity(self, entity: Entity) -> Entity: @@ -127,7 +166,7 @@ def upsert_entity(self, entity: Entity) -> Entity: (entity.id, entity.namespace, entity.name, entity.type, entity.summary, entity.resolved_id, entity.created_at), ) - self._db.commit() + self._commit() return entity def get_entity(self, entity_id: str) -> Optional[Entity]: @@ -163,7 +202,7 @@ def add_fact(self, fact: Fact, embedding: np.ndarray, embedding_256: np.ndarray) "INSERT INTO fact_fts(fact_id, fact_text) VALUES (?,?)", (fact.id, fact.fact_text), ) - db.commit() + self._commit() def supersede_fact(self, old_fact_id: str, new_fact_id: str, valid_to: int) -> None: db = self._db @@ -173,7 +212,7 @@ def supersede_fact(self, old_fact_id: str, new_fact_id: str, valid_to: int) -> N ) # keep the vec0 metadata filter column in sync so superseded facts drop out db.execute("UPDATE fact_vec SET is_latest=0 WHERE fact_id=?", (old_fact_id,)) - db.commit() + self._commit() def get_fact(self, fact_id: str) -> Optional[Fact]: row = self._db.execute("SELECT * FROM fact WHERE id=?", (fact_id,)).fetchone() @@ -311,7 +350,93 @@ def touch(self, fact_id: str, when_ms: int) -> None: "UPDATE fact SET last_access=?, access_count=access_count+1 WHERE id=?", (when_ms, fact_id), ) - self._db.commit() + self._commit() + + # ── maintenance mutation surface (sleep-time job; design spec §4.0) ── + def retire_duplicate(self, loser_id: str, survivor_id: str) -> None: + """Retire an exact duplicate onto its survivor (two-surface, one txn). + + Flips loser is_latest=0 + superseded_by=survivor on fact AND fact_vec; + valid_to is UNTOUCHED (verb (c) — the is_latest_only=False as-of surface is + unchanged). Maintains the chain invariant that every OPEN retired duplicate + (superseded_by set, valid_to NULL) points DIRECTLY at an is_latest=1 survivor: + (i) resolve `survivor_id` to its live canonical (depth 1) — a retired + survivor arg would otherwise leave the loser pointing at a non-latest row; + (ii) re-point existing open losers of `loser_id` at that canonical, so a + transitive B→A→D retirement never resurrects B when D is later superseded + (rev-3 blocker, §4.0 / §10.2). + """ + if loser_id == survivor_id: + raise ValueError("retire_duplicate: survivor must differ from loser") + db = self._db + # (i) resolve the survivor arg to its live canonical (depth 1). + row = db.execute( + "SELECT superseded_by, is_latest FROM fact WHERE id=?", (survivor_id,) + ).fetchone() + if row is not None and not row["is_latest"] and row["superseded_by"]: + survivor_id = row["superseded_by"] + if loser_id == survivor_id: # resolution collapsed onto the loser — reject + raise ValueError("retire_duplicate: survivor resolves to the loser") + + db.execute( + "UPDATE fact SET superseded_by=?, is_latest=0 WHERE id=?", + (survivor_id, loser_id), + ) + db.execute("UPDATE fact_vec SET is_latest=0 WHERE fact_id=?", (loser_id,)) + # (ii) re-point existing OPEN losers of the loser directly at the survivor. + db.execute( + "UPDATE fact SET superseded_by=? WHERE superseded_by=? AND valid_to IS NULL", + (survivor_id, loser_id), + ) + self._commit() + + def set_tier(self, fact_id: str, tier: str) -> None: + """Move a fact between the hot/cold tiers — fact.tier + fact_vec.tier, one txn.""" + db = self._db + db.execute("UPDATE fact SET tier=? WHERE id=?", (tier, fact_id)) + db.execute("UPDATE fact_vec SET tier=? WHERE fact_id=?", (tier, fact_id)) + self._commit() + + def get_embedding(self, fact_id: str) -> Optional[np.ndarray]: + """Read a fact's stored full-dim vector back (no re-embed). None if absent. + + Mirrors add_fact's serialization: vec0 holds float32 blobs, read via + np.frombuffer — an exact round-trip.""" + row = self._db.execute( + "SELECT embedding FROM fact_vec WHERE fact_id=?", (fact_id,) + ).fetchone() + if row is None or row["embedding"] is None: + return None + return np.frombuffer(row["embedding"], dtype=np.float32) + + def iter_latest_facts(self, after_id: Optional[str] = None) -> Iterator[Fact]: + """Id high-water scan over is_latest=1 rows (evict/summarize candidates). + Ids are time-sortable (types.new_id), so id order is ingestion order.""" + if after_id is None: + rows = self._db.execute( + "SELECT * FROM fact WHERE is_latest=1 ORDER BY id" + ).fetchall() + else: + rows = self._db.execute( + "SELECT * FROM fact WHERE is_latest=1 AND id > ? ORDER BY id", + (after_id,), + ).fetchall() + for r in rows: + yield _row_to_fact(r) + + def iter_slots_touched_since(self, cursor_id: str) -> Iterator[tuple[str, str]]: + """DISTINCT (subject_id, predicate) slots that GAINED a member since the cursor. + + Keyed on any fact with id > cursor — so a duplicate landing on a long-quiet + slot resurfaces it (the verified cursor gap a bare latest-scan would miss). + Slot transforms then read the full slot via find_latest_in_slot.""" + rows = self._db.execute( + "SELECT DISTINCT subject_id, predicate FROM fact WHERE id > ? " + "ORDER BY subject_id, predicate", + (cursor_id,), + ).fetchall() + for r in rows: + yield (r["subject_id"], r["predicate"]) def close(self) -> None: self._db.close() diff --git a/tests/test_store_maintenance_verbs.py b/tests/test_store_maintenance_verbs.py new file mode 100644 index 0000000..5f6afcc --- /dev/null +++ b/tests/test_store_maintenance_verbs.py @@ -0,0 +1,257 @@ +"""Task 1 — store maintenance plumbing: busy_timeout, batch(), new verbs. + +All offline (FakeEmbedder), against a real SqliteStore. These pin the store-level +foundation the sleep-time maintenance job stands on (design spec §4.0, §7.1): + - busy_timeout PRAGMA is applied on connect + - batch() is an atomic unit-of-work: one COMMIT at exit, ROLLBACK on exception + - retire_duplicate / set_tier keep the fact and fact_vec surfaces in sync + - get_embedding round-trips the stored float32 vector (no re-embed) + - the two cursor iterators find the right rows, incl. the verified cursor gap + - the retire_duplicate chain invariant survives a transitive B→A→D retirement +""" + +from __future__ import annotations + +import numpy as np +import pytest + +from lean_memory.embed.fake import FakeEmbedder +from lean_memory.store.sqlite_store import SqliteStore +from lean_memory.types import Entity, Episode, Fact + + +@pytest.fixture +def store(tmp_path): + emb = FakeEmbedder() + s = SqliteStore(tmp_path / "ns.db", dim=emb.dim, coarse_dim=emb.coarse_dim) + ep = Episode(namespace="ns", raw="seed", t_ref=1_000) + s.add_episode(ep) + ent = s.upsert_entity(Entity(namespace="ns", name="user", type="person")) + s._emb = emb + s._ep = ep + s._ent = ent + yield s + s.close() + + +def _add_fact(store, text, *, valid_at, is_latest=1, valid_to=None, tier="hot"): + f = Fact( + namespace="ns", subject_id=store._ent.id, predicate="works_at", + fact_text=text, valid_at=valid_at, episode_id=store._ep.id, + is_latest=is_latest, valid_to=valid_to, tier=tier, + ) + full, coarse = store._emb.embed_with_coarse(text) + store.add_fact(f, full, coarse) + return f + + +# ── Step 1: busy_timeout ── +def test_busy_timeout_default_applied(tmp_path): + s = SqliteStore(tmp_path / "d.db") + assert s._db.execute("PRAGMA busy_timeout").fetchone()[0] == 1500 + s.close() + + +def test_busy_timeout_override(tmp_path): + s = SqliteStore(tmp_path / "m.db", busy_timeout_ms=5000) + assert s._db.execute("PRAGMA busy_timeout").fetchone()[0] == 5000 + s.close() + + +# ── Step 2: batch() atomicity ── +def test_batch_single_commit_visible(store): + with store.batch(): + _add_fact(store, "user works at acme", valid_at=1_000) + _add_fact(store, "user works at globex", valid_at=2_000) + n = store._db.execute("SELECT COUNT(*) FROM fact").fetchone()[0] + assert n == 2, "both rows committed once at batch exit" + + +def test_batch_rolls_back_on_exception(store): + _add_fact(store, "user works at acme", valid_at=1_000) # committed pre-batch + with pytest.raises(RuntimeError): + with store.batch(): + _add_fact(store, "user works at globex", valid_at=2_000) + raise RuntimeError("boom mid-batch") + rows = store._db.execute("SELECT fact_text FROM fact").fetchall() + texts = [r["fact_text"] for r in rows] + assert texts == ["user works at acme"], "mid-batch work rolled back, pre-batch survives" + + +def test_batch_not_reentrant(store): + with pytest.raises(RuntimeError): + with store.batch(): + with store.batch(): + pass + # the connection recovered — a fresh batch still works + with store.batch(): + _add_fact(store, "user works at acme", valid_at=1_000) + assert store._db.execute("SELECT COUNT(*) FROM fact").fetchone()[0] == 1 + + +def test_batch_suppresses_per_call_commit(store): + """Inside a batch, a mutator's own commit() is suppressed — the row is not yet + durable to a second connection until the batch's single COMMIT.""" + with store.batch(): + _add_fact(store, "user works at acme", valid_at=1_000) + other = SqliteStore(store.path) + try: + seen = other._db.execute("SELECT COUNT(*) FROM fact").fetchone()[0] + finally: + other.close() + assert seen == 0, "uncommitted batch write invisible to another connection" + other = SqliteStore(store.path) + try: + seen = other._db.execute("SELECT COUNT(*) FROM fact").fetchone()[0] + finally: + other.close() + assert seen == 1, "visible after the batch COMMIT" + + +# ── Step 3: retire_duplicate two-surface sync ── +def test_retire_duplicate_two_surface(store): + survivor = _add_fact(store, "user works at acme", valid_at=1_000) + loser = _add_fact(store, "user works at acme", valid_at=2_000) + store.retire_duplicate(loser.id, survivor.id) + + row = store._db.execute( + "SELECT is_latest, superseded_by, valid_to FROM fact WHERE id=?", (loser.id,) + ).fetchone() + assert row["is_latest"] == 0 + assert row["superseded_by"] == survivor.id + assert row["valid_to"] is None, "valid_to UNTOUCHED (verb (c) as-of-safe)" + + vec = store._db.execute( + "SELECT is_latest FROM fact_vec WHERE fact_id=?", (loser.id,) + ).fetchone() + assert vec["is_latest"] == 0, "fact_vec mirrored — loser drops out of latest search" + + surv = store._db.execute( + "SELECT is_latest FROM fact WHERE id=?", (survivor.id,) + ).fetchone() + assert surv["is_latest"] == 1, "survivor stays latest" + + +def test_retire_duplicate_survivor_resolved_to_canonical(store): + """(i) If the survivor arg is itself retired, resolve to its live canonical.""" + canonical = _add_fact(store, "user works at acme", valid_at=1_000) + mid = _add_fact(store, "user works at acme", valid_at=2_000) + store.retire_duplicate(mid.id, canonical.id) # mid → canonical + loser = _add_fact(store, "user works at acme", valid_at=3_000) + store.retire_duplicate(loser.id, mid.id) # arg is mid (retired) → must resolve to canonical + + row = store._db.execute( + "SELECT superseded_by FROM fact WHERE id=?", (loser.id,) + ).fetchone() + assert row["superseded_by"] == canonical.id, "survivor resolved to live canonical (depth 1)" + + +def test_retire_duplicate_chain_invariant_transitive(store): + """(ii) rev-3 blocker: retire B→A, then A→D re-points B to D. After it, ZERO open + retired duplicates point at a non-latest row.""" + A = _add_fact(store, "user works at acme", valid_at=1_000) + B = _add_fact(store, "user works at acme", valid_at=2_000) + D = _add_fact(store, "user works at acme", valid_at=3_000) + + store.retire_duplicate(B.id, A.id) # B → A + store.retire_duplicate(A.id, D.id) # A → D ; must re-point B → D too + + b = store._db.execute("SELECT superseded_by FROM fact WHERE id=?", (B.id,)).fetchone() + a = store._db.execute("SELECT superseded_by FROM fact WHERE id=?", (A.id,)).fetchone() + assert a["superseded_by"] == D.id + assert b["superseded_by"] == D.id, "B re-pointed to the live canonical D" + + # Standing invariant: no open retired duplicate points at a non-latest row. + orphans = store._db.execute( + """SELECT f.id FROM fact f + JOIN fact t ON t.id = f.superseded_by + WHERE f.superseded_by IS NOT NULL AND f.valid_to IS NULL AND t.is_latest = 0""" + ).fetchall() + assert orphans == [], "zero open duplicates pointing at a non-latest row" + + +def test_retire_duplicate_survivor_equals_loser_is_rejected(store): + f = _add_fact(store, "user works at acme", valid_at=1_000) + with pytest.raises(ValueError): + store.retire_duplicate(f.id, f.id) + row = store._db.execute("SELECT is_latest FROM fact WHERE id=?", (f.id,)).fetchone() + assert row["is_latest"] == 1, "no-op on the store — the fact stays latest" + + +# ── Step 3: set_tier two-surface sync ── +def test_set_tier_two_surface(store): + f = _add_fact(store, "user works at acme", valid_at=1_000, tier="hot") + store.set_tier(f.id, "cold") + frow = store._db.execute("SELECT tier FROM fact WHERE id=?", (f.id,)).fetchone() + vrow = store._db.execute("SELECT tier FROM fact_vec WHERE fact_id=?", (f.id,)).fetchone() + assert frow["tier"] == "cold" + assert vrow["tier"] == "cold", "fact_vec tier metadata mirrored" + + +# ── Step 3: get_embedding round-trip ── +def test_get_embedding_roundtrip(store): + f = _add_fact(store, "user works at acme", valid_at=1_000) + full, _ = store._emb.embed_with_coarse("user works at acme") + got = store.get_embedding(f.id) + assert got is not None + assert got.dtype == np.float32 + assert np.array_equal(got, full.astype(np.float32)), "exact float32 round-trip, no re-embed" + + +def test_get_embedding_missing_returns_none(store): + assert store.get_embedding("nope-does-not-exist") is None + + +# ── Step 3: iterators ── +def test_iter_latest_facts_high_water(store): + f1 = _add_fact(store, "user works at acme", valid_at=1_000) + f2 = _add_fact(store, "user works at globex", valid_at=2_000) + store.supersede_fact(f1.id, f2.id, valid_to=2_000) # f1 no longer latest + f3 = _add_fact(store, "user works at zorbex", valid_at=3_000) + + all_latest = [f.id for f in store.iter_latest_facts()] + assert f1.id not in all_latest, "superseded fact excluded (is_latest=0)" + assert set(all_latest) == {f2.id, f3.id} + # yielded ascending by id (the scan's high-water order) + assert all_latest == sorted(all_latest) + + # The scan is a strict id high-water cut. Ids are time-sortable but the same-ms + # tiebreak is random, so key off the actual max id, not insertion order. + latest_ids = sorted({f2.id, f3.id}) + after = [f.id for f in store.iter_latest_facts(after_id=latest_ids[0])] + assert after == [latest_ids[1]], "returns exactly the is_latest ids strictly greater" + assert list(store.iter_latest_facts(after_id=latest_ids[1])) == [], "cursor at max → empty" + + +def _add_with_id(store, fact_id, text, predicate, *, valid_at): + f = Fact( + namespace="ns", subject_id=store._ent.id, predicate=predicate, + fact_text=text, valid_at=valid_at, episode_id=store._ep.id, id=fact_id, + ) + full, coarse = store._emb.embed_with_coarse(text) + store.add_fact(f, full, coarse) + return f + + +def test_iter_slots_touched_since_finds_quiet_slot_duplicate(store): + """The verified cursor gap: a duplicate landing on a long-quiet slot must be + surfaced by joining new-facts-since-cursor → DISTINCT slots. + + Explicit monotonic ids so the cursor deterministically precedes the later inserts + (new_id()'s same-ms tiebreak is random — see test_iter_latest_facts_high_water).""" + # Slot `works_at` gets an early fact; the cursor is taken just after it. + early = _add_with_id(store, "0001-early", "user works at acme", "works_at", valid_at=1_000) + cursor = early.id # "0001-early" + + # A different, busy predicate churns; then MUCH later a duplicate lands on the + # original quiet slot (same subject+predicate as `early`). + _add_with_id(store, "0002-hobby", "user likes jazz", "likes", valid_at=2_000) + late_dup = _add_with_id(store, "0003-dup", "user works at acme", "works_at", valid_at=9_000) + assert late_dup.id > cursor + + slots = set(store.iter_slots_touched_since(cursor)) + assert (store._ent.id, "works_at") in slots, "quiet works_at slot resurfaced by its new dup" + assert (store._ent.id, "likes") in slots, "the busy slot is included too" + + # A cursor at/after every fact yields nothing (no slot gained a member since). + assert set(store.iter_slots_touched_since(late_dup.id)) == set() From 5dc0984f0301981231e994cc39f0ae72c8743fb1 Mon Sep 17 00:00:00 2001 From: Wuesteon Date: Thu, 16 Jul 2026 19:32:47 +0800 Subject: [PATCH 03/14] =?UTF-8?q?feat(store):=20schema=20v2=20=E2=80=94=20?= =?UTF-8?q?versioned=20migration,=20ledger,=20proposal=20CRUD?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First persisted-format change. Restructures _init_schema into a user_version-gated migration: the always-run SCHEMA_SQL blob (all IF-NOT-EXISTS) now carries the three maintenance tables (fact_derivation, maintenance_run, maintenance_proposal) + indexes, while the non-idempotent `ALTER TABLE fact ADD COLUMN record_kind` lives ONLY in the `if user_version < 2:` branch, stamping 2 after and never lowering a newer stamp. Threads record_kind through the Fact dataclass, add_fact's INSERT, and _row_to_fact. Adds pure ledger/proposal CRUD to the Store ABC + SqliteStore (create_run/heartbeat_run/finish_run/get_live_run, stage_proposal/ get_proposal/list_proposals) — no decide/apply logic. The maintenance_run INSERT is the atomic lease claim via the ux_run_live partial-unique index. Ships the WP6 migration obligation: a checked-in tiny v1-format fixture DB (built by a committed deterministic script) + a migration regression test that opens it, upgrades once, and REOPENS cleanly — pinning the ALTER-idempotence trap with a real second open. Updates the console EXPECTED_SCHEMA_FINGERPRINT for the new CREATE TABLE lines. Core: 169 passed, 1 skipped. Console: 125 passed. Co-Authored-By: Claude Fable 5 --- .../src/lean_memory_console/inspect_sql.py | 2 +- src/lean_memory/store/base.py | 61 ++++ src/lean_memory/store/schema.py | 42 +++ src/lean_memory/store/sqlite_store.py | 137 +++++++- src/lean_memory/types.py | 1 + tests/fixtures/make_v1_fixture.py | 211 ++++++++++++ tests/fixtures/v1_format.db | Bin 0 -> 253952 bytes tests/test_schema_migration.py | 304 ++++++++++++++++++ tests/test_schema_version.py | 38 ++- 9 files changed, 772 insertions(+), 24 deletions(-) create mode 100644 tests/fixtures/make_v1_fixture.py create mode 100644 tests/fixtures/v1_format.db create mode 100644 tests/test_schema_migration.py diff --git a/console/src/lean_memory_console/inspect_sql.py b/console/src/lean_memory_console/inspect_sql.py index e1c3580..7df2ba9 100644 --- a/console/src/lean_memory_console/inspect_sql.py +++ b/console/src/lean_memory_console/inspect_sql.py @@ -311,7 +311,7 @@ def list_entities(db_path: Path, page: int = 1, page_size: int = 50) -> dict: # Filled once from the first run's printed digests (Step 5), then a test pins # equality so engine drift turns the suite red. EXPECTED_SCHEMA_FINGERPRINT = ( - "c12a9560c065a6cc9be19b91b71b4ebee1b74eceff477040073f85943c43742f" + "a6c7f41188a196929ff4e7c257a181c100af9f0f7091aa93d428a9a7c0eec8ef" ) EXPECTED_SANITIZER_FINGERPRINT = ( "dfef8699dd9519f2ef6592be577d25fdb2ae761e298d7063ced3323a14e2cf77" diff --git a/src/lean_memory/store/base.py b/src/lean_memory/store/base.py index 22516b1..2e842b1 100644 --- a/src/lean_memory/store/base.py +++ b/src/lean_memory/store/base.py @@ -116,6 +116,67 @@ def iter_slots_touched_since(self, cursor_id: str) -> Iterator[tuple[str, str]]: id > cursor) since the cursor — including duplicates landing on long-quiet slots (the verified cursor gap).""" + # ── maintenance ledger + proposal CRUD (schema v2, design spec §4.0/§5) ── + # Pure row CRUD — no decide/apply logic (that is the proposal lifecycle, a + # later task). The create-half is needed by the maintenance runner. + @abstractmethod + def create_run( + self, namespace: str, trigger: str, started_at: int, config_hash: Optional[str] + ) -> str: + """Claim the maintenance lease: INSERT a status='running' maintenance_run and + return its id. The ux_run_live partial-unique index makes this the atomic + lease claim — a second live run for the same namespace raises + sqlite3.IntegrityError (§7.2).""" + + @abstractmethod + def heartbeat_run(self, run_id: str, at: int) -> None: + """Bump maintenance_run.heartbeat_at — proof the run is still alive (§7.2).""" + + @abstractmethod + def finish_run( + self, + run_id: str, + status: str, + finished_at: int, + stats_json: Optional[str], + cursor_id: Optional[str], + ) -> None: + """Close a run: stamp status ('ok'|'aborted'), finished_at, stats_json, + cursor_id. Clearing status='running' releases the ux_run_live lease.""" + + @abstractmethod + def get_live_run(self, namespace: str) -> Optional[dict]: + """The status='running' run for a namespace, or None. (At most one exists — + ux_run_live enforces it.)""" + + @abstractmethod + def stage_proposal( + self, + run_id: str, + namespace: str, + kind: str, + payload_json: str, + created_at: int, + expires_at: int, + evidence_backend: Optional[str] = None, + ) -> str: + """INSERT a status='pending' maintenance_proposal and return its id.""" + + @abstractmethod + def get_proposal(self, proposal_id: str) -> Optional[dict]: + """A single proposal row as a dict, or None.""" + + @abstractmethod + def list_proposals( + self, + namespace: str, + status: Optional[str] = None, + kind: Optional[str] = None, + limit: int = 100, + ) -> list[dict]: + """Proposals for a namespace, newest first, optionally filtered by status + and/or kind.""" + # ── lifecycle ── @abstractmethod def close(self) -> None: ... diff --git a/src/lean_memory/store/schema.py b/src/lean_memory/store/schema.py index 2ee918f..b190f1c 100644 --- a/src/lean_memory/store/schema.py +++ b/src/lean_memory/store/schema.py @@ -80,4 +80,46 @@ fact_id UNINDEXED, fact_text ); + +-- ── MAINTENANCE LAYER (schema v2; sleep-time job, design spec §5) ──── +-- All IF-NOT-EXISTS so they belong in the always-run blob. The non-idempotent +-- v2 DDL (ALTER TABLE fact ADD COLUMN record_kind) lives ONLY in the versioned +-- `if user_version < 2:` branch of _init_schema — never here — because ADD +-- COLUMN raises 'duplicate column name' on reopen. +CREATE TABLE IF NOT EXISTS fact_derivation ( + summary_id TEXT NOT NULL REFERENCES fact(id), + source_id TEXT NOT NULL REFERENCES fact(id), + run_id TEXT NOT NULL, + created_at INTEGER NOT NULL, + PRIMARY KEY (summary_id, source_id) +); +CREATE INDEX IF NOT EXISTS ix_derivation_source ON fact_derivation(source_id); + -- the staleness cascade's lookup path (§4.3) + +CREATE TABLE IF NOT EXISTS maintenance_run ( + id TEXT PRIMARY KEY, namespace TEXT NOT NULL, + started_at INTEGER NOT NULL, finished_at INTEGER, + heartbeat_at INTEGER, + trigger TEXT NOT NULL, -- 'cli'|'mcp'|'auto'|'console' + cursor_id TEXT, + config_hash TEXT, stats_json TEXT, + status TEXT NOT NULL DEFAULT 'running' -- 'running'|'ok'|'aborted' +); +CREATE UNIQUE INDEX IF NOT EXISTS ux_run_live + ON maintenance_run(namespace) WHERE status='running'; + -- the INSERT is the atomic lease claim: a second runner gets a constraint + -- error, not a silent second row (verified race gap in rev 1) + +CREATE TABLE IF NOT EXISTS maintenance_proposal ( + id TEXT PRIMARY KEY, run_id TEXT NOT NULL REFERENCES maintenance_run(id), + namespace TEXT NOT NULL, + kind TEXT NOT NULL, -- 'dedup_near'|'summarize'|'evict' + payload_json TEXT NOT NULL, + status TEXT NOT NULL DEFAULT 'pending', -- 'pending'|'approved'|'rejected'|'edited'|'expired' + expiry_reason TEXT, -- 'timeout'|'stale_target' + created_at INTEGER NOT NULL, expires_at INTEGER NOT NULL, + decided_at INTEGER, decided_by TEXT, -- 'console'|'mcp'|'expiry' + applied_at INTEGER, edited_text TEXT, + evidence_backend TEXT -- 'stub'|'ollama:'|embedder id +); """ diff --git a/src/lean_memory/store/sqlite_store.py b/src/lean_memory/store/sqlite_store.py index e920f39..140941f 100644 --- a/src/lean_memory/store/sqlite_store.py +++ b/src/lean_memory/store/sqlite_store.py @@ -23,7 +23,7 @@ import numpy as np from sqlite_vec import serialize_float32 -from ..types import Entity, Episode, Fact +from ..types import Entity, Episode, Fact, new_id from .base import Store from .schema import SCHEMA_SQL @@ -101,14 +101,35 @@ def batch(self) -> Iterator[None]: def _init_schema(self) -> None: self._check_existing_dims() + # Always-run blob: every statement is CREATE ... IF NOT EXISTS, so it is a + # no-op on an already-created DB (fresh or migrated) and creates the v2 + # tables/indexes on any DB that lacks them. sql = SCHEMA_SQL.format(dim=self.dim, coarse_dim=self.coarse_dim) self._db.executescript(sql) + # Schema-version stamp — the migration anchor for future releases. # Version 1 == the 0.1.x layout; pre-stamp files (0.1.0–0.1.2, version 0) - # have an identical schema and are upgraded in place. Never write over a + # have an identical spine and are treated as version 1. Never write over a # NEWER release's stamp. - if self._db.execute("PRAGMA user_version").fetchone()[0] == 0: + version = self._db.execute("PRAGMA user_version").fetchone()[0] + if version == 0: + version = 1 self._db.execute("PRAGMA user_version = 1") + + # ── Versioned migrations. Each branch runs the NON-idempotent DDL for its + # version exactly once, keyed off user_version. A fresh DB is version 1 + # here (just stamped above), so it flows through the same `< 2` branch and + # gains record_kind via the SAME ALTER — there is no separate fresh path. + # ADD COLUMN is not idempotent (raises 'duplicate column name' on reopen), + # so it MUST live here and never in the always-run blob. + if version < 2: + self._db.execute( + "ALTER TABLE fact ADD COLUMN record_kind TEXT NOT NULL " + "DEFAULT 'fact'" # 'fact'|'summary' + ) + self._db.execute("PRAGMA user_version = 2") + version = 2 + self._db.commit() def _check_existing_dims(self) -> None: @@ -182,15 +203,15 @@ def add_fact(self, fact: Fact, embedding: np.ndarray, embedding_256: np.ndarray) valid_at, valid_to, superseded_by, is_latest, ingested_at, expired_at, invalidated_by, confidence, salience, last_access, access_count, is_inference, tier, - episode_id, created_at) - VALUES (?,?,?,?,?,?,?, ?,?,?,?, ?,?,?, ?,?,?,?,?,?, ?,?)""", + record_kind, episode_id, created_at) + VALUES (?,?,?,?,?,?,?, ?,?,?,?, ?,?,?, ?,?,?,?,?,?, ?,?,?)""", (fact.id, fact.namespace, fact.subject_id, fact.predicate, fact.object_id, fact.object_literal, fact.fact_text, fact.valid_at, fact.valid_to, fact.superseded_by, fact.is_latest, fact.ingested_at, fact.expired_at, fact.invalidated_by, fact.confidence, fact.salience, fact.last_access, fact.access_count, fact.is_inference, fact.tier, - fact.episode_id, fact.created_at), + fact.record_kind, fact.episode_id, fact.created_at), ) db.execute( "INSERT INTO fact_vec(fact_id, namespace, is_latest, tier, embedding, embedding_256) " @@ -438,6 +459,107 @@ def iter_slots_touched_since(self, cursor_id: str) -> Iterator[tuple[str, str]]: for r in rows: yield (r["subject_id"], r["predicate"]) + # ── maintenance ledger + proposal CRUD (schema v2; design spec §4.0/§5) ── + # Pure row CRUD. No decide/apply logic lives here (that is the proposal + # lifecycle, a later task) — these just read and write the ledger tables. + def create_run( + self, namespace: str, trigger: str, started_at: int, config_hash: Optional[str] + ) -> str: + run_id = new_id() + # The INSERT is the atomic lease claim: ux_run_live (partial-unique on + # namespace WHERE status='running') makes a second live run for the same + # namespace raise sqlite3.IntegrityError, not a silent second row. + self._db.execute( + "INSERT INTO maintenance_run(" + "id, namespace, started_at, finished_at, heartbeat_at, trigger, " + "cursor_id, config_hash, stats_json, status) " + "VALUES (?,?,?,?,?,?,?,?,?, 'running')", + (run_id, namespace, started_at, None, started_at, trigger, + None, config_hash, None), + ) + self._commit() + return run_id + + def heartbeat_run(self, run_id: str, at: int) -> None: + self._db.execute( + "UPDATE maintenance_run SET heartbeat_at=? WHERE id=?", (at, run_id) + ) + self._commit() + + def finish_run( + self, + run_id: str, + status: str, + finished_at: int, + stats_json: Optional[str], + cursor_id: Optional[str], + ) -> None: + # Clearing status='running' releases the ux_run_live lease. + self._db.execute( + "UPDATE maintenance_run SET status=?, finished_at=?, stats_json=?, " + "cursor_id=? WHERE id=?", + (status, finished_at, stats_json, cursor_id, run_id), + ) + self._commit() + + def get_live_run(self, namespace: str) -> Optional[dict]: + row = self._db.execute( + "SELECT * FROM maintenance_run WHERE namespace=? AND status='running'", + (namespace,), + ).fetchone() + return dict(row) if row else None + + def stage_proposal( + self, + run_id: str, + namespace: str, + kind: str, + payload_json: str, + created_at: int, + expires_at: int, + evidence_backend: Optional[str] = None, + ) -> str: + proposal_id = new_id() + self._db.execute( + "INSERT INTO maintenance_proposal(" + "id, run_id, namespace, kind, payload_json, status, " + "created_at, expires_at, evidence_backend) " + "VALUES (?,?,?,?,?, 'pending', ?,?,?)", + (proposal_id, run_id, namespace, kind, payload_json, + created_at, expires_at, evidence_backend), + ) + self._commit() + return proposal_id + + def get_proposal(self, proposal_id: str) -> Optional[dict]: + row = self._db.execute( + "SELECT * FROM maintenance_proposal WHERE id=?", (proposal_id,) + ).fetchone() + return dict(row) if row else None + + def list_proposals( + self, + namespace: str, + status: Optional[str] = None, + kind: Optional[str] = None, + limit: int = 100, + ) -> list[dict]: + where = ["namespace=?"] + args: list = [namespace] + if status is not None: + where.append("status=?") + args.append(status) + if kind is not None: + where.append("kind=?") + args.append(kind) + clause = " AND ".join(where) + rows = self._db.execute( + f"SELECT * FROM maintenance_proposal WHERE {clause} " + f"ORDER BY created_at DESC, id DESC LIMIT ?", + (*args, limit), + ).fetchall() + return [dict(r) for r in rows] + def close(self) -> None: self._db.close() @@ -476,5 +598,6 @@ def _row_to_fact(row: sqlite3.Row) -> Fact: invalidated_by=row["invalidated_by"], confidence=row["confidence"], salience=row["salience"], last_access=row["last_access"], access_count=row["access_count"], is_inference=row["is_inference"], - tier=row["tier"], episode_id=row["episode_id"], created_at=row["created_at"], + tier=row["tier"], record_kind=row["record_kind"], + episode_id=row["episode_id"], created_at=row["created_at"], ) diff --git a/src/lean_memory/types.py b/src/lean_memory/types.py index 6dddf0d..09d4bb0 100644 --- a/src/lean_memory/types.py +++ b/src/lean_memory/types.py @@ -87,6 +87,7 @@ class Fact: access_count: int = 0 is_inference: int = 0 tier: str = "hot" # 'hot'|'cold' + record_kind: str = "fact" # 'fact'|'summary' (schema v2; sleep-time maintenance) id: str = field(default_factory=new_id) created_at: int = field(default_factory=now_ms) diff --git a/tests/fixtures/make_v1_fixture.py b/tests/fixtures/make_v1_fixture.py new file mode 100644 index 0000000..6c61645 --- /dev/null +++ b/tests/fixtures/make_v1_fixture.py @@ -0,0 +1,211 @@ +"""Build the checked-in v1-format fixture DB (tests/fixtures/v1_format.db). + +This reproduces the schema **as it existed at user_version=1** — the 0.1.x layout, +BEFORE the schema-v2 migration (no `fact.record_kind`, no maintenance tables). It +does NOT import the current SqliteStore, whose _init_schema would immediately +migrate the file to v2 and defeat the purpose. Instead it lays down the v1 DDL by +hand and inserts a few rows through plain INSERTs that mirror add_fact's v1 column +list. + +Determinism: fixed ids, fixed timestamps, fixed tiny vectors — so re-running the +script byte-reproduces the same DB (modulo SQLite page layout, which is stable for +this fixed input). Run from the repo root: + + .venv/bin/python tests/fixtures/make_v1_fixture.py + +The migration regression test (tests/test_schema_migration.py) opens the result +with the CURRENT code and asserts a clean, once-only 1→2 upgrade. +""" + +from __future__ import annotations + +import struct +from pathlib import Path + +import sqlite_vec + +# ── v1 SCHEMA (verbatim from the store/schema.py layout at user_version=1; +# fact has NO record_kind column, and there are NO maintenance tables) ── +DIM = 8 +COARSE_DIM = 4 + +V1_SCHEMA = f""" +CREATE TABLE IF NOT EXISTS episode ( + id TEXT PRIMARY KEY, + namespace TEXT NOT NULL, + raw TEXT NOT NULL, + source TEXT, + t_ref INTEGER NOT NULL, + created_at INTEGER NOT NULL +); + +CREATE TABLE IF NOT EXISTS entity ( + id TEXT PRIMARY KEY, + namespace TEXT NOT NULL, + name TEXT NOT NULL, + type TEXT, + summary TEXT, + resolved_id TEXT, + created_at INTEGER NOT NULL +); +CREATE INDEX IF NOT EXISTS ix_entity_lookup ON entity(namespace, name, type); + +CREATE TABLE IF NOT EXISTS fact ( + id TEXT PRIMARY KEY, + namespace TEXT NOT NULL, + subject_id TEXT NOT NULL REFERENCES entity(id), + predicate TEXT NOT NULL, + object_id TEXT REFERENCES entity(id), + object_literal TEXT, + fact_text TEXT NOT NULL, + + valid_at INTEGER NOT NULL, + valid_to INTEGER, + superseded_by TEXT REFERENCES fact(id), + is_latest INTEGER NOT NULL DEFAULT 1, + + ingested_at INTEGER NOT NULL, + expired_at INTEGER, + invalidated_by TEXT REFERENCES fact(id), + + confidence REAL NOT NULL DEFAULT 1.0, + salience REAL NOT NULL DEFAULT 0.0, + last_access INTEGER, + access_count INTEGER NOT NULL DEFAULT 0, + is_inference INTEGER NOT NULL DEFAULT 0, + tier TEXT NOT NULL DEFAULT 'hot', + episode_id TEXT NOT NULL REFERENCES episode(id), + created_at INTEGER NOT NULL +); +CREATE INDEX IF NOT EXISTS ix_fact_ns_latest ON fact(namespace, is_latest); +CREATE INDEX IF NOT EXISTS ix_fact_slot ON fact(namespace, subject_id, predicate); +CREATE INDEX IF NOT EXISTS ix_fact_valid ON fact(namespace, valid_at, valid_to); + +CREATE VIRTUAL TABLE IF NOT EXISTS fact_vec USING vec0( + fact_id TEXT PRIMARY KEY, + is_latest INTEGER, + tier TEXT, + namespace TEXT, + embedding FLOAT[{DIM}], + embedding_256 FLOAT[{COARSE_DIM}] +); + +CREATE VIRTUAL TABLE IF NOT EXISTS fact_fts USING fts5( + fact_id UNINDEXED, + fact_text +); +""" + +FIXTURE_PATH = Path(__file__).with_name("v1_format.db") + +NS = "v1user" +EP_ID = "0000ep000000-0000000000000001" +ENT_ID = "0000en000000-0000000000000001" + +# Two facts in one (subject, predicate) slot: an older superseded row + the latest. +FACTS = [ + { + "id": "0000fa000000-0000000000000001", + "predicate": "works_at", + "object_literal": "Acme", + "fact_text": "The user works at Acme.", + "valid_at": 1_700_000_000_000, + "valid_to": 1_700_000_100_000, + "superseded_by": "0000fa000000-0000000000000002", + "is_latest": 0, + }, + { + "id": "0000fa000000-0000000000000002", + "predicate": "works_at", + "object_literal": "Globex", + "fact_text": "The user works at Globex.", + "valid_at": 1_700_000_100_000, + "valid_to": None, + "superseded_by": None, + "is_latest": 1, + }, +] + + +def _vec(seed: int, dim: int) -> bytes: + """A fixed, L2-normalized float32 vector — vec0's float32 wire format.""" + raw = [float((seed * (i + 1)) % 7 + 1) for i in range(dim)] + norm = sum(x * x for x in raw) ** 0.5 + unit = [x / norm for x in raw] + return struct.pack(f"{dim}f", *unit) + + +def build(path: Path = FIXTURE_PATH) -> Path: + import sqlite3 + + if path.exists(): + path.unlink() + db = sqlite3.connect(path) + db.enable_load_extension(True) + sqlite_vec.load(db) + db.enable_load_extension(False) + db.executescript(V1_SCHEMA) + + db.execute( + "INSERT INTO episode(id, namespace, raw, source, t_ref, created_at) " + "VALUES (?,?,?,?,?,?)", + (EP_ID, NS, "seed episode", "user", 1_700_000_000_000, 1_700_000_000_000), + ) + db.execute( + "INSERT INTO entity(id, namespace, name, type, summary, resolved_id, created_at) " + "VALUES (?,?,?,?,?,?,?)", + (ENT_ID, NS, "user", "person", None, None, 1_700_000_000_000), + ) + + for i, f in enumerate(FACTS, start=1): + # v1 fact column list — deliberately WITHOUT record_kind. + db.execute( + """INSERT INTO fact( + id, namespace, subject_id, predicate, object_id, object_literal, fact_text, + valid_at, valid_to, superseded_by, is_latest, + ingested_at, expired_at, invalidated_by, + confidence, salience, last_access, access_count, is_inference, tier, + episode_id, created_at) + VALUES (?,?,?,?,?,?,?, ?,?,?,?, ?,?,?, ?,?,?,?,?,?, ?,?)""", + (f["id"], NS, ENT_ID, f["predicate"], None, f["object_literal"], + f["fact_text"], + f["valid_at"], f["valid_to"], f["superseded_by"], f["is_latest"], + f["valid_at"], None, None, + 1.0, 1.0, None, 0, 0, "hot", + EP_ID, f["valid_at"]), + ) + db.execute( + "INSERT INTO fact_vec(fact_id, namespace, is_latest, tier, embedding, embedding_256) " + "VALUES (?,?,?,?,?,?)", + (f["id"], NS, f["is_latest"], "hot", _vec(i, DIM), _vec(i, COARSE_DIM)), + ) + db.execute( + "INSERT INTO fact_fts(fact_id, fact_text) VALUES (?,?)", + (f["id"], f["fact_text"]), + ) + + # The v1 stamp — this is what makes the file "v1-format" for the migration. + db.execute("PRAGMA user_version = 1") + db.commit() + db.close() + return path + + +if __name__ == "__main__": + out = build() + import sqlite3 + + check = sqlite3.connect(out) + version = check.execute("PRAGMA user_version").fetchone()[0] + cols = [r[1] for r in check.execute("PRAGMA table_info(fact)").fetchall()] + tables = [ + r[0] + for r in check.execute( + "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name" + ).fetchall() + ] + check.close() + assert version == 1, version + assert "record_kind" not in cols, "fixture must be v1-format (no record_kind)" + assert "maintenance_run" not in tables, "fixture must be v1-format (no v2 tables)" + print(f"built {out} — user_version={version}, fact cols={len(cols)}, tables={tables}") diff --git a/tests/fixtures/v1_format.db b/tests/fixtures/v1_format.db new file mode 100644 index 0000000000000000000000000000000000000000..ca6debe99dba959fd4e8a744f43a9476e70a1b18 GIT binary patch literal 253952 zcmeI*eQX=&eZcWMzKW8ibL=Q*MRjvg>Pf6*hjQySiDxTHLMv)zDORX7MvyQxd9qnc zq(Yu-tHlsADOzA13Jk;k=+~wYFSvtdD4WHC@*p zwJc53y2byK;=g}6AnwHc2jW)u{x@yfaJO4K^2a}j%Z$c5GVwm+oxbn(R#M;U&8FtN ze$n;$J~orJhC1XL8mUz;mrK@D^V6p% z=g(T7%$^;yWZ70O`)n?KdN5%Yj_X>vQnIhqUs@7g`NFDGbsy(Dl$r0;q`^ktfuuPx zp*Kd`2UNVUTDe$n6Ox~BV3VNye4WF3Y-a8;Yj!SY%`VK$q|Xc_OnYL>@%hEKiO^3P zX!PB)l^8P{@VpW-_iQc3%$8#`ZzRS{=NL0)qxbHldE$iLI2`Urad0}-THdz^LTuT7 zcZUdDvb76PWvN@?Zk+2+n8g#Dc4JHVO%b-F^f!9F z&d%n>#5f~I6>DL3`piN$I19(z<0UUry&}8~)y2iS?f4thC+I0!B`mOWY_005qZeeo9dyexHt%awjXQwQ2$Y(~nEz8|34leKF9)ZnIcOpx< zo?jBrrgcZ<@ef0h;*{;0zo)pBJCBLY@))SE6pFS=^Ecbe=j>9cTsf~cpPHGQ%suze z^Ky?MDS!0X$1F=Fef0V6bg^6XMZexK-RIQu74KP9_NHukj~D9n>=W5%tx&RC$=s|P zpnFuhEpj`cT*?<5|DID#+o_~^NK|q#R7riQ>c~2}+twjdRUB>VsIQ*;v>5EgNjoNb z-!7Gl!Y}A=Rhy1fYLb}4^uO{BYLPg65x1N@kSe)7ukpAZ7AC$gs|7iMzSxZ5&unus$_6znywHM^|d zzOqslN0*<)Z@o+^SKQFVVW*nbxn&}Av07Oym*h~d+9qBa&jdrK6}0O}#*JB=d+S*( z(aG&HXH@2;Lfy$1isCt>?iSrNcBrlSVs*9R^0x*U)a%wN^u)+#*Zi8Vox5dR3LjrV z!NV7-&amsaQm$7^wtS9tFD_QlFHQYvtBih^7HhT(>>od^jp|MhClluJ{%el$?%RQ` zLOf|69uUtt{@9>SnEX<;dU16{JxH_;4xv19Z1B#qVEi3(hvG5ISzEEw4Lh1N2L|+4 z)f2MHtFB|AAwq52Fi5y6El&+WO~NgtCRt^o0&SBho79}>9$Kr*?h~clE=!Z!YIeQ4 zbV(e=qI3Lh&F&k`M`H3JRO-9*AHs`KBwORaF( zk9;Fqy}DZS&qF_6CttG{ySykmddi#e*^sUPg=b>EwRM>D>o zws4IA0tg_000IagfB*srAbY#YWly!S^zo! zFTc%HtuX(8ha4Y-i2wo!Ab8i>7$K&y- zt}Qm%`x(A1HF>hQY;SC^+$x)^ezs3ET9(yONM+nYM4~u3{~s~FCw~7=Tya4F0R#|0 z009ILKmY**5I_Kd-6wE=bV|SGCjk+E{eP#=3dYs@|DS2b{~ABreF4%U1Q0*~0R#|0 z009ILKmY**5V$UZp4dQ2S8FFC-LZi#@0$SX{r^`r<449K>z^+ z5I_I{1Q0*~0R#|00D+wpI1q~-(;uJDPUf=tXR`B8PtVQfpPI}){$$FGMvtXhQch3L z&dr;V=&`QWgvn2c1U;G_2q#?192r0Iu$=#o8Gor6Zy9eGe=F{AK>z^+5I_I{1Q0*~ z0R#|00D+q!aG=A-0I@a+?xz6Jb_w!RfXH3Zbj#-ka{gcM+pCFxTo6D20R#|0009IL zKmY**?jV7`@7BAt$SZpqSNFuU=uHk>_v(+_VpXkL-&?M^nMZ;)T`9#kz>} z?)7adqn!Vb7~j#1e-~F=5I_I{1Q0*~0R#|0009ILKwwu13`OGFR4OHZL$28I;6`yU2;RGJmc#3|G%dhKQX?ys{*7k2q1s}0tg_000IagfB*srAfN>9k51{= zdUt;`VnwH}`5D2u`u@M6^%+#g*uI_p58y`NY zX;)9a_Kk0xTv&KR+}9=+7WyZ|;}bG1{@Q0xKKR`qpRD|F-^AJP51%~y{l601-jTN_ zWZE5Mb}NUPmh=C*@vbKRaX|n91Q0*~0R#|0009ILKmdW;P#_vjrI`P}4Lg>Q5I_I{ z1Q0*~0R#|00D;W~8t(l6D_8eCdG?DZe;NPr$tTbDO#I1%M<$N^_190n`$%e{aNna7 zfAojXF~7gL<*ksz{r*4m|F=R{(MALiKmY**5I_I{1Q0*~0R(PLV1537uhzHs#>!wX z1Q0*~0R#|0009ILKmdVTQ{eC9_x~GL_xP>X{X5*=jsnN{{{QXRt;B=?0tg_000Iag zfB*srAb`NS!20|@zyH6k!aW2KKmY**5I_I{1Q0;rjt~gG{~s0MUZ^@j(5;Vy1n%6* z0v~7o|5olk+KvDM2q1s}0tg_000IagfWYPg{`|kjcmFrXA_oEpAb{Ao%`&LPT|Ge6?=ZRCukAZ>ON|;8>3L8o$+gl3&sKero(JRb_7x|yad)-(-pA*&lew&wn|y30YYlCvMCBS9sZ}qROV(5K z)2Aor&sv|%o*lDf^Q>I<*<9NAZDOr(e9JMZGTTI?5(XQ62a@K*gx(nS(ZT_FCL2TY z6Ao+=l%KD2SdY!jJ!Z|$<*eC-nHdpZ{L9zhXX9-m^lKhy^xd#490l&(>nh zY(2)2LJz+d_creHh+9vu{fQw(Y@A0Y?7kx&v(P!j=Z^Z)El+&CyZ) z6*Js}?r2aCI!-;@+aVm4>UY!hY%V*Moo^mWtjUGk-1My2{d9IVHzvjzIjUF-v(sl5 zvcXw6<{mG3k?IxUZKy6T)@{e%D9%;=zt4O?|62-1U@^kO7}{Rk_DnZE-lq%?g$>o=9~i2KeW5bfhhvpGG}3Bt4aR!;Gq$g? zj(DlCR4$dBweYDirlJpTOxt^s=J2rINQE0-t}IrAX3GbAIy73Q4Ot@>?KMk|?AA)H zyj-ZQSz^POB^t77TZPJ6`r+P$d3t!$9+Rat^_P6u+c@8oG!Gu^di8-&6aChC?RsW< zKDRJA<2}duiPpl?)3Z~SIOH=U-InEU76+GiagV^}r#q3QT+c5R9J}tQJpN%QQk=3~ z^Y;|Da_2FzSsnxRl|s>0Y5rz=`J7!Ul`H4f=2J6sley;}dS31kB;}7D`nz| zoi28ZzUbE*ru#afe8qdEAbV4`yvGZ5diIIzvsNhCtz>T24bVL*-4?kWP%h;Qj(^Xo zrtMVHJR~YP7^-EHfTsVa^(b<|hSeOe6m;-noDy>FMwMd6ozp(|-VD5{Bv zs;T(8vaVFyx`HfLo4SJDNq=E)B5C&b>tBinM};gi>6NKo>c}mY?D+LQU+<;n)2dBJ zs`1Hxc_1^a2ijDKpTU~Xp32T=XCKc#ZP^t^oP8tZQd;g7n5exrmRt4xnlp4P&dV)d zQ`QPg>&KytHSeB+ZCW?hEgIqu3*un#s@XVh1bLilOCA;eie0PQCA*YAwUq>)bMtxmc|%mP>M|S8Wq7jc0`&b{@lmgwYmnKLT$QlakT3q|o9 zQg@4P8ave1e6hM(ars*V4C-}j6?$Uivul3M*UsHCE`^V;py1&PRcF|BTq)P9C0jnn zx)&EK=$EGcv{gpGON%wz1@@1h)<$)whm#5Oc>guWc=zReS0SD>4-bgv9Di(3Cro~+ zTD`csq8=n#2ZvA|IW~A_Sup;NxkK@o<*cpP>4qImngav+tLh0^_@&4tzKQN`RAb@uamFY zi{37+@k*m>dmN4=%*nxy9^SD(?!_#B&aq2=?!oW>-NB#!5IO<~Abg`q$J#D$%vt zHI)2B@}Cm_k+>XxJN^gpgJLHx2q1vKjtZ0yxf9OK^R|mlUCc0u*^Tq=T+sT%-wA!R zJKT~wJaREK6=q#ZTc4WFJvq0Kv*zbMHT}f$hs4MAhxJBpFw>ToWtXZ&IsF`z<~=^t zp)xOdSj;E*b2s5?>z{F_rqfFgsOoyc)rrU)@i}%-o%i@d9jfz^J6BiCj9JC;^k;{} zbm$?y@o>1da_LG?leqsthgw9^@JQW0FXukReB82_0X<)-jx`6?oSbfvJ%7KNd~cN% zi>`v&uXmkVyOnpUvirqJ?amNv!;-#6-v_p`JZz(uS}2k^`hiWn{Cu6o!fU5a4_-fA zy9IDA-T2^5U$S^+;6|2Vwu{J<92ZOacG>b7i;=<*P_L_{WlO`b03BS%C{Qm!rlT#rA2q1s}0tg_000Iag zfB*tJE+FUsqsH%P#+$~k#0?h&5I_I{1Q0*~0R#|0009ILc)taTpSvn9dQ5A4LKJ;A zo+%WUZ9SpuiHPpV+qidieyMuSz7mlsdeph#=84F(h#r%Fqe0${s;l_rYVBfO-jBIg zv7X*uW3Sjv+ZWNgjqa%SAsL3Q>+-*@e*f=n&3OC$4vz8=KmY**5I_I{1Q0*~0R#|0 z;N}a&W2rrQM9%+5jNjLce-Kw(5I_I{1Q0*~0R#|0009ILKwz5%9*M-XsfmoZEEeQ- zCUaQb1lLD%7i>#T_gn59za{=4<&?WH;K)N_4Z+lLnX%Kp;~V##6pIaxsQ3TB){Or% ze!a~BQal0(Ab69S*adB%2?!v700IagfB*srAb int: + return store._db.execute("PRAGMA user_version").fetchone()[0] + + +@pytest.fixture +def v1_db(tmp_path): + """A writable copy of the checked-in v1-format fixture.""" + assert FIXTURE_SRC.exists(), ( + f"missing {FIXTURE_SRC} — rebuild it with " + "`.venv/bin/python tests/fixtures/make_v1_fixture.py`" + ) + dst = tmp_path / "v1user.db" + shutil.copy(FIXTURE_SRC, dst) + return dst + + +# ── Migration: the v1-format fixture ── +def test_v1_fixture_is_genuinely_v1(v1_db): + """Guard the fixture itself: it must be a v1 file (version 1, no record_kind, + no maintenance tables) — otherwise the migration test proves nothing.""" + db = sqlite3.connect(v1_db) + try: + assert db.execute("PRAGMA user_version").fetchone()[0] == 1 + cols = [r[1] for r in db.execute("PRAGMA table_info(fact)").fetchall()] + assert "record_kind" not in cols + tables = { + r[0] + for r in db.execute( + "SELECT name FROM sqlite_master WHERE type='table'" + ).fetchall() + } + assert "maintenance_run" not in tables + assert "fact_derivation" not in tables + assert "maintenance_proposal" not in tables + finally: + db.close() + + +def test_v1_migrates_once_to_v2(v1_db): + store = SqliteStore(v1_db, dim=FIXTURE_DIM, coarse_dim=FIXTURE_COARSE_DIM) + try: + assert _user_version(store) == 2, "upgraded to schema v2" + # record_kind added, default 'fact' backfilled on the existing rows. + cols = [r[1] for r in store._db.execute("PRAGMA table_info(fact)").fetchall()] + assert "record_kind" in cols + kinds = { + r[0] for r in store._db.execute("SELECT record_kind FROM fact").fetchall() + } + assert kinds == {"fact"}, "pre-existing v1 rows backfilled to record_kind='fact'" + # v2 tables now exist. + tables = { + r[0] + for r in store._db.execute( + "SELECT name FROM sqlite_master WHERE type='table'" + ).fetchall() + } + assert {"fact_derivation", "maintenance_run", "maintenance_proposal"} <= tables + finally: + store.close() + + +def test_v1_reopens_cleanly_after_migration(v1_db): + """The ALTER-idempotence trap: a SECOND open must not raise 'duplicate column + name'. This is the whole point of gating the ADD COLUMN behind `user_version < 2` + instead of putting it in the always-run schema blob.""" + SqliteStore(v1_db, dim=FIXTURE_DIM, coarse_dim=FIXTURE_COARSE_DIM).close() + + # A real second open of the now-migrated file — must be clean. + reopened = SqliteStore(v1_db, dim=FIXTURE_DIM, coarse_dim=FIXTURE_COARSE_DIM) + try: + assert _user_version(reopened) == 2 + finally: + reopened.close() + + +def test_migrated_v1_search_roundtrips(v1_db): + """After migration the old rows are still queryable — a dense search over the + fixture's stored vectors returns the latest fact.""" + store = SqliteStore(v1_db, dim=FIXTURE_DIM, coarse_dim=FIXTURE_COARSE_DIM) + try: + # Read a stored vector back and search with it — must find its own row. + latest = store._db.execute( + "SELECT id FROM fact WHERE is_latest=1" + ).fetchone()["id"] + full = store.get_embedding(latest) + assert full is not None + coarse = store._db.execute( + "SELECT embedding_256 FROM fact_vec WHERE fact_id=?", (latest,) + ).fetchone()["embedding_256"] + coarse_vec = np.frombuffer(coarse, dtype=np.float32) + + hits = store.dense_search(coarse_vec, full, k=3, is_latest_only=True) + assert any(fid == latest for fid, _ in hits), "migrated fact is retrievable" + + # And it round-trips through the Fact dataclass with record_kind populated. + fact = store.get_fact(latest) + assert fact is not None + assert fact.record_kind == "fact" + finally: + store.close() + + +# ── Migration: fresh-create + no-downgrade (the same versioned branch) ── +def test_fresh_create_stamps_v2_and_reopens_clean(tmp_path): + path = tmp_path / "fresh.db" + store = SqliteStore(path, dim=768) + assert _user_version(store) == 2 + # record_kind present on a fresh DB too (same ALTER branch, fresh is version 1 + # at that point). + cols = [r[1] for r in store._db.execute("PRAGMA table_info(fact)").fetchall()] + assert "record_kind" in cols + store.close() + + reopened = SqliteStore(path, dim=768) # must not raise + assert _user_version(reopened) == 2 + reopened.close() + + +def test_newer_version_db_not_downgraded(tmp_path): + path = tmp_path / "future.db" + store = SqliteStore(path, dim=768) + store._db.execute("PRAGMA user_version = 5") # a hypothetical v>2 release + store._db.commit() + store.close() + + reopened = SqliteStore(path, dim=768) + assert _user_version(reopened) == 5, "a newer stamp is never lowered" + reopened.close() + + +# ── Ledger/proposal CRUD (§4.0, §5) ── +@pytest.fixture +def store(tmp_path): + s = SqliteStore(tmp_path / "ns.db", dim=768) + yield s + s.close() + + +def test_run_lifecycle_roundtrip(store): + run_id = store.create_run( + namespace="ns", trigger="cli", started_at=1_000, config_hash="cfg-abc" + ) + assert run_id + + live = store.get_live_run("ns") + assert live is not None + assert live["id"] == run_id + assert live["status"] == "running" + assert live["trigger"] == "cli" + assert live["config_hash"] == "cfg-abc" + assert live["started_at"] == 1_000 + assert live["heartbeat_at"] == 1_000 + + store.heartbeat_run(run_id, at=2_000) + assert store.get_live_run("ns")["heartbeat_at"] == 2_000 + + store.finish_run( + run_id, status="ok", finished_at=3_000, + stats_json='{"deduped": 2}', cursor_id="cur-1", + ) + # No live run once finished — the lease is released. + assert store.get_live_run("ns") is None + row = store._db.execute( + "SELECT * FROM maintenance_run WHERE id=?", (run_id,) + ).fetchone() + assert row["status"] == "ok" + assert row["finished_at"] == 3_000 + assert row["stats_json"] == '{"deduped": 2}' + assert row["cursor_id"] == "cur-1" + + +def test_get_live_run_none_when_no_run(store): + assert store.get_live_run("ns") is None + + +def test_ux_run_live_second_live_run_raises(store): + """The lease: a second status='running' INSERT for the same namespace hits the + partial-unique index ux_run_live, raising IntegrityError — not a silent second + live row.""" + store.create_run(namespace="ns", trigger="cli", started_at=1_000, config_hash=None) + with pytest.raises(sqlite3.IntegrityError): + store.create_run(namespace="ns", trigger="mcp", started_at=1_100, config_hash=None) + + +def test_ux_run_live_allows_new_run_after_finish(store): + """Finishing the first run releases the lease so a second run can claim it.""" + r1 = store.create_run(namespace="ns", trigger="cli", started_at=1_000, config_hash=None) + store.finish_run(r1, status="ok", finished_at=2_000, stats_json=None, cursor_id=None) + r2 = store.create_run(namespace="ns", trigger="cli", started_at=3_000, config_hash=None) + assert r2 != r1 + assert store.get_live_run("ns")["id"] == r2 + + +def test_ux_run_live_is_per_namespace(store): + """The lease is per-namespace — concurrent live runs in DIFFERENT namespaces are + fine. (This store is one file, but the index keys on the namespace column.)""" + store.create_run(namespace="ns", trigger="cli", started_at=1_000, config_hash=None) + other = store.create_run(namespace="other", trigger="cli", started_at=1_000, config_hash=None) + assert store.get_live_run("other")["id"] == other + + +def test_proposal_roundtrip(store): + run_id = store.create_run(namespace="ns", trigger="cli", started_at=1_000, config_hash=None) + pid = store.stage_proposal( + run_id=run_id, namespace="ns", kind="summarize", + payload_json='{"subject": "user"}', created_at=1_000, expires_at=99_000, + evidence_backend="stub", + ) + assert pid + + got = store.get_proposal(pid) + assert got is not None + assert got["id"] == pid + assert got["run_id"] == run_id + assert got["namespace"] == "ns" + assert got["kind"] == "summarize" + assert got["payload_json"] == '{"subject": "user"}' + assert got["status"] == "pending" + assert got["created_at"] == 1_000 + assert got["expires_at"] == 99_000 + assert got["evidence_backend"] == "stub" + # Lifecycle columns default empty — no decide/apply has run. + assert got["decided_at"] is None + assert got["applied_at"] is None + assert got["expiry_reason"] is None + + +def test_get_proposal_missing_returns_none(store): + assert store.get_proposal("nope-does-not-exist") is None + + +def test_list_proposals_filters_and_orders(store): + run_id = store.create_run(namespace="ns", trigger="cli", started_at=1_000, config_hash=None) + p1 = store.stage_proposal( + run_id=run_id, namespace="ns", kind="summarize", + payload_json="{}", created_at=1_000, expires_at=99_000, + ) + p2 = store.stage_proposal( + run_id=run_id, namespace="ns", kind="dedup_near", + payload_json="{}", created_at=2_000, expires_at=99_000, + ) + p3 = store.stage_proposal( + run_id=run_id, namespace="ns", kind="evict", + payload_json="{}", created_at=3_000, expires_at=99_000, + ) + # a proposal in a different namespace must never leak in + store.stage_proposal( + run_id=run_id, namespace="other", kind="evict", + payload_json="{}", created_at=4_000, expires_at=99_000, + ) + + all_ns = store.list_proposals("ns") + assert [p["id"] for p in all_ns] == [p3, p2, p1], "newest first" + + by_kind = store.list_proposals("ns", kind="dedup_near") + assert [p["id"] for p in by_kind] == [p2] + + pending = store.list_proposals("ns", status="pending") + assert {p["id"] for p in pending} == {p1, p2, p3} + + assert store.list_proposals("ns", status="approved") == [] + assert len(store.list_proposals("ns", limit=1)) == 1 + assert store.list_proposals("other")[0]["namespace"] == "other" diff --git a/tests/test_schema_version.py b/tests/test_schema_version.py index f4d779b..2b08aa5 100644 --- a/tests/test_schema_version.py +++ b/tests/test_schema_version.py @@ -1,34 +1,40 @@ -"""Namespace DBs carry a schema-version stamp so future releases can migrate. - -v0.1.0-0.1.2 shipped files with user_version=0 and no migration anchor; a 0.2.0 -schema change would have had no way to tell an old-but-valid file from a foreign -SQLite database. Version 1 == the 0.1.x layout, and pre-stamp files are upgraded -to 1 in place (their schema is identical). A file stamped by a NEWER release is -never downgraded. +"""Namespace DBs carry a schema-version stamp so releases can migrate. + +v0.1.0-0.1.2 shipped files with user_version=0 and no migration anchor; a schema +change would have had no way to tell an old-but-valid file from a foreign SQLite +database. Version 1 == the 0.1.x layout; version 2 == the sleep-time-maintenance +layout (adds fact.record_kind + the maintenance tables). A fresh DB is stamped at +the current version (2); older files migrate in place; a file stamped by a NEWER +release is never downgraded. + +The genuine v1-format-file → v2 upgrade path (incl. the ALTER-idempotence reopen +trap) is pinned end-to-end in test_schema_migration.py against a checked-in +v1-format fixture DB. This module pins the stamp arithmetic. """ from lean_memory.store.sqlite_store import SqliteStore +CURRENT_SCHEMA_VERSION = 2 + def _user_version(store: SqliteStore) -> int: return store._db.execute("PRAGMA user_version").fetchone()[0] -def test_fresh_store_stamps_user_version_1(tmp_path): +def test_fresh_store_stamps_current_version(tmp_path): store = SqliteStore(tmp_path / "ns.db", dim=768) - assert _user_version(store) == 1 + assert _user_version(store) == CURRENT_SCHEMA_VERSION store.close() -def test_prestamp_file_upgraded_in_place(tmp_path): +def test_migrated_file_reopens_at_current_version(tmp_path): + """A DB already migrated to the current version reopens without re-running the + non-idempotent migration DDL (the ALTER-idempotence trap).""" path = tmp_path / "ns.db" - store = SqliteStore(path, dim=768) - store._db.execute("PRAGMA user_version = 0") # simulate a 0.1.0-0.1.2 file - store._db.commit() - store.close() + SqliteStore(path, dim=768).close() # fresh → stamped current - reopened = SqliteStore(path, dim=768) - assert _user_version(reopened) == 1 + reopened = SqliteStore(path, dim=768) # must NOT raise 'duplicate column name' + assert _user_version(reopened) == CURRENT_SCHEMA_VERSION reopened.close() From 5e46364fa458339fd00fe77b774c8c49745afd70 Mon Sep 17 00:00:00 2001 From: Wuesteon Date: Thu, 16 Jul 2026 19:42:43 +0800 Subject: [PATCH 04/14] =?UTF-8?q?feat(engine):=20ingest=20commutation=20ho?= =?UTF-8?q?oks=20=E2=80=94=20duplicate=20+=20staleness=20cascades?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restore the ingest-commutation condition (spec §3.1.3), fixing the two empirically-demonstrated rev-1 wrong answers (resurrection, stale summary; §14). Hook 1 — duplicate-cascade in supersede_fact (§4.0): after closing `old`, close its OPEN retired duplicates (superseded_by=old, valid_to NULL) at the same V and RETURN [old_id] + cascade-closed ids. Single level suffices — Task 1's chain invariant keeps every open duplicate pointing directly at the live survivor. One transaction. supersede_fact return type None → list[str] (ABC updated). Hook 2 — summary-staleness cascade in _apply_supersession (§4.3): feed the FULL closed set returned by every supersede_fact call (explicit targets PLUS duplicate-cascade-closed rows — the load-bearing seam) into find_summaries_derived_from; each still-latest derived summary flips is_latest=0, valid_to=new.valid_at, invalidated_by=new.id (+vec mirror). memory.py stays free of raw SQL. New store methods (schema-v2 derivation surface): add_derivation, find_summaries_derived_from, invalidate_summary — all double-guarded on is_latest=1 so a re-fire is a no-op. Both hooks are exact no-ops until maintenance has produced rows — first-run path byte-unchanged (§10.10 pinned). Tests: new tests/test_ingest_commutation.py (9) — resurrection, transitive B→A→D, stale summary (vacuity-guarded), cascade-closed-duplicate seam, no-op pin. Suite: 178 passed, 1 skipped; four named pins green untouched. Co-Authored-By: Claude Fable 5 --- src/lean_memory/memory.py | 20 +- src/lean_memory/store/base.py | 31 ++- src/lean_memory/store/sqlite_store.py | 85 +++++- tests/test_ingest_commutation.py | 385 ++++++++++++++++++++++++++ 4 files changed, 517 insertions(+), 4 deletions(-) create mode 100644 tests/test_ingest_commutation.py diff --git a/src/lean_memory/memory.py b/src/lean_memory/memory.py index 611ec26..f345f88 100644 --- a/src/lean_memory/memory.py +++ b/src/lean_memory/memory.py @@ -162,6 +162,16 @@ def _apply_supersession(store, decision, fact, slot_latest) -> None: employers. A replacement on a functional slot therefore retires EVERY latest fact in the slot. Multi-valued slots (likes/uses/...) keep the single-target behavior: the user's other co-valid values survive. + + Ingest hook 2 — SUMMARY-STALENESS CASCADE (§4.3): every supersede_fact + returns the FULL closed set — its explicit target PLUS any duplicate-cascade- + closed rows (§4.0). Any summary still is_latest=1 derived from ANY closed + source is stale and must leave the default surface (is_latest=0, + valid_to=new.valid_at, invalidated_by=new.id) — else the design ships a live + contradiction (empirically demonstrated, §14). Feeding the RETURNED ids + (not merely the loop's own targets) is load-bearing: a summary derived from + a retired duplicate would otherwise never flip (rev-3 seam fix). No-op until + fact_derivation has rows — the first-run path is unchanged. """ if decision.label != SUPERSEDES or decision.target is None: return @@ -169,8 +179,16 @@ def _apply_supersession(store, decision, fact, slot_latest) -> None: targets = [decision.target] else: targets = [f for f in slot_latest if f.id != fact.id] + + closed_ids: list[str] = [] for old in targets: - store.supersede_fact(old.id, fact.id, valid_to=fact.valid_at) + closed_ids += store.supersede_fact(old.id, fact.id, valid_to=fact.valid_at) + + # Staleness cascade over the full closed set (explicit + cascade-closed). + for summary_id in store.find_summaries_derived_from(closed_ids): + store.invalidate_summary( + summary_id, valid_to=fact.valid_at, invalidated_by=fact.id + ) def _build_fact( self, tf: TypedFact, *, namespace: str, episode_id: str, store: SqliteStore diff --git a/src/lean_memory/store/base.py b/src/lean_memory/store/base.py index 2e842b1..7e77573 100644 --- a/src/lean_memory/store/base.py +++ b/src/lean_memory/store/base.py @@ -38,9 +38,14 @@ def add_fact(self, fact: Fact, embedding: np.ndarray, embedding_256: np.ndarray) """Insert a fact row + its vec0 vectors + its FTS row, in one transaction.""" @abstractmethod - def supersede_fact(self, old_fact_id: str, new_fact_id: str, valid_to: int) -> None: + def supersede_fact( + self, old_fact_id: str, new_fact_id: str, valid_to: int + ) -> list[str]: """ADD-only supersession: point old→new, set old.valid_to, flip old.is_latest=0. - Never deletes.""" + Never deletes. Additionally cascade-closes old's OPEN retired duplicates + (superseded_by=old, valid_to NULL) at the same valid_to — ingest hook 1, + §4.0. RETURNS [old_id] + cascade-closed ids so the summary-staleness cascade + (§4.3) keys on every closed row.""" @abstractmethod def get_fact(self, fact_id: str) -> Optional[Fact]: ... @@ -116,6 +121,28 @@ def iter_slots_touched_since(self, cursor_id: str) -> Iterator[tuple[str, str]]: id > cursor) since the cursor — including duplicates landing on long-quiet slots (the verified cursor gap).""" + # ── derivation lineage + staleness cascade (schema v2, design spec §4.3) ── + @abstractmethod + def add_derivation( + self, summary_id: str, source_id: str, run_id: str, created_at: int + ) -> None: + """Record one summary←source lineage edge (fact_derivation). Idempotent on + the (summary_id, source_id) PK. The staleness cascade reads these via + ix_derivation_source (§4.3).""" + + @abstractmethod + def find_summaries_derived_from(self, source_ids: Sequence[str]) -> list[str]: + """DISTINCT still-latest (is_latest=1) summary ids derived from any of + `source_ids` — the staleness cascade's lookup (ingest hook 2, §4.3).""" + + @abstractmethod + def invalidate_summary( + self, summary_id: str, valid_to: int, invalidated_by: str + ) -> None: + """Retire a summary stale-invalidated by live ingest: is_latest=0, valid_to, + invalidated_by on fact + is_latest=0 mirror on fact_vec (ingest hook 2, + §4.3). Scoped to is_latest=1 so a re-fire is a no-op.""" + # ── maintenance ledger + proposal CRUD (schema v2, design spec §4.0/§5) ── # Pure row CRUD — no decide/apply logic (that is the proposal lifecycle, a # later task). The create-half is needed by the maintenance runner. diff --git a/src/lean_memory/store/sqlite_store.py b/src/lean_memory/store/sqlite_store.py index 140941f..75d5a56 100644 --- a/src/lean_memory/store/sqlite_store.py +++ b/src/lean_memory/store/sqlite_store.py @@ -225,7 +225,28 @@ def add_fact(self, fact: Fact, embedding: np.ndarray, embedding_256: np.ndarray) ) self._commit() - def supersede_fact(self, old_fact_id: str, new_fact_id: str, valid_to: int) -> None: + def supersede_fact( + self, old_fact_id: str, new_fact_id: str, valid_to: int + ) -> list[str]: + """Close `old` at world-time `valid_to`, cascade-close its OPEN retired + duplicates at the same V, and return the full closed-id list. + + Ingest hook 1 — DUPLICATE-CASCADE (§4.0): retire_duplicate leaves a + duplicate with superseded_by=survivor but valid_to NULL (verb (c), as-of + safe at dedup time). Such an open duplicate is invisible to + find_latest_in_slot, so ordinary ingest supersession never closes it — + after the survivor is superseded it would resurrect as a permanently-open + interval on the pure as-of surface (empirically demonstrated wrong answer, + §14). Closing `WHERE superseded_by=old AND valid_to IS NULL` at the same V + restores commutation. A SINGLE level suffices because retire_duplicate's + chain invariant keeps every open duplicate pointing DIRECTLY at the live + survivor (§4.0) — no recursion. Whole thing is ONE transaction. + + RETURNS [old_id] + cascade-closed ids so the summary-staleness cascade + (§4.3) keys on every closed row, not just the explicit target — the + cascade-closed ids are collected by SELECT before the UPDATE (rev-3 seam + fix). No-op cascade until retire_duplicate has produced such rows. + """ db = self._db db.execute( "UPDATE fact SET superseded_by=?, valid_to=?, is_latest=0 WHERE id=?", @@ -233,6 +254,68 @@ def supersede_fact(self, old_fact_id: str, new_fact_id: str, valid_to: int) -> N ) # keep the vec0 metadata filter column in sync so superseded facts drop out db.execute("UPDATE fact_vec SET is_latest=0 WHERE fact_id=?", (old_fact_id,)) + # Duplicate-cascade: collect the open retired duplicates FIRST (for the + # returned set), then close them at the same V — same transaction. + cascade_ids = [ + r["id"] + for r in db.execute( + "SELECT id FROM fact WHERE superseded_by=? AND valid_to IS NULL", + (old_fact_id,), + ).fetchall() + ] + if cascade_ids: + db.execute( + "UPDATE fact SET valid_to=? WHERE superseded_by=? AND valid_to IS NULL", + (valid_to, old_fact_id), + ) + self._commit() + return [old_fact_id, *cascade_ids] + + # ── derivation lineage + staleness cascade support (schema v2; §4.3) ── + def add_derivation( + self, summary_id: str, source_id: str, run_id: str, created_at: int + ) -> None: + """Record one summary←source lineage edge (fact_derivation). The staleness + cascade reads these via ix_derivation_source to find summaries to invalidate + when a source is closed by ingest (§4.3). INSERT OR IGNORE keeps the + (summary_id, source_id) PK idempotent.""" + self._db.execute( + "INSERT OR IGNORE INTO fact_derivation(summary_id, source_id, run_id, created_at) " + "VALUES (?,?,?,?)", + (summary_id, source_id, run_id, created_at), + ) + self._commit() + + def find_summaries_derived_from(self, source_ids: Sequence[str]) -> list[str]: + """DISTINCT still-latest summary ids derived from any of `source_ids` — the + staleness cascade's lookup (§4.3), served by ix_derivation_source. Restricted + to is_latest=1 so the caller never re-invalidates an already-retired summary.""" + ids = list(source_ids) + if not ids: + return [] + placeholders = ",".join("?" * len(ids)) + rows = self._db.execute( + f"""SELECT DISTINCT d.summary_id AS summary_id + FROM fact_derivation d JOIN fact f ON f.id = d.summary_id + WHERE d.source_id IN ({placeholders}) AND f.is_latest = 1""", + ids, + ).fetchall() + return [r["summary_id"] for r in rows] + + def invalidate_summary( + self, summary_id: str, valid_to: int, invalidated_by: str + ) -> None: + """Ingest hook 2 write — retire a summary stale-invalidated by live ingest + (§4.3): is_latest=0, valid_to, invalidated_by on fact + is_latest=0 mirror on + fact_vec, one txn. Scoped to is_latest=1 so a re-fire is a no-op. As-of + windows [t_a, valid_to) still show it — accurate: it was the believed state.""" + db = self._db + db.execute( + "UPDATE fact SET is_latest=0, valid_to=?, invalidated_by=? " + "WHERE id=? AND is_latest=1", + (valid_to, invalidated_by, summary_id), + ) + db.execute("UPDATE fact_vec SET is_latest=0 WHERE fact_id=?", (summary_id,)) self._commit() def get_fact(self, fact_id: str) -> Optional[Fact]: diff --git a/tests/test_ingest_commutation.py b/tests/test_ingest_commutation.py new file mode 100644 index 0000000..0a0485f --- /dev/null +++ b/tests/test_ingest_commutation.py @@ -0,0 +1,385 @@ +"""Task 3 — the two ingest hooks that restore ingest commutation (design spec +§3.1 condition 3, §4.0 duplicate-cascade, §4.3 staleness cascade). + +Offline (FakeEmbedder/StubTyper), against a real Memory + SqliteStore. These pin +the two empirically-demonstrated rev-1 wrong answers the hooks fix (§14): + + - RESURRECTION (§10.2): a duplicate retired by maintenance (valid_to NULL) is + invisible to find_latest_in_slot, so ordinary ingest supersession never + closes it — after the survivor is superseded the retired duplicate resurfaces + as a permanently-open interval on the pure as-of surface. The duplicate-cascade + in supersede_fact closes it at the same world-time as the survivor. + - TRANSITIVE resurrection (§10.2, the rev-3 killer): B→A→D retirement; when D is + superseded, BOTH A and B must close at V (Task 1's chain invariant re-points B + to D, so the single-level cascade reaches it). + - STALE SUMMARY (§10.3): a summary derived from a source fact must leave the + default surface the moment ordinary ingest contradicts the source. The + staleness cascade flips the derived summary is_latest=0. + +Plus the no-op pin (§10.10): on a maintenance-naive DB both cascades' triggering +sets are provably empty, so a full ingest+search sequence is unchanged. +""" + +from __future__ import annotations + +from lean_memory import Memory +from lean_memory.embed.fake import FakeEmbedder +from lean_memory.store.sqlite_store import SqliteStore +from lean_memory.types import Entity, Episode, Fact + + +# ── low-level store fixture helpers (mirror test_store_maintenance_verbs.py) ── +def _fresh_store(tmp_path, name="ns.db"): + emb = FakeEmbedder() + s = SqliteStore(tmp_path / name, dim=emb.dim, coarse_dim=emb.coarse_dim) + ep = Episode(namespace="ns", raw="seed", t_ref=1_000) + s.add_episode(ep) + ent = s.upsert_entity(Entity(namespace="ns", name="user", type="person")) + s._emb = emb + s._ep = ep + s._ent = ent + return s + + +def _add_fact(store, text, *, valid_at, predicate="works_at", is_inference=0, + record_kind="fact", valid_to=None): + f = Fact( + namespace="ns", subject_id=store._ent.id, predicate=predicate, + fact_text=text, valid_at=valid_at, episode_id=store._ep.id, + is_inference=is_inference, record_kind=record_kind, valid_to=valid_to, + ) + full, coarse = store._emb.embed_with_coarse(text) + store.add_fact(f, full, coarse) + return f + + +def _row(store, fact_id): + return store._db.execute( + "SELECT is_latest, valid_to, superseded_by, invalidated_by " + "FROM fact WHERE id=?", + (fact_id,), + ).fetchone() + + +# ══════════════════════════════════════════════════════════════════════════ +# Step 1 — duplicate-cascade in supersede_fact (store-level unit) +# ══════════════════════════════════════════════════════════════════════════ +def test_supersede_fact_returns_closed_ids_no_duplicates(tmp_path): + """The new return value: with zero retired duplicates, supersede_fact returns + exactly [old_id].""" + s = _fresh_store(tmp_path) + old = _add_fact(s, "user works at acme", valid_at=1_000) + new = _add_fact(s, "user works at zorbex", valid_at=3_000) + + closed = s.supersede_fact(old.id, new.id, valid_to=3_000) + assert closed == [old.id], "no duplicates → just the explicit target" + assert _row(s, old.id)["valid_to"] == 3_000 + s.close() + + +def test_supersede_fact_cascade_closes_open_duplicate(tmp_path): + """A retired duplicate (valid_to NULL, superseded_by=survivor) closes at the + SAME world-time when its survivor is superseded, and is in the returned set.""" + s = _fresh_store(tmp_path) + survivor = _add_fact(s, "user works at acme", valid_at=1_000) + dup = _add_fact(s, "user works at acme", valid_at=2_000) + s.retire_duplicate(dup.id, survivor.id) # dup → survivor, valid_to NULL + assert _row(s, dup.id)["valid_to"] is None # precondition: open duplicate + + new = _add_fact(s, "user works at zorbex", valid_at=5_000) + closed = s.supersede_fact(survivor.id, new.id, valid_to=5_000) + + assert set(closed) == {survivor.id, dup.id}, "cascade set includes the duplicate" + assert _row(s, survivor.id)["valid_to"] == 5_000 + assert _row(s, dup.id)["valid_to"] == 5_000, "duplicate closed at the same V" + # vec mirror already 0 from retire_duplicate; survivor's flips to 0 here. + vec = s._db.execute( + "SELECT is_latest FROM fact_vec WHERE fact_id=?", (survivor.id,) + ).fetchone() + assert vec["is_latest"] == 0 + s.close() + + +def test_supersede_fact_cascade_only_touches_open_duplicates(tmp_path): + """The cascade UPDATE is scoped to superseded_by=old AND valid_to IS NULL, so + an already-closed duplicate (from a prior supersession) is not re-closed.""" + s = _fresh_store(tmp_path) + survivor = _add_fact(s, "user works at acme", valid_at=1_000) + # A duplicate that was already world-time-closed (valid_to set) — NOT a + # retire_duplicate row. It must be left alone by the cascade. + already_closed = _add_fact(s, "user works at acme", valid_at=2_000, valid_to=2_500) + s._db.execute( + "UPDATE fact SET superseded_by=?, is_latest=0 WHERE id=?", + (survivor.id, already_closed.id), + ) + s._db.commit() + + new = _add_fact(s, "user works at zorbex", valid_at=5_000) + closed = s.supersede_fact(survivor.id, new.id, valid_to=5_000) + + assert closed == [survivor.id], "already-closed duplicate not in the cascade set" + assert _row(s, already_closed.id)["valid_to"] == 2_500, "its valid_to untouched" + s.close() + + +# ══════════════════════════════════════════════════════════════════════════ +# Step 3a — RESURRECTION (§10.2): dedup then supersede via ORDINARY ingest +# ══════════════════════════════════════════════════════════════════════════ +def _latest_texts(mem, ns, predicate): + store = mem._store(ns) + rows = store._db.execute( + "SELECT fact_text FROM fact WHERE predicate=? AND is_latest=1", + (predicate,), + ).fetchall() + return [r["fact_text"] for r in rows] + + +def test_resurrection_dedup_then_ordinary_ingest_supersession(tmp_path): + """Retire a duplicate B→A, then supersede A via a contradicting Memory.add. + The cascade must close B at the same world-time as A, and a pure point-in-time + as_of search after the supersession must return only the new fact — B must NOT + resurrect on the is_latest_only=False surface.""" + mem = Memory(root=tmp_path) + mem.add("ns", "I work at Acme.", t_ref=1_000) + store = mem._store("ns") + + A = store._db.execute( + "SELECT id, subject_id, valid_at FROM fact WHERE predicate='works_at' " + "AND is_latest=1" + ).fetchone() + # Hand-inject an exact duplicate B in the same slot, then retire it onto A. + B = Fact( + namespace="ns", subject_id=A["subject_id"], predicate="works_at", + fact_text="I work at Acme.", valid_at=2_000, + episode_id=store._db.execute("SELECT id FROM episode LIMIT 1").fetchone()["id"], + ) + full, coarse = mem.embedder.embed_with_coarse(B.fact_text) + store.add_fact(B, full, coarse) + store.retire_duplicate(B.id, A["id"]) + assert _row(store, B.id)["valid_to"] is None # open retired duplicate + + # Ordinary ingest of a contradicting fact in the same functional slot. + new_ids = mem.add("ns", "I work at Zorbex now.", t_ref=5_000) + new_id = new_ids[0] + new_valid_at = store.get_fact(new_id).valid_at + + # A closed by ordinary supersession; B closed by the duplicate-cascade at V. + assert _row(store, A["id"])["valid_to"] == new_valid_at + assert _row(store, B.id)["valid_to"] == new_valid_at, ( + "retired duplicate resurrected — cascade did not close it" + ) + + # Pure point-in-time as-of surface AFTER the supersession: only the new fact. + hits = mem.search( + "ns", "where do I work", k=10, as_of=new_valid_at + 1, is_latest_only=False + ) + texts = {h.fact.fact_text for h in hits} + assert "I work at Zorbex now." in texts + assert "I work at Acme." not in texts, "Acme (A or B) visible after supersession" + mem.close() + + +def test_transitive_resurrection_B_to_A_to_D_then_supersede(tmp_path): + """The rev-3 killer: retire B→A, then A→D (chain re-points B→D), then supersede + D via ordinary ingest. The single-level cascade must close BOTH A and B at V, + because Task 1's chain invariant left both pointing directly at D.""" + mem = Memory(root=tmp_path) + mem.add("ns", "I work at Acme.", t_ref=1_000) + store = mem._store("ns") + ep_id = store._db.execute("SELECT id FROM episode LIMIT 1").fetchone()["id"] + D_row = store._db.execute( + "SELECT id, subject_id FROM fact WHERE predicate='works_at' AND is_latest=1" + ).fetchone() + subj = D_row["subject_id"] + + def _inject(text, valid_at): + f = Fact(namespace="ns", subject_id=subj, predicate="works_at", + fact_text=text, valid_at=valid_at, episode_id=ep_id) + full, coarse = mem.embedder.embed_with_coarse(text) + store.add_fact(f, full, coarse) + return f + + A = _inject("I work at Acme.", 2_000) + B = _inject("I work at Acme.", 3_000) + store.retire_duplicate(B.id, A.id) # B → A + store.retire_duplicate(A.id, D_row["id"]) # A → D ; re-points B → D + + # Precondition (Task 1 behavior): B was re-pointed to D. + assert _row(store, B.id)["superseded_by"] == D_row["id"], "B re-pointed to D" + assert _row(store, A.id)["superseded_by"] == D_row["id"] + + new_ids = mem.add("ns", "I work at Zorbex now.", t_ref=6_000) + new_valid_at = store.get_fact(new_ids[0]).valid_at + + assert _row(store, D_row["id"])["valid_to"] == new_valid_at + assert _row(store, A.id)["valid_to"] == new_valid_at, "A not closed by cascade" + assert _row(store, B.id)["valid_to"] == new_valid_at, "B not closed by cascade" + + # Standing invariant (spec §10.2): no open retired duplicate points at a + # non-latest row. + orphans = store._db.execute( + """SELECT f.id FROM fact f JOIN fact t ON t.id = f.superseded_by + WHERE f.superseded_by IS NOT NULL AND f.valid_to IS NULL + AND t.is_latest = 0""" + ).fetchall() + assert orphans == [] + mem.close() + + +# ══════════════════════════════════════════════════════════════════════════ +# Step 3c — STALE SUMMARY (§10.3): hand-inserted fixture +# ══════════════════════════════════════════════════════════════════════════ +def test_stale_summary_flips_on_ordinary_ingest_supersession(tmp_path): + """Hand-insert a summary fact derived from a source, then contradict the source + via ordinary ingest. The staleness cascade must flip the summary is_latest=0, + valid_to=new.valid_at, invalidated_by=new.id.""" + mem = Memory(root=tmp_path) + mem.add("ns", "I work at Acme.", t_ref=1_000) + store = mem._store("ns") + ep_id = store._db.execute("SELECT id FROM episode LIMIT 1").fetchone()["id"] + source = store._db.execute( + "SELECT id, subject_id FROM fact WHERE predicate='works_at' AND is_latest=1" + ).fetchone() + + # Hand-insert a summary row (record_kind='summary', predicate='summary', + # is_inference=1) in its OWN slot, plus a derivation edge to the source. + summary = Fact( + namespace="ns", subject_id=source["subject_id"], predicate="summary", + fact_text="The user has worked at Acme.", valid_at=1_500, episode_id=ep_id, + is_inference=1, record_kind="summary", + ) + full, coarse = mem.embedder.embed_with_coarse(summary.fact_text) + store.add_fact(summary, full, coarse) + store.add_derivation(summary.id, source["id"], run_id="run-x", created_at=1_500) + + # Vacuity guard FIRST: derivation must be non-empty, else the test is hollow. + n_deriv = store._db.execute( + "SELECT COUNT(*) AS c FROM fact_derivation WHERE source_id=?", (source["id"],) + ).fetchone()["c"] + assert n_deriv == 1, "fixture derivation row missing — test would pass vacuously" + assert _row(store, summary.id)["is_latest"] == 1 # summary starts latest + + # Ordinary ingest contradicts the source (functional works_at slot). + new_ids = mem.add("ns", "I work at Zorbex now.", t_ref=5_000) + new_id = new_ids[0] + new_valid_at = store.get_fact(new_id).valid_at + + srow = _row(store, summary.id) + assert srow["is_latest"] == 0, "stale summary did not leave the default surface" + assert srow["valid_to"] == new_valid_at + assert srow["invalidated_by"] == new_id + # vec mirror flipped too. + vec = store._db.execute( + "SELECT is_latest FROM fact_vec WHERE fact_id=?", (summary.id,) + ).fetchone() + assert vec["is_latest"] == 0 + mem.close() + + +def test_stale_summary_fires_for_cascade_closed_duplicate(tmp_path): + """The load-bearing seam (spec §4.3): the staleness cascade keys off the FULL + closed set returned by supersede_fact — including duplicate-cascade-closed + rows — not just the loop's explicit targets. A summary derived from a retired + DUPLICATE must still flip when the survivor is superseded.""" + mem = Memory(root=tmp_path) + mem.add("ns", "I work at Acme.", t_ref=1_000) + store = mem._store("ns") + ep_id = store._db.execute("SELECT id FROM episode LIMIT 1").fetchone()["id"] + survivor = store._db.execute( + "SELECT id, subject_id FROM fact WHERE predicate='works_at' AND is_latest=1" + ).fetchone() + + # A retired duplicate of the survivor (valid_to NULL). + dup = Fact(namespace="ns", subject_id=survivor["subject_id"], predicate="works_at", + fact_text="I work at Acme.", valid_at=2_000, episode_id=ep_id) + full, coarse = mem.embedder.embed_with_coarse(dup.fact_text) + store.add_fact(dup, full, coarse) + store.retire_duplicate(dup.id, survivor["id"]) + + # A summary derived ONLY from the retired duplicate (not the survivor). + summary = Fact(namespace="ns", subject_id=survivor["subject_id"], predicate="summary", + fact_text="The user has worked at Acme.", valid_at=2_500, + episode_id=ep_id, is_inference=1, record_kind="summary") + sfull, scoarse = mem.embedder.embed_with_coarse(summary.fact_text) + store.add_fact(summary, sfull, scoarse) + store.add_derivation(summary.id, dup.id, run_id="run-y", created_at=2_500) + assert _row(store, summary.id)["is_latest"] == 1 + + new_ids = mem.add("ns", "I work at Zorbex now.", t_ref=5_000) + new_valid_at = store.get_fact(new_ids[0]).valid_at + + srow = _row(store, summary.id) + assert srow["is_latest"] == 0, ( + "summary derived from a cascade-closed duplicate did not flip — the " + "returned-set plumbing regressed" + ) + assert srow["valid_to"] == new_valid_at + assert srow["invalidated_by"] == new_ids[0] + mem.close() + + +def test_staleness_cascade_leaves_unrelated_latest_summary_alone(tmp_path): + """Guard: the cascade only invalidates summaries derived from the CLOSED + sources. A summary derived from an UNAFFECTED source stays is_latest=1.""" + mem = Memory(root=tmp_path) + mem.add("ns", "I work at Acme.", t_ref=1_000) + mem.add("ns", "I live in Boston.", t_ref=1_000) + store = mem._store("ns") + ep_id = store._db.execute("SELECT id FROM episode LIMIT 1").fetchone()["id"] + lives = store._db.execute( + "SELECT id, subject_id FROM fact WHERE predicate='lives_in' AND is_latest=1" + ).fetchone() + + # Summary derived from the lives_in source (untouched by a works_at contradiction). + summary = Fact(namespace="ns", subject_id=lives["subject_id"], predicate="summary", + fact_text="The user lives in Boston.", valid_at=1_500, + episode_id=ep_id, is_inference=1, record_kind="summary") + full, coarse = mem.embedder.embed_with_coarse(summary.fact_text) + store.add_fact(summary, full, coarse) + store.add_derivation(summary.id, lives["id"], run_id="run-z", created_at=1_500) + + mem.add("ns", "I work at Zorbex now.", t_ref=5_000) # closes works_at, not lives_in + + assert _row(store, summary.id)["is_latest"] == 1, "unrelated summary wrongly flipped" + mem.close() + + +# ══════════════════════════════════════════════════════════════════════════ +# Step 4 — NO-OP PIN (§10.10): both cascades provably do nothing on a +# maintenance-naive DB +# ══════════════════════════════════════════════════════════════════════════ +def test_hooks_are_noops_on_maintenance_naive_db(tmp_path): + """A full multi-fact ingest with real supersessions on a fresh DB where + maintenance NEVER ran: (i) no fact row has superseded_by set with valid_to + NULL (so the duplicate-cascade had nothing to close), and (ii) fact_derivation + is empty (so the staleness cascade had nothing to look up). Both cascades + provably had no effect — the first-run path is unchanged.""" + mem = Memory(root=tmp_path) + # A corpus with genuine ingest supersessions (functional slot replacements). + mem.add("ns", "I work at Acme.", t_ref=1_000) + mem.add("ns", "I also work at Globex.", t_ref=2_000) # co-valid extend + mem.add("ns", "I work at Zorbex now.", t_ref=3_000) # supersedes the slot + mem.add("ns", "I live in Boston.", t_ref=4_000) + mem.add("ns", "I live in Denver now.", t_ref=5_000) # supersedes lives_in + store = mem._store("ns") + + # (i) No open retired duplicate exists — the duplicate-cascade's trigger set is + # empty (retire_duplicate is the ONLY writer of superseded_by-with-NULL-valid_to, + # and maintenance never ran). Ordinary supersession always sets both together. + open_supersedes = store._db.execute( + "SELECT COUNT(*) AS c FROM fact WHERE superseded_by IS NOT NULL " + "AND valid_to IS NULL" + ).fetchone()["c"] + assert open_supersedes == 0, "ordinary ingest left an open-superseded row" + + # (ii) fact_derivation is empty — the staleness cascade's lookup set is empty. + n_deriv = store._db.execute("SELECT COUNT(*) AS c FROM fact_derivation").fetchone()["c"] + assert n_deriv == 0, "no summarize ran, yet a derivation row exists" + + # And the ordinary supersessions still worked correctly (spine intact). + assert _latest_texts(mem, "ns", "works_at") == ["I work at Zorbex now."] + assert _latest_texts(mem, "ns", "lives_in") == ["I live in Denver now."] + + hits = mem.search("ns", "where do I work", k=5) + assert any("Zorbex" in h.fact.fact_text for h in hits) + mem.close() From 022e3017200f8f4f420529a2f0ee6b61cbc68466 Mon Sep 17 00:00:00 2001 From: Wuesteon Date: Thu, 16 Jul 2026 20:00:45 +0800 Subject: [PATCH 05/14] =?UTF-8?q?feat(maintain):=20config,=20scoring,=20tr?= =?UTF-8?q?ansforms=20=E2=80=94=20the=20sleep-time=20job's=20heart?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New src/lean_memory/maintain/ package (design spec §3.6, §4.1–§4.4): - MaintenanceConfig: frozen dataclass, all §3.6/§6.6 defaults; evict_threshold a documented conservative/tunable 0.15 (spec pins none); config_hash() = sha256 of canonical JSON. - score.value(fact, now): §4.4 standing value; recency anchor VERBATIM the retriever's (last_access or valid_at), DECAY_LAMBDA imported (never re-derived) — backfills score stale exactly as the ranker de-ranks them. - Summarizer seam: ExtractiveStubSummarizer (deterministic, pinned) + OllamaSummarizer behind an [llm] import guard; LM_FORCE_STUBS honored. - Four transforms as pure Store-ABC functions: dedup_exact (AUTO — value- preserving normalization only, argmin(valid_at)/min-id survivor, cluster usage-stats merge, per-slot batch()), dedup_near + summarize + evict_propose (PROPOSE — zero spine writes), evict_auto (strict band). run_transforms enforces the §4.4 intra-run ordering: stage all proposals over the pre- transform snapshot first, then autos excluding staged targets; shared proposal budget with reported (never silent) truncation. Store: add merge_usage_stats (ABC + SqliteStore) for the DEDUP-EXACT survivor stats merge — plain UPDATE, batch-aware. Tests (offline, +34): test_maintenance_transforms.py (config/score/summarizer/ normalization/dedup/evict/ordering/budget/zero-spine-delta) and test_maintenance_asof_grid.py (§10.1 store-predicate invariance over a T grid under autos+staging; §10.8 ranking-delta pin; no inverted intervals). Full suite: 212 passed, 1 skipped. Co-Authored-By: Claude Fable 5 --- src/lean_memory/maintain/__init__.py | 50 +++ src/lean_memory/maintain/config.py | 84 ++++ src/lean_memory/maintain/score.py | 51 +++ src/lean_memory/maintain/summarize.py | 137 +++++++ src/lean_memory/maintain/transforms.py | 515 +++++++++++++++++++++++++ src/lean_memory/store/base.py | 9 + src/lean_memory/store/sqlite_store.py | 14 + tests/test_maintenance_asof_grid.py | 218 +++++++++++ tests/test_maintenance_transforms.py | 489 +++++++++++++++++++++++ 9 files changed, 1567 insertions(+) create mode 100644 src/lean_memory/maintain/__init__.py create mode 100644 src/lean_memory/maintain/config.py create mode 100644 src/lean_memory/maintain/score.py create mode 100644 src/lean_memory/maintain/summarize.py create mode 100644 src/lean_memory/maintain/transforms.py create mode 100644 tests/test_maintenance_asof_grid.py create mode 100644 tests/test_maintenance_transforms.py diff --git a/src/lean_memory/maintain/__init__.py b/src/lean_memory/maintain/__init__.py new file mode 100644 index 0000000..6a4a744 --- /dev/null +++ b/src/lean_memory/maintain/__init__.py @@ -0,0 +1,50 @@ +"""Sleep-time maintenance — config, scoring, summarizer seam, and transforms. + +The heart of the offline maintenance job (design spec §3–§4): pure functions over +the Store ABC that deduplicate, summarize, and evict stored memory while preserving +the ADD-only spine and as-of semantics. Auto transforms apply provably-safe verbs; +propose transforms stage judgment calls into the human review queue. +""" + +from __future__ import annotations + +from .config import MS_PER_DAY, MaintenanceConfig +from .score import value +from .summarize import ( + ExtractiveStubSummarizer, + OllamaSummarizer, + Summarizer, + default_summarizer, +) +from .transforms import ( + Merge, + StagedProposal, + TransformReport, + dedup_exact, + dedup_near, + evict_auto, + evict_propose, + normalize_text, + run_transforms, + summarize, +) + +__all__ = [ + "MaintenanceConfig", + "MS_PER_DAY", + "value", + "Summarizer", + "ExtractiveStubSummarizer", + "OllamaSummarizer", + "default_summarizer", + "normalize_text", + "dedup_exact", + "dedup_near", + "summarize", + "evict_propose", + "evict_auto", + "run_transforms", + "Merge", + "StagedProposal", + "TransformReport", +] diff --git a/src/lean_memory/maintain/config.py b/src/lean_memory/maintain/config.py new file mode 100644 index 0000000..659a220 --- /dev/null +++ b/src/lean_memory/maintain/config.py @@ -0,0 +1,84 @@ +"""MaintenanceConfig — the frozen knob set for the sleep-time maintenance job. + +The engine has no config mechanism today; per the design spec (§3.6) this frozen +dataclass IS it. Every default here is spec-pinned (§3.6, §6.6) except +`evict_threshold`, which the spec leaves tunable — see its field comment. + +Its canonical-JSON hash (`config_hash()`) is recorded per run in +`maintenance_run.config_hash`, matching the repo's frozen-config discipline: a +number produced under one config is traceable to the exact knobs that produced +it. Frozen (immutable, hashable) so a run's config can't drift mid-run. +""" + +from __future__ import annotations + +import hashlib +import json +from dataclasses import asdict, dataclass + +# One epoch-ms day — ages in the transforms are computed in ms (the same unit as +# `now`, `valid_at`, `last_access`), so the day-valued knobs convert through this. +MS_PER_DAY = 1000 * 60 * 60 * 24 + + +@dataclass(frozen=True) +class MaintenanceConfig: + """Immutable configuration for a maintenance run (design spec §3.6, §6.6). + + Day-valued thresholds (`*_days`) are stored in days for human legibility and + converted to ms by the transforms via `MS_PER_DAY`. + """ + + # ── DEDUP-NEAR (§4.2) ── + #: Cosine floor on stored embeddings for a same-slot pair to be *proposed* as a + #: near-duplicate. 0.95 per §3.6/§4.2 — below it the multivalued co-valid band + #: ("likes jazz"/"likes blues" ~0.6-0.95) and near-but-distinct literals live. + tau_near: float = 0.95 + + # ── SUMMARIZE (§4.3) / EVICT (§4.4) age gate ── + #: A fact must be older than this (in days) before SUMMARIZE or EVICT will touch + #: it (§3.6). A hard guard in EVICT — never demote a fact younger than this. + age_floor_days: int = 90 + #: A (subject, predicate) slot must hold at least this many latest facts before + #: SUMMARIZE considers it a cluster worth consolidating (§3.6). + min_cluster: int = 5 + + # ── EVICT (§4.4) ── + #: Standing value-score floor below which a still-latest fact is *proposed* for + #: demotion. The spec (§3.6, §4.4) pins NO value here — it is explicitly left + #: tunable. 0.15 is a conservative default: with salience alone contributing up + #: to 0.5 to the score, only genuinely low-salience, stale, unaccessed facts fall + #: below it. Documented as tunable per the spec. + evict_threshold: float = 0.15 + #: The strict AUTO-band (§4.4): a fact matching ALL of (salience < this, + #: access_count == 0, age > `auto_evict_age_days`) is demoted to 'cold' WITHOUT + #: review. §3.6 pins the band as `salience<2 AND access_count=0 AND age>180d`. + auto_evict_salience: float = 2.0 + auto_evict_age_days: int = 180 + + # ── proposal queue (§3.6, §5) ── + #: Proposals not decided within this many days expire (never auto-apply; silence + #: ≠ consent, §0/§12). Sets each proposal's `expires_at = now + this` (§4.2). + proposal_expiry_days: int = 30 + #: Max proposals a single run may stage. Beyond it the run truncates and REPORTS + #: the drop (§8.1) — never a silent cap. Keeps the review queue small (§2.4). + proposal_budget_per_run: int = 50 + + # ── work thresholds (§6.6) — read by the runner (Task 5), housed here ── + #: Below ALL of these, a run is a no-op. Facts since cursor ≥ this, OR cumulative + #: salience of new facts ≥ `min_new_salience`, OR ≥ `max_days_between_runs` since + #: the last run (generative-agents reflection trigger, rescaled — §6.6). + min_new_facts: int = 200 + min_new_salience: float = 300.0 + max_days_between_runs: int = 7 + + def config_hash(self) -> str: + """SHA-256 of this config's canonical JSON (sorted keys, stable separators). + + Deterministic across processes/machines for identical field values — the + traceability anchor stamped in `maintenance_run.config_hash`. + """ + canonical = json.dumps( + asdict(self), sort_keys=True, separators=(",", ":") + ) + return hashlib.sha256(canonical.encode("utf-8")).hexdigest() diff --git a/src/lean_memory/maintain/score.py b/src/lean_memory/maintain/score.py new file mode 100644 index 0000000..644e43e --- /dev/null +++ b/src/lean_memory/maintain/score.py @@ -0,0 +1,51 @@ +"""Standing (query-free) value score for a fact — the EVICT signal (design spec §4.4). + +This is NOT the retriever's query-time score; it is a query-independent "how much +is this fact worth keeping on the hot surface" number in [0, 1]. EVICT proposes / +auto-demotes facts that score low (§4.4). + + value = 0.5·(salience/10) + + 0.3·exp(−DECAY_LAMBDA·(now − (last_access or valid_at))) + + 0.2·min(1, log1p(access_count)/log1p(10)) + +The recency anchor is VERBATIM the retriever's — `(last_access or valid_at)` +(`retrieve/retriever.py:97`), reusing its `DECAY_LAMBDA` (imported, never +re-derived). This alignment is load-bearing: "stale" must mean the same thing to +EVICT as to the ranker. In particular a BACKFILLED fact — old `valid_at`, no +`last_access` yet — scores stale the moment it lands, exactly as the retriever +de-ranks it, even though it was just ingested. Using `created_at`/ingest time +instead (rev 1's bug) would score it fresh and diverge from the ranker (§4.4). +""" + +from __future__ import annotations + +import math + +from ..retrieve.retriever import DECAY_LAMBDA # reuse the ranker's decay — never re-derive +from ..types import Fact + +# Weights (§4.4). Sum to 1.0, so `value` ∈ [0, 1]. +_W_SALIENCE = 0.5 +_W_RECENCY = 0.3 +_W_ACCESS = 0.2 + +# Access-count saturation reference: log1p(10) — 10 hits saturates the term to ~1. +_ACCESS_SAT = math.log1p(10) + + +def value(fact: Fact, now: int) -> float: + """Standing value of `fact` at wall-clock `now` (epoch ms), in [0, 1]. + + Recency anchor is the retriever's `(last_access or valid_at)` verbatim; age is + clamped at 0 (a future-dated fact is treated as maximally recent, matching the + retriever's `max(0, ...)`). + """ + salience_term = _W_SALIENCE * (fact.salience / 10.0) + + anchor = fact.last_access if fact.last_access else fact.valid_at + age = max(0, now - anchor) + recency_term = _W_RECENCY * math.exp(-DECAY_LAMBDA * age) + + access_term = _W_ACCESS * min(1.0, math.log1p(fact.access_count) / _ACCESS_SAT) + + return salience_term + recency_term + access_term diff --git a/src/lean_memory/maintain/summarize.py b/src/lean_memory/maintain/summarize.py new file mode 100644 index 0000000..aa1234e --- /dev/null +++ b/src/lean_memory/maintain/summarize.py @@ -0,0 +1,137 @@ +"""Summarizer seam for SUMMARIZE (design spec §3.5, §4.3). + +Offline-by-default discipline: the default summarizer is a deterministic extractive +stub (top-salience `fact_text`s, honestly labeled — NOT abstractive prose). `[llm]` +upgrades to an Ollama abstractive backend, import-guarded so this module imports +with no `ollama` installed and only fails on instantiation/use. Every proposal +records the backend that produced its evidence via `backend_id` — 'stub' or +'ollama:' (§3.5). + +`LM_FORCE_STUBS` is honored (as elsewhere in the codebase): `default_summarizer()` +returns the stub whenever it is set, even if `[llm]` is installed — the test suite +and CI never touch a model. +""" + +from __future__ import annotations + +import os +from typing import Optional, Protocol, Sequence, runtime_checkable + +from ..types import Fact + + +@runtime_checkable +class Summarizer(Protocol): + """Turns a cluster of source facts into one summary sentence (SUMMARIZE, §4.3). + + Implementations MUST be deterministic given a fixed backend (temperature 0 for + a model) so a staged proposal is reproducible. `backend_id` labels the evidence + backend for the proposal payload ('stub' | 'ollama:'). + """ + + #: Identifies the backend for `evidence_backend` on the staged proposal. + backend_id: str + + def summarize(self, facts: Sequence[Fact]) -> str: ... + + +def _stable_top_salience(facts: Sequence[Fact]) -> list[Fact]: + """Facts ordered by descending salience, tie-broken by ascending id (stable, + deterministic regardless of input order).""" + return sorted(facts, key=lambda f: (-f.salience, f.id)) + + +class ExtractiveStubSummarizer: + """Deterministic, dependency-free default — the offline SUMMARIZE backend. + + NOT abstractive: it selects the top-salience source `fact_text`s and joins them + in a stable order behind an honest label, so a reviewer sees exactly which + originals it stands for (no invented prose). Reproducible byte-for-byte given + the same facts — pinned by test. + """ + + #: The label prefix — honest about being extractive, not model-written. + LABEL = "Summary (extractive):" + + def __init__(self, max_facts: int = 5) -> None: + #: Cap on how many source texts the extractive summary carries. + self.max_facts = max_facts + self.backend_id = "stub" + + def summarize(self, facts: Sequence[Fact]) -> str: + ranked = _stable_top_salience(facts)[: self.max_facts] + joined = " ".join(f.fact_text.strip() for f in ranked) + return f"{self.LABEL} {joined}".rstrip() + + +class OllamaSummarizer: + """Real abstractive backend behind the `[llm]` extra — import-guarded. + + Importing THIS module never requires `ollama` (mirrors OllamaTyper): the import + lives inside `_client`, so instantiation of a bare-environment install fails with + a clear, actionable error pointing at `lean-memory[llm]`, and the module itself + stays import-clean for the offline suite. + """ + + DEFAULT_MODEL = "qwen2.5:3b" + + def __init__( + self, + model: str = DEFAULT_MODEL, + *, + host: Optional[str] = None, + temperature: float = 0.0, + ) -> None: + self.model = model + self.host = host + self.temperature = temperature + self.backend_id = f"ollama:{model}" + self._client_obj = None + + def _client(self): + if self._client_obj is not None: + return self._client_obj + try: + import ollama # type: ignore + except ImportError as e: # optional dep absent → clear, actionable failure + raise RuntimeError( + "OllamaSummarizer needs the 'llm' extra: install with " + "`pip install lean-memory[llm]`, or use ExtractiveStubSummarizer " + "for offline summarization." + ) from e + self._client_obj = ollama.Client(host=self.host) if self.host else ollama + return self._client_obj + + def summarize(self, facts: Sequence[Fact]) -> str: + client = self._client() + bullet = "\n".join(f"- {f.fact_text.strip()}" for f in _stable_top_salience(facts)) + prompt = ( + "Consolidate these memory facts about one subject into a single, concise " + "factual summary sentence. Do not invent facts; only compress what is " + "stated.\n\n" + bullet + ) + resp = client.chat( + model=self.model, + messages=[{"role": "user", "content": prompt}], + options={"temperature": self.temperature}, + ) + msg = getattr(resp, "message", None) + content = getattr(msg, "content", None) if msg is not None else None + if content is None: + try: + content = resp["message"]["content"] # type: ignore[index] + except (TypeError, KeyError) as e: + raise RuntimeError(f"unexpected Ollama response shape: {resp!r}") from e + return content.strip() + + +def default_summarizer() -> Summarizer: + """The summarizer to use when a caller supplies none. + + Honors `LM_FORCE_STUBS` (returns the extractive stub even if `[llm]` is + installed) — the offline default. Today it always returns the stub; wiring the + Ollama upgrade on `[llm]` is a v2 item (§9.2), but the seam is here. + """ + if os.environ.get("LM_FORCE_STUBS"): + return ExtractiveStubSummarizer() + return ExtractiveStubSummarizer() diff --git a/src/lean_memory/maintain/transforms.py b/src/lean_memory/maintain/transforms.py new file mode 100644 index 0000000..1733b5d --- /dev/null +++ b/src/lean_memory/maintain/transforms.py @@ -0,0 +1,515 @@ +"""The four maintenance transforms + the intra-run orchestrator (design spec §4). + +Pure functions over the Store ABC. Two transforms AUTO-APPLY provably-safe verbs; +three PROPOSE judgment calls into the review queue with zero spine writes. The +orchestrator (`run_transforms`) enforces the load-bearing intra-run ordering +(§4.4): stage every proposal over the PRE-transform snapshot FIRST, then run the +auto transforms EXCLUDING any fact referenced by a staged proposal — so reviewer +evidence never drifts mid-run. + +Offline & batch discipline (§7.1): all embedding reads and summarizer text are +computed BEFORE any `store.batch()` window — a batch holds only row writes. +""" + +from __future__ import annotations + +import json +import unicodedata +from collections import defaultdict +from dataclasses import dataclass, field +from typing import Iterable, Optional + +import numpy as np + +from ..extract.contradiction import is_multivalued +from ..store.base import Store +from ..types import Fact +from . import score +from .config import MS_PER_DAY, MaintenanceConfig +from .summarize import Summarizer + +Slot = tuple[str, str] # (subject_id, predicate) + + +# ── value-preserving text normalization (DEDUP-EXACT, §4.1) ────────────────── +def normalize_text(s: str) -> str: + """Value-PRESERVING normalization for exact-duplicate detection (§4.1). + + NFC (canonical Unicode composition) + case-fold + whitespace collapse — and + NOTHING else. Never stemming, never synonyms: a lossy normalization could merge + genuinely distinct values ('salary 100k' vs 'salary 110k', 'likes jazz' vs + 'likes blues') — the verified risk that makes DEDUP-EXACT safe to auto-apply. + Two texts share a normal form iff they are the same value written differently + (case / spacing / Unicode form). + """ + nfc = unicodedata.normalize("NFC", s) + folded = nfc.casefold() + return " ".join(folded.split()) + + +# ── reports (what each transform did / would do) ───────────────────────────── +@dataclass +class Merge: + """One exact-duplicate merge that DEDUP-EXACT performed.""" + + slot: Slot + survivor_id: str + loser_ids: list[str] + merged_access_count: int + merged_last_access: Optional[int] + + +@dataclass +class StagedProposal: + """A proposal DEDUP-NEAR / SUMMARIZE / EVICT staged into the review queue.""" + + proposal_id: str + kind: str + fact_ids: list[str] # every fact this proposal references (for exclusion + budget) + + +@dataclass +class TransformReport: + """Aggregate outcome of a maintenance run's transform phase.""" + + merges: list[Merge] = field(default_factory=list) + demoted_ids: list[str] = field(default_factory=list) # auto-band evictions + proposals: list[StagedProposal] = field(default_factory=list) + dropped_proposals: int = 0 # truncated by proposal_budget_per_run (never silent) + + @property + def staged_fact_ids(self) -> set[str]: + """Union of every fact referenced by a staged proposal — the auto-phase + exclusion set (§4.4).""" + out: set[str] = set() + for p in self.proposals: + out.update(p.fact_ids) + return out + + +# ── helpers ────────────────────────────────────────────────────────────────── +def _coalesced_last_access(f: Fact) -> int: + """The retriever's recency anchor for a fact: last_access or valid_at (§4.1/§4.4).""" + return f.last_access if f.last_access else f.valid_at + + +def _cosine(a: np.ndarray, b: np.ndarray) -> float: + """Cosine over stored float32 vectors. Copies out of the read-only frombuffer + view implicitly via numpy ops (no in-place mutation).""" + na = float(np.linalg.norm(a)) + nb = float(np.linalg.norm(b)) + if na == 0.0 or nb == 0.0: + return 0.0 + return float(np.dot(a, b) / (na * nb)) + + +def _latest_nonsummary_in_slot(store: Store, slot: Slot) -> list[Fact]: + """Co-valid is_latest=1 non-summary facts in a slot (DEDUP targets, §4.1/§4.2).""" + subject_id, predicate = slot + return [ + f + for f in store.find_latest_in_slot(subject_id, predicate) + if f.record_kind != "summary" + ] + + +# ── DEDUP-EXACT (auto-apply, §4.1) ─────────────────────────────────────────── +def dedup_exact( + store: Store, + config: MaintenanceConfig, + now: int, + slots: Iterable[Slot], + *, + exclude_ids: Optional[set[str]] = None, +) -> list[Merge]: + """Auto-retire exact duplicates within each slot (§4.1). + + Target: co-valid is_latest=1 non-summary facts in one (subject, predicate) slot + whose NORMALIZED fact_text is identical (value-preserving normalization only). + Survivor = argmin(valid_at), tiebreak min id. Each loser is retired onto the + survivor via `retire_duplicate` (verb (c) — the is_latest_only=False as-of + surface is bit-identical); the survivor inherits the cluster's usage stats: + access_count SUMMED, last_access = max over cluster of coalesce(last_access, + valid_at) — the §4.1 rule that keeps the deduped fact's recency anchor so it is + not de-ranked on the latest surface. + + `exclude_ids`: facts referenced by a staged proposal — never touched here (the + intra-run ordering exclusion, §4.4). Each slot's mutations run in one batch(). + """ + exclude_ids = exclude_ids or set() + merges: list[Merge] = [] + + for slot in slots: + facts = [f for f in _latest_nonsummary_in_slot(store, slot) if f.id not in exclude_ids] + if len(facts) < 2: + continue + + # Group by value-preserving normal form. + clusters: dict[str, list[Fact]] = defaultdict(list) + for f in facts: + clusters[normalize_text(f.fact_text)].append(f) + + for norm, members in clusters.items(): + if len(members) < 2: + continue # a single fact is not a duplicate + + # Survivor = argmin(valid_at), tiebreak min id — preserves "since when". + survivor = min(members, key=lambda f: (f.valid_at, f.id)) + losers = [f for f in members if f.id != survivor.id] + + # Merge usage stats over the WHOLE cluster (survivor included), computed + # BEFORE the batch window (§7.1 — no logic-heavy reads inside the lock). + merged_access = sum(f.access_count for f in members) + merged_last_access = max(_coalesced_last_access(f) for f in members) + + with store.batch(): + for loser in losers: + store.retire_duplicate(loser.id, survivor.id) + store.merge_usage_stats(survivor.id, merged_access, merged_last_access) + + merges.append( + Merge( + slot=slot, + survivor_id=survivor.id, + loser_ids=[f.id for f in losers], + merged_access_count=merged_access, + merged_last_access=merged_last_access, + ) + ) + + return merges + + +# ── DEDUP-NEAR (propose only, §4.2) ────────────────────────────────────────── +def dedup_near( + store: Store, + config: MaintenanceConfig, + now: int, + slots: Iterable[Slot], + *, + run_id: str, + budget: int = 1_000_000, +) -> tuple[list[StagedProposal], int]: + """Stage near-duplicate merge PROPOSALS — zero spine writes (§4.2). + + Target: same-slot co-valid pairs whose stored-embedding cosine >= tau_near and + that are NOT textually identical (post-normalization — those are DEDUP-EXACT's + job). Never auto-applied: the multivalued co-valid band and near-but-distinct + literals make it a judgment call. Stages one proposal per qualifying pair with + both fact ids/texts, cosine, slot, the multivalued flag, and the proposed + survivor (argmin valid_at). evidence_backend='stored' (embeddings read back, not + re-embedded). Respects `budget`; returns (staged, dropped_count). + """ + staged: list[StagedProposal] = [] + dropped = 0 + tau = config.tau_near + expires_at = now + config.proposal_expiry_days * MS_PER_DAY + + for slot in slots: + facts = _latest_nonsummary_in_slot(store, slot) + if len(facts) < 2: + continue + # Read stored embeddings ONCE per fact (no re-embed, no batch window). + embs: dict[str, np.ndarray] = {} + for f in facts: + v = store.get_embedding(f.id) + if v is not None: + embs[f.id] = v + + multivalued = is_multivalued(slot[1]) + # Deterministic pair order: by (valid_at, id) so proposals are reproducible. + ordered = sorted(facts, key=lambda f: (f.valid_at, f.id)) + for i in range(len(ordered)): + for j in range(i + 1, len(ordered)): + a, b = ordered[i], ordered[j] + if a.id not in embs or b.id not in embs: + continue + if normalize_text(a.fact_text) == normalize_text(b.fact_text): + continue # exact dup — DEDUP-EXACT owns it + cos = _cosine(embs[a.id], embs[b.id]) + if cos < tau: + continue + # Proposed survivor = argmin(valid_at) (a already sorts first). + survivor = min((a, b), key=lambda f: (f.valid_at, f.id)) + payload = { + "slot": {"subject_id": slot[0], "predicate": slot[1]}, + "fact_ids": [a.id, b.id], + "fact_texts": {a.id: a.fact_text, b.id: b.fact_text}, + "cosine": round(cos, 6), + "multivalued": multivalued, + "proposed_survivor": survivor.id, + "evidence_backend": "stored", + } + if len(staged) >= budget: + dropped += 1 + continue + pid = store.stage_proposal( + run_id=run_id, + namespace=a.namespace, + kind="dedup_near", + payload_json=json.dumps(payload, sort_keys=True), + created_at=now, + expires_at=expires_at, + evidence_backend="stored", + ) + staged.append( + StagedProposal(proposal_id=pid, kind="dedup_near", fact_ids=[a.id, b.id]) + ) + + return staged, dropped + + +# ── SUMMARIZE (propose only — STAGING side; apply is Task 6, §4.3) ─────────── +def summarize( + store: Store, + config: MaintenanceConfig, + now: int, + summarizer: Summarizer, + *, + run_id: str, + budget: int = 1_000_000, +) -> tuple[list[StagedProposal], int]: + """Stage SUMMARIZE proposals — zero spine writes, no embedding at stage time (§4.3). + + Per subject entity: gather latest non-summary facts older than age_floor_days in + slots holding >= min_cluster such facts. Subjects are ranked by cluster HEAT — + documented as the SUM of score.value() over the subject's qualifying facts (a + query-free standing-value proxy; hotter subjects consolidate first). Payload + carries the source fact ids + texts, the proposed summary text from `summarizer`, + and evidence_backend = the summarizer's backend id. Respects `budget`; returns + (staged, dropped_count). + """ + staged: list[StagedProposal] = [] + dropped = 0 + age_floor_ms = config.age_floor_days * MS_PER_DAY + expires_at = now + config.proposal_expiry_days * MS_PER_DAY + + # Group qualifying (old-enough) latest non-summary facts by subject, then slot. + by_subject: dict[str, dict[Slot, list[Fact]]] = defaultdict(lambda: defaultdict(list)) + for f in store.iter_latest_facts(): + if f.record_kind == "summary": + continue + if (now - f.valid_at) < age_floor_ms: + continue + by_subject[f.subject_id][(f.subject_id, f.predicate)].append(f) + + # Rank subjects by cluster heat = sum of value() over their qualifying facts. + def subject_heat(slots_map: dict[Slot, list[Fact]]) -> float: + return sum(score.value(f, now) for fs in slots_map.values() for f in fs) + + ranked_subjects = sorted( + by_subject.items(), key=lambda kv: (-subject_heat(kv[1]), kv[0]) + ) + + for subject_id, slots_map in ranked_subjects: + # Only slots meeting the cluster-size floor qualify (§4.3). + qualifying = {s: fs for s, fs in slots_map.items() if len(fs) >= config.min_cluster} + if not qualifying: + continue + # Sources = every qualifying fact for the subject, in a stable order. + sources: list[Fact] = sorted( + (f for fs in qualifying.values() for f in fs), + key=lambda f: (f.valid_at, f.id), + ) + # Summarizer text computed BEFORE any batch window (§7.1). Stage-time only — + # no embedding here (embedding is the Task-6 apply path, §4.3). + summary_text = summarizer.summarize(sources) + payload = { + "subject_id": subject_id, + "source_fact_ids": [f.id for f in sources], + "source_fact_texts": {f.id: f.fact_text for f in sources}, + "summary_text": summary_text, + "evidence_backend": summarizer.backend_id, + } + if len(staged) >= budget: + dropped += 1 + continue + pid = store.stage_proposal( + run_id=run_id, + namespace=sources[0].namespace, + kind="summarize", + payload_json=json.dumps(payload, sort_keys=True), + created_at=now, + expires_at=expires_at, + evidence_backend=summarizer.backend_id, + ) + staged.append( + StagedProposal( + proposal_id=pid, kind="summarize", fact_ids=[f.id for f in sources] + ) + ) + + return staged, dropped + + +# ── EVICT (auto strict band + propose, §4.4) ───────────────────────────────── +def _evict_guarded(f: Fact, config: MaintenanceConfig, now: int) -> bool: + """True iff `f` is a legal EVICT candidate — passes every §4.4 guard. + + Never propose/demote: salience >= 6, age < age_floor_days, record_kind='summary'. + (Staged-proposal referencing is handled by the caller's exclusion set.) + access_count==0 alone is NEVER sufficient (implicit — it is not a guard here; the + value threshold and the auto-band decide). + """ + if f.record_kind == "summary": + return False + if f.salience >= 6: + return False + age_ms = now - f.valid_at + if age_ms < config.age_floor_days * MS_PER_DAY: + return False + return True + + +def evict_propose( + store: Store, + config: MaintenanceConfig, + now: int, + *, + run_id: str, + budget: int = 1_000_000, + exclude_ids: Optional[set[str]] = None, +) -> tuple[list[StagedProposal], int]: + """Stage EVICT (demotion) proposals for still-latest facts below the value floor + but NOT in the strict auto-band — zero spine writes (§4.4). + + A guarded fact whose standing value() < evict_threshold and that is not eligible + for the auto-band is proposed for demotion. Respects `budget`; returns + (staged, dropped_count). + """ + exclude_ids = exclude_ids or set() + staged: list[StagedProposal] = [] + dropped = 0 + expires_at = now + config.proposal_expiry_days * MS_PER_DAY + + for f in store.iter_latest_facts(): + if f.id in exclude_ids: + continue + if not _evict_guarded(f, config, now): + continue + if _in_auto_band(f, config, now): + continue # the auto-band demotes it without review + v = score.value(f, now) + if v >= config.evict_threshold: + continue + payload = { + "fact_id": f.id, + "fact_text": f.fact_text, + "value": round(v, 6), + "salience": f.salience, + "access_count": f.access_count, + "evidence_backend": "score", + } + if len(staged) >= budget: + dropped += 1 + continue + pid = store.stage_proposal( + run_id=run_id, + namespace=f.namespace, + kind="evict", + payload_json=json.dumps(payload, sort_keys=True), + created_at=now, + expires_at=expires_at, + evidence_backend="score", + ) + staged.append(StagedProposal(proposal_id=pid, kind="evict", fact_ids=[f.id])) + + return staged, dropped + + +def _in_auto_band(f: Fact, config: MaintenanceConfig, now: int) -> bool: + """The strict auto-demote band (§4.4/§3.6): salience < auto_evict_salience AND + access_count == 0 AND age > auto_evict_age_days. Guards are checked separately.""" + if f.salience >= config.auto_evict_salience: + return False + if f.access_count != 0: + return False + age_ms = now - f.valid_at + return age_ms > config.auto_evict_age_days * MS_PER_DAY + + +def evict_auto( + store: Store, + config: MaintenanceConfig, + now: int, + *, + exclude_ids: Optional[set[str]] = None, +) -> list[str]: + """Auto-demote the strict-band facts to 'cold' without review (§4.4). + + Band (config): salience < auto_evict_salience AND access_count == 0 AND age > + auto_evict_age_days, on a guarded, still-latest, non-excluded fact. Uses + set_tier (verb (c), predicate-invisible). Returns the demoted fact ids. + """ + exclude_ids = exclude_ids or set() + demoted: list[str] = [] + for f in store.iter_latest_facts(): + if f.id in exclude_ids: + continue + if not _evict_guarded(f, config, now): + continue + if not _in_auto_band(f, config, now): + continue + store.set_tier(f.id, "cold") + demoted.append(f.id) + return demoted + + +# ── intra-run orchestrator (the load-bearing ordering, §4.4) ───────────────── +def run_transforms( + store: Store, + config: MaintenanceConfig, + now: int, + *, + run_id: str, + slots: Iterable[Slot], + summarizer: Optional[Summarizer] = None, +) -> TransformReport: + """Run all four transforms in the spec-mandated intra-run order (§4.4). + + ORDER (load-bearing): stage ALL proposals (dedup_near, summarize, evict-propose) + over the PRE-transform snapshot FIRST — so reviewer evidence never drifts — THEN + run the auto transforms (dedup_exact, evict auto-band) EXCLUDING any fact + referenced by a staged proposal. The global proposal budget + (`proposal_budget_per_run`) is shared across the three propose transforms; + truncation is REPORTED in `dropped_proposals`, never silent. + + `slots` is materialized once (the pre-transform snapshot of touched slots) and + reused for both the near-dup proposals and the exact-dup autos. + """ + from .summarize import default_summarizer + + if summarizer is None: + summarizer = default_summarizer() + + slots = list(slots) # materialize once — reused across phases + report = TransformReport() + + # ── Phase 1: STAGE ALL PROPOSALS over the pre-transform snapshot ── + budget = config.proposal_budget_per_run + remaining = budget + + near, dropped = dedup_near(store, config, now, slots, run_id=run_id, budget=remaining) + report.proposals.extend(near) + report.dropped_proposals += dropped + remaining = budget - len(report.proposals) + + summ, dropped = summarize( + store, config, now, summarizer, run_id=run_id, budget=max(0, remaining) + ) + report.proposals.extend(summ) + report.dropped_proposals += dropped + remaining = budget - len(report.proposals) + + ev_prop, dropped = evict_propose( + store, config, now, run_id=run_id, budget=max(0, remaining) + ) + report.proposals.extend(ev_prop) + report.dropped_proposals += dropped + + # ── Phase 2: AUTO transforms, EXCLUDING every staged-proposal target ── + staged_ids = report.staged_fact_ids + report.merges = dedup_exact(store, config, now, slots, exclude_ids=staged_ids) + report.demoted_ids = evict_auto(store, config, now, exclude_ids=staged_ids) + + return report diff --git a/src/lean_memory/store/base.py b/src/lean_memory/store/base.py index 7e77573..8eb0e30 100644 --- a/src/lean_memory/store/base.py +++ b/src/lean_memory/store/base.py @@ -107,6 +107,15 @@ def retire_duplicate(self, loser_id: str, survivor_id: str) -> None: def set_tier(self, fact_id: str, tier: str) -> None: """Move a fact between the hot/cold tiers — fact.tier + fact_vec.tier, one txn.""" + @abstractmethod + def merge_usage_stats( + self, fact_id: str, access_count: int, last_access: Optional[int] + ) -> None: + """Overwrite a fact's usage stats (access_count, last_access) — the + DEDUP-EXACT survivor-merge write (§4.1): the survivor inherits the + cluster-summed access_count and the max coalesce(last_access, valid_at). + Plain UPDATE on fact; no vec/FTS surface involved.""" + @abstractmethod def get_embedding(self, fact_id: str) -> Optional[np.ndarray]: """Read a fact's stored full-dim vector back (no re-embed). None if absent.""" diff --git a/src/lean_memory/store/sqlite_store.py b/src/lean_memory/store/sqlite_store.py index 75d5a56..cb61a34 100644 --- a/src/lean_memory/store/sqlite_store.py +++ b/src/lean_memory/store/sqlite_store.py @@ -501,6 +501,20 @@ def set_tier(self, fact_id: str, tier: str) -> None: db.execute("UPDATE fact_vec SET tier=? WHERE fact_id=?", (tier, fact_id)) self._commit() + def merge_usage_stats( + self, fact_id: str, access_count: int, last_access: Optional[int] + ) -> None: + """DEDUP-EXACT survivor-merge write (§4.1): OVERWRITE the survivor's + usage stats with the cluster-merged values (access_count summed over the + cluster, last_access = max coalesce(last_access, valid_at)). Plain UPDATE + on fact — no vec/FTS surface. Commits through _commit so it participates + in an open batch() window.""" + self._db.execute( + "UPDATE fact SET access_count=?, last_access=? WHERE id=?", + (access_count, last_access, fact_id), + ) + self._commit() + def get_embedding(self, fact_id: str) -> Optional[np.ndarray]: """Read a fact's stored full-dim vector back (no re-embed). None if absent. diff --git a/tests/test_maintenance_asof_grid.py b/tests/test_maintenance_asof_grid.py new file mode 100644 index 0000000..801f601 --- /dev/null +++ b/tests/test_maintenance_asof_grid.py @@ -0,0 +1,218 @@ +"""Task 4 — the as-of grid invariance test at the STORE predicate (design spec §10.1). + +The headline invariance argument, executable and scoped to what runs at THIS task: +the AUTO transforms (dedup_exact, evict auto-band) plus staging (which must be a +ZERO spine delta). The propose-transforms' spine effects only exist after the +Task-6 apply path; Task 6 re-runs this grid post-apply (§10.1 rev-3 note). + +Corpus: backfills (id-order != valid_at-order), a functional slot with a +supersession, and a multivalued slot with exact duplicates. We snapshot the id-set +satisfying the store-level visibility predicate + valid_at <= T AND (valid_to IS NULL OR valid_to > T) +(the pure point-in-time surface, is_latest_only=False) over a T grid, run the auto +transforms + all staging, and assert: + - identical sets for every T < t_m (verb (c) + append-only ⇒ predicate-invariant), + - staging alone is a ZERO spine delta (full fact-table dump unchanged), + - no inverted intervals post-run (valid_to > valid_at), + - a ranking-delta pin (§10.8): DEDUP-EXACT's last_access merge keeps the deduped + fact's latest-mode top-k rank. +""" + +from __future__ import annotations + +import pytest + +from lean_memory.embed.fake import FakeEmbedder +from lean_memory.maintain import ( + MaintenanceConfig, + MS_PER_DAY, + dedup_exact, + dedup_near, + evict_auto, + evict_propose, + summarize, +) +from lean_memory.maintain.summarize import ExtractiveStubSummarizer +from lean_memory.store.sqlite_store import SqliteStore +from lean_memory.types import Entity, Episode, Fact + +T_M = 2_000_000_000_000 # maintenance time (epoch ms) +DAY = MS_PER_DAY + + +@pytest.fixture +def store(tmp_path): + emb = FakeEmbedder() + s = SqliteStore(tmp_path / "grid.db", dim=emb.dim, coarse_dim=emb.coarse_dim) + ep = Episode(namespace="ns", raw="seed", t_ref=1_000) + s.add_episode(ep) + s._emb = emb + s._ep = ep + s._subj = s.upsert_entity(Entity(namespace="ns", name="user", type="person")) + yield s + s.close() + + +def _add(store, text, *, predicate, valid_at, valid_to=None, superseded_by=None, + is_latest=1, salience=5.0, access_count=0, last_access=None, + record_kind="fact", embed_text=None): + f = Fact( + namespace="ns", subject_id=store._subj.id, predicate=predicate, + fact_text=text, valid_at=valid_at, valid_to=valid_to, + superseded_by=superseded_by, is_latest=is_latest, episode_id=store._ep.id, + salience=salience, access_count=access_count, last_access=last_access, + record_kind=record_kind, + ) + full, coarse = store._emb.embed_with_coarse(embed_text if embed_text is not None else text) + store.add_fact(f, full, coarse) + return f + + +def _visible_at(store, T): + """The STORE-level visibility predicate id-set at world-time T (is_latest_only=False).""" + rows = store._db.execute( + "SELECT id FROM fact WHERE valid_at <= ? AND (valid_to IS NULL OR valid_to > ?)", + (T, T), + ).fetchall() + return frozenset(r["id"] for r in rows) + + +def _fact_dump(store): + return [ + tuple(r) + for r in store._db.execute( + "SELECT id, is_latest, valid_at, valid_to, superseded_by, tier, " + "access_count, last_access, record_kind FROM fact ORDER BY id" + ).fetchall() + ] + + +def _build_corpus(store): + """A corpus with backfills, a functional supersession, and exact dups on a + multivalued slot. Returns a dict of the notable facts.""" + # Functional slot 'works_at': acme (valid_at t0) superseded by globex (valid_at t1). + t0 = T_M - 500 * DAY + t1 = T_M - 300 * DAY + acme = _add(store, "user works at acme", predicate="works_at", valid_at=t0, + valid_to=t1, superseded_by=None, is_latest=0) + globex = _add(store, "user works at globex", predicate="works_at", valid_at=t1) + store._db.execute("UPDATE fact SET superseded_by=? WHERE id=?", (globex.id, acme.id)) + store._db.commit() + + # Multivalued slot 'likes' with EXACT duplicates (case/whitespace variants) — + # BACKFILLED so id-order != valid_at-order (the later-ingested row is older). + jazz_a = _add(store, "likes jazz", predicate="likes", valid_at=T_M - 400 * DAY, + access_count=1, last_access=T_M - 350 * DAY) + jazz_b = _add(store, "LIKES jazz", predicate="likes", valid_at=T_M - 450 * DAY, + access_count=4, last_access=T_M - 10 * DAY) # older world-time, newer access + # A distinct multivalued value that must NEVER merge. + blues = _add(store, "likes blues", predicate="likes", valid_at=T_M - 420 * DAY) + + # An auto-band eviction candidate (salience<2, access 0, age>180d) on its own slot. + trivial = _add(store, "trivial ancient note", predicate="notes", valid_at=T_M - 300 * DAY, + salience=1.0, access_count=0) + + return { + "acme": acme, "globex": globex, + "jazz_a": jazz_a, "jazz_b": jazz_b, "blues": blues, "trivial": trivial, + } + + +def test_asof_grid_invariant_under_autos_and_staging(store): + facts = _build_corpus(store) + cfg = MaintenanceConfig() + + grid = [T_M - t * DAY for t in (480, 460, 430, 410, 350, 250, 100, 1)] + before_sets = {T: _visible_at(store, T) for T in grid} + dump_before_staging = _fact_dump(store) + + # ── Phase 1: STAGE all proposals (must be ZERO spine delta) ── + slots = [(store._subj.id, p) for p in ("works_at", "likes", "notes")] + run_id = store.create_run("ns", "cli", T_M, cfg.config_hash()) + dedup_near(store, cfg, T_M, slots, run_id=run_id) + summarize(store, cfg, T_M, ExtractiveStubSummarizer(), run_id=run_id) + evict_propose(store, cfg, T_M, run_id=run_id) + assert _fact_dump(store) == dump_before_staging, "staging wrote ZERO spine changes" + + # ── Phase 2: AUTO transforms (dedup_exact + evict auto-band) ── + merges = dedup_exact(store, cfg, T_M, slots) + demoted = evict_auto(store, cfg, T_M) + + # The exact jazz duplicates merged; blues never did. + assert len(merges) == 1 + merged_ids = set(merges[0].loser_ids) | {merges[0].survivor_id} + assert merged_ids == {facts["jazz_a"].id, facts["jazz_b"].id} + assert facts["blues"].id not in merged_ids + # The trivial note was auto-demoted. + assert demoted == [facts["trivial"].id] + + # ── Invariance: the store predicate is bit-identical for every T < t_m ── + for T in grid: + assert T < T_M + assert _visible_at(store, T) == before_sets[T], f"predicate changed at T={T}" + + # ── No inverted intervals anywhere post-run ── + inverted = store._db.execute( + "SELECT id FROM fact WHERE valid_to IS NOT NULL AND valid_to <= valid_at" + ).fetchall() + assert inverted == [], "no inverted (valid_to <= valid_at) intervals post-run" + + +def test_asof_grid_dedup_exact_leaves_valid_to_untouched(store): + """DEDUP-EXACT is verb (c): the retired duplicate's valid_to stays NULL, so it + stays visible on the pure as-of surface for every T after its valid_at.""" + facts = _build_corpus(store) + cfg = MaintenanceConfig() + slots = [(store._subj.id, "likes")] + dedup_exact(store, cfg, T_M, slots) + + loser = facts["jazz_a"] if facts["jazz_a"].valid_at > facts["jazz_b"].valid_at else facts["jazz_b"] + row = store._db.execute( + "SELECT is_latest, valid_to FROM fact WHERE id=?", (loser.id,) + ).fetchone() + assert row["is_latest"] == 0 # dropped from the latest surface + assert row["valid_to"] is None # but as-of-visible: valid_to untouched + # Visible at a T after its own valid_at (pure as-of surface). + T_after = max(facts["jazz_a"].valid_at, facts["jazz_b"].valid_at) + 1 + assert loser.id in _visible_at(store, T_after) + + +def test_dedup_exact_ranking_delta_pin(store): + """§10.8 ranking honesty: after DEDUP-EXACT the survivor keeps the deduped + fact's RECENCY via the last_access merge, so it holds its latest-mode top-k + rank. Constructed so that WITHOUT the merge the survivor would be de-ranked. + + We score the survivor's retriever recency term before vs after the merge and + assert it rose to the cluster's freshest last_access (the merge rule), which is + what keeps its rank.""" + from lean_memory.retrieve.retriever import DECAY_LAMBDA + import math + + # Survivor = oldest valid_at, but STALE last_access; loser = newer restatement + # with a FRESH last_access. Without the merge the survivor stays stale. + survivor = _add(store, "user works at acme", predicate="works_at", + valid_at=T_M - 500 * DAY, access_count=1, + last_access=T_M - 490 * DAY) # very stale + loser = _add(store, "USER WORKS AT ACME", predicate="works_at", + valid_at=T_M - 400 * DAY, access_count=1, + last_access=T_M - 2 * DAY) # fresh restatement + + def recency(last_access): + age = max(0, T_M - last_access) + return math.exp(-DECAY_LAMBDA * age) + + rec_before = recency(survivor.last_access) # stale survivor recency + + dedup_exact(store, MaintenanceConfig(), T_M, [(store._subj.id, "works_at")]) + + merged = store._db.execute( + "SELECT access_count, last_access FROM fact WHERE id=?", (survivor.id,) + ).fetchone() + # last_access = max coalesce over cluster = loser's fresh last_access. + assert merged["last_access"] == loser.last_access + assert merged["access_count"] == 2 # 1 + 1 summed + rec_after = recency(merged["last_access"]) + # The merge lifted the survivor's recency term toward "fresh" — it is NOT + # de-ranked. Dropping the merge would have left it at the stale rec_before. + assert rec_after > rec_before + assert rec_after > 0.9 # ~2 days of decay → near-fresh diff --git a/tests/test_maintenance_transforms.py b/tests/test_maintenance_transforms.py new file mode 100644 index 0000000..9185ed6 --- /dev/null +++ b/tests/test_maintenance_transforms.py @@ -0,0 +1,489 @@ +"""Task 4 — MaintenanceConfig, scoring, summarizer, and the four transforms. + +All offline (FakeEmbedder, deterministic stub summarizer) against a real +SqliteStore. Pins the design-spec §3.6/§4.1–§4.4 behavior: + - config hash stability + - value() recency anchor incl. the backfill case + - extractive stub summarizer pinned output + - value-preserving normalization (case/whitespace/NFC) + NEVER-merge distinct values + - DEDUP-EXACT survivor rule + tiebreak + usage-stats merge + - multivalued slots never auto-merged; only DEDUP-NEAR proposes them + - EVICT guards + strict auto-band + - intra-run ordering: a staged-proposal target is excluded from the autos + - proposal budget truncation is reported, not silent + - all three propose transforms make ZERO spine writes +""" + +from __future__ import annotations + +import unicodedata + +import pytest + +from lean_memory.embed.fake import FakeEmbedder +from lean_memory.maintain import ( + ExtractiveStubSummarizer, + MaintenanceConfig, + MS_PER_DAY, + dedup_exact, + dedup_near, + evict_auto, + evict_propose, + normalize_text, + run_transforms, + summarize, + value, +) +from lean_memory.store.sqlite_store import SqliteStore +from lean_memory.types import Entity, Episode, Fact + +NOW = 2_000_000_000_000 # fixed wall-clock for reproducibility (epoch ms) + + +# ── fixtures / helpers ──────────────────────────────────────────────────────── +@pytest.fixture +def store(tmp_path): + emb = FakeEmbedder() + s = SqliteStore(tmp_path / "ns.db", dim=emb.dim, coarse_dim=emb.coarse_dim) + ep = Episode(namespace="ns", raw="seed", t_ref=1_000) + s.add_episode(ep) + subj = s.upsert_entity(Entity(namespace="ns", name="user", type="person")) + s._emb = emb + s._ep = ep + s._subj = subj + yield s + s.close() + + +def _run(store): + """Claim a maintenance run and return its id (proposals need a run_id).""" + return store.create_run("ns", "cli", NOW, MaintenanceConfig().config_hash()) + + +def _add( + store, + text, + *, + predicate="works_at", + valid_at, + salience=5.0, + access_count=0, + last_access=None, + is_latest=1, + record_kind="fact", + subject_id=None, + embed_text=None, +): + """Insert a fact. `embed_text` overrides what gets embedded (to force cosine bands).""" + f = Fact( + namespace="ns", + subject_id=subject_id or store._subj.id, + predicate=predicate, + fact_text=text, + valid_at=valid_at, + episode_id=store._ep.id, + salience=salience, + access_count=access_count, + last_access=last_access, + is_latest=is_latest, + record_kind=record_kind, + ) + full, coarse = store._emb.embed_with_coarse(embed_text if embed_text is not None else text) + store.add_fact(f, full, coarse) + return f + + +def _fact_table_dump(store): + """Full ordered dump of the fact table — the ZERO-spine-delta oracle.""" + return store._db.execute( + "SELECT id, is_latest, valid_at, valid_to, superseded_by, tier, " + "access_count, last_access, record_kind FROM fact ORDER BY id" + ).fetchall() + + +def _dump_tuples(store): + return [tuple(r) for r in _fact_table_dump(store)] + + +# ── config ──────────────────────────────────────────────────────────────────── +def test_config_defaults_match_spec(): + c = MaintenanceConfig() + assert c.tau_near == 0.95 + assert c.age_floor_days == 90 + assert c.min_cluster == 5 + assert c.proposal_expiry_days == 30 + assert c.proposal_budget_per_run == 50 + assert c.auto_evict_salience == 2.0 + assert c.auto_evict_age_days == 180 + assert c.min_new_facts == 200 + assert c.min_new_salience == 300.0 + assert c.max_days_between_runs == 7 + + +def test_config_hash_is_stable_and_sensitive(): + a = MaintenanceConfig() + b = MaintenanceConfig() + assert a.config_hash() == b.config_hash(), "same fields → same hash" + assert len(a.config_hash()) == 64, "sha256 hex digest" + c = MaintenanceConfig(tau_near=0.9) + assert c.config_hash() != a.config_hash(), "different field → different hash" + + +def test_config_is_frozen(): + c = MaintenanceConfig() + with pytest.raises(Exception): + c.tau_near = 0.5 # frozen dataclass + + +# ── score.value ─────────────────────────────────────────────────────────────── +def test_value_in_unit_range_and_weighted(): + f = Fact( + namespace="ns", subject_id="s", predicate="p", fact_text="x", + valid_at=NOW, episode_id="e", salience=10.0, access_count=10, last_access=NOW, + ) + v = value(f, NOW) + assert 0.0 <= v <= 1.0 + # salience 10 → 0.5, recency (age 0) → 0.3, access log1p(10)/log1p(10)=1 → 0.2 + assert v == pytest.approx(1.0, abs=1e-9) + + +def test_value_backfill_anchor_scores_stale(store): + """A BACKFILLED fact — old valid_at, NO last_access — must score stale even though + it was just ingested (§4.4: anchor is (last_access or valid_at), never ingest time).""" + old_valid = NOW - 400 * MS_PER_DAY # >1yr in the world-time past + backfilled = Fact( + namespace="ns", subject_id="s", predicate="p", fact_text="old news", + valid_at=old_valid, episode_id="e", salience=0.0, access_count=0, + last_access=None, + ) + fresh = Fact( + namespace="ns", subject_id="s", predicate="p", fact_text="new news", + valid_at=NOW, episode_id="e", salience=0.0, access_count=0, last_access=None, + ) + # Both have salience 0 & 0 accesses → only the recency term differs. + assert value(backfilled, NOW) < value(fresh, NOW) + # The stale one's recency term is ~0 (400 days of decay), so value ≈ 0. + assert value(backfilled, NOW) < 0.05 + + +def test_value_last_access_overrides_valid_at(): + """last_access present → it is the anchor, NOT valid_at.""" + old_valid = NOW - 400 * MS_PER_DAY + recently_read = Fact( + namespace="ns", subject_id="s", predicate="p", fact_text="x", + valid_at=old_valid, episode_id="e", salience=0.0, access_count=0, + last_access=NOW, # read just now → fresh despite ancient valid_at + ) + assert value(recently_read, NOW) > 0.25 # recency term near its 0.3 max + + +# ── extractive stub summarizer ──────────────────────────────────────────────── +def test_stub_summarizer_pinned_output(store): + f1 = _add(store, "user likes tea", valid_at=1_000, salience=3.0) + f2 = _add(store, "user likes coffee", valid_at=2_000, salience=8.0) + f3 = _add(store, "user likes water", valid_at=3_000, salience=5.0) + out = ExtractiveStubSummarizer().summarize([f1, f2, f3]) + # Ordered by DESC salience, tie-break ASC id: coffee(8) water(5) tea(3). + assert out == "Summary (extractive): user likes coffee user likes water user likes tea" + + +def test_stub_summarizer_backend_id(): + assert ExtractiveStubSummarizer().backend_id == "stub" + + +# ── normalization (value-preserving ONLY) ───────────────────────────────────── +def test_normalize_case_and_whitespace(): + assert normalize_text("User Likes\tCoffee") == normalize_text("user likes coffee") + + +def test_normalize_nfc_equivalence(): + # 'é' as NFC single codepoint vs NFD 'e' + combining accent → same normal form. + nfc = "café" # café (composed) + nfd = "café" # café (decomposed) + assert unicodedata.normalize("NFC", nfd) == nfc # sanity on the fixture + assert normalize_text(nfc) == normalize_text(nfd) + + +def test_normalize_never_merges_distinct_values(): + """The load-bearing safety property: distinct values NEVER share a normal form.""" + assert normalize_text("salary 100k") != normalize_text("salary 110k") + assert normalize_text("likes jazz") != normalize_text("likes blues") + + +# ── DEDUP-EXACT ──────────────────────────────────────────────────────────────── +def test_dedup_exact_survivor_argmin_valid_at(store): + survivor = _add(store, "user works at acme", valid_at=1_000) + loser = _add(store, "USER works at ACME", valid_at=2_000) # same value, diff case/ws + slot = (store._subj.id, "works_at") + merges = dedup_exact(store, MaintenanceConfig(), NOW, [slot]) + assert len(merges) == 1 + assert merges[0].survivor_id == survivor.id # argmin(valid_at) + assert merges[0].loser_ids == [loser.id] + # loser retired (verb (c)): is_latest=0, superseded_by=survivor, valid_to UNTOUCHED + row = store._db.execute( + "SELECT is_latest, superseded_by, valid_to FROM fact WHERE id=?", (loser.id,) + ).fetchone() + assert row["is_latest"] == 0 + assert row["superseded_by"] == survivor.id + assert row["valid_to"] is None + + +def test_dedup_exact_tiebreak_min_id(store): + """Equal valid_at → survivor is min id.""" + a = _add(store, "user works at acme", valid_at=5_000) + b = _add(store, "user works at acme", valid_at=5_000) + survivor_id = min(a.id, b.id) + slot = (store._subj.id, "works_at") + merges = dedup_exact(store, MaintenanceConfig(), NOW, [slot]) + assert merges[0].survivor_id == survivor_id + + +def test_dedup_exact_merges_usage_stats(store): + """access_count summed over cluster; last_access = max coalesce(last_access, valid_at).""" + survivor = _add(store, "user works at acme", valid_at=1_000, access_count=2, last_access=1_500) + loser = _add(store, "user works at acme", valid_at=9_000, access_count=3, last_access=None) + slot = (store._subj.id, "works_at") + dedup_exact(store, MaintenanceConfig(), NOW, [slot]) + row = store._db.execute( + "SELECT access_count, last_access FROM fact WHERE id=?", (survivor.id,) + ).fetchone() + assert row["access_count"] == 5 # 2 + 3 + # max(coalesce(1500, 1000), coalesce(None→9000, 9000)) = 9000 + assert row["last_access"] == 9_000 + + +def test_dedup_exact_excludes_summaries(store): + """record_kind='summary' rows are never a dedup target.""" + _add(store, "user works at acme", valid_at=1_000, record_kind="summary", predicate="summary") + _add(store, "user works at acme", valid_at=2_000, record_kind="summary", predicate="summary") + slot = (store._subj.id, "summary") + merges = dedup_exact(store, MaintenanceConfig(), NOW, [slot]) + assert merges == [] + + +def test_dedup_exact_distinct_values_never_merged(store): + """Different values in the same slot are NEVER merged (normalization is safe).""" + _add(store, "salary 100k", predicate="salary", valid_at=1_000) + _add(store, "salary 110k", predicate="salary", valid_at=2_000) + slot = (store._subj.id, "salary") + merges = dedup_exact(store, MaintenanceConfig(), NOW, [slot]) + assert merges == [] + + +def test_dedup_exact_multivalued_distinct_never_merged(store): + """'likes jazz' / 'likes blues' — distinct multivalued values — never auto-merged.""" + _add(store, "likes jazz", predicate="likes", valid_at=1_000) + _add(store, "likes blues", predicate="likes", valid_at=2_000) + slot = (store._subj.id, "likes") + merges = dedup_exact(store, MaintenanceConfig(), NOW, [slot]) + assert merges == [] + + +# ── DEDUP-NEAR (propose) ─────────────────────────────────────────────────────── +def test_dedup_near_proposes_high_cosine_pair(store): + """Two non-identical texts with cosine >= tau_near → one proposal, no spine write.""" + # Force cosine >= 0.95 by embedding both near-identically but keeping text distinct. + f1 = _add(store, "likes jazz music", predicate="likes", valid_at=1_000, embed_text="MUSICVEC") + f2 = _add(store, "likes jazz tunes", predicate="likes", valid_at=2_000, embed_text="MUSICVEC") + # identical embed_text → cosine 1.0 (>= tau); texts differ → not exact dup. + run_id = _run(store) + before = _dump_tuples(store) + staged, dropped = dedup_near(store, MaintenanceConfig(), NOW, [(store._subj.id, "likes")], run_id=run_id) + assert dropped == 0 + assert len(staged) == 1 + assert set(staged[0].fact_ids) == {f1.id, f2.id} + # ZERO spine writes. + assert _dump_tuples(store) == before + prop = store.get_proposal(staged[0].proposal_id) + assert prop["kind"] == "dedup_near" + assert prop["evidence_backend"] == "stored" + import json + payload = json.loads(prop["payload_json"]) + assert payload["multivalued"] is True # 'likes' is a multivalued predicate + assert payload["cosine"] >= 0.95 + assert payload["proposed_survivor"] == f1.id # argmin valid_at + + +def test_dedup_near_ignores_low_cosine(store): + _add(store, "likes jazz", predicate="likes", valid_at=1_000, embed_text="AAA") + _add(store, "likes opera", predicate="likes", valid_at=2_000, embed_text="ZZZ") + run_id = _run(store) + staged, _ = dedup_near(store, MaintenanceConfig(), NOW, [(store._subj.id, "likes")], run_id=run_id) + assert staged == [] # FakeEmbedder gives near-orthogonal vectors for distinct text + + +def test_dedup_near_skips_exact_duplicates(store): + """Textually identical (post-normalize) pairs belong to DEDUP-EXACT, not near.""" + _add(store, "likes jazz", predicate="likes", valid_at=1_000, embed_text="SAME") + _add(store, "LIKES JAZZ", predicate="likes", valid_at=2_000, embed_text="SAME") + run_id = _run(store) + staged, _ = dedup_near(store, MaintenanceConfig(), NOW, [(store._subj.id, "likes")], run_id=run_id) + assert staged == [] + + +# ── SUMMARIZE (propose) ──────────────────────────────────────────────────────── +def test_summarize_proposes_old_cluster(store): + """>= min_cluster old-enough facts in a slot → one summarize proposal, no spine write.""" + old = NOW - 200 * MS_PER_DAY # older than age_floor (90d) + for i in range(5): + _add(store, f"note number {i}", predicate="notes", valid_at=old + i) + run_id = _run(store) + before = _dump_tuples(store) + staged, dropped = summarize(store, MaintenanceConfig(), NOW, ExtractiveStubSummarizer(), run_id=run_id) + assert dropped == 0 + assert len(staged) == 1 + assert _dump_tuples(store) == before # ZERO spine writes + prop = store.get_proposal(staged[0].proposal_id) + assert prop["kind"] == "summarize" + assert prop["evidence_backend"] == "stub" + + +def test_summarize_skips_small_or_young_clusters(store): + old = NOW - 200 * MS_PER_DAY + # too small (4 < min_cluster 5) + for i in range(4): + _add(store, f"a{i}", predicate="small", valid_at=old + i) + # big but too young (< age_floor) + for i in range(6): + _add(store, f"b{i}", predicate="young", valid_at=NOW - 1_000 + i) + run_id = _run(store) + staged, _ = summarize(store, MaintenanceConfig(), NOW, ExtractiveStubSummarizer(), run_id=run_id) + assert staged == [] + + +# ── EVICT ────────────────────────────────────────────────────────────────────── +def test_evict_auto_band_demotes(store): + """salience<2 AND access_count==0 AND age>180d → demoted to cold without review.""" + old = NOW - 200 * MS_PER_DAY + f = _add(store, "trivial old note", valid_at=old, salience=1.0, access_count=0) + demoted = evict_auto(store, MaintenanceConfig(), NOW) + assert demoted == [f.id] + row = store._db.execute("SELECT tier FROM fact WHERE id=?", (f.id,)).fetchone() + assert row["tier"] == "cold" + vrow = store._db.execute("SELECT tier FROM fact_vec WHERE fact_id=?", (f.id,)).fetchone() + assert vrow["tier"] == "cold" # two-surface sync + + +def test_evict_guard_high_salience(store): + """salience >= 6 is never demoted, even if old + unaccessed.""" + old = NOW - 400 * MS_PER_DAY + _add(store, "important old fact", valid_at=old, salience=9.0, access_count=0) + run_id = _run(store) + assert evict_auto(store, MaintenanceConfig(), NOW) == [] + staged, _ = evict_propose(store, MaintenanceConfig(), NOW, run_id=run_id) + assert staged == [] + + +def test_evict_guard_young(store): + """age < age_floor_days is never demoted/proposed regardless of salience.""" + _add(store, "recent low note", valid_at=NOW - 1_000, salience=0.0, access_count=0) + run_id = _run(store) + assert evict_auto(store, MaintenanceConfig(), NOW) == [] + staged, _ = evict_propose(store, MaintenanceConfig(), NOW, run_id=run_id) + assert staged == [] + + +def test_evict_guard_summary(store): + """record_kind='summary' is never demoted/proposed.""" + old = NOW - 400 * MS_PER_DAY + _add(store, "old summary", predicate="summary", valid_at=old, salience=0.0, + access_count=0, record_kind="summary") + run_id = _run(store) + assert evict_auto(store, MaintenanceConfig(), NOW) == [] + staged, _ = evict_propose(store, MaintenanceConfig(), NOW, run_id=run_id) + assert staged == [] + + +def test_evict_access_count_zero_not_sufficient_alone(store): + """access_count==0 alone never demotes: a salient, recent-ish, unaccessed fact + with salience above the auto band and value above threshold is left alone.""" + old = NOW - 100 * MS_PER_DAY # past age_floor(90) but before auto band(180) + f = _add(store, "unaccessed but salient", valid_at=old, salience=5.0, access_count=0) + run_id = _run(store) + assert evict_auto(store, MaintenanceConfig(), NOW) == [] # not in auto band (age<180, sal>2) + staged, _ = evict_propose(store, MaintenanceConfig(), NOW, run_id=run_id) + # value: 0.5*(5/10)=0.25 salience alone >= evict_threshold(0.15) → no proposal + assert value(f, NOW) >= MaintenanceConfig().evict_threshold + assert staged == [] + + +def test_evict_proposes_below_threshold_out_of_band(store): + """Low value, guarded, but NOT in the strict auto-band → proposal (no spine write).""" + # age between age_floor(90) and auto band(180): 120d. salience 0.5 → value ~0.25*... low. + old = NOW - 120 * MS_PER_DAY + f = _add(store, "meh note", valid_at=old, salience=0.5, access_count=0) + assert not (NOW - f.valid_at > 180 * MS_PER_DAY) # confirm out of auto band + assert value(f, NOW) < MaintenanceConfig().evict_threshold + run_id = _run(store) + before = _dump_tuples(store) + staged, dropped = evict_propose(store, MaintenanceConfig(), NOW, run_id=run_id) + assert len(staged) == 1 and staged[0].fact_ids == [f.id] + assert _dump_tuples(store) == before # ZERO spine writes + assert store.get_proposal(staged[0].proposal_id)["kind"] == "evict" + + +# ── intra-run ordering + budget (run_transforms) ─────────────────────────────── +def test_run_transforms_excludes_staged_from_autos(store): + """A fact referenced by a staged proposal is EXCLUDED from the auto transforms (§4.4). + + Construct a fact that would BOTH be proposed (near-dup) AND auto-band-demotable, + and assert staging wins: it is NOT demoted.""" + old = NOW - 200 * MS_PER_DAY # auto-band eligible (age>180, sal<2, access 0) + # Near-dup pair on a 'likes' slot, both low-salience/old/unaccessed. + a = _add(store, "likes vintage jazz", predicate="likes", valid_at=old, salience=1.0, + access_count=0, embed_text="NEARDUP") + b = _add(store, "likes classic jazz", predicate="likes", valid_at=old + 1, salience=1.0, + access_count=0, embed_text="NEARDUP") + run_id = _run(store) + slots = [(store._subj.id, "likes")] + report = run_transforms(store, MaintenanceConfig(), NOW, run_id=run_id, slots=slots) + # Both facts were staged in the near-dup proposal → excluded from auto-evict. + assert len(report.proposals) >= 1 + staged_ids = report.staged_fact_ids + assert a.id in staged_ids and b.id in staged_ids + assert a.id not in report.demoted_ids and b.id not in report.demoted_ids + # And they were NOT auto-deduped either (excluded from dedup_exact). + for f in (a, b): + row = store._db.execute("SELECT is_latest, tier FROM fact WHERE id=?", (f.id,)).fetchone() + assert row["is_latest"] == 1 # untouched by autos + assert row["tier"] == "hot" + + +def test_run_transforms_budget_truncation_reported(store): + """When staging exceeds proposal_budget_per_run, the drop is REPORTED, not silent.""" + cfg = MaintenanceConfig(proposal_budget_per_run=2) + old = NOW - 200 * MS_PER_DAY + # Build 4 near-dup pairs on 4 distinct slots → 4 candidate proposals, budget 2. + slots = [] + for p in range(4): + pred = f"likes{p}" + _add(store, f"likes thing {p} alpha", predicate=pred, valid_at=old, embed_text=f"DUP{p}") + _add(store, f"likes thing {p} beta", predicate=pred, valid_at=old + 1, embed_text=f"DUP{p}") + slots.append((store._subj.id, pred)) + run_id = _run(store) + report = run_transforms(store, cfg, NOW, run_id=run_id, slots=slots) + assert len(report.proposals) == 2, "budget caps staged proposals" + assert report.dropped_proposals == 2, "the 2 dropped are reported, not silent" + + +def test_run_transforms_all_propose_transforms_zero_spine_delta(store): + """Staging (all three propose transforms via run_transforms) writes ZERO spine + changes — the full fact-table dump is unchanged by the propose phase. + + (Autos here have nothing to do: the near-dup pair is excluded from dedup_exact, + and no fact is in the auto-evict band once staged.)""" + old = NOW - 200 * MS_PER_DAY + _add(store, "likes deep jazz", predicate="likes", valid_at=old, salience=1.0, embed_text="Z") + _add(store, "likes cool jazz", predicate="likes", valid_at=old + 1, salience=1.0, embed_text="Z") + for i in range(5): + _add(store, f"old cluster note {i}", predicate="notes", valid_at=old + i, salience=1.0) + run_id = _run(store) + before = _dump_tuples(store) + report = run_transforms( + store, MaintenanceConfig(), NOW, run_id=run_id, + slots=[(store._subj.id, "likes"), (store._subj.id, "notes")], + ) + assert report.proposals, "something was staged" + # No auto merges/demotions happened here (all candidates were staged/excluded), + # so the entire fact table is byte-identical to before the run. + assert _dump_tuples(store) == before From a7094789a781deee036ba59b1312f30774e79976 Mon Sep 17 00:00:00 2001 From: Wuesteon Date: Thu, 16 Jul 2026 20:16:54 +0800 Subject: [PATCH 06/14] =?UTF-8?q?feat(maintain):=20runner=20=E2=80=94=20le?= =?UTF-8?q?ase,=20thresholds,=20cursor,=20crash-resume?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MaintenanceRunner drives one maintenance run on one namespace store: atomic lease claim (ux_run_live) with fresh-skip / stale-takeover / lost-race paths; OR-combined work thresholds (facts, salience, age) with a cheap below-threshold no-op; advance-before-write cursor high-water and the iter_slots_touched_since cursor-gap fix; cross-run exclusion of facts under review plus an identical-pending dedupe guard for crash idempotence; heartbeats at phase and batch boundaries; takeover never rolls back. run_transforms gains extra_exclude_ids + pending_signatures (threaded to dedup_near/summarize as skip_signatures) — the runner's cross-run coupling, no logic duplicated. 16 new tests (real subprocess lease race, cursor-gap dedup, crash-resume convergence, heartbeat cadence); suite 228 passed, 1 skipped. Co-Authored-By: Claude Fable 5 --- src/lean_memory/maintain/__init__.py | 3 + src/lean_memory/maintain/runner.py | 438 +++++++++++++++++++ src/lean_memory/maintain/transforms.py | 47 +- tests/test_maintenance_runner.py | 572 +++++++++++++++++++++++++ 4 files changed, 1056 insertions(+), 4 deletions(-) create mode 100644 src/lean_memory/maintain/runner.py create mode 100644 tests/test_maintenance_runner.py diff --git a/src/lean_memory/maintain/__init__.py b/src/lean_memory/maintain/__init__.py index 6a4a744..9dcd968 100644 --- a/src/lean_memory/maintain/__init__.py +++ b/src/lean_memory/maintain/__init__.py @@ -16,6 +16,7 @@ Summarizer, default_summarizer, ) +from .runner import MaintenanceRunner, RunReport from .transforms import ( Merge, StagedProposal, @@ -47,4 +48,6 @@ "Merge", "StagedProposal", "TransformReport", + "MaintenanceRunner", + "RunReport", ] diff --git a/src/lean_memory/maintain/runner.py b/src/lean_memory/maintain/runner.py new file mode 100644 index 0000000..a782840 --- /dev/null +++ b/src/lean_memory/maintain/runner.py @@ -0,0 +1,438 @@ +"""MaintenanceRunner — the concurrency-and-crash-safety driver (design spec §6.6, §7). + +One runner drives ONE namespace store through a single maintenance run: + + 1. LEASE (§7.2): the `maintenance_run` INSERT is the atomic claim — the partial + unique index `ux_run_live` makes a second live run for the same namespace raise + IntegrityError, never a silent second row. Before claiming, a live run with a + FRESH heartbeat means we cleanly skip; a live run with a STALE heartbeat is + marked 'aborted' and taken over. Takeover NEVER rolls anything back — transforms + are idempotent and every batch co-commits (§7.4), so a partial prior run is safe + to resume from DB state. + 2. WORK THRESHOLDS (§6.6, OR-combined): facts since cursor >= min_new_facts, OR + cumulative salience of new facts >= min_new_salience, OR >= max_days_between_runs + since the last finished run. Below all of them the run is a cheap no-op: it + releases the lease with status 'ok' and a `below_threshold` stat, writing no + proposals and invoking no transforms. The first-ever run (no prior finished run) + is always over threshold. + 3. CURSOR (§6.6, advance-before-write): the new cursor is the max fact id snapshotted + at run START, BEFORE any transform output is written — so the run's own outputs + (summaries with higher ids, maintenance episodes) land after it and are excluded + from the next run's "new facts" delta by id order. Slot-level transforms are + driven by `iter_slots_touched_since(previous_cursor)`; `iter_latest_facts` feeds + evict/summarize. Candidate scans already exclude record_kind='summary' (verified + in transforms.py — we rely on it, never duplicate it). + 4. CROSS-RUN EXCLUSION (§4.4): the auto phase excludes not just this run's staged + ids but every fact id referenced by a PRIOR run's still-pending proposal, and a + propose transform never re-stages an identical-evidence proposal (§7.4). + +Heartbeats fire at every batch boundary via the `on_batch` hook the runner threads +into the store, and the runner heartbeats around the transform phase. The staleness +threshold is `max(300s, 10 * longest observed single-batch duration this process)` — +`observed` defaults to 0, so a fresh process uses the 5-minute floor; nothing exotic +is persisted. + +Store construction (§7.1): the runner does NOT construct stores — the CLI/Memory +wiring (Tasks 6-7) opens a dedicated maintenance `SqliteStore` with +`busy_timeout_ms=5000` for the run's duration and passes it in. The runner only +drives whatever store it is handed. Tests open that second/maintenance store directly. +""" + +from __future__ import annotations + +import json +import time +from dataclasses import dataclass, field +from typing import Callable, Optional + +from ..store.base import Store +from .config import MS_PER_DAY, MaintenanceConfig +from .summarize import Summarizer +from .transforms import TransformReport, run_transforms + +# Staleness floor: a live run whose heartbeat is older than this (or older than +# 10x the longest observed single-batch duration, whichever is larger) is abandoned +# and may be taken over (§7.2). +_STALE_FLOOR_S = 300.0 # 5 minutes + + +@dataclass +class RunReport: + """The outcome of one `MaintenanceRunner.run()` — a small, JSON-able record. + + `status` is one of: + - 'ok' : the run claimed the lease, did its work (possibly a no-op), and + released the lease cleanly. + - 'skipped' : the run did NOT claim the lease; `skipped_reason` says why. + Note the ledger row's own status is 'ok'|'aborted'|'running'; a 'skipped' RunReport + writes no ledger row (it never held the lease). + """ + + status: str + run_id: Optional[str] = None + skipped_reason: Optional[str] = None + below_threshold: bool = False + threshold_stats: dict = field(default_factory=dict) + transform_report: Optional[TransformReport] = None + cursor_id: Optional[str] = None + config_hash: Optional[str] = None + took_over_run_id: Optional[str] = None # id of a stale run this run aborted, if any + + +def _heartbeat_is_fresh(live_run: dict, now_s: float, stale_after_s: float) -> bool: + """True iff the live run's heartbeat is within `stale_after_s` of `now_s`. + + A run with NO heartbeat_at (shouldn't happen — create_run stamps started_at into + it — but be defensive) is treated as stale so a wedged run can always be reclaimed. + """ + hb = live_run.get("heartbeat_at") + if hb is None: + return False + age_s = now_s - (hb / 1000.0) + return age_s < stale_after_s + + +def _proposal_fact_ids(proposal: dict) -> list[str]: + """Parse the fact ids a pending proposal references, kind-aware (§4.4/§5 payloads). + + Payload shapes differ by kind: + - dedup_near : payload['fact_ids'] -> [a, b] + - summarize : payload['source_fact_ids']-> [...] + - evict : payload['fact_id'] -> single id + A malformed/absent payload yields [] (defensive — never crash the exclusion scan). + """ + try: + payload = json.loads(proposal["payload_json"]) + except (KeyError, TypeError, ValueError): + return [] + kind = proposal.get("kind") + if kind == "dedup_near": + return list(payload.get("fact_ids", [])) + if kind == "summarize": + return list(payload.get("source_fact_ids", [])) + if kind == "evict": + fid = payload.get("fact_id") + return [fid] if fid else [] + # Unknown kind: union whatever id-ish keys are present, defensively. + ids: list[str] = [] + ids.extend(payload.get("fact_ids", [])) + ids.extend(payload.get("source_fact_ids", [])) + if payload.get("fact_id"): + ids.append(payload["fact_id"]) + return ids + + +class MaintenanceRunner: + """Drives one maintenance run on ONE namespace store (design spec §6.6, §7). + + Stateful only in `_max_observed_batch_s` — the longest single-batch duration this + process has seen, feeding the staleness threshold (§7.2). Nothing exotic is + persisted; a fresh process starts at 0 and uses the 5-minute floor. + """ + + def __init__( + self, + store: Store, + namespace: str, + config: Optional[MaintenanceConfig] = None, + *, + trigger: str = "cli", + summarizer: Optional[Summarizer] = None, + clock: Optional[Callable[[], float]] = None, + now_ms: Optional[Callable[[], int]] = None, + ) -> None: + self.store = store + self.namespace = namespace + self.config = config or MaintenanceConfig() + self.trigger = trigger + self.summarizer = summarizer + # `clock` is wall-clock seconds (for heartbeat freshness/staleness); `now_ms` + # is world/wall time in epoch ms (for `valid_at` deltas + ledger timestamps). + # Both injectable so tests never sleep on real time. + self._clock = clock or time.time + self._now_ms = now_ms or (lambda: int(self._clock() * 1000)) + # Longest single-batch duration observed this process (seconds); feeds the + # staleness threshold. Persisted nowhere — a fresh process starts at 0. + self._max_observed_batch_s = 0.0 + + # ── staleness threshold (§7.2) ── + def _stale_after_s(self) -> float: + """max(5 min, 10x longest observed single-batch duration this process).""" + return max(_STALE_FLOOR_S, 10.0 * self._max_observed_batch_s) + + # ── cursor high-water (§6.6, advance-before-write) ── + def _max_fact_id(self) -> Optional[str]: + """Max fact id among currently-latest facts — the pre-write cursor high-water. + + Snapshotted at run START, before any transform writes. Ids are time-sortable + (types.new_id), so `iter_latest_facts` yields in id order and the last is the + max. The run's own later outputs (summaries, maintenance episodes) get HIGHER + ids and thus fall after this cursor — excluded from the next run's delta by id + order, as §6.6 requires. None on an empty namespace. + """ + last: Optional[str] = None + for f in self.store.iter_latest_facts(): + last = f.id + return last + + # ── work thresholds (§6.6) ── + def _threshold_check(self, previous_cursor: Optional[str], last_finished_at: Optional[int]): + """Decide whether this run is over threshold (§6.6), returning (over, stats). + + Over threshold iff ANY of (OR): + - facts since `previous_cursor` >= config.min_new_facts + - cumulative salience of those new facts >= config.min_new_salience + - >= config.max_days_between_runs since the last finished run + The first-ever run (previous_cursor is None) is unconditionally over threshold + — there is no cursor yet and nothing to gate on. + """ + if previous_cursor is None: + return True, {"reason": "first_run"} + + new_facts = 0 + new_salience = 0.0 + for f in self.store.iter_latest_facts(after_id=previous_cursor): + if f.record_kind == "summary": + continue # the job's own outputs never count as new work (§6.6) + new_facts += 1 + new_salience += f.salience + + now_ms = self._now_ms() + days_since = ( + (now_ms - last_finished_at) / MS_PER_DAY if last_finished_at is not None else None + ) + + by_facts = new_facts >= self.config.min_new_facts + by_salience = new_salience >= self.config.min_new_salience + by_age = days_since is not None and days_since >= self.config.max_days_between_runs + + stats = { + "new_facts": new_facts, + "new_salience": round(new_salience, 6), + "days_since_last_run": round(days_since, 6) if days_since is not None else None, + "triggered_by_facts": by_facts, + "triggered_by_salience": by_salience, + "triggered_by_age": by_age, + } + return (by_facts or by_salience or by_age), stats + + # ── prior finished run (previous cursor + last-run age) ── + def _last_finished_run(self) -> Optional[dict]: + """The most recent status='ok' run row for the namespace, or None. + + `list_proposals` is proposals; for runs we read the ledger directly through the + store's own query surface. There is no `list_runs` verb (Task 2 exposed only + the create/heartbeat/finish/get-live half), so the runner reads the ledger via + the concrete store — documented coupling, isolated to this one accessor. + """ + # Prefer a public accessor if a future store grows one; today we read the row + # via the concrete SqliteStore's connection (the only Store impl). + db = getattr(self.store, "_db", None) + if db is None: # pragma: no cover - non-sqlite store + return None + row = db.execute( + "SELECT * FROM maintenance_run WHERE namespace=? AND status='ok' " + "ORDER BY finished_at DESC, id DESC LIMIT 1", + (self.namespace,), + ).fetchone() + return dict(row) if row else None + + # ── cross-run exclusion (§4.4) ── + def _prior_pending(self): + """(exclude_ids, pending_signatures) from prior-run still-pending proposals. + + exclude_ids: union of every fact id referenced by a pending proposal — the + auto phase must not touch a fact a human is still reviewing. + pending_signatures: {kind: {frozenset(fact_ids)}} so a propose transform can + skip re-staging an identical-evidence proposal (crash idempotence, §7.4). + """ + exclude_ids: set[str] = set() + signatures: dict[str, set[frozenset[str]]] = {} + for p in self.store.list_proposals(self.namespace, status="pending", limit=1_000_000): + fids = _proposal_fact_ids(p) + exclude_ids.update(fids) + kind = p.get("kind") + if kind: + signatures.setdefault(kind, set()).add(frozenset(fids)) + return exclude_ids, signatures + + # ── lease (§7.2) ── + def _try_claim_lease(self, now_ms: int) -> tuple[Optional[str], Optional[str], Optional[str]]: + """Attempt the atomic lease claim. Returns (run_id, skipped_reason, took_over). + + - Fresh live lease held -> (None, 'lease held', None): clean skip. + - Stale live lease -> abort it, then claim: (run_id, None, aborted_id). + - No live lease -> claim: (run_id, None, None). + - Lost race on INSERT -> (None, 'lost race', None): a concurrent claimer won. + + The `create_run` INSERT is the atomic claim (ux_run_live); a second live INSERT + raises IntegrityError — caught here as the lost-race path (§7.2). + """ + import sqlite3 + + took_over: Optional[str] = None + live = self.store.get_live_run(self.namespace) + if live is not None: + now_s = now_ms / 1000.0 + if _heartbeat_is_fresh(live, now_s, self._stale_after_s()): + return None, "lease held", None + # Stale: mark it 'aborted' (takeover NEVER rolls anything back — §7.2/§7.4) + # then fall through to claim. finish_run on a live row clears status= + # 'running', releasing ux_run_live so our claim can succeed. + self.store.finish_run( + live["id"], "aborted", now_ms, None, live.get("cursor_id") + ) + took_over = live["id"] + + try: + run_id = self.store.create_run( + self.namespace, self.trigger, now_ms, self.config.config_hash() + ) + except sqlite3.IntegrityError: + # A concurrent claimer inserted a live row between our check and INSERT. + return None, "lost race", None + return run_id, None, took_over + + # ── the run ── + def run(self, *, on_batch: Optional[Callable[[], None]] = None) -> RunReport: + """Execute one maintenance run end to end (lease → thresholds → transforms). + + `on_batch` is a test hook invoked at every batch boundary (after the runner's + own heartbeat) — used by the heartbeat/crash tests to observe cadence and to + simulate a mid-run crash. Production passes None. + """ + cfg_hash = self.config.config_hash() + claim_ms = self._now_ms() + run_id, skipped_reason, took_over = self._try_claim_lease(claim_ms) + if run_id is None: + return RunReport(status="skipped", skipped_reason=skipped_reason, config_hash=cfg_hash) + + # We hold the lease. Everything below runs under it; we release it in finally. + # A heartbeat fires at each PHASE boundary (here) and at every BATCH boundary + # (via the batch hook installed below) — the §7.2 "every batch commit and at + # least every 30 s" cadence. The propose phase is bounded by + # proposal_budget_per_run, so it never runs long without a phase-boundary beat. + def _beat() -> None: + self.store.heartbeat_run(run_id, self._now_ms()) + if on_batch is not None: + on_batch() + + cursor_id: Optional[str] = None + stats: dict = {} + report: Optional[TransformReport] = None + below = False + try: + prior = self._last_finished_run() + previous_cursor = prior.get("cursor_id") if prior else None + last_finished_at = prior.get("finished_at") if prior else None + + over, threshold_stats = self._threshold_check(previous_cursor, last_finished_at) + stats = dict(threshold_stats) + + # Advance-before-write: snapshot the cursor high-water NOW, before any + # transform output is written (§6.6). On a no-op we still record it so the + # next run's delta is measured from here. + cursor_id = self._max_fact_id() + + if not over: + below = True + stats["below_threshold"] = True + # Cheap no-op: no transforms, no proposals. Release with 'ok'. + return self._finish_ok(run_id, stats, cursor_id, cfg_hash, took_over, + below=True, report=None) + + # Over threshold: run the transform phase under a heartbeat cadence. + _beat() # heartbeat at the phase boundary + + # Slot-level transforms scan every slot that gained a member since the + # PREVIOUS cursor (iter_slots_touched_since) — the cursor-gap fix (§6.6). + if previous_cursor is None: + # First-ever run: every slot is "new". Drive off all latest facts. + slots = sorted({(f.subject_id, f.predicate) for f in self.store.iter_latest_facts()}) + else: + slots = list(self.store.iter_slots_touched_since(previous_cursor)) + + exclude_ids, pending_sigs = self._prior_pending() + + # The store batches (dedup_exact) fire on_batch/heartbeat at each boundary. + self._install_batch_hook(_beat) + try: + report = run_transforms( + self.store, + self.config, + self._now_ms(), + run_id=run_id, + slots=slots, + summarizer=self.summarizer, + extra_exclude_ids=exclude_ids, + pending_signatures=pending_sigs, + ) + finally: + self._uninstall_batch_hook() + + _beat() # final heartbeat before finishing + stats["merges"] = len(report.merges) + stats["staged"] = len(report.proposals) + stats["demoted"] = len(report.demoted_ids) + stats["dropped_proposals"] = report.dropped_proposals + return self._finish_ok(run_id, stats, cursor_id, cfg_hash, took_over, + below=False, report=report) + except BaseException: + # A failure DURING our run: mark our own row aborted (consistent DB per + # per-batch commits) and re-raise. The next runner takes over from state. + try: + self.store.finish_run(run_id, "aborted", self._now_ms(), None, cursor_id) + except Exception: + pass + raise + + def _finish_ok( + self, run_id, stats, cursor_id, cfg_hash, took_over, *, below, report + ) -> RunReport: + self.store.finish_run(run_id, "ok", self._now_ms(), json.dumps(stats, sort_keys=True), cursor_id) + return RunReport( + status="ok", + run_id=run_id, + below_threshold=below, + threshold_stats=stats, + transform_report=report, + cursor_id=cursor_id, + config_hash=cfg_hash, + took_over_run_id=took_over, + ) + + # ── batch-boundary heartbeat wiring ── + def _install_batch_hook(self, beat: Callable[[], None]) -> None: + """Wrap the store's batch() so each batch COMMIT fires a heartbeat (§7.2). + + The store exposes `batch()` as a context manager (Task 1). We wrap it so that + on successful exit (a committed batch) we bump the heartbeat and fire on_batch. + This is how heartbeat_at advances across batch boundaries during a long run and + how the crash test observes a batch boundary. Restored in `_uninstall_batch_hook`. + """ + store = self.store + original = store.batch + self._original_batch = original + + import contextlib + + @contextlib.contextmanager + def _batch_with_beat(): + import time as _t + + t0 = _t.perf_counter() + with original(): + yield + # Committed successfully — record the batch duration and heartbeat. + dt = _t.perf_counter() - t0 + if dt > self._max_observed_batch_s: + self._max_observed_batch_s = dt + beat() + + store.batch = _batch_with_beat # type: ignore[method-assign] + + def _uninstall_batch_hook(self) -> None: + original = getattr(self, "_original_batch", None) + if original is not None: + self.store.batch = original # type: ignore[method-assign] + self._original_batch = None diff --git a/src/lean_memory/maintain/transforms.py b/src/lean_memory/maintain/transforms.py index 1733b5d..ee708fa 100644 --- a/src/lean_memory/maintain/transforms.py +++ b/src/lean_memory/maintain/transforms.py @@ -189,6 +189,7 @@ def dedup_near( *, run_id: str, budget: int = 1_000_000, + skip_signatures: Optional[set[frozenset[str]]] = None, ) -> tuple[list[StagedProposal], int]: """Stage near-duplicate merge PROPOSALS — zero spine writes (§4.2). @@ -199,7 +200,12 @@ def dedup_near( both fact ids/texts, cosine, slot, the multivalued flag, and the proposed survivor (argmin valid_at). evidence_backend='stored' (embeddings read back, not re-embedded). Respects `budget`; returns (staged, dropped_count). + + `skip_signatures`: `{frozenset(fact_ids)}` of dedup_near proposals ALREADY pending + (from a prior run) — a pair matching one is not re-staged, so a crash-resumed run + converges without double-staging identical evidence (§7.4 idempotence). """ + skip_signatures = skip_signatures or set() staged: list[StagedProposal] = [] dropped = 0 tau = config.tau_near @@ -229,6 +235,8 @@ def dedup_near( cos = _cosine(embs[a.id], embs[b.id]) if cos < tau: continue + if frozenset((a.id, b.id)) in skip_signatures: + continue # an identical pair is already pending — don't double-stage # Proposed survivor = argmin(valid_at) (a already sorts first). survivor = min((a, b), key=lambda f: (f.valid_at, f.id)) payload = { @@ -268,6 +276,7 @@ def summarize( *, run_id: str, budget: int = 1_000_000, + skip_signatures: Optional[set[frozenset[str]]] = None, ) -> tuple[list[StagedProposal], int]: """Stage SUMMARIZE proposals — zero spine writes, no embedding at stage time (§4.3). @@ -278,7 +287,12 @@ def summarize( carries the source fact ids + texts, the proposed summary text from `summarizer`, and evidence_backend = the summarizer's backend id. Respects `budget`; returns (staged, dropped_count). + + `skip_signatures`: `{frozenset(source_fact_ids)}` of SUMMARIZE proposals ALREADY + pending (from a prior run) — a subject whose source set exactly matches one is not + re-staged, so a crash-resumed run does not double-stage the same summary (§7.4). """ + skip_signatures = skip_signatures or set() staged: list[StagedProposal] = [] dropped = 0 age_floor_ms = config.age_floor_days * MS_PER_DAY @@ -311,6 +325,8 @@ def subject_heat(slots_map: dict[Slot, list[Fact]]) -> float: (f for fs in qualifying.values() for f in fs), key=lambda f: (f.valid_at, f.id), ) + if frozenset(f.id for f in sources) in skip_signatures: + continue # this exact source set is already pending — don't double-stage # Summarizer text computed BEFORE any batch window (§7.1). Stage-time only — # no embedding here (embedding is the Task-6 apply path, §4.3). summary_text = summarizer.summarize(sources) @@ -464,6 +480,8 @@ def run_transforms( run_id: str, slots: Iterable[Slot], summarizer: Optional[Summarizer] = None, + extra_exclude_ids: Optional[set[str]] = None, + pending_signatures: Optional[dict[str, set[frozenset[str]]]] = None, ) -> TransformReport: """Run all four transforms in the spec-mandated intra-run order (§4.4). @@ -476,12 +494,24 @@ def run_transforms( `slots` is materialized once (the pre-transform snapshot of touched slots) and reused for both the near-dup proposals and the exact-dup autos. + + Cross-run coupling (the runner supplies these; §4.4 'facts referenced by any + staged proposal' + §7.4 crash idempotence): + - `extra_exclude_ids`: fact ids referenced by PRIOR-run pending proposals — + unioned into the Phase-2 auto exclusion so an auto transform never touches a + fact a human is still reviewing. + - `pending_signatures`: `{kind: {frozenset(fact_ids)}}` of prior-run pending + proposals — a propose transform skips re-staging an identical-evidence + proposal, so a crash-resumed run converges without duplicate proposals. """ from .summarize import default_summarizer if summarizer is None: summarizer = default_summarizer() + extra_exclude_ids = extra_exclude_ids or set() + pending_signatures = pending_signatures or {} + slots = list(slots) # materialize once — reused across phases report = TransformReport() @@ -489,26 +519,35 @@ def run_transforms( budget = config.proposal_budget_per_run remaining = budget - near, dropped = dedup_near(store, config, now, slots, run_id=run_id, budget=remaining) + near, dropped = dedup_near( + store, config, now, slots, run_id=run_id, budget=remaining, + skip_signatures=pending_signatures.get("dedup_near"), + ) report.proposals.extend(near) report.dropped_proposals += dropped remaining = budget - len(report.proposals) summ, dropped = summarize( - store, config, now, summarizer, run_id=run_id, budget=max(0, remaining) + store, config, now, summarizer, run_id=run_id, budget=max(0, remaining), + skip_signatures=pending_signatures.get("summarize"), ) report.proposals.extend(summ) report.dropped_proposals += dropped remaining = budget - len(report.proposals) + # A fact referenced by a prior-run pending proposal is never re-proposed for + # eviction (excluding its id covers both re-propose and auto-demote — §4.4). ev_prop, dropped = evict_propose( - store, config, now, run_id=run_id, budget=max(0, remaining) + store, config, now, run_id=run_id, budget=max(0, remaining), + exclude_ids=extra_exclude_ids, ) report.proposals.extend(ev_prop) report.dropped_proposals += dropped # ── Phase 2: AUTO transforms, EXCLUDING every staged-proposal target ── - staged_ids = report.staged_fact_ids + # This run's staged ids UNION prior runs' still-pending referenced ids: neither a + # fact a human is reviewing now nor one staged this run is auto-mutated. + staged_ids = report.staged_fact_ids | extra_exclude_ids report.merges = dedup_exact(store, config, now, slots, exclude_ids=staged_ids) report.demoted_ids = evict_auto(store, config, now, exclude_ids=staged_ids) diff --git a/tests/test_maintenance_runner.py b/tests/test_maintenance_runner.py new file mode 100644 index 0000000..4136013 --- /dev/null +++ b/tests/test_maintenance_runner.py @@ -0,0 +1,572 @@ +"""Task 5 — MaintenanceRunner: lease, work thresholds, cursor, crash-resume. + +All offline (FakeEmbedder, deterministic stub summarizer) against real SqliteStores. +Pins design-spec §6.6 (thresholds + cursor), §7.2 (lease), §7.4 (crash safety): + + - Lease: fresh-lease clean skip; stale-heartbeat takeover marks 'aborted' + proceeds; + a real two-process race (subprocess claims first → in-process runner skips); + lost-race IntegrityError path. + - Thresholds: below-threshold run is a cheap no-op (no transforms, lease released + 'ok', below_threshold stat); >=7-days-since-last-run triggers with 0 new facts; + first-ever run triggers. + - Cursor: run 1 finishes with cursor C1; a duplicate lands on a long-quiet slot; + run 2 sees it via iter_slots_touched_since(C1) and dedups it (the cursor-gap case). + - Cross-run exclusion: a fact referenced by a PRIOR run's pending proposal is not + auto-demoted/deduped this run; identical-pending proposals are not double-staged. + - Crash/resume: abandon a 'running' row with a stale heartbeat + partial work; the + next runner takes over (aborts it), re-runs, and converges (no duplicate proposals, + no double summary). + - Heartbeat: heartbeat_at advances across batch boundaries during a run (observed + via the on_batch hook + get_live_run mid-run). + +Time is injected everywhere (clock / now_ms) so no test sleeps on real wall-clock. +""" + +from __future__ import annotations + +import json +import subprocess +import sys + +import pytest + +from lean_memory.embed.fake import FakeEmbedder +from lean_memory.maintain import MaintenanceConfig, MaintenanceRunner +from lean_memory.maintain.config import MS_PER_DAY +from lean_memory.store.sqlite_store import SqliteStore +from lean_memory.types import Entity, Episode, Fact + +NOW = 2_000_000_000_000 # fixed wall-clock (epoch ms) +NOW_S = NOW / 1000.0 + +# A tiny config so tests trip thresholds with a handful of facts (defaults are 200). +def _cfg(**over): + base = dict( + min_new_facts=3, + min_new_salience=1000.0, # high, so the facts-count threshold is the live one + max_days_between_runs=7, + age_floor_days=90, + min_cluster=5, + proposal_budget_per_run=50, + ) + base.update(over) + return MaintenanceConfig(**base) + + +# ── clock injection helpers ─────────────────────────────────────────────────── +class Clock: + """A hand-cranked wall-clock (seconds). now_ms derives from it.""" + + def __init__(self, t_s: float = NOW_S) -> None: + self.t = t_s + + def __call__(self) -> float: + return self.t + + def advance(self, seconds: float) -> None: + self.t += seconds + + def now_ms(self) -> int: + return int(self.t * 1000) + + +# ── fixtures / helpers ──────────────────────────────────────────────────────── +def _make_store(path): + emb = FakeEmbedder() + s = SqliteStore(path, dim=emb.dim, coarse_dim=emb.coarse_dim) + ep = Episode(namespace="ns", raw="seed", t_ref=1_000) + s.add_episode(ep) + subj = s.upsert_entity(Entity(namespace="ns", name="user", type="person")) + s._emb = emb + s._ep = ep + s._subj = subj + return s + + +@pytest.fixture +def store(tmp_path): + s = _make_store(tmp_path / "ns.db") + yield s + s.close() + + +def _add( + store, + text, + *, + predicate="works_at", + valid_at, + salience=5.0, + access_count=0, + last_access=None, + subject_id=None, + embed_text=None, +): + f = Fact( + namespace="ns", + subject_id=subject_id or store._subj.id, + predicate=predicate, + fact_text=text, + valid_at=valid_at, + episode_id=store._ep.id, + salience=salience, + access_count=access_count, + last_access=last_access, + ) + full, coarse = store._emb.embed_with_coarse(embed_text if embed_text is not None else text) + store.add_fact(f, full, coarse) + return f + + +def _runner(store, clock=None, **kw): + clock = clock or Clock() + return MaintenanceRunner( + store, "ns", _cfg(**kw.pop("cfg", {})), + clock=clock, now_ms=clock.now_ms, **kw + ) + + +# ══════════════════════════════════════════════════════════════════════════════ +# Lease (§7.2) +# ══════════════════════════════════════════════════════════════════════════════ +def test_second_runner_skips_while_fresh_lease_held(tmp_path): + """A second store handle on the SAME file cleanly skips while the first holds a + fresh lease (§7.2).""" + s1 = _make_store(tmp_path / "ns.db") + clock = Clock() + # First runner claims and heartbeats — simulate a held lease by claiming directly. + run_id = s1.create_run("ns", "cli", clock.now_ms(), _cfg().config_hash()) + s1.heartbeat_run(run_id, clock.now_ms()) + + # Second store handle on the same path (busy_timeout=5000 for maintenance). + s2 = SqliteStore(tmp_path / "ns.db", dim=s1._emb.dim, coarse_dim=s1._emb.coarse_dim, + busy_timeout_ms=5000) + r2 = MaintenanceRunner(s2, "ns", _cfg(), clock=clock, now_ms=clock.now_ms) + rep = r2.run() + assert rep.status == "skipped" + assert rep.skipped_reason == "lease held" + # The first lease is untouched (still running). + assert s1.get_live_run("ns")["id"] == run_id + s1.close() + s2.close() + + +def test_stale_heartbeat_takeover_marks_aborted_and_proceeds(store): + """A live run with a STALE heartbeat is marked 'aborted' and taken over (§7.2).""" + clock = Clock() + # Plant a live run with an OLD heartbeat (older than the 5-min stale floor). + stale_id = store.create_run("ns", "cli", clock.now_ms(), _cfg().config_hash()) + store.heartbeat_run(stale_id, clock.now_ms()) + clock.advance(10 * 60) # 10 minutes — past the 300s floor + + _add(store, "a", valid_at=NOW - 400 * MS_PER_DAY) + _add(store, "b", valid_at=NOW - 400 * MS_PER_DAY) + _add(store, "c", valid_at=NOW - 400 * MS_PER_DAY) + + r = MaintenanceRunner(store, "ns", _cfg(), clock=clock, now_ms=clock.now_ms) + rep = r.run() + assert rep.status == "ok" + assert rep.took_over_run_id == stale_id + # The stale run is now 'aborted', not 'running'. + aborted = store._db.execute( + "SELECT status FROM maintenance_run WHERE id=?", (stale_id,) + ).fetchone()["status"] + assert aborted == "aborted" + # Our own run finished 'ok'. + assert store._db.execute( + "SELECT status FROM maintenance_run WHERE id=?", (rep.run_id,) + ).fetchone()["status"] == "ok" + + +def test_stale_takeover_never_rolls_back_prior_work(store): + """Takeover marks the stale run aborted but never rolls back its committed work + (§7.2/§7.4) — a proposal staged by the crashed run survives.""" + clock = Clock() + stale_id = store.create_run("ns", "cli", clock.now_ms(), _cfg().config_hash()) + store.heartbeat_run(stale_id, clock.now_ms()) + # The crashed run had staged a proposal before dying. + pid = store.stage_proposal( + run_id=stale_id, namespace="ns", kind="evict", + payload_json=json.dumps({"fact_id": "zzz", "evidence_backend": "score"}), + created_at=clock.now_ms(), expires_at=clock.now_ms() + MS_PER_DAY, + ) + clock.advance(10 * 60) + + r = MaintenanceRunner(store, "ns", _cfg(), clock=clock, now_ms=clock.now_ms) + r.run() + # The prior proposal is untouched — takeover rolled nothing back. + assert store.get_proposal(pid)["status"] == "pending" + + +def test_two_process_lease_race_subprocess_claims_first(tmp_path): + """A real subprocess claims the lease on a shared temp DB first; the in-process + runner then cleanly skips (§7.2). Deterministic: the child claims before we try.""" + db_path = tmp_path / "ns.db" + # Build the DB (schema + seed) in-process so the child only claims the lease. + s = _make_store(db_path) + s.close() + + # Child claims a live lease and heartbeats, WITHOUT finishing — it leaves the row + # 'running' with a fresh heartbeat, then exits. The open row is the held lease. + child = ( + "import sys;" + "from lean_memory.store.sqlite_store import SqliteStore;" + f"s=SqliteStore({str(db_path)!r}, busy_timeout_ms=5000);" + "rid=s.create_run('ns','cli',2000000000000, None);" + "s.heartbeat_run(rid, 2000000000000);" + "s.close();" + "print(rid)" + ) + proc = subprocess.run( + [sys.executable, "-c", child], capture_output=True, text=True, timeout=60 + ) + assert proc.returncode == 0, proc.stderr + child_run_id = proc.stdout.strip() + assert child_run_id + + # In-process runner: same wall-clock as the child's heartbeat → lease is fresh. + s2 = _make_store(db_path) + clock = Clock(t_s=NOW_S) # equals the child's heartbeat time → fresh + r = MaintenanceRunner(s2, "ns", _cfg(), clock=clock, now_ms=clock.now_ms) + rep = r.run() + assert rep.status == "skipped" + assert rep.skipped_reason == "lease held" + assert s2.get_live_run("ns")["id"] == child_run_id + s2.close() + + +def test_lost_race_integrity_error_path(store, monkeypatch): + """If a concurrent claimer inserts a live row between our check and INSERT, the + create_run IntegrityError surfaces as 'lost race' (§7.2).""" + import sqlite3 + + clock = Clock() + r = MaintenanceRunner(store, "ns", _cfg(), clock=clock, now_ms=clock.now_ms) + + # get_live_run says 'nobody' (None), but create_run raises IntegrityError — exactly + # the race window where another process claimed between check and INSERT. + monkeypatch.setattr(store, "get_live_run", lambda ns: None) + def _boom(*a, **k): + raise sqlite3.IntegrityError("UNIQUE constraint failed: ux_run_live") + monkeypatch.setattr(store, "create_run", _boom) + + rep = r.run() + assert rep.status == "skipped" + assert rep.skipped_reason == "lost race" + + +# ══════════════════════════════════════════════════════════════════════════════ +# Work thresholds (§6.6) +# ══════════════════════════════════════════════════════════════════════════════ +def _finish_a_prior_run(store, clock, cursor_id, at_ms): + """Record a finished 'ok' run with a given cursor + finished_at (test scaffold).""" + rid = store.create_run("ns", "cli", at_ms, _cfg().config_hash()) + store.finish_run(rid, "ok", at_ms, json.dumps({}), cursor_id) + return rid + + +def test_below_threshold_is_clean_noop(store): + """Below all thresholds: no transforms, no proposals, lease released 'ok' with a + below_threshold stat (§6.6/§10.9).""" + clock = Clock() + # A prior run finished recently with a cursor at the current high-water. + _add(store, "seed", valid_at=NOW - 400 * MS_PER_DAY) + hw = None + for f in store.iter_latest_facts(): + hw = f.id + _finish_a_prior_run(store, clock, hw, clock.now_ms()) + + # No new facts since the cursor; recent last run → below all thresholds. + r = MaintenanceRunner(store, "ns", _cfg(), clock=clock, now_ms=clock.now_ms) + rep = r.run() + assert rep.status == "ok" + assert rep.below_threshold is True + assert rep.threshold_stats.get("below_threshold") is True + assert rep.transform_report is None # no transforms invoked + # Zero new proposals. + assert store.list_proposals("ns", status="pending") == [] + # Lease released. + assert store.get_live_run("ns") is None + + +def test_age_threshold_triggers_with_zero_new_facts(store): + """>= max_days_between_runs since the last finished run triggers even with 0 new + facts (§6.6).""" + clock = Clock() + _add(store, "seed", valid_at=NOW - 400 * MS_PER_DAY) + hw = list(store.iter_latest_facts())[-1].id + # Prior run finished 8 days ago (> max_days_between_runs=7). + _finish_a_prior_run(store, clock, hw, clock.now_ms() - 8 * MS_PER_DAY) + + r = MaintenanceRunner(store, "ns", _cfg(), clock=clock, now_ms=clock.now_ms) + rep = r.run() + assert rep.status == "ok" + assert rep.below_threshold is False + assert rep.threshold_stats["triggered_by_age"] is True + assert rep.threshold_stats["new_facts"] == 0 + + +def test_first_ever_run_triggers(store): + """The first-ever run (no prior finished run) is always over threshold (§6.6).""" + clock = Clock() + r = MaintenanceRunner(store, "ns", _cfg(), clock=clock, now_ms=clock.now_ms) + rep = r.run() + assert rep.status == "ok" + assert rep.below_threshold is False + assert rep.threshold_stats.get("reason") == "first_run" + + +def test_facts_count_threshold_triggers(store): + """facts since cursor >= min_new_facts triggers (§6.6).""" + clock = Clock() + _add(store, "seed", valid_at=NOW - 400 * MS_PER_DAY) + hw = list(store.iter_latest_facts())[-1].id + _finish_a_prior_run(store, clock, hw, clock.now_ms()) # recent, so age won't trip + + # min_new_facts=3 → add 3 new facts on distinct slots. + _add(store, "n1", predicate="p1", valid_at=NOW) + _add(store, "n2", predicate="p2", valid_at=NOW) + _add(store, "n3", predicate="p3", valid_at=NOW) + + r = MaintenanceRunner(store, "ns", _cfg(min_new_salience=1e9), clock=clock, now_ms=clock.now_ms) + rep = r.run() + assert rep.below_threshold is False + assert rep.threshold_stats["triggered_by_facts"] is True + assert rep.threshold_stats["new_facts"] == 3 + + +# ══════════════════════════════════════════════════════════════════════════════ +# Cursor semantics (§6.6) — the verified cursor-gap scenario +# ══════════════════════════════════════════════════════════════════════════════ +def test_cursor_gap_duplicate_on_quiet_slot_deduped_next_run(store): + """Run 1 finishes with cursor C1; a duplicate lands on a long-quiet slot; run 2 + sees that slot via iter_slots_touched_since(C1) and dedups it (§6.6).""" + clock = Clock() + # A quiet slot with one old fact, seeded before run 1. + orig = _add(store, "Acme Corp", predicate="works_at", valid_at=NOW - 400 * MS_PER_DAY) + + # Run 1 (first-ever): processes the corpus, records cursor C1 = current high-water. + r1 = MaintenanceRunner(store, "ns", _cfg(), clock=clock, now_ms=clock.now_ms) + rep1 = r1.run() + assert rep1.status == "ok" + c1 = rep1.cursor_id + assert c1 is not None + + # A duplicate lands on that SAME long-quiet slot AFTER run 1 (id > C1). + clock.advance(8 * 24 * 3600) # 8 days later so the age threshold also trips + dup = _add(store, "Acme Corp", predicate="works_at", valid_at=clock.now_ms()) + assert dup.id > c1 # newer id than the run-1 high-water + + # Run 2: the duplicate's slot must be re-scanned via iter_slots_touched_since(C1) + # and the exact duplicate retired. Before: 2 latest in the slot; after: 1. + before = [f for f in store.find_latest_in_slot(orig.subject_id, "works_at")] + assert len(before) == 2 + + r2 = MaintenanceRunner(store, "ns", _cfg(), clock=clock, now_ms=clock.now_ms) + rep2 = r2.run() + assert rep2.status == "ok" + after = [f for f in store.find_latest_in_slot(orig.subject_id, "works_at")] + assert len(after) == 1, "the cursor-gap duplicate was deduped" + assert after[0].id == orig.id # survivor = argmin(valid_at) = the original + + +def test_cursor_advances_before_transform_output(store): + """The stored cursor is snapshotted BEFORE transform outputs are written: it equals + the pre-run max fact id, never a summary/maintenance-episode id created mid-run.""" + clock = Clock() + pre_ids = [f.id for f in store.iter_latest_facts()] + _add(store, "x", valid_at=NOW - 400 * MS_PER_DAY) + hw = list(store.iter_latest_facts())[-1].id + + r = MaintenanceRunner(store, "ns", _cfg(), clock=clock, now_ms=clock.now_ms) + rep = r.run() + # Cursor is the pre-write high-water — the id of the last fact that existed at start. + assert rep.cursor_id == hw + + +# ══════════════════════════════════════════════════════════════════════════════ +# Cross-run exclusion (§4.4) + identical-pending dedupe guard (§7.4) +# ══════════════════════════════════════════════════════════════════════════════ +def test_fact_in_prior_pending_proposal_not_auto_demoted(store): + """A fact referenced by a PRIOR run's pending EVICT proposal is not auto-demoted + this run (§4.4 cross-run exclusion).""" + clock = Clock() + # An auto-band-eligible fact: salience<2, access_count=0, very old. + victim = _add( + store, "low value old fact", predicate="note", + valid_at=NOW - 400 * MS_PER_DAY, salience=0.0, access_count=0, + ) + # A prior run staged a pending EVICT proposal that references this exact fact. + prior_run = store.create_run("ns", "cli", clock.now_ms() - MS_PER_DAY, _cfg().config_hash()) + store.finish_run(prior_run, "ok", clock.now_ms() - MS_PER_DAY, json.dumps({}), None) + store.stage_proposal( + run_id=prior_run, namespace="ns", kind="evict", + payload_json=json.dumps({"fact_id": victim.id, "evidence_backend": "score"}), + created_at=clock.now_ms() - MS_PER_DAY, expires_at=clock.now_ms() + 10 * MS_PER_DAY, + ) + + r = MaintenanceRunner(store, "ns", _cfg(), clock=clock, now_ms=clock.now_ms) + rep = r.run() + assert rep.status == "ok" + # The victim was NOT auto-demoted — its tier is still hot. + tier = store._db.execute("SELECT tier FROM fact WHERE id=?", (victim.id,)).fetchone()["tier"] + assert tier == "hot", "a fact under review is never auto-demoted" + assert victim.id not in (rep.transform_report.demoted_ids if rep.transform_report else []) + + +def test_identical_pending_dedup_near_not_double_staged(store): + """A dedup_near proposal identical to one already pending (same fact-id pair) is not + re-staged on the next run — even when the slot is RE-SCANNED (§7.4 idempotence). + + The guard is load-bearing precisely when run 2 revisits the pair: we land a third + member on the same slot after run 1's cursor so iter_slots_touched_since re-surfaces + it, forcing dedup_near to re-evaluate the a/b pair. The skip-signature is then the + only thing preventing a second identical proposal. + """ + clock = Clock() + # Two near-duplicate facts in one slot (same forced embedding, different text → + # cosine 1.0 >= tau, but not textually identical → DEDUP-NEAR, a proposal). + a = _add(store, "salary is 100k", predicate="salary", valid_at=NOW - 400 * MS_PER_DAY, + embed_text="SALARY") + b = _add(store, "earns 100000", predicate="salary", valid_at=NOW - 399 * MS_PER_DAY, + embed_text="SALARY") + + # Run 1 stages the near-dup proposal for the a/b pair. + r1 = MaintenanceRunner(store, "ns", _cfg(), clock=clock, now_ms=clock.now_ms) + rep1 = r1.run() + pending1 = store.list_proposals("ns", status="pending", kind="dedup_near") + assert len(pending1) == 1 + payload = json.loads(pending1[0]["payload_json"]) + assert set(payload["fact_ids"]) == {a.id, b.id} + + # A THIRD, textually-distinct member lands on the SAME slot after C1 — this both + # trips the age/facts threshold AND re-touches the slot so run 2 re-scans it. + clock.advance(8 * 24 * 3600) + c = _add(store, "compensation 100k", predicate="salary", valid_at=clock.now_ms(), + embed_text="SALARY") + assert c.id > rep1.cursor_id + + r2 = MaintenanceRunner(store, "ns", _cfg(), clock=clock, now_ms=clock.now_ms) + r2.run() + pending2 = store.list_proposals("ns", status="pending", kind="dedup_near") + # The a/b pair is NOT re-staged (skip-signature); only NEW pairs involving c are. + ab = frozenset((a.id, b.id)) + ab_count = sum( + 1 for p in pending2 if frozenset(json.loads(p["payload_json"])["fact_ids"]) == ab + ) + assert ab_count == 1, "identical near-dup evidence is not double-staged" + + +# ══════════════════════════════════════════════════════════════════════════════ +# Crash / resume (§7.4) +# ══════════════════════════════════════════════════════════════════════════════ +def test_crash_resume_converges_no_duplicate_proposals(store): + """Simulate a crash: a 'running' row with a stale heartbeat + a partially-staged + proposal, abandoned WITHOUT finish_run. The next runner takes over (aborts it), + re-runs, and converges — no duplicate proposals for the same evidence (§7.4).""" + clock = Clock() + a = _add(store, "salary is 100k", predicate="salary", valid_at=NOW - 400 * MS_PER_DAY, + embed_text="SALARY") + b = _add(store, "earns 100000", predicate="salary", valid_at=NOW - 399 * MS_PER_DAY, + embed_text="SALARY") + + # The crashed run: claimed the lease, staged the near-dup proposal, then died — + # left 'running' with an OLD heartbeat (no finish_run). + crashed = store.create_run("ns", "cli", clock.now_ms(), _cfg().config_hash()) + store.heartbeat_run(crashed, clock.now_ms()) + store.stage_proposal( + run_id=crashed, namespace="ns", kind="dedup_near", + payload_json=json.dumps({ + "slot": {"subject_id": a.subject_id, "predicate": "salary"}, + "fact_ids": [a.id, b.id], + "fact_texts": {a.id: a.fact_text, b.id: b.fact_text}, + "cosine": 1.0, "multivalued": False, + "proposed_survivor": a.id, "evidence_backend": "stored", + }), + created_at=clock.now_ms(), expires_at=clock.now_ms() + 30 * MS_PER_DAY, + ) + clock.advance(10 * 60) # heartbeat now stale (> 300s floor) + + # The recovery runner takes over. + r = MaintenanceRunner(store, "ns", _cfg(), clock=clock, now_ms=clock.now_ms) + rep = r.run() + assert rep.status == "ok" + assert rep.took_over_run_id == crashed + # The crashed run is 'aborted'; its staged proposal survives (never rolled back). + assert store._db.execute( + "SELECT status FROM maintenance_run WHERE id=?", (crashed,) + ).fetchone()["status"] == "aborted" + # Convergence: exactly ONE dedup_near proposal for this pair — no double-stage. + near = store.list_proposals("ns", status="pending", kind="dedup_near") + assert len(near) == 1 + + +def test_crash_mid_run_leaves_consistent_db_and_marks_aborted(store): + """A crash DURING the transform phase (raised at a batch boundary via on_batch) + leaves a consistent DB; the runner marks its own row aborted before re-raising, and + the next runner takes over (§7.4).""" + clock = Clock() + # Exact-dup cluster so dedup_exact runs a batch we can crash inside. + _add(store, "Acme", predicate="works_at", valid_at=NOW - 400 * MS_PER_DAY) + _add(store, "acme", predicate="works_at", valid_at=NOW - 399 * MS_PER_DAY) # exact-dup + + r = MaintenanceRunner(store, "ns", _cfg(), clock=clock, now_ms=clock.now_ms) + + boom_calls = {"n": 0} + + def _crash_on_second_batch(): + boom_calls["n"] += 1 + if boom_calls["n"] >= 2: # let ≥1 batch commit, then crash at a boundary + raise RuntimeError("simulated crash mid-run") + + with pytest.raises(RuntimeError, match="simulated crash"): + r.run(on_batch=_crash_on_second_batch) + + # The runner marked its own row 'aborted' (no lingering 'running' lease it owns). + live = store.get_live_run("ns") + assert live is None, "crashed runner released/aborted its own live row" + + # The DB is consistent — no inverted intervals from the partial run. + bad = store._db.execute( + "SELECT COUNT(*) c FROM fact WHERE valid_to IS NOT NULL AND valid_to < valid_at" + ).fetchone()["c"] + assert bad == 0 + + # A fresh runner takes over cleanly and finishes ok. + clock.advance(1.0) + r2 = MaintenanceRunner(store, "ns", _cfg(), clock=clock, now_ms=clock.now_ms) + rep2 = r2.run() + assert rep2.status == "ok" + + +# ══════════════════════════════════════════════════════════════════════════════ +# Heartbeat cadence (§7.2) +# ══════════════════════════════════════════════════════════════════════════════ +def test_heartbeat_advances_across_batch_boundaries(store): + """heartbeat_at advances across batch boundaries during a run — observed via the + on_batch hook reading get_live_run mid-run (§7.2).""" + clock = Clock() + # A corpus with multiple exact-dup clusters on distinct slots → multiple batches. + for i in range(3): + pred = f"slot{i}" + _add(store, "DupVal", predicate=pred, valid_at=NOW - 400 * MS_PER_DAY) + _add(store, "dupval", predicate=pred, valid_at=NOW - 399 * MS_PER_DAY) # exact-dup + + r = MaintenanceRunner(store, "ns", _cfg(), clock=clock, now_ms=clock.now_ms) + + seen_heartbeats = [] + + def _observe(): + # Each batch boundary advances the clock a hair, then the runner has already + # heartbeated; capture the live row's heartbeat_at as observed mid-run. + clock.advance(0.01) + live = store.get_live_run("ns") + if live is not None: + seen_heartbeats.append(live["heartbeat_at"]) + + rep = r.run(on_batch=_observe) + assert rep.status == "ok" + # We crossed >= 2 batch boundaries and the heartbeat strictly advanced. + assert len(seen_heartbeats) >= 2 + assert seen_heartbeats == sorted(seen_heartbeats) + assert seen_heartbeats[-1] > seen_heartbeats[0] From 6f14d8396e204c3b3a0ed1696ffadea41f64bab2 Mon Sep 17 00:00:00 2001 From: Wuesteon Date: Thu, 16 Jul 2026 20:27:07 +0800 Subject: [PATCH 07/14] fix(maintain): lease claim in one transaction; no-op preserves cursor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three review findings on the maintenance runner: 1. Lease claim is now ONE transaction (spec §7.2): check → optional stale-abort → INSERT claim are wrapped in store.batch() (single BEGIN IMMEDIATE / COMMIT), IntegrityError caught OUTSIDE the batch after __exit__ rolls back. A failed claim now rolls back the stale-abort UPDATE too (prior run stays 'running', not stranded 'aborted') — pinned by a new atomicity test. 2. A below-threshold no-op no longer advances the cursor: it carries the last work-doing run's cursor forward, so a duplicate landing on a quiet slot stays in the next real run's iter_slots_touched_since window and is eventually deduped — pinned by a new regression test. 3. Added a last_finished_run(namespace) read verb to the Store ABC + SqliteStore; the runner uses it instead of reaching into store._db. Suite: 230 passed, 1 skipped (18 runner tests). Co-Authored-By: Claude Fable 5 --- src/lean_memory/maintain/runner.py | 107 ++++++++++++++------------ src/lean_memory/store/base.py | 7 ++ src/lean_memory/store/sqlite_store.py | 11 +++ tests/test_maintenance_runner.py | 80 +++++++++++++++++++ 4 files changed, 157 insertions(+), 48 deletions(-) diff --git a/src/lean_memory/maintain/runner.py b/src/lean_memory/maintain/runner.py index a782840..d2774a7 100644 --- a/src/lean_memory/maintain/runner.py +++ b/src/lean_memory/maintain/runner.py @@ -216,27 +216,6 @@ def _threshold_check(self, previous_cursor: Optional[str], last_finished_at: Opt } return (by_facts or by_salience or by_age), stats - # ── prior finished run (previous cursor + last-run age) ── - def _last_finished_run(self) -> Optional[dict]: - """The most recent status='ok' run row for the namespace, or None. - - `list_proposals` is proposals; for runs we read the ledger directly through the - store's own query surface. There is no `list_runs` verb (Task 2 exposed only - the create/heartbeat/finish/get-live half), so the runner reads the ledger via - the concrete store — documented coupling, isolated to this one accessor. - """ - # Prefer a public accessor if a future store grows one; today we read the row - # via the concrete SqliteStore's connection (the only Store impl). - db = getattr(self.store, "_db", None) - if db is None: # pragma: no cover - non-sqlite store - return None - row = db.execute( - "SELECT * FROM maintenance_run WHERE namespace=? AND status='ok' " - "ORDER BY finished_at DESC, id DESC LIMIT 1", - (self.namespace,), - ).fetchone() - return dict(row) if row else None - # ── cross-run exclusion (§4.4) ── def _prior_pending(self): """(exclude_ids, pending_signatures) from prior-run still-pending proposals. @@ -265,33 +244,59 @@ def _try_claim_lease(self, now_ms: int) -> tuple[Optional[str], Optional[str], O - No live lease -> claim: (run_id, None, None). - Lost race on INSERT -> (None, 'lost race', None): a concurrent claimer won. - The `create_run` INSERT is the atomic claim (ux_run_live); a second live INSERT - raises IntegrityError — caught here as the lost-race path (§7.2). + Spec §7.2 binds the whole sequence as ONE transaction: + `BEGIN IMMEDIATE → check live-heartbeat row → (optional) abort stale → INSERT + (loser hits ux_run_live) → COMMIT`. `store.batch()` supplies the BEGIN IMMEDIATE + and single COMMIT, suppressing the CRUD verbs' per-call commits so + get_live_run/finish_run/create_run all land in the one transaction. The + create_run INSERT is the atomic claim; a second live INSERT raises + IntegrityError — batch's __exit__ rolls the whole transaction back first, THEN + we catch it OUTSIDE the batch context and return the lost-race skip. """ import sqlite3 - took_over: Optional[str] = None + # Fresh-lease skip must NOT open a write transaction — read first, and only if a + # takeover/claim is actually warranted do we enter BEGIN IMMEDIATE. The batch + # below re-reads under the write lock (the authoritative §7.2 check); this + # pre-check just short-circuits the common held-lease case cheaply. live = self.store.get_live_run(self.namespace) - if live is not None: - now_s = now_ms / 1000.0 - if _heartbeat_is_fresh(live, now_s, self._stale_after_s()): - return None, "lease held", None - # Stale: mark it 'aborted' (takeover NEVER rolls anything back — §7.2/§7.4) - # then fall through to claim. finish_run on a live row clears status= - # 'running', releasing ux_run_live so our claim can succeed. - self.store.finish_run( - live["id"], "aborted", now_ms, None, live.get("cursor_id") - ) - took_over = live["id"] - + if live is not None and _heartbeat_is_fresh( + live, now_ms / 1000.0, self._stale_after_s() + ): + return None, "lease held", None + + # One transaction (§7.2): BEGIN IMMEDIATE → check → optional stale-abort → + # INSERT claim → COMMIT. batch() suppresses the CRUD verbs' per-call commits so + # they co-commit; IntegrityError on the INSERT is caught OUTSIDE the batch, after + # __exit__ has rolled the whole transaction back. + result: dict = {} try: - run_id = self.store.create_run( - self.namespace, self.trigger, now_ms, self.config.config_hash() - ) + with self.store.batch(): + live = self.store.get_live_run(self.namespace) + if live is not None: + if _heartbeat_is_fresh( + live, now_ms / 1000.0, self._stale_after_s() + ): + # Became fresh between the pre-check and the lock: abandon the + # claim (the write-free transaction commits empty on return). + return None, "lease held", None + # Stale: mark it 'aborted' in the SAME transaction. Takeover NEVER + # rolls anything back (§7.2/§7.4) — the prior run's own work is + # already durably committed; this only clears status='running' to + # release ux_run_live for the claim below. + self.store.finish_run( + live["id"], "aborted", now_ms, None, live.get("cursor_id") + ) + result["took_over"] = live["id"] + # The atomic claim: a concurrent live row makes this raise IntegrityError. + result["run_id"] = self.store.create_run( + self.namespace, self.trigger, now_ms, self.config.config_hash() + ) except sqlite3.IntegrityError: - # A concurrent claimer inserted a live row between our check and INSERT. + # __exit__ already rolled back the whole transaction (including any + # stale-abort UPDATE), leaving the concurrent claimer's live row intact. return None, "lost race", None - return run_id, None, took_over + return result["run_id"], None, result.get("took_over") # ── the run ── def run(self, *, on_batch: Optional[Callable[[], None]] = None) -> RunReport: @@ -322,25 +327,31 @@ def _beat() -> None: report: Optional[TransformReport] = None below = False try: - prior = self._last_finished_run() + prior = self.store.last_finished_run(self.namespace) previous_cursor = prior.get("cursor_id") if prior else None last_finished_at = prior.get("finished_at") if prior else None over, threshold_stats = self._threshold_check(previous_cursor, last_finished_at) stats = dict(threshold_stats) - # Advance-before-write: snapshot the cursor high-water NOW, before any - # transform output is written (§6.6). On a no-op we still record it so the - # next run's delta is measured from here. - cursor_id = self._max_fact_id() - if not over: below = True stats["below_threshold"] = True - # Cheap no-op: no transforms, no proposals. Release with 'ok'. + # A below-threshold run does NO work, so it must PRESERVE the cursor of + # the last run that did (§6.6). Advancing it would strand an un-processed + # duplicate on a quiet slot: the next real run scans + # iter_slots_touched_since(previous_cursor) and still sees it. Carry the + # previous ok-run's cursor forward unchanged (None on a first no-op). + cursor_id = previous_cursor return self._finish_ok(run_id, stats, cursor_id, cfg_hash, took_over, below=True, report=None) + # Advance-before-write: snapshot the cursor high-water NOW, before any + # transform output is written (§6.6). Only a run that DOES work advances the + # cursor — the run's own later outputs (summaries, maintenance episodes) get + # higher ids and fall after this snapshot, excluded from the next delta. + cursor_id = self._max_fact_id() + # Over threshold: run the transform phase under a heartbeat cadence. _beat() # heartbeat at the phase boundary diff --git a/src/lean_memory/store/base.py b/src/lean_memory/store/base.py index 8eb0e30..5bd6d3f 100644 --- a/src/lean_memory/store/base.py +++ b/src/lean_memory/store/base.py @@ -185,6 +185,13 @@ def get_live_run(self, namespace: str) -> Optional[dict]: """The status='running' run for a namespace, or None. (At most one exists — ux_run_live enforces it.)""" + @abstractmethod + def last_finished_run(self, namespace: str) -> Optional[dict]: + """The most recent status='ok' run row for a namespace as a dict, or None. + Newest first by (finished_at, id). Pure read — the runner's previous-cursor + + last-run-age source (§6.6). Excludes 'aborted'/'running' rows: a run that did no + work (aborted) must not advance the cursor another run reasons from.""" + @abstractmethod def stage_proposal( self, diff --git a/src/lean_memory/store/sqlite_store.py b/src/lean_memory/store/sqlite_store.py index cb61a34..e8e5f9d 100644 --- a/src/lean_memory/store/sqlite_store.py +++ b/src/lean_memory/store/sqlite_store.py @@ -606,6 +606,17 @@ def get_live_run(self, namespace: str) -> Optional[dict]: ).fetchone() return dict(row) if row else None + def last_finished_run(self, namespace: str) -> Optional[dict]: + # Most recent status='ok' run — the runner's previous-cursor + last-run-age + # source (§6.6). 'aborted'/'running' rows are excluded so a no-work run never + # advances the cursor another run reasons from. Pure read. + row = self._db.execute( + "SELECT * FROM maintenance_run WHERE namespace=? AND status='ok' " + "ORDER BY finished_at DESC, id DESC LIMIT 1", + (namespace,), + ).fetchone() + return dict(row) if row else None + def stage_proposal( self, run_id: str, diff --git a/tests/test_maintenance_runner.py b/tests/test_maintenance_runner.py index 4136013..42549b9 100644 --- a/tests/test_maintenance_runner.py +++ b/tests/test_maintenance_runner.py @@ -255,6 +255,38 @@ def _boom(*a, **k): assert rep.skipped_reason == "lost race" +def test_lease_claim_is_one_transaction_abort_rolls_back_on_failed_insert(store, monkeypatch): + """The §7.2 sequence (check → stale-abort → INSERT) is ONE transaction: if the claim + INSERT fails (IntegrityError), the stale-abort UPDATE in the same transaction MUST + roll back too — the prior run stays 'running', not stranded 'aborted'. + + This is the behavioral fingerprint of single-transaction semantics: three separate + autocommit statements would leave the prior run permanently aborted (a lost lease). + """ + import sqlite3 + + clock = Clock() + # A stale live run (heartbeat 10 min old → past the 300s floor). + stale_id = store.create_run("ns", "cli", clock.now_ms(), _cfg().config_hash()) + store.heartbeat_run(stale_id, clock.now_ms()) + clock.advance(10 * 60) + + r = MaintenanceRunner(store, "ns", _cfg(), clock=clock, now_ms=clock.now_ms) + # The claim INSERT loses a race (a concurrent process claimed) → IntegrityError. + def _boom(*a, **k): + raise sqlite3.IntegrityError("UNIQUE constraint failed: ux_run_live") + monkeypatch.setattr(store, "create_run", _boom) + + rep = r.run() + assert rep.status == "skipped" + assert rep.skipped_reason == "lost race" + # The stale run's abort was rolled back with the failed INSERT — still 'running'. + status = store._db.execute( + "SELECT status FROM maintenance_run WHERE id=?", (stale_id,) + ).fetchone()["status"] + assert status == "running", "abort+claim not atomic — the abort UPDATE leaked" + + # ══════════════════════════════════════════════════════════════════════════════ # Work thresholds (§6.6) # ══════════════════════════════════════════════════════════════════════════════ @@ -370,6 +402,54 @@ def test_cursor_gap_duplicate_on_quiet_slot_deduped_next_run(store): assert after[0].id == orig.id # survivor = argmin(valid_at) = the original +def test_below_threshold_noop_preserves_cursor_so_quiet_dup_survives(store): + """A below-threshold no-op must NOT advance the cursor (§6.6 regression). + + Otherwise it strands an un-processed duplicate on a quiet slot: run 1 dedups with + cursor C1; a duplicate D lands on quiet slot Q (id > C1); a below-threshold run + would advance the cursor to C2 > D.id without processing Q; a later real run scans + iter_slots_touched_since(C2) and NEVER sees Q. Preserving C1 across the no-op keeps + Q in the next real run's scan window, so D is eventually deduped. + """ + clock = Clock() + orig = _add(store, "Acme Corp", predicate="works_at", valid_at=NOW - 400 * MS_PER_DAY) + + # Run 1 (first-ever, over threshold): finishes with cursor C1 = current high-water. + r1 = MaintenanceRunner(store, "ns", _cfg(), clock=clock, now_ms=clock.now_ms) + rep1 = r1.run() + assert rep1.status == "ok" + c1 = rep1.cursor_id + + # A single duplicate lands on the quiet slot AFTER C1 — one new fact is BELOW the + # facts threshold (min_new_facts=3) and salience is capped out, so the very next run + # is a no-op. + dup = _add(store, "Acme Corp", predicate="works_at", valid_at=clock.now_ms()) + assert dup.id > c1 + + # The below-threshold no-op run: recent last run (age won't trip), 1 new fact + # (< min_new_facts), high salience floor (won't trip). + r2 = MaintenanceRunner(store, "ns", _cfg(min_new_salience=1e12), clock=clock, now_ms=clock.now_ms) + rep2 = r2.run() + assert rep2.status == "ok" + assert rep2.below_threshold is True + # THE FIX: the no-op preserved the last work-doing run's cursor (C1), not the + # post-D high-water — so slot Q stays in the next run's scan window. + assert rep2.cursor_id == c1 + # The duplicate is still un-deduped (the no-op did no work). + assert len(store.find_latest_in_slot(orig.subject_id, "works_at")) == 2 + + # A later over-threshold run (age trips after 8 days) MUST still see slot Q via + # iter_slots_touched_since(C1) and dedup D. + clock.advance(8 * 24 * 3600) + r3 = MaintenanceRunner(store, "ns", _cfg(min_new_salience=1e12), clock=clock, now_ms=clock.now_ms) + rep3 = r3.run() + assert rep3.status == "ok" + assert rep3.below_threshold is False + after = store.find_latest_in_slot(orig.subject_id, "works_at") + assert len(after) == 1, "the quiet-slot duplicate survived the no-op and was deduped" + assert after[0].id == orig.id + + def test_cursor_advances_before_transform_output(store): """The stored cursor is snapshotted BEFORE transform outputs are written: it equals the pre-run max fact id, never a summary/maintenance-episode id created mid-run.""" From 7263cc09af321aae9f445f0fa2877674b833cf7b Mon Sep 17 00:00:00 2001 From: Wuesteon Date: Thu, 16 Jul 2026 20:51:02 +0800 Subject: [PATCH 08/14] =?UTF-8?q?feat(maintain):=20proposal=20lifecycle,?= =?UTF-8?q?=20Memory=20fa=C3=A7ade,=20tier=20retrieval?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Proposal decide+apply lifecycle (maintain/lifecycle.py, new): approve|reject|edit| promote with the §5 CAS, one-batch approve-and-apply (CAS -> re-validate targets -> per-kind verbs -> applied_at), stale-target expiry that rolls the approve back so only the expiry commits (spine untouched), lazy timeout expiry, and model/embedding work computed before the batch window (§7.1). Store gains cas_decide_proposal / mark_proposal_applied / expire_proposal. Memory façade: maintain (dedicated 5000ms store, dry-run default = zero writes/no lease), review_queue (grouped by entity, lazy timeout expiry), decide, promote, and search(include_cold=). Runner + run_transforms + all five transforms gain a dry_run flag (default False; Task 4/5 behavior byte-identical). Retrieval: dense/sparse tier='hot' filter applied ONLY in default latest-mode (as_of never filters tier; include_cold opts out); find_latest_in_slot deliberately un-filtered so ingest supersession still sees cold facts. Tests: proposal lifecycle (CAS double-decide across two handles, re-apply -> already-applied, timeout + stale-target expiry with full-DB-hash spine invariance, edited-approve human provenance + source=user re-score, summarize apply end-to-end + old-summary supersession, evict+promote round-trip, dim-mismatch refusal, the §10.1 apply-path as-of grid re-run + a backdating fault-injection guard) and the as_of x include_cold x tier matrix. Full offline suite: 251 passed, 1 skipped. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_0198CpP3tf1SbtDGaRX2NWvJ --- src/lean_memory/maintain/__init__.py | 3 + src/lean_memory/maintain/lifecycle.py | 467 ++++++++++++++++ src/lean_memory/maintain/runner.py | 58 +- src/lean_memory/maintain/transforms.py | 42 +- src/lean_memory/memory.py | 173 +++++- src/lean_memory/retrieve/retriever.py | 7 +- src/lean_memory/store/base.py | 39 +- src/lean_memory/store/sqlite_store.py | 76 ++- tests/test_proposal_lifecycle.py | 745 +++++++++++++++++++++++++ tests/test_tier_retrieval.py | 167 ++++++ 10 files changed, 1750 insertions(+), 27 deletions(-) create mode 100644 src/lean_memory/maintain/lifecycle.py create mode 100644 tests/test_proposal_lifecycle.py create mode 100644 tests/test_tier_retrieval.py diff --git a/src/lean_memory/maintain/__init__.py b/src/lean_memory/maintain/__init__.py index 9dcd968..5490000 100644 --- a/src/lean_memory/maintain/__init__.py +++ b/src/lean_memory/maintain/__init__.py @@ -9,6 +9,7 @@ from __future__ import annotations from .config import MS_PER_DAY, MaintenanceConfig +from .lifecycle import decide, promote_fact from .score import value from .summarize import ( ExtractiveStubSummarizer, @@ -34,6 +35,8 @@ "MaintenanceConfig", "MS_PER_DAY", "value", + "decide", + "promote_fact", "Summarizer", "ExtractiveStubSummarizer", "OllamaSummarizer", diff --git a/src/lean_memory/maintain/lifecycle.py b/src/lean_memory/maintain/lifecycle.py new file mode 100644 index 0000000..d1accef --- /dev/null +++ b/src/lean_memory/maintain/lifecycle.py @@ -0,0 +1,467 @@ +"""Proposal decide + apply — the maintenance review lifecycle (design spec §4.2-§4.4, §5). + +DEVIATION (deliberate, approved): the spec's §9.3 file table does not name this +module — it folds "decide/apply" into `memory.py`. We host it HERE instead so +`memory.py` stays a thin, raw-SQL-free façade: the approve-and-apply transaction is +non-trivial (CAS → re-validation → per-kind verbs → applied_at, all in one +`store.batch()` window, with model/embedding work computed BEFORE the window per +§7.1), and belongs beside the transforms it replays, not inside the public API +object. `Memory.decide/promote` are one-line delegations to `decide()` here. + +Contract (spec §5, §3.4): + - Proposals are INVISIBLE until approved: staging wrote zero spine changes, so + apply is the ONLY place a proposal touches the spine. + - APPROVE-AND-APPLY is ONE transaction: CAS decide → RE-VALIDATE every referenced + target (still is_latest=1; dedup pairs additionally still co-valid in the same + slot) → replay the transform's verbs at apply-time t_a (the visibility theorem + holds with t_a) → stamp applied_at. + - Any STALE target ⇒ the whole proposal flips to status='expired', + expiry_reason='stale_target' and the spine is left BYTE-IDENTICAL: the approve + transaction is rolled back, and the expiry is what commits. + - CAS 0-rows ⇒ already decided: an informative result, never an error, never a + re-apply. A retry after a committed apply returns "already applied". + - PROMOTION is explicit-only (§4.4): set_tier(fact, 'hot'). As a decision on an + evict proposal it REJECTS the proposal and promotes; as a direct verb it is + Memory.promote(fact_id). No automatic promotion anywhere. + - TIMEOUT expiry is LAZY: decide() (and review_queue) expire a pending proposal + whose expires_at < now instead of deciding it (status='expired', + expiry_reason='timeout'). + +Offline & batch discipline (§7.1): the summary embedding (the only model/embedding +work in this module) is computed BEFORE the batch window; the lock-hold span holds +only row writes. +""" + +from __future__ import annotations + +import json +from typing import Optional + +import numpy as np + +from ..embed.base import Embedder, matryoshka_truncate +from ..extract.salience import score_salience +from ..store.base import Store +from ..types import Episode, Fact +from .transforms import _coalesced_last_access + +# Decision verbs accepted by decide(). +APPROVE = "approve" +REJECT = "reject" +EDIT = "edit" +PROMOTE = "promote" +_DECISIONS = frozenset((APPROVE, REJECT, EDIT, PROMOTE)) + + +class _StaleTarget(Exception): + """Internal signal: a re-validation failed inside the approve batch. Raising it + rolls the batch back (batch() ROLLBACKs on exception) so the spine is untouched; + the caller then commits the stale-target expiry in a fresh transaction.""" + + +class _LostRace(Exception): + """Internal signal: the CAS decide updated 0 rows (a concurrent writer already + decided this proposal). Rolls the batch back; the caller reports already-decided.""" + + +def decide( + store: Store, + embedder: Embedder, + proposal_id: str, + decision: str, + *, + now: int, + decided_by: str, + edited_text: Optional[str] = None, + run_id: Optional[str] = None, +) -> dict: + """Decide a proposal (approve | reject | edit | promote) and apply on approval (§5). + + Returns a JSON-friendly result dict; never raises for the ordinary "already + decided" / "stale" / "timed out" outcomes (those are reported in the dict). Raises + only for genuine caller errors: an unknown decision, or a dim-mismatched embedder + on a summarize apply (a config error that must not silently corrupt the DB). + """ + if decision not in _DECISIONS: + raise ValueError( + f"unknown decision {decision!r}; expected one of " + f"{sorted(_DECISIONS)}" + ) + + proposal = store.get_proposal(proposal_id) + if proposal is None: + return {"proposal_id": proposal_id, "outcome": "not_found"} + + # Lazy timeout expiry (§5): a pending proposal past expires_at is expired here + # instead of decided. A non-pending proposal is never re-expired. + if proposal["status"] == "pending" and proposal["expires_at"] < now: + expired = store.expire_proposal(proposal_id, "timeout") + if expired: + return { + "proposal_id": proposal_id, + "outcome": "expired", + "expiry_reason": "timeout", + } + # Lost the race to a concurrent decider — fall through to the status report. + proposal = store.get_proposal(proposal_id) or proposal + + if proposal["status"] != "pending": + return _already_decided(proposal) + + if decision == REJECT: + return _reject(store, proposal, now=now, decided_by=decided_by) + if decision == PROMOTE: + return _promote_decision(store, proposal, now=now, decided_by=decided_by) + # approve | edit + return _approve( + store, embedder, proposal, + now=now, decided_by=decided_by, + edited_text=edited_text if decision == EDIT else None, + is_edit=(decision == EDIT), + run_id=run_id, + ) + + +# ── promote as a direct verb (Memory.promote) ──────────────────────────────── +def promote_fact(store: Store, fact_id: str, *, now: int) -> dict: + """Explicit promotion of a single fact to the hot tier (§4.4) — the direct verb + behind Memory.promote(). No proposal involved; set_tier is verb (c), + predicate-invisible. A missing fact is reported, not raised.""" + fact = store.get_fact(fact_id) + if fact is None: + return {"fact_id": fact_id, "outcome": "not_found"} + with store.batch(): + store.set_tier(fact_id, "hot") + return {"fact_id": fact_id, "outcome": "promoted", "tier": "hot"} + + +# ── result helpers ──────────────────────────────────────────────────────────── +def _already_decided(proposal: dict) -> dict: + """Informative result for a proposal that is no longer pending (§5). Distinguishes + an applied proposal ('already applied') from a merely-decided one — never an + error, never a re-apply.""" + applied = proposal.get("applied_at") is not None + return { + "proposal_id": proposal["id"], + "outcome": "already_applied" if applied else "already_decided", + "status": proposal["status"], + "applied_at": proposal.get("applied_at"), + } + + +# ── reject (§3.4: zero spine trace) ────────────────────────────────────────── +def _reject(store: Store, proposal: dict, *, now: int, decided_by: str) -> dict: + won = store.cas_decide_proposal(proposal["id"], "rejected", now, decided_by) + if not won: + return _already_decided(store.get_proposal(proposal["id"]) or proposal) + return {"proposal_id": proposal["id"], "outcome": "rejected"} + + +# ── promote decision (on an evict proposal: reject + promote, §4.4) ─────────── +def _promote_decision(store: Store, proposal: dict, *, now: int, decided_by: str) -> dict: + """A PROMOTE decision on an evict proposal: REJECT the eviction and PROMOTE the + referenced fact to hot (§4.4). One batch so the reject CAS and the set_tier + co-commit. Re-validates that the fact is still latest (promoting a superseded row + to hot is meaningless) — a stale target expires stale, spine untouched.""" + if proposal["kind"] != "evict": + return { + "proposal_id": proposal["id"], + "outcome": "invalid_decision", + "reason": f"promote is only valid on an evict proposal, not {proposal['kind']!r}", + } + payload = json.loads(proposal["payload_json"]) + fact_id = payload["fact_id"] + try: + with store.batch(): + won = store.cas_decide_proposal(proposal["id"], "rejected", now, decided_by) + if not won: + raise _LostRace() + fact = store.get_fact(fact_id) + if fact is None or not fact.is_latest: + raise _StaleTarget() + store.set_tier(fact_id, "hot") + except _LostRace: + return _already_decided(store.get_proposal(proposal["id"]) or proposal) + except _StaleTarget: + store.expire_proposal(proposal["id"], "stale_target") + return { + "proposal_id": proposal["id"], + "outcome": "expired", + "expiry_reason": "stale_target", + } + return { + "proposal_id": proposal["id"], + "outcome": "promoted", + "fact_id": fact_id, + "tier": "hot", + } + + +# ── approve / edit (the approve-and-apply transaction, §5) ──────────────────── +def _approve( + store: Store, + embedder: Embedder, + proposal: dict, + *, + now: int, + decided_by: str, + edited_text: Optional[str], + is_edit: bool, + run_id: Optional[str], +) -> dict: + kind = proposal["kind"] + payload = json.loads(proposal["payload_json"]) + + # `edit` = edited_text + approve semantics with human provenance. Only summarize + # carries editable text; an edit on any other kind is a caller error. + if is_edit and kind != "summarize": + return { + "proposal_id": proposal["id"], + "outcome": "invalid_decision", + "reason": f"edit is only valid on a summarize proposal, not {kind!r}", + } + + # Status enum (§5): an edit-approve records status='edited' (human provenance); + # a plain approve records status='approved'. + decided_status = "edited" if is_edit else "approved" + + # ── Model/embedding work computed BEFORE the batch window (§7.1). Only + # summarize embeds; the dim guard refuses a mismatched embedder up front. ── + precomputed: dict = {} + if kind == "summarize": + summary_text = edited_text if is_edit else payload["summary_text"] + precomputed["summary_text"] = summary_text + precomputed["embedding"] = _embed_or_refuse(store, embedder, summary_text) + precomputed["coarse"] = matryoshka_truncate( + precomputed["embedding"], embedder.coarse_dim + ) + + try: + with store.batch(): + won = store.cas_decide_proposal( + proposal["id"], decided_status, now, decided_by, + edited_text=edited_text, + ) + if not won: + raise _LostRace() + if kind == "dedup_near": + applied = _apply_dedup_near(store, payload, now=now) + elif kind == "summarize": + applied = _apply_summarize( + store, proposal, payload, precomputed, + now=now, is_edit=is_edit, run_id=run_id, + ) + elif kind == "evict": + applied = _apply_evict(store, payload) + else: # pragma: no cover — schema constrains kind to the three above + raise ValueError(f"unknown proposal kind {kind!r}") + store.mark_proposal_applied(proposal["id"], now) + except _LostRace: + return _already_decided(store.get_proposal(proposal["id"]) or proposal) + except _StaleTarget: + # The approve batch rolled back — status is back to 'pending' — so this CAS + # to 'expired' wins and IS the write that commits. Spine untouched. + store.expire_proposal(proposal["id"], "stale_target") + return { + "proposal_id": proposal["id"], + "outcome": "expired", + "expiry_reason": "stale_target", + } + + return { + "proposal_id": proposal["id"], + "outcome": "applied", + "kind": kind, + "status": decided_status, + "applied_at": now, + **applied, + } + + +def _embed_or_refuse(store: Store, embedder: Embedder, text: str) -> np.ndarray: + """Embed the summary text with the apply-owner's embedder, refusing if its dim + does not match the namespace's baked vec0 dims (§4.3 apply-ownership rule). + + The apply process OWNS the embedding; embedding with a mismatched embedder would + corrupt the vec0 table (or fail deep in the insert). We check up front and raise a + clear, actionable ValueError — the same class of guard as _check_existing_dims.""" + store_dim = getattr(store, "dim", None) + store_coarse = getattr(store, "coarse_dim", None) + if store_dim is not None and embedder.dim != store_dim: + raise ValueError( + f"cannot apply summarize: the apply embedder produces {embedder.dim}-dim " + f"vectors but this namespace's vec0 table was baked at {store_dim} dims. " + f"Apply with an embedder matching the namespace's dims, or the summary " + f"vector would corrupt the index." + ) + if store_coarse is not None and embedder.coarse_dim != store_coarse: + raise ValueError( + f"cannot apply summarize: the apply embedder's coarse dim " + f"{embedder.coarse_dim} does not match this namespace's {store_coarse}." + ) + return embedder.embed_one(text) + + +# ── per-kind apply mechanics (all inside the batch; §4.2-§4.4) ──────────────── +def _apply_dedup_near(store: Store, payload: dict, *, now: int) -> dict: + """DEDUP-NEAR approve → DEDUP-EXACT mechanics (§4.2): re-validate the pair is + still co-valid in one slot, then retire the loser onto the survivor and merge + usage stats per the §4.1 rule. Reuses transforms._coalesced_last_access — the + merge formula is NOT re-derived here.""" + a_id, b_id = payload["fact_ids"] + survivor_id = payload["proposed_survivor"] + loser_id = b_id if survivor_id == a_id else a_id + + a = store.get_fact(a_id) + b = store.get_fact(b_id) + # Both must still be latest AND still co-valid in the same slot (both open, same + # subject+predicate) — the §5 dedup re-validation. + if a is None or b is None or not a.is_latest or not b.is_latest: + raise _StaleTarget() + if a.valid_to is not None or b.valid_to is not None: + raise _StaleTarget() + if (a.subject_id, a.predicate) != (b.subject_id, b.predicate): + raise _StaleTarget() + + survivor = a if survivor_id == a_id else b + loser = b if survivor_id == a_id else a + # Merge usage stats over the pair (survivor + loser), the §4.1 rule: access_count + # SUMMED, last_access = max coalesce(last_access, valid_at). Reuse the transforms' + # coalesce helper so the recency-anchor rule has one definition. + merged_access = survivor.access_count + loser.access_count + merged_last_access = max( + _coalesced_last_access(survivor), _coalesced_last_access(loser) + ) + + store.retire_duplicate(loser.id, survivor.id) + store.merge_usage_stats(survivor.id, merged_access, merged_last_access) + return { + "survivor_id": survivor.id, + "loser_id": loser.id, + "merged_access_count": merged_access, + "merged_last_access": merged_last_access, + } + + +def _apply_summarize( + store: Store, + proposal: dict, + payload: dict, + precomputed: dict, + *, + now: int, + is_edit: bool, + run_id: Optional[str], +) -> dict: + """SUMMARIZE approve — the §4.3 apply steps (embedding already precomputed): + (1) re-validate every source still is_latest=1; + (2) insert a maintenance EPISODE (source='maintenance', raw=report JSON); + (3) insert the summary fact (predicate='summary', record_kind='summary', + is_inference=1, valid_at=t_a — NEVER backdated — valid_to=NULL, tier='hot'); + (4) add_derivation(summary, source, run_id) rows; + (5) set_tier(source, 'cold') for each source; + (6) if a previous is_latest summary exists for the subject, supersede it at t_a. + """ + source_ids = payload["source_fact_ids"] + subject_id = payload["subject_id"] + + # (1) Re-validate: every source still latest, else the proposal expires stale. + sources: list[Fact] = [] + for sid in source_ids: + f = store.get_fact(sid) + if f is None or not f.is_latest: + raise _StaleTarget() + sources.append(f) + + namespace = sources[0].namespace + summary_text = precomputed["summary_text"] + embedding = precomputed["embedding"] + coarse = precomputed["coarse"] + + # (2) Maintenance episode — satisfies the fact.episode_id NOT NULL FK. + report_raw = json.dumps( + { + "kind": "summarize", + "proposal_id": proposal["id"], + "run_id": proposal["run_id"], + "source_fact_ids": list(source_ids), + "applied_at": now, + "edited": is_edit, + }, + sort_keys=True, + ) + episode = Episode( + namespace=namespace, raw=report_raw, t_ref=now, source="maintenance" + ) + store.add_episode(episode) + + # Salience: re-score via the salience module. An edit-approve carries human + # provenance → source='user' (which the stub favors); a machine approve scores as + # a non-user maintenance source, so a human-curated summary naturally outranks a + # machine one with zero new ranking code (§4.3). + salience = score_salience( + summary_text, + source="user" if is_edit else "maintenance", + is_inference=True, + ) + + # (3) The summary fact. valid_at = now (t_a) — NEVER backdated, so it appears in + # no past window (§4.3). is_inference=1, tier='hot', its own 'summary' slot. + summary = Fact( + namespace=namespace, + subject_id=subject_id, + predicate="summary", + fact_text=summary_text, + valid_at=now, + valid_to=None, + is_latest=1, + episode_id=episode.id, + salience=salience, + is_inference=1, + tier="hot", + record_kind="summary", + ingested_at=now, + created_at=now, + ) + store.add_fact(summary, embedding, coarse) + + # (4) Derivation lineage rows (summary ← each source). + derive_run_id = run_id or proposal["run_id"] + for sid in source_ids: + store.add_derivation(summary.id, sid, derive_run_id, now) + + # (5) Demote each source to cold — they stay is_latest=1 and fully as-of visible; + # they leave the default hot surface, where the summary now represents them. + for sid in source_ids: + store.set_tier(sid, "cold") + + # (6) Supersede a previous is_latest summary for the subject at t_a, if any. The + # old summary is NOT a source, so the staleness cascade does not misfire on it. + superseded_prev: Optional[str] = None + prev = [ + f + for f in store.find_latest_in_slot(subject_id, "summary") + if f.id != summary.id + ] + if prev: + old = min(prev, key=lambda f: f.id) # deterministic; there is normally one + store.supersede_fact(old.id, summary.id, valid_to=now) + superseded_prev = old.id + + return { + "summary_id": summary.id, + "episode_id": episode.id, + "source_ids": list(source_ids), + "superseded_prev_summary_id": superseded_prev, + } + + +def _apply_evict(store: Store, payload: dict) -> dict: + """EVICT approve → set_tier(fact, 'cold') (§4.4). Re-validate the fact is still + latest; a stale target expires stale.""" + fact_id = payload["fact_id"] + fact = store.get_fact(fact_id) + if fact is None or not fact.is_latest: + raise _StaleTarget() + store.set_tier(fact_id, "cold") + return {"fact_id": fact_id, "tier": "cold"} diff --git a/src/lean_memory/maintain/runner.py b/src/lean_memory/maintain/runner.py index d2774a7..6a3ab74 100644 --- a/src/lean_memory/maintain/runner.py +++ b/src/lean_memory/maintain/runner.py @@ -298,14 +298,70 @@ def _try_claim_lease(self, now_ms: int) -> tuple[Optional[str], Optional[str], O return None, "lost race", None return result["run_id"], None, result.get("took_over") + # ── dry run (§6.3 — report-only, zero writes, no lease) ── + def dry_run(self) -> RunReport: + """Compute the full would-do report with ZERO writes and NO lease (§6.3). + + A dry-run mutates nothing — no proposals, no auto transforms, not even a lease + row (it writes nothing, so it needs no lease). It still runs the threshold gate + and, if over threshold, the transforms in report-only mode (`dry_run=True` + suppresses every store write and counts instead). The returned RunReport has + status 'ok' and run_id=None (no ledger row exists). This is the `apply=False` + default path for `Memory.maintain()` and the CLI/MCP tools. + """ + cfg_hash = self.config.config_hash() + prior = self.store.last_finished_run(self.namespace) + previous_cursor = prior.get("cursor_id") if prior else None + last_finished_at = prior.get("finished_at") if prior else None + + over, threshold_stats = self._threshold_check(previous_cursor, last_finished_at) + stats = dict(threshold_stats) + if not over: + stats["below_threshold"] = True + return RunReport( + status="ok", run_id=None, below_threshold=True, + threshold_stats=stats, transform_report=None, + cursor_id=previous_cursor, config_hash=cfg_hash, + ) + + if previous_cursor is None: + slots = sorted( + {(f.subject_id, f.predicate) for f in self.store.iter_latest_facts()} + ) + else: + slots = list(self.store.iter_slots_touched_since(previous_cursor)) + + exclude_ids, pending_sigs = self._prior_pending() + report = run_transforms( + self.store, self.config, self._now_ms(), run_id="dry-run", slots=slots, + summarizer=self.summarizer, extra_exclude_ids=exclude_ids, + pending_signatures=pending_sigs, dry_run=True, + ) + stats["merges"] = len(report.merges) + stats["staged"] = len(report.proposals) + stats["demoted"] = len(report.demoted_ids) + stats["dropped_proposals"] = report.dropped_proposals + return RunReport( + status="ok", run_id=None, below_threshold=False, + threshold_stats=stats, transform_report=report, + cursor_id=self._max_fact_id(), config_hash=cfg_hash, + ) + # ── the run ── - def run(self, *, on_batch: Optional[Callable[[], None]] = None) -> RunReport: + def run( + self, *, on_batch: Optional[Callable[[], None]] = None, dry_run: bool = False + ) -> RunReport: """Execute one maintenance run end to end (lease → thresholds → transforms). `on_batch` is a test hook invoked at every batch boundary (after the runner's own heartbeat) — used by the heartbeat/crash tests to observe cadence and to simulate a mid-run crash. Production passes None. + + `dry_run=True` delegates to `dry_run()` — report-only, zero writes, no lease. """ + if dry_run: + return self.dry_run() + cfg_hash = self.config.config_hash() claim_ms = self._now_ms() run_id, skipped_reason, took_over = self._try_claim_lease(claim_ms) diff --git a/src/lean_memory/maintain/transforms.py b/src/lean_memory/maintain/transforms.py index ee708fa..ec73802 100644 --- a/src/lean_memory/maintain/transforms.py +++ b/src/lean_memory/maintain/transforms.py @@ -121,6 +121,7 @@ def dedup_exact( slots: Iterable[Slot], *, exclude_ids: Optional[set[str]] = None, + dry_run: bool = False, ) -> list[Merge]: """Auto-retire exact duplicates within each slot (§4.1). @@ -162,10 +163,12 @@ def dedup_exact( merged_access = sum(f.access_count for f in members) merged_last_access = max(_coalesced_last_access(f) for f in members) - with store.batch(): - for loser in losers: - store.retire_duplicate(loser.id, survivor.id) - store.merge_usage_stats(survivor.id, merged_access, merged_last_access) + # dry_run: compute the merge record but write NOTHING (no batch, no verbs). + if not dry_run: + with store.batch(): + for loser in losers: + store.retire_duplicate(loser.id, survivor.id) + store.merge_usage_stats(survivor.id, merged_access, merged_last_access) merges.append( Merge( @@ -190,6 +193,7 @@ def dedup_near( run_id: str, budget: int = 1_000_000, skip_signatures: Optional[set[frozenset[str]]] = None, + dry_run: bool = False, ) -> tuple[list[StagedProposal], int]: """Stage near-duplicate merge PROPOSALS — zero spine writes (§4.2). @@ -251,7 +255,8 @@ def dedup_near( if len(staged) >= budget: dropped += 1 continue - pid = store.stage_proposal( + # dry_run: count the would-stage proposal, write NOTHING. + pid = "dry-run" if dry_run else store.stage_proposal( run_id=run_id, namespace=a.namespace, kind="dedup_near", @@ -277,6 +282,7 @@ def summarize( run_id: str, budget: int = 1_000_000, skip_signatures: Optional[set[frozenset[str]]] = None, + dry_run: bool = False, ) -> tuple[list[StagedProposal], int]: """Stage SUMMARIZE proposals — zero spine writes, no embedding at stage time (§4.3). @@ -340,7 +346,8 @@ def subject_heat(slots_map: dict[Slot, list[Fact]]) -> float: if len(staged) >= budget: dropped += 1 continue - pid = store.stage_proposal( + # dry_run: count the would-stage proposal, write NOTHING. + pid = "dry-run" if dry_run else store.stage_proposal( run_id=run_id, namespace=sources[0].namespace, kind="summarize", @@ -385,6 +392,7 @@ def evict_propose( run_id: str, budget: int = 1_000_000, exclude_ids: Optional[set[str]] = None, + dry_run: bool = False, ) -> tuple[list[StagedProposal], int]: """Stage EVICT (demotion) proposals for still-latest facts below the value floor but NOT in the strict auto-band — zero spine writes (§4.4). @@ -419,7 +427,8 @@ def evict_propose( if len(staged) >= budget: dropped += 1 continue - pid = store.stage_proposal( + # dry_run: count the would-stage proposal, write NOTHING. + pid = "dry-run" if dry_run else store.stage_proposal( run_id=run_id, namespace=f.namespace, kind="evict", @@ -450,6 +459,7 @@ def evict_auto( now: int, *, exclude_ids: Optional[set[str]] = None, + dry_run: bool = False, ) -> list[str]: """Auto-demote the strict-band facts to 'cold' without review (§4.4). @@ -466,7 +476,8 @@ def evict_auto( continue if not _in_auto_band(f, config, now): continue - store.set_tier(f.id, "cold") + if not dry_run: # dry_run: count the would-demote, write NOTHING. + store.set_tier(f.id, "cold") demoted.append(f.id) return demoted @@ -482,6 +493,7 @@ def run_transforms( summarizer: Optional[Summarizer] = None, extra_exclude_ids: Optional[set[str]] = None, pending_signatures: Optional[dict[str, set[frozenset[str]]]] = None, + dry_run: bool = False, ) -> TransformReport: """Run all four transforms in the spec-mandated intra-run order (§4.4). @@ -521,7 +533,7 @@ def run_transforms( near, dropped = dedup_near( store, config, now, slots, run_id=run_id, budget=remaining, - skip_signatures=pending_signatures.get("dedup_near"), + skip_signatures=pending_signatures.get("dedup_near"), dry_run=dry_run, ) report.proposals.extend(near) report.dropped_proposals += dropped @@ -529,7 +541,7 @@ def run_transforms( summ, dropped = summarize( store, config, now, summarizer, run_id=run_id, budget=max(0, remaining), - skip_signatures=pending_signatures.get("summarize"), + skip_signatures=pending_signatures.get("summarize"), dry_run=dry_run, ) report.proposals.extend(summ) report.dropped_proposals += dropped @@ -539,7 +551,7 @@ def run_transforms( # eviction (excluding its id covers both re-propose and auto-demote — §4.4). ev_prop, dropped = evict_propose( store, config, now, run_id=run_id, budget=max(0, remaining), - exclude_ids=extra_exclude_ids, + exclude_ids=extra_exclude_ids, dry_run=dry_run, ) report.proposals.extend(ev_prop) report.dropped_proposals += dropped @@ -548,7 +560,11 @@ def run_transforms( # This run's staged ids UNION prior runs' still-pending referenced ids: neither a # fact a human is reviewing now nor one staged this run is auto-mutated. staged_ids = report.staged_fact_ids | extra_exclude_ids - report.merges = dedup_exact(store, config, now, slots, exclude_ids=staged_ids) - report.demoted_ids = evict_auto(store, config, now, exclude_ids=staged_ids) + report.merges = dedup_exact( + store, config, now, slots, exclude_ids=staged_ids, dry_run=dry_run + ) + report.demoted_ids = evict_auto( + store, config, now, exclude_ids=staged_ids, dry_run=dry_run + ) return report diff --git a/src/lean_memory/memory.py b/src/lean_memory/memory.py index f345f88..92b1972 100644 --- a/src/lean_memory/memory.py +++ b/src/lean_memory/memory.py @@ -10,10 +10,11 @@ from __future__ import annotations +import json import re import sys from pathlib import Path -from typing import Optional +from typing import Optional, TYPE_CHECKING from .embed.base import Embedder from .embed.fake import FakeEmbedder @@ -22,11 +23,23 @@ from .extract.llm_typer import StubTyper, TypedFact, Typer, TyperError from .extract.router import RecallBiasedRouter from .extract.salience import score_salience +from .maintain import lifecycle +from .maintain.config import MaintenanceConfig +from .maintain.runner import MaintenanceRunner +from .maintain.summarize import Summarizer from .retrieve.rerank import IdentityReranker, Reranker from .retrieve.retriever import Retriever from .store.sqlite_store import SqliteStore from .types import Entity, Episode, Fact, RetrievedFact, new_id, now_ms +if TYPE_CHECKING: + from .maintain.runner import RunReport + +# busy_timeout for a dedicated maintenance connection (§7.1): the serving store stays +# at 1500 ms; a maintenance run (and the apply path) opens at 5000 so the 5000 budget +# reaches in-process callers without re-tuning the serving connection. +_MAINT_BUSY_TIMEOUT_MS = 5000 + # domain predicate the rules/stub passes use when none is guessed _DEFAULT_PREDICATE = "about" @@ -39,6 +52,33 @@ _SAFE_NS = re.compile(r"[^A-Za-z0-9_.-]") +def _safe_json(raw: Optional[str]) -> dict: + """Parse a proposal payload defensively — a malformed/absent payload yields {} + rather than crashing the review queue.""" + if not raw: + return {} + try: + parsed = json.loads(raw) + except (TypeError, ValueError): + return {} + return parsed if isinstance(parsed, dict) else {} + + +def _proposal_subject_id(proposal: dict, payload: dict) -> str: + """The subject entity a proposal groups under, kind-aware (§6.3 grouping): + - summarize : payload['subject_id'] + - dedup_near: payload['slot']['subject_id'] + - evict : no subject in the payload — group under '' (ungrouped bucket). + Missing keys degrade to '' so the queue never crashes on a surprising payload.""" + kind = proposal.get("kind") + if kind == "summarize": + return payload.get("subject_id", "") or "" + if kind == "dedup_near": + slot = payload.get("slot") or {} + return slot.get("subject_id", "") or "" + return "" + + class Memory: def __init__( self, @@ -237,16 +277,143 @@ def search( as_of: Optional[int] = None, is_latest_only: bool = True, now: Optional[int] = None, + include_cold: bool = False, ) -> list[RetrievedFact]: """`now` (epoch ms) anchors the recency-decay term — pass the corpus's present when querying historical data, else the wall clock is used and - recency is ≈0 for everything old (the term de-ranks nothing).""" + recency is ≈0 for everything old (the term de-ranks nothing). + + `include_cold=True` opts a default latest-mode search out of the tier filter, + so tier='cold' (maintenance-demoted) facts are searchable again (§8). Has no + effect on as_of queries, which never filter tier.""" store = self._store(namespace) retriever = Retriever(store, self.embedder, self.reranker) return retriever.retrieve( - query, k, as_of=as_of, is_latest_only=is_latest_only, now=now + query, k, as_of=as_of, is_latest_only=is_latest_only, now=now, + include_cold=include_cold, ) + # ── sleep-time maintenance façade (design spec §7.1, §8) ── + def _namespace_path(self, namespace: str) -> Path: + safe = _SAFE_NS.sub("_", namespace) or "default" + return self.root / f"{safe}.db" + + def _maintenance_store(self, namespace: str) -> SqliteStore: + """Open a DEDICATED maintenance SqliteStore on the namespace file with the + 5000 ms busy_timeout (§7.1) — this is how the 5000 budget reaches in-process + callers WITHOUT re-tuning the serving store's connection (which stays at 1500). + Caller MUST close it when the run/apply finishes.""" + return SqliteStore( + self._namespace_path(namespace), + dim=self.embedder.dim, + coarse_dim=self.embedder.coarse_dim, + busy_timeout_ms=_MAINT_BUSY_TIMEOUT_MS, + ) + + def maintain( + self, + namespace: str, + *, + config: Optional[MaintenanceConfig] = None, + apply: bool = False, + trigger: str = "cli", + summarizer: Optional[Summarizer] = None, + ) -> "RunReport": + """Run one sleep-time maintenance pass on `namespace` (§7.1). + + Opens a dedicated maintenance store (5000 ms budget), drives the + MaintenanceRunner, closes the store, and returns its RunReport. DRY-RUN by + default: `apply=False` stages NOTHING and mutates nothing — it computes the + full would-do report with zero writes and takes no lease (a dry-run writes + nothing, so it needs no lease). `apply=True` claims the lease, runs the auto + band, and stages proposals. + """ + store = self._maintenance_store(namespace) + try: + runner = MaintenanceRunner( + store, namespace, config=config, trigger=trigger, + summarizer=summarizer, + ) + return runner.run(dry_run=not apply) + finally: + store.close() + + def review_queue( + self, namespace: str, *, kind: Optional[str] = None, limit: int = 20 + ) -> list[dict]: + """Pending proposals with their evidence payloads, grouped by subject entity + (§6.3). Lazily expires any pending proposal past its expires_at (status + 'expired', reason 'timeout') before returning — silence must not surface as a + live proposal. + + Returns a list of {entity_id, entity_name, proposals: [...]} groups; each + proposal carries its parsed `payload` alongside the row fields, so a caller + (console / MCP) renders evidence without re-parsing JSON.""" + store = self._maintenance_store(namespace) + try: + now = now_ms() + pending = store.list_proposals( + namespace, status="pending", kind=kind, limit=limit * 4 + ) + live: list[dict] = [] + for p in pending: + if p["expires_at"] < now: + store.expire_proposal(p["id"], "timeout") # lazy timeout expiry + continue + live.append(p) + if len(live) >= limit: + break + + groups: dict[str, dict] = {} + order: list[str] = [] + for p in live: + payload = _safe_json(p.get("payload_json")) + subject_id = _proposal_subject_id(p, payload) + if subject_id not in groups: + entity = store.get_entity(subject_id) if subject_id else None + groups[subject_id] = { + "entity_id": subject_id, + "entity_name": entity.name if entity else None, + "proposals": [], + } + order.append(subject_id) + item = dict(p) + item["payload"] = payload + groups[subject_id]["proposals"].append(item) + return [groups[s] for s in order] + finally: + store.close() + + def decide( + self, + proposal_id: str, + decision: str, + *, + namespace: str, + edited_text: Optional[str] = None, + decided_by: str = "console", + ) -> dict: + """Decide a proposal (approve | reject | edit | promote), applying on approval + through the lifecycle (§5). Uses a dedicated 5000 ms store for the apply — the + same rule as maintain() — and the server's embedder for the summary vector.""" + store = self._maintenance_store(namespace) + try: + return lifecycle.decide( + store, self.embedder, proposal_id, decision, + now=now_ms(), decided_by=decided_by, edited_text=edited_text, + ) + finally: + store.close() + + def promote(self, fact_id: str, *, namespace: str) -> dict: + """Explicit promotion of a fact to the hot tier (§4.4). Explicit-only — there + is no automatic promotion anywhere. Uses a dedicated 5000 ms store.""" + store = self._maintenance_store(namespace) + try: + return lifecycle.promote_fact(store, fact_id, now=now_ms()) + finally: + store.close() + def close(self) -> None: for s in self._stores.values(): s.close() diff --git a/src/lean_memory/retrieve/retriever.py b/src/lean_memory/retrieve/retriever.py index b337095..e7a4f5b 100644 --- a/src/lean_memory/retrieve/retriever.py +++ b/src/lean_memory/retrieve/retriever.py @@ -41,6 +41,7 @@ def retrieve( as_of: Optional[int] = None, is_latest_only: bool = True, now: Optional[int] = None, + include_cold: bool = False, ) -> list[RetrievedFact]: now = now if now is not None else now_ms() @@ -48,13 +49,15 @@ def retrieve( q_full = self.embedder.embed_one(query) q_coarse = matryoshka_truncate(q_full, self.embedder.coarse_dim) - # 2+3. dense and sparse arms, each over-retrieved + # 2+3. dense and sparse arms, each over-retrieved. include_cold threads to both + # arms; the store applies the tier filter only in default latest-mode (§8). dense = self.store.dense_search( q_coarse, q_full, OVER_RETRIEVE, - is_latest_only=is_latest_only, as_of=as_of, + is_latest_only=is_latest_only, as_of=as_of, include_cold=include_cold, ) sparse = self.store.sparse_search( query, OVER_RETRIEVE, is_latest_only=is_latest_only, as_of=as_of, + include_cold=include_cold, ) # 4. RRF fuse (k=10). score(d) = Σ 1/(k + rank_i(d)), rank 1-based. diff --git a/src/lean_memory/store/base.py b/src/lean_memory/store/base.py index 5bd6d3f..2c09268 100644 --- a/src/lean_memory/store/base.py +++ b/src/lean_memory/store/base.py @@ -55,7 +55,11 @@ def find_latest_in_slot( self, subject_id: str, predicate: str ) -> Sequence[Fact]: """All currently-latest facts in a (subject, predicate) slot — for contradiction - detection / supersession lookup.""" + detection / supersession lookup. + + MUST NOT filter on tier (§8): ingest supersession lookup runs through here and + must see COLD facts, else a demoted-then-superseded fact would never close. + The tier filter lives only in the two retrieval arms, never here.""" # ── retrieval primitives (the Retriever composes these) ── @abstractmethod @@ -67,16 +71,20 @@ def dense_search( *, is_latest_only: bool = True, as_of: Optional[int] = None, + include_cold: bool = False, ) -> list[tuple[str, float]]: - """Two-stage Matryoshka dense search. Returns [(fact_id, distance)] best-first.""" + """Two-stage Matryoshka dense search. Returns [(fact_id, distance)] best-first. + Filters tier='hot' ONLY in default latest-mode (is_latest_only and as_of is + None and not include_cold); as_of never filters tier (§8).""" @abstractmethod def sparse_search( self, query_text: str, k: int, *, is_latest_only: bool = True, - as_of: Optional[int] = None, + as_of: Optional[int] = None, include_cold: bool = False, ) -> list[tuple[str, float]]: """BM25 lexical search. Returns [(fact_id, score)] best-first. - as_of applies the same interval predicate as the dense arm.""" + as_of applies the same interval predicate as the dense arm; the tier filter + follows the same default-latest-mode-only rule (§8).""" @abstractmethod def hydrate(self, fact_ids: Sequence[str]) -> dict[str, Fact]: @@ -209,6 +217,29 @@ def stage_proposal( def get_proposal(self, proposal_id: str) -> Optional[dict]: """A single proposal row as a dict, or None.""" + @abstractmethod + def cas_decide_proposal( + self, + proposal_id: str, + status: str, + decided_at: int, + decided_by: str, + edited_text: Optional[str] = None, + ) -> int: + """CAS a pending proposal into a decided state (§5). Only flips a row still + `status='pending'`; returns rows updated (1 == won the decision, 0 == already + decided). The race-safe decide primitive.""" + + @abstractmethod + def mark_proposal_applied(self, proposal_id: str, applied_at: int) -> None: + """Stamp applied_at after a proposal's verbs committed (§5). Co-commits with + the spine writes when called inside the approve-and-apply batch().""" + + @abstractmethod + def expire_proposal(self, proposal_id: str, expiry_reason: str) -> int: + """Expire a still-pending proposal (§5): status='expired' + expiry_reason + ('timeout'|'stale_target'), CAS on status='pending'. Returns rows updated.""" + @abstractmethod def list_proposals( self, diff --git a/src/lean_memory/store/sqlite_store.py b/src/lean_memory/store/sqlite_store.py index e8e5f9d..9497e95 100644 --- a/src/lean_memory/store/sqlite_store.py +++ b/src/lean_memory/store/sqlite_store.py @@ -338,6 +338,7 @@ def dense_search( *, is_latest_only: bool = True, as_of: Optional[int] = None, + include_cold: bool = False, ) -> list[tuple[str, float]]: """Two-stage Matryoshka: coarse 256-dim KNN over a wider pool, then re-score the survivors at full 768-dim. The coarse pool is k*COARSE_FACTOR wide so the @@ -346,10 +347,21 @@ def dense_search( coarse_k = max(k * COARSE_FACTOR, k) latest_clause = "AND is_latest = 1" if is_latest_only else "" + # Tier filter (§8): drop cold facts from the default hot surface ONLY — the + # vec0 'tier' metadata column ANDed into the KNN WHERE. Applied EXCLUSIVELY in + # default latest-mode: as_of queries NEVER filter tier (historical reads see + # everything), and include_cold=True opts out explicitly. Every existing row is + # tier='hot', so this clause is byte-identical for anyone who never runs + # maintenance (regression pin). + tier_clause = ( + "AND tier = 'hot'" + if (is_latest_only and as_of is None and not include_cold) + else "" + ) # Stage 1: coarse KNN. vec0 KNN must use a single MATCH + LIMIT. coarse_rows = self._db.execute( f"""SELECT fact_id, distance FROM fact_vec - WHERE embedding_256 MATCH ? {latest_clause} + WHERE embedding_256 MATCH ? {latest_clause} {tier_clause} ORDER BY distance LIMIT ?""", (_serialize(query_256), coarse_k), ).fetchall() @@ -398,10 +410,15 @@ def _apply_as_of(self, scored: list[tuple[str, float]], as_of: int) -> list[tupl def sparse_search( self, query_text: str, k: int, *, is_latest_only: bool = True, - as_of: Optional[int] = None, + as_of: Optional[int] = None, include_cold: bool = False, ) -> list[tuple[str, float]]: # FTS5 BM25: lower bm25() is better, so we negate to "higher is better". - needs_row_check = is_latest_only or as_of is not None + # Tier filter (§8): drop cold facts in default latest-mode ONLY — same + # condition as the dense arm, applied in the existing per-row recheck so both + # arms read the single flag set_tier writes. as_of NEVER filters tier; + # include_cold=True opts out. Byte-identical when nothing is cold. + filter_tier = is_latest_only and as_of is None and not include_cold + needs_row_check = is_latest_only or as_of is not None or filter_tier try: rows = self._db.execute( """SELECT f.fact_id AS fact_id, bm25(fact_fts) AS score @@ -423,13 +440,15 @@ def sparse_search( for r in rows: if needs_row_check: row = self._db.execute( - "SELECT is_latest, valid_at, valid_to FROM fact WHERE id=?", + "SELECT is_latest, valid_at, valid_to, tier FROM fact WHERE id=?", (r["fact_id"],), ).fetchone() if not row: continue if is_latest_only and not row["is_latest"]: continue + if filter_tier and row["tier"] != "hot": + continue if as_of is not None and not ( row["valid_at"] <= as_of and (row["valid_to"] is None or row["valid_to"] > as_of) @@ -645,6 +664,55 @@ def get_proposal(self, proposal_id: str) -> Optional[dict]: ).fetchone() return dict(row) if row else None + # ── proposal decide (CAS) + apply-time stamps (design spec §5) ── + def cas_decide_proposal( + self, + proposal_id: str, + status: str, + decided_at: int, + decided_by: str, + edited_text: Optional[str] = None, + ) -> int: + """Compare-and-swap a pending proposal into a decided state (§5). + + The EXACT §5 CAS: only flips a row still `status='pending'`, so two concurrent + writers race for the single UPDATE and exactly one wins. Returns the rows + updated — 1 == this caller won the decision, 0 == the proposal was already + decided (report, never re-apply). Commits through _commit so it participates + in an open approve-and-apply batch() window (§5).""" + cur = self._db.execute( + "UPDATE maintenance_proposal SET status=?, decided_at=?, decided_by=?, " + "edited_text=? WHERE id=? AND status='pending'", + (status, decided_at, decided_by, edited_text, proposal_id), + ) + self._commit() + return cur.rowcount + + def mark_proposal_applied(self, proposal_id: str, applied_at: int) -> None: + """Stamp applied_at on a proposal after its verbs committed (§5). Plain UPDATE + keyed on id — the apply happens inside the same batch() as the CAS, so the + stamp co-commits with the spine writes.""" + self._db.execute( + "UPDATE maintenance_proposal SET applied_at=? WHERE id=?", + (applied_at, proposal_id), + ) + self._commit() + + def expire_proposal(self, proposal_id: str, expiry_reason: str) -> int: + """Expire a still-pending proposal (§5): status='expired' + expiry_reason + ('timeout' | 'stale_target'), CAS on status='pending' so it can never clobber + an already-decided row. Returns rows updated (1 == expired here, 0 == already + decided). Commits through _commit — the stale-target expiry co-commits with an + open batch() window so it is the write that survives while the spine stays + untouched (§5).""" + cur = self._db.execute( + "UPDATE maintenance_proposal SET status='expired', expiry_reason=? " + "WHERE id=? AND status='pending'", + (expiry_reason, proposal_id), + ) + self._commit() + return cur.rowcount + def list_proposals( self, namespace: str, diff --git a/tests/test_proposal_lifecycle.py b/tests/test_proposal_lifecycle.py new file mode 100644 index 0000000..8b0cd49 --- /dev/null +++ b/tests/test_proposal_lifecycle.py @@ -0,0 +1,745 @@ +"""Proposal lifecycle — decide + apply, the maintenance review path (spec §4.2-§4.4, §5, §10). + +Covers, TDD-first: + - CAS double-decide across two store handles on ONE file (second returns + already-decided, no re-apply); re-apply retry after commit returns "already + applied", not an error (§5, §10.7). + - Lazy timeout expiry: a pending proposal past expires_at is expired by decide(). + - STALE-TARGET expiry: a dedup target superseded between stage and approve flips the + proposal to expired/stale_target while the SPINE stays byte-identical (full fact- + table hash) (§5, §10.7). + - REJECT leaves the spine byte-identical (§10.7). + - EDITED-approve records human provenance (edited_text stored, status='edited') and + re-scores the summary with source='user', outranking a machine-scored sibling + (salience delta) (§4.3). + - SUMMARIZE apply end-to-end: maintenance episode + summary fact (valid_at=t_a, tier + hot, record_kind summary) + derivation rows + sources cold + old-summary + supersession on a second approve (§4.3). + - EVICT apply + PROMOTE verb round-trip (§4.4). + - Dim-mismatch embedder refuses to apply with a clear error (§4.3 apply-ownership). + - THE HEADLINE: apply-path as-of grid re-run — approve one summarize + one dedup_near + + one evict via a real runner pass; the store predicate is bit-identical for all + T < t_a and shows only the intended deltas at T >= t_a (§10.1). + +Offline on FakeEmbedder; LM_FORCE_STUBS honored by the default summarizer. +""" + +from __future__ import annotations + +import hashlib +import json + +import pytest + +from lean_memory.embed.fake import FakeEmbedder +from lean_memory.maintain import lifecycle +from lean_memory.maintain.config import MS_PER_DAY, MaintenanceConfig +from lean_memory.store.sqlite_store import SqliteStore +from lean_memory.types import Entity, Episode, Fact + +T_A = 2_000_000_000_000 # apply time (epoch ms) +DAY = MS_PER_DAY + + +# ── fixtures / helpers ───────────────────────────────────────────────────────── +@pytest.fixture +def emb(): + return FakeEmbedder() + + +@pytest.fixture +def db_path(tmp_path): + return tmp_path / "lifecycle.db" + + +def _open(db_path, emb, *, busy_timeout_ms=5000): + s = SqliteStore(db_path, dim=emb.dim, coarse_dim=emb.coarse_dim, + busy_timeout_ms=busy_timeout_ms) + return s + + +def _seed(store, emb): + ep = Episode(namespace="ns", raw="seed", t_ref=1_000) + store.add_episode(ep) + store._emb = emb + store._ep = ep + store._subj = store.upsert_entity(Entity(namespace="ns", name="user", type="person")) + return store + + +def _add(store, text, *, predicate="likes", valid_at=T_A - 300 * DAY, valid_to=None, + superseded_by=None, is_latest=1, salience=5.0, access_count=0, + last_access=None, tier="hot", record_kind="fact"): + f = Fact( + namespace="ns", subject_id=store._subj.id, predicate=predicate, + fact_text=text, valid_at=valid_at, valid_to=valid_to, + superseded_by=superseded_by, is_latest=is_latest, episode_id=store._ep.id, + salience=salience, access_count=access_count, last_access=last_access, + tier=tier, record_kind=record_kind, + ) + full, coarse = store._emb.embed_with_coarse(text) + store.add_fact(f, full, coarse) + return f + + +def _run_id(store): + return store.create_run("ns", "cli", T_A, "cfg") + + +def _fact_hash(store): + """A stable hash of the ENTIRE fact table — the §10.7 spine-invariance probe.""" + rows = store._db.execute( + "SELECT id, is_latest, valid_at, valid_to, superseded_by, invalidated_by, " + "tier, access_count, last_access, record_kind, salience " + "FROM fact ORDER BY id" + ).fetchall() + blob = json.dumps([tuple(r) for r in rows], sort_keys=True).encode() + return hashlib.sha256(blob).hexdigest() + + +def _visible_at(store, T): + rows = store._db.execute( + "SELECT id FROM fact WHERE valid_at <= ? AND (valid_to IS NULL OR valid_to > ?)", + (T, T), + ).fetchall() + return frozenset(r["id"] for r in rows) + + +def _stage_dedup_near(store, run_id, a, b, survivor): + payload = { + "slot": {"subject_id": a.subject_id, "predicate": a.predicate}, + "fact_ids": [a.id, b.id], + "fact_texts": {a.id: a.fact_text, b.id: b.fact_text}, + "cosine": 0.99, + "multivalued": True, + "proposed_survivor": survivor.id, + "evidence_backend": "stored", + } + return store.stage_proposal( + run_id=run_id, namespace="ns", kind="dedup_near", + payload_json=json.dumps(payload, sort_keys=True), + created_at=T_A, expires_at=T_A + 30 * DAY, evidence_backend="stored", + ) + + +def _stage_summarize(store, run_id, sources, summary_text, subject_id=None): + payload = { + "subject_id": subject_id or sources[0].subject_id, + "source_fact_ids": [f.id for f in sources], + "source_fact_texts": {f.id: f.fact_text for f in sources}, + "summary_text": summary_text, + "evidence_backend": "stub", + } + return store.stage_proposal( + run_id=run_id, namespace="ns", kind="summarize", + payload_json=json.dumps(payload, sort_keys=True), + created_at=T_A, expires_at=T_A + 30 * DAY, evidence_backend="stub", + ) + + +def _pick(proposals, predicate): + """The first proposal matching `predicate` — raises if none (keeps the grid test + honest: a corpus change that stops producing the target proposal fails loudly).""" + for p in proposals: + if predicate(p): + return p + raise AssertionError("no proposal matched the picker predicate") + + +def _stage_evict(store, run_id, fact): + payload = { + "fact_id": fact.id, "fact_text": fact.fact_text, "value": 0.1, + "salience": fact.salience, "access_count": fact.access_count, + "evidence_backend": "score", + } + return store.stage_proposal( + run_id=run_id, namespace="ns", kind="evict", + payload_json=json.dumps(payload, sort_keys=True), + created_at=T_A, expires_at=T_A + 30 * DAY, evidence_backend="score", + ) + + +# ── CAS double-decide across two handles ─────────────────────────────────────── +def test_cas_double_decide_across_two_handles(db_path, emb): + """Two store handles on the SAME file race to approve one evict proposal. Exactly + one wins the CAS; the second returns already-decided and does NOT re-apply (§5).""" + s1 = _seed(_open(db_path, emb), emb) + fact = _add(s1, "trivial ancient note", predicate="notes", salience=1.0) + run_id = _run_id(s1) + pid = _stage_evict(s1, run_id, fact) + s1.close() + + a = _open(db_path, emb) + b = _open(db_path, emb) + r1 = lifecycle.decide(a, emb, pid, "approve", now=T_A, decided_by="console") + r2 = lifecycle.decide(b, emb, pid, "approve", now=T_A, decided_by="mcp") + + assert r1["outcome"] == "applied" + assert r2["outcome"] == "already_applied" + # Exactly ONE apply happened: the fact is cold, and applied_at is set once. + assert b.get_fact(fact.id).tier == "cold" + row = b.get_proposal(pid) + assert row["status"] == "approved" + assert row["applied_at"] == T_A + a.close() + b.close() + + +def test_reapply_retry_after_commit_returns_already_applied(db_path, emb): + """A retry of the SAME approve after it committed returns 'already applied', never + an error and never a second apply (§5).""" + s = _seed(_open(db_path, emb), emb) + fact = _add(s, "trivial ancient note", predicate="notes", salience=1.0) + run_id = _run_id(s) + pid = _stage_evict(s, run_id, fact) + + first = lifecycle.decide(s, emb, pid, "approve", now=T_A, decided_by="console") + assert first["outcome"] == "applied" + + # Retry on the same handle. + second = lifecycle.decide(s, emb, pid, "approve", now=T_A + 1, decided_by="console") + assert second["outcome"] == "already_applied" + assert second["applied_at"] == T_A # unchanged — no re-apply + s.close() + + +# ── lazy timeout expiry ──────────────────────────────────────────────────────── +def test_timeout_expiry_on_decide(db_path, emb): + """decide() on a pending proposal past expires_at expires it (timeout) instead of + deciding it — silence is never consent (§5).""" + s = _seed(_open(db_path, emb), emb) + fact = _add(s, "trivial ancient note", predicate="notes", salience=1.0) + run_id = _run_id(s) + pid = _stage_evict(s, run_id, fact) + # expires_at = T_A + 30d; decide at a time PAST that. + now = T_A + 31 * DAY + r = lifecycle.decide(s, emb, pid, "approve", now=now, decided_by="console") + assert r["outcome"] == "expired" + assert r["expiry_reason"] == "timeout" + row = s.get_proposal(pid) + assert row["status"] == "expired" and row["expiry_reason"] == "timeout" + assert row["applied_at"] is None + assert s.get_fact(fact.id).tier == "hot" # never applied + s.close() + + +# ── stale-target expiry (dedup) — spine byte-identical ───────────────────────── +def test_stale_target_dedup_expires_and_spine_untouched(db_path, emb): + """Stage a dedup_near, supersede one target via ordinary ingest, then approve → + the proposal flips to expired/stale_target and the SPINE is byte-identical to the + pre-approve state (the approve transaction rolled back; only the expiry committed) + (§5, §10.7).""" + s = _seed(_open(db_path, emb), emb) + a = _add(s, "likes jazz music", valid_at=T_A - 300 * DAY, access_count=1) + b = _add(s, "enjoys jazz tunes", valid_at=T_A - 250 * DAY, access_count=2) + run_id = _run_id(s) + pid = _stage_dedup_near(s, run_id, a, b, survivor=a) + + # Supersede `b` via an ordinary supersession (b is now is_latest=0, closed). + newer = _add(s, "no longer into jazz", valid_at=T_A - 100 * DAY, is_latest=1) + s.supersede_fact(b.id, newer.id, valid_to=newer.valid_at) + + hash_before = _fact_hash(s) + r = lifecycle.decide(s, emb, pid, "approve", now=T_A, decided_by="console") + + assert r["outcome"] == "expired" + assert r["expiry_reason"] == "stale_target" + row = s.get_proposal(pid) + assert row["status"] == "expired" and row["expiry_reason"] == "stale_target" + assert row["applied_at"] is None + # THE SPINE IS BYTE-IDENTICAL — the approve verbs never touched it. + assert _fact_hash(s) == hash_before + s.close() + + +# ── reject leaves the spine byte-identical ───────────────────────────────────── +def test_reject_leaves_spine_byte_identical(db_path, emb): + s = _seed(_open(db_path, emb), emb) + a = _add(s, "likes jazz music", valid_at=T_A - 300 * DAY) + b = _add(s, "enjoys jazz tunes", valid_at=T_A - 250 * DAY) + run_id = _run_id(s) + pid = _stage_dedup_near(s, run_id, a, b, survivor=a) + + hash_before = _fact_hash(s) + r = lifecycle.decide(s, emb, pid, "reject", now=T_A, decided_by="console") + assert r["outcome"] == "rejected" + assert s.get_proposal(pid)["status"] == "rejected" + assert _fact_hash(s) == hash_before # zero spine trace (§3.4/§10.7) + s.close() + + +# ── edited-approve: human provenance + source='user' re-score ────────────────── +def test_edited_approve_records_human_provenance_and_outranks_machine(db_path, emb): + """An edit-approve stores edited_text, marks status='edited', and re-scores the + summary with source='user' — so it outranks the same text scored as a machine + (maintenance) summary. Asserts the salience delta directly (§4.3).""" + s = _seed(_open(db_path, emb), emb) + sources = [ + _add(s, f"note number {i} about the project", predicate="notes", + valid_at=T_A - (200 - i) * DAY) + for i in range(5) + ] + run_id = _run_id(s) + machine_text = "Consolidated project status summary line" + + # (1) Machine approve of a first proposal → summary scored as maintenance source. + pid_machine = _stage_summarize(s, run_id, sources, machine_text) + r_machine = lifecycle.decide(s, emb, pid_machine, "approve", now=T_A, + decided_by="console") + machine_summary = s.get_fact(r_machine["summary_id"]) + machine_salience = machine_summary.salience + + # (2) Edit-approve of a second proposal with the SAME text → summary scored as + # source='user' (human provenance). + pid_edit = _stage_summarize(s, run_id, sources, "stub text that gets overridden") + r_edit = lifecycle.decide( + s, emb, pid_edit, "edit", now=T_A + DAY, decided_by="console", + edited_text=machine_text, + ) + edited_summary = s.get_fact(r_edit["summary_id"]) + + # Human provenance recorded on the proposal row. + row = s.get_proposal(pid_edit) + assert row["status"] == "edited" + assert row["edited_text"] == machine_text + # The edited summary uses the edited text. + assert edited_summary.fact_text == machine_text + # source='user' beats source='maintenance' for the SAME text → higher salience. + assert edited_summary.salience > machine_salience + s.close() + + +# ── summarize apply end-to-end ───────────────────────────────────────────────── +def test_summarize_apply_end_to_end(db_path, emb): + """SUMMARIZE approve: maintenance episode + summary fact (valid_at=t_a, tier hot, + record_kind summary, is_inference) + derivation rows + sources demoted cold; a + SECOND approve supersedes the previous summary at t_a (§4.3).""" + s = _seed(_open(db_path, emb), emb) + sources = [ + _add(s, f"historical note {i} on topic", predicate="notes", + valid_at=T_A - (200 - i) * DAY) + for i in range(5) + ] + run_id = _run_id(s) + pid = _stage_summarize(s, run_id, sources, "Summary line one") + + r = lifecycle.decide(s, emb, pid, "approve", now=T_A, decided_by="console") + assert r["outcome"] == "applied" and r["kind"] == "summarize" + + summary = s.get_fact(r["summary_id"]) + assert summary.predicate == "summary" + assert summary.record_kind == "summary" + assert summary.is_inference == 1 + assert summary.tier == "hot" + assert summary.valid_at == T_A # t_a — NEVER backdated + assert summary.valid_to is None + assert summary.is_latest == 1 + + # Maintenance episode exists with source='maintenance'. + ep = s._db.execute( + "SELECT source, raw FROM episode WHERE id=?", (r["episode_id"],) + ).fetchone() + assert ep["source"] == "maintenance" + assert json.loads(ep["raw"])["kind"] == "summarize" + + # Derivation rows: one per source. + drows = s._db.execute( + "SELECT source_id FROM fact_derivation WHERE summary_id=?", (summary.id,) + ).fetchall() + assert {d["source_id"] for d in drows} == {f.id for f in sources} + + # Sources demoted to cold; still is_latest=1 and as-of visible. + for f in sources: + row = s.get_fact(f.id) + assert row.tier == "cold" + assert row.is_latest == 1 + assert row.valid_to is None + + # A SECOND summarize approve supersedes the previous summary at t_a. + t_a2 = T_A + 10 * DAY + pid2 = _stage_summarize(s, run_id, sources, "Summary line two") + r2 = lifecycle.decide(s, emb, pid2, "approve", now=t_a2, decided_by="console") + assert r2["superseded_prev_summary_id"] == summary.id + old = s.get_fact(summary.id) + assert old.is_latest == 0 + assert old.valid_to == t_a2 + assert old.superseded_by == r2["summary_id"] + # No misfire: the old summary is not a SOURCE, so the staleness cascade did not + # invalidate the new summary — the new one is live. + assert s.get_fact(r2["summary_id"]).is_latest == 1 + s.close() + + +def test_summarize_apply_valid_at_not_backdated_grid(db_path, emb): + """The summary appears in NO past window: it is invisible for every T < t_a and + visible only at T >= t_a (valid_at=t_a). A guard against a backdated valid_at.""" + s = _seed(_open(db_path, emb), emb) + sources = [ + _add(s, f"aged note {i}", predicate="notes", valid_at=T_A - (200 - i) * DAY) + for i in range(5) + ] + run_id = _run_id(s) + pid = _stage_summarize(s, run_id, sources, "Consolidated") + r = lifecycle.decide(s, emb, pid, "approve", now=T_A, decided_by="console") + sid = r["summary_id"] + + for T in (T_A - 150 * DAY, T_A - 1, T_A - 0 - 1): + assert sid not in _visible_at(s, T) + assert sid in _visible_at(s, T_A) + assert sid in _visible_at(s, T_A + 50 * DAY) + s.close() + + +# ── evict apply + promote round-trip ─────────────────────────────────────────── +def test_evict_apply_then_promote_round_trip(db_path, emb): + """EVICT approve demotes to cold; the PROMOTE verb restores hot; a PROMOTE + decision on an evict proposal rejects it and promotes (§4.4).""" + s = _seed(_open(db_path, emb), emb) + fact = _add(s, "trivial ancient note", predicate="notes", salience=1.0) + run_id = _run_id(s) + pid = _stage_evict(s, run_id, fact) + + r = lifecycle.decide(s, emb, pid, "approve", now=T_A, decided_by="console") + assert r["outcome"] == "applied" + assert s.get_fact(fact.id).tier == "cold" + + # Direct promote verb restores hot. + pr = lifecycle.promote_fact(s, fact.id, now=T_A + DAY) + assert pr["outcome"] == "promoted" + assert s.get_fact(fact.id).tier == "hot" + + # A PROMOTE *decision* on a fresh evict proposal: rejects it AND promotes. + fact2 = _add(s, "another ancient note", predicate="notes", salience=1.0, tier="cold") + pid2 = _stage_evict(s, run_id, fact2) + dr = lifecycle.decide(s, emb, pid2, "promote", now=T_A + 2 * DAY, decided_by="console") + assert dr["outcome"] == "promoted" + assert s.get_proposal(pid2)["status"] == "rejected" + assert s.get_fact(fact2.id).tier == "hot" + s.close() + + +# ── dim-mismatch embedder refuses to apply ───────────────────────────────────── +def test_dim_mismatch_embedder_refuses_summarize_apply(db_path, emb): + """A summarize apply with an embedder whose dim != the namespace's baked dims is + refused with a clear error, BEFORE any spine write (§4.3 apply-ownership).""" + s = _seed(_open(db_path, emb), emb) + sources = [ + _add(s, f"note {i}", predicate="notes", valid_at=T_A - (200 - i) * DAY) + for i in range(5) + ] + run_id = _run_id(s) + pid = _stage_summarize(s, run_id, sources, "Consolidated") + + wrong = FakeEmbedder(dim=512, coarse_dim=256) # dim mismatch (store baked at 768) + hash_before = _fact_hash(s) + with pytest.raises(ValueError, match="dim"): + lifecycle.decide(s, wrong, pid, "approve", now=T_A, decided_by="console") + # Nothing written; the proposal stays pending. + assert s.get_proposal(pid)["status"] == "pending" + assert _fact_hash(s) == hash_before + s.close() + + +# ── THE HEADLINE: apply-path as-of grid re-run (spec §10.1) ──────────────────── +def _grid_corpus(store): + """Backfills (id-order != valid_at-order), a functional supersession, a + multivalued slot with a near-dup pair, plus an evict candidate. Returns notable + facts. Ages are wide (>90d) so SUMMARIZE/EVICT age gates pass under a tuned cfg.""" + # Functional slot 'works_at': acme (t0) superseded by globex (t1). + t0 = T_A - 500 * DAY + t1 = T_A - 300 * DAY + acme = _add(store, "user works at acme", predicate="works_at", valid_at=t0, + valid_to=t1, is_latest=0) + globex = _add(store, "user works at globex", predicate="works_at", valid_at=t1) + store._db.execute("UPDATE fact SET superseded_by=? WHERE id=?", (globex.id, acme.id)) + store._db.commit() + + # Multivalued slot 'likes': a NEAR-dup pair (distinct text) BACKFILLED so the + # later-ingested row is older in world-time. + jazz = _add(store, "likes jazz music a lot", predicate="likes", + valid_at=T_A - 400 * DAY, access_count=1, last_access=T_A - 350 * DAY) + jazz2 = _add(store, "enjoys jazz very much indeed", predicate="likes", + valid_at=T_A - 450 * DAY, access_count=3, last_access=T_A - 20 * DAY) + + # Enough aged 'notes' facts for a SUMMARIZE cluster (>= min_cluster) on the subject. + notes = [ + _add(store, f"aged project note number {i}", predicate="notes", + valid_at=T_A - (300 - i) * DAY, salience=5.0) + for i in range(5) + ] + + # A low-value aged fact for an EVICT proposal (not in the strict auto-band because + # access_count != 0, so it PROPOSES rather than auto-demotes). + evictable = _add(store, "low value aged remark", predicate="chatter", + valid_at=T_A - 300 * DAY, salience=1.0, access_count=2, + last_access=T_A - 300 * DAY) + + return { + "acme": acme, "globex": globex, "jazz": jazz, "jazz2": jazz2, + "notes": notes, "evictable": evictable, + } + + +def test_apply_path_as_of_grid_rerun(db_path, emb): + """THE HEADLINE (§10.1): stage proposals via a REAL runner pass, approve one of + each kind (summarize, dedup_near, evict), and assert the store predicate + (is_latest_only=False) is BIT-IDENTICAL for every T < t_a, with only the intended + deltas at T >= t_a: + - the summary fact appears ONLY at T >= t_a (valid_at=t_a, never backdated); + - the retired near-dup loser stays predicate-visible (valid_to untouched); + - no inverted intervals. + """ + from lean_memory.maintain.runner import MaintenanceRunner + + s = _seed(_open(db_path, emb), emb) + facts = _grid_corpus(s) + + # A tuned config so the FakeEmbedder corpus actually stages each kind: tau_near=0 + # (any distinct same-slot pair proposes), min_cluster=2, small age floor, a high + # evict_threshold so the low-value fact proposes, and a strict auto-band it evades + # (access_count != 0). Work thresholds are moot: the first run is always over. + cfg = MaintenanceConfig( + tau_near=0.0, min_cluster=2, age_floor_days=1, + evict_threshold=0.9, auto_evict_salience=0.0, auto_evict_age_days=100000, + proposal_budget_per_run=100, + ) + + # Snapshot the predicate id-set over a T grid BEFORE any apply. + grid = [T_A - t * DAY for t in (480, 460, 430, 410, 350, 250, 100, 1)] + before_sets = {T: _visible_at(s, T) for T in grid} + + # ── Real runner pass stages the proposals (apply=False semantics: staging only) ── + runner = MaintenanceRunner(s, "ns", config=cfg, now_ms=lambda: T_A) + report = runner.run() + assert report.status == "ok" + + pending = s.list_proposals("ns", status="pending", limit=1000) + by_kind: dict[str, list[dict]] = {} + for p in pending: + by_kind.setdefault(p["kind"], []).append(p) + assert "summarize" in by_kind, "runner staged no summarize proposal" + assert "dedup_near" in by_kind, "runner staged no dedup_near proposal" + assert "evict" in by_kind, "runner staged no evict proposal" + + # Pick NON-CONFLICTING proposals of each kind (the propose transforms all run over + # the same pre-transform snapshot, so a fact can appear in several proposals; we + # approve a disjoint trio so the three applies don't invalidate each other): + # - summarize : the one proposal (its sources are the notes + jazz pair); + # - dedup_near: the pair on the 'likes' slot (jazz / jazz2); + # - evict : the 'chatter' fact, referenced by no other approved proposal. + dn = _pick(by_kind["dedup_near"], + lambda p: set(json.loads(p["payload_json"])["fact_ids"]) + == {facts["jazz"].id, facts["jazz2"].id}) + ev = _pick(by_kind["evict"], + lambda p: json.loads(p["payload_json"])["fact_id"] == facts["evictable"].id) + sm = by_kind["summarize"][0] + + # Approve at apply-time t_a. summarize demotes its notes sources to cold (still + # is_latest=1), which does NOT invalidate the jazz dedup or the chatter evict. + t_a = T_A + 1 * DAY + r_sum = lifecycle.decide(s, emb, sm["id"], "approve", now=t_a, decided_by="console") + r_dn = lifecycle.decide(s, emb, dn["id"], "approve", now=t_a, decided_by="console") + r_ev = lifecycle.decide(s, emb, ev["id"], "approve", now=t_a, decided_by="console") + assert r_sum["outcome"] == "applied" + assert r_dn["outcome"] == "applied" + assert r_ev["outcome"] == "applied" + summary_id = r_sum["summary_id"] + dn_loser = r_dn["loser_id"] + + # ── Invariance: the store predicate is BIT-IDENTICAL for every T < t_a ── + for T in grid: + assert T < t_a + assert _visible_at(s, T) == before_sets[T], f"predicate changed at T={T}" + + # ── Intended deltas at T >= t_a ── + # The summary is invisible before t_a and visible at/after it. + assert summary_id not in _visible_at(s, t_a - 1) + assert summary_id in _visible_at(s, t_a) + # The retired near-dup loser keeps predicate visibility (valid_to untouched): + # visible at any T after its own valid_at, INCLUDING at/after t_a. + loser_row = s.get_fact(dn_loser) + assert loser_row.is_latest == 0 # dropped from the latest surface + assert loser_row.valid_to is None # but as-of-visible + assert dn_loser in _visible_at(s, t_a + 10 * DAY) + + # ── No inverted intervals anywhere post-apply ── + inverted = s._db.execute( + "SELECT id FROM fact WHERE valid_to IS NOT NULL AND valid_to <= valid_at" + ).fetchall() + assert inverted == [], "no inverted (valid_to <= valid_at) intervals post-apply" + s.close() + + +def test_apply_path_grid_fails_if_summary_backdated(db_path, emb, monkeypatch): + """Fault-injection guard for §10.1: if the summarize apply BACKDATED valid_at (used + an old source time instead of t_a), the summary would leak into a past window and + the T < t_a invariance would break. We patch the apply to backdate and assert the + grid check would catch it — then the real code (unpatched) does not backdate.""" + from lean_memory.maintain import lifecycle as lc + + s = _seed(_open(db_path, emb), emb) + sources = [ + _add(s, f"aged note {i}", predicate="notes", valid_at=T_A - (200 - i) * DAY) + for i in range(3) + ] + run_id = _run_id(s) + pid = _stage_summarize(s, run_id, sources, "Consolidated") + + grid = [T_A - t * DAY for t in (150, 100, 50, 1)] + before = {T: _visible_at(s, T) for T in grid} + + # Fault-inject: force the summary's valid_at to a BACKDATED source time. + backdate = T_A - 200 * DAY + orig_add_fact = s.add_fact + + def _backdating_add_fact(fact, full, coarse): + if fact.record_kind == "summary": + fact.valid_at = backdate # the bug we are guarding against + return orig_add_fact(fact, full, coarse) + + monkeypatch.setattr(s, "add_fact", _backdating_add_fact) + t_a = T_A + DAY + lc.decide(s, emb, pid, "approve", now=t_a, decided_by="console") + + # With the injected backdate, the past-window predicate CHANGED — the grid test + # would have caught the regression. + changed = any(_visible_at(s, T) != before[T] for T in grid) + assert changed, "fault injection did not perturb a past window — test is inert" + s.close() + + +# ── Memory façade wiring (maintain / review_queue / decide / promote) ────────── +def _full_db_hash(store): + out = {} + for t in ("fact", "episode", "entity", "maintenance_proposal", + "maintenance_run", "fact_derivation"): + out[t] = [tuple(r) for r in store._db.execute(f"SELECT * FROM {t}").fetchall()] + return hashlib.sha256(json.dumps(out, default=str, sort_keys=True).encode()).hexdigest() + + +def _facade_corpus(store, now): + """A controlled corpus written straight to the namespace file so the façade tests + are deterministic regardless of the offline extraction stub. Timestamps are anchored + to `now` (the real wall clock the façade's runner uses) so the age gates pass — + every fact is 300+ days old relative to the run's `now`.""" + for i in range(5): + _add(store, f"aged note {i} about the trip", predicate="notes", + valid_at=now - (300 - i) * DAY, salience=5.0) + _add(store, "low value aged remark", predicate="chatter", + valid_at=now - 300 * DAY, salience=1.0, access_count=2, + last_access=now - 300 * DAY) + + +def test_facade_dry_run_writes_nothing(db_path, emb): + """Memory.maintain(apply=False) computes the report but mutates NOTHING — the full + DB is byte-identical, and no lease/proposal row is written (§7.1 dry-run).""" + from lean_memory.memory import Memory + + # Seed the namespace file the Memory will open. + from lean_memory.types import now_ms + + now = now_ms() + s = _seed(_open(db_path, emb), emb) + _facade_corpus(s, now) + before = _full_db_hash(s) + s.close() + + m = Memory(root=db_path.parent, embedder=emb) + # The seeded facts live in namespace 'ns'; point the façade at the same file. + ns = "ns" + target = m._namespace_path(ns) + import shutil + + shutil.copy(db_path, target) + + cfg = MaintenanceConfig(tau_near=0.0, min_cluster=2, age_floor_days=1, + min_new_facts=1, proposal_budget_per_run=100) + rep = m.maintain(ns, config=cfg, apply=False) + assert rep.status == "ok" + assert rep.run_id is None # dry-run takes no lease + + check = _open(target, emb) + assert _full_db_hash(check) == before, "dry-run mutated the DB" + check.close() + m.close() + + +def test_facade_apply_then_queue_then_decide_and_promote(db_path, emb): + """The end-to-end façade path: maintain(apply=True) stages, review_queue groups by + entity, decide(approve) applies, promote() restores hot.""" + from lean_memory.memory import Memory + + from lean_memory.types import now_ms + + now = now_ms() + s = _seed(_open(db_path, emb), emb) + _facade_corpus(s, now) + s.close() + + m = Memory(root=db_path.parent, embedder=emb) + ns = "ns" + import shutil + + shutil.copy(db_path, m._namespace_path(ns)) + + cfg = MaintenanceConfig(tau_near=0.0, min_cluster=2, age_floor_days=1, + evict_threshold=0.9, auto_evict_salience=0.0, + auto_evict_age_days=100000, min_new_facts=1, + proposal_budget_per_run=100) + rep = m.maintain(ns, config=cfg, apply=True) + assert rep.status == "ok" and rep.run_id is not None + + groups = m.review_queue(ns, limit=50) + assert groups, "review_queue returned no groups" + # Every group is JSON-friendly with a proposals list carrying parsed payloads. + all_props = [p for g in groups for p in g["proposals"]] + assert all_props + assert all("payload" in p for p in all_props) + + # Approve an evict proposal through the façade → the fact goes cold. + evict = next(p for p in all_props if p["kind"] == "evict") + fact_id = evict["payload"]["fact_id"] + r = m.decide(evict["id"], "approve", namespace=ns) + assert r["outcome"] == "applied" + assert m._maintenance_store(ns).get_fact(fact_id).tier == "cold" + + # Promote it back through the façade. + pr = m.promote(fact_id, namespace=ns) + assert pr["outcome"] == "promoted" + assert m._maintenance_store(ns).get_fact(fact_id).tier == "hot" + m.close() + + +def test_facade_review_queue_lazy_timeout_expiry(db_path, emb): + """review_queue expires any pending proposal past its expires_at (timeout) and does + not return it (§5 lazy expiry).""" + from lean_memory.memory import Memory + + ns = "ns" # the seeded facts + proposal live in namespace 'ns' + s = _seed(_open(db_path, emb), emb) + fact = _add(s, "trivial ancient note", predicate="notes", salience=1.0) + run_id = s.create_run(ns, "cli", T_A, "cfg") + # Stage a proposal (in namespace `ns`) that already expired. + payload = {"fact_id": fact.id, "fact_text": fact.fact_text, "value": 0.1, + "salience": 1.0, "access_count": 0, "evidence_backend": "score"} + # review_queue lazily expires against the real wall clock (now_ms), so the proposal + # must expire in the REAL past — created_at/expires_at at epoch ~0 do exactly that. + pid = s.stage_proposal( + run_id=run_id, namespace=ns, kind="evict", + payload_json=json.dumps(payload, sort_keys=True), + created_at=1, expires_at=1, # long expired against any real now_ms() + evidence_backend="score", + ) + s.close() + + m = Memory(root=db_path.parent, embedder=emb) + import shutil + + shutil.copy(db_path, m._namespace_path(ns)) + + groups = m.review_queue(ns, limit=50) + assert groups == [] # the expired proposal is not surfaced + row = m._maintenance_store(ns).get_proposal(pid) + assert row["status"] == "expired" and row["expiry_reason"] == "timeout" + m.close() + diff --git a/tests/test_tier_retrieval.py b/tests/test_tier_retrieval.py new file mode 100644 index 0000000..7ec6b90 --- /dev/null +++ b/tests/test_tier_retrieval.py @@ -0,0 +1,167 @@ +"""Tier-filter retrieval matrix (design spec §8, §10.6). + +The tier filter drops cold (maintenance-demoted) facts from the DEFAULT hot search +surface ONLY: + - default latest-mode: cold facts are absent unless include_cold=True; + - as_of queries NEVER filter tier (historical reads see everything); + - byte-identical behavior when every row is tier='hot' (regression pin); + - dense and sparse arms agree on the tier flag (both read what set_tier writes). + +All offline on FakeEmbedder — no model, no network. +""" + +from __future__ import annotations + +import pytest + +from lean_memory.embed.fake import FakeEmbedder +from lean_memory.store.sqlite_store import SqliteStore +from lean_memory.types import Entity, Episode, Fact + +T0 = 1_000_000_000_000 + + +@pytest.fixture +def store(tmp_path): + emb = FakeEmbedder() + s = SqliteStore(tmp_path / "tier.db", dim=emb.dim, coarse_dim=emb.coarse_dim) + ep = Episode(namespace="ns", raw="seed", t_ref=T0) + s.add_episode(ep) + s._emb = emb + s._ep = ep + s._subj = s.upsert_entity(Entity(namespace="ns", name="user", type="person")) + yield s + s.close() + + +def _add(store, text, *, valid_at=T0, predicate="about", tier="hot", is_latest=1): + f = Fact( + namespace="ns", subject_id=store._subj.id, predicate=predicate, + fact_text=text, valid_at=valid_at, episode_id=store._ep.id, + tier=tier, is_latest=is_latest, salience=5.0, + ) + full, coarse = store._emb.embed_with_coarse(text) + store.add_fact(f, full, coarse) + return f + + +def _dense(store, query, **kw): + q = store._emb.embed_one(query) + from lean_memory.embed.base import matryoshka_truncate + + qc = matryoshka_truncate(q, store._emb.coarse_dim) + return {fid for fid, _ in store.dense_search(qc, q, 20, **kw)} + + +def _sparse(store, query, **kw): + return {fid for fid, _ in store.sparse_search(query, 20, **kw)} + + +# ── default surface drops cold; include_cold opts back in ────────────────────── +def test_cold_fact_absent_from_default_search_present_with_include_cold(store): + hot = _add(store, "the sky is blue today") + cold = _add(store, "the sky is blue today anciently", tier="cold") + + # Default latest-mode: cold is filtered from BOTH arms. + assert cold.id not in _dense(store, "sky blue") + assert cold.id not in _sparse(store, "sky blue") + assert hot.id in _dense(store, "sky blue") + assert hot.id in _sparse(store, "sky blue") + + # include_cold=True: cold reappears on both arms. + assert cold.id in _dense(store, "sky blue", include_cold=True) + assert cold.id in _sparse(store, "sky blue", include_cold=True) + + +def test_set_tier_then_search_round_trip(store): + """set_tier writes both surfaces; a demoted fact then drops from the default arms + and returns under include_cold — the two-surface sync (§8/§10.6).""" + f = _add(store, "quantum widget calibration notes") + assert f.id in _dense(store, "quantum widget") + assert f.id in _sparse(store, "quantum widget") + + with store.batch(): + store.set_tier(f.id, "cold") + + assert f.id not in _dense(store, "quantum widget") + assert f.id not in _sparse(store, "quantum widget") + assert f.id in _dense(store, "quantum widget", include_cold=True) + assert f.id in _sparse(store, "quantum widget", include_cold=True) + + +# ── as_of NEVER filters tier ─────────────────────────────────────────────────── +def test_as_of_ignores_tier(store): + """A cold fact valid at T is visible in an as_of=T search regardless of tier — the + as_of surface never filters tier (§8).""" + cold = _add(store, "historical cold record about widgets", valid_at=T0, tier="cold") + T_after = T0 + 1 + + assert cold.id in _dense(store, "historical widgets", as_of=T_after) + assert cold.id in _sparse(store, "historical widgets", as_of=T_after) + # include_cold is irrelevant on as_of (already unfiltered) — same result. + assert cold.id in _dense(store, "historical widgets", as_of=T_after, include_cold=True) + assert cold.id in _sparse(store, "historical widgets", as_of=T_after, include_cold=True) + + +# ── dense/sparse arm agreement on the tier flag ──────────────────────────────── +def test_dense_sparse_arm_agreement_on_tier(store): + """Both arms make the SAME tier decision for the same fact across the matrix.""" + f = _add(store, "arm agreement widget probe", tier="cold") + T_after = T0 + 1 + + # Default: both exclude. + assert (f.id in _dense(store, "arm agreement widget")) == ( + f.id in _sparse(store, "arm agreement widget") + ) == False + # include_cold: both include. + assert (f.id in _dense(store, "arm agreement widget", include_cold=True)) == ( + f.id in _sparse(store, "arm agreement widget", include_cold=True) + ) == True + # as_of: both include. + assert (f.id in _dense(store, "arm agreement widget", as_of=T_after)) == ( + f.id in _sparse(store, "arm agreement widget", as_of=T_after) + ) == True + + +# ── byte-identical default when nothing is cold (regression pin) ─────────────── +def test_byte_identical_default_when_nothing_cold(store): + """When every row is tier='hot', default search results are exactly what they were + before the tier filter existed — the filter is a no-op (§8 regression pin).""" + ids = [_add(store, f"hot fact number {i} about widgets").id for i in range(5)] + + dense_default = _dense(store, "widgets") + sparse_default = _sparse(store, "widgets") + # include_cold on an all-hot store returns the SAME set (nothing cold to add). + assert _dense(store, "widgets", include_cold=True) == dense_default + assert _sparse(store, "widgets", include_cold=True) == sparse_default + # All the hot facts are present. + assert set(ids) <= dense_default + assert set(ids) <= sparse_default + + +# ── the full as_of × include_cold × tier grid (§10.6) ────────────────────────── +def test_asof_include_cold_tier_matrix(store): + """The matrix: a cold fact valid at T_valid, queried across + {default, include_cold, as_of} — expected presence per §8.""" + T_valid = T0 + T_query = T0 + 1 + cold = _add(store, "matrix cold widget entry", valid_at=T_valid, tier="cold") + hot = _add(store, "matrix hot widget entry", valid_at=T_valid, tier="hot") + + # (default, cold) -> absent ; (default, hot) -> present + d = _dense(store, "matrix widget") + s = _sparse(store, "matrix widget") + assert cold.id not in d and cold.id not in s + assert hot.id in d and hot.id in s + + # (include_cold, cold) -> present ; (include_cold, hot) -> present + d = _dense(store, "matrix widget", include_cold=True) + s = _sparse(store, "matrix widget", include_cold=True) + assert cold.id in d and cold.id in s + assert hot.id in d and hot.id in s + + # (as_of, cold) -> present ; (as_of, hot) -> present (tier ignored) + d = _dense(store, "matrix widget", as_of=T_query) + s = _sparse(store, "matrix widget", as_of=T_query) + assert cold.id in d and cold.id in s + assert hot.id in d and hot.id in s From 532b3755596b4ba39921cabd1433af5ef7455fb6 Mon Sep 17 00:00:00 2001 From: Wuesteon Date: Thu, 16 Jul 2026 20:58:17 +0800 Subject: [PATCH 09/14] =?UTF-8?q?fix(console):=20bump=20sanitizer=20finger?= =?UTF-8?q?print=20for=20Task=206=20fa=C3=A7ade?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Memory._maintenance_store() reuses the identical namespace-sanitize expression, adding a third line matching the tripwire's pattern. Verified semantics unchanged (same _SAFE_NS regex, same 'or "default"' fallback), so the console config.py mirror stays valid — this is the tripwire's designed verify-then-bump workflow. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_0198CpP3tf1SbtDGaRX2NWvJ --- console/src/lean_memory_console/inspect_sql.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/console/src/lean_memory_console/inspect_sql.py b/console/src/lean_memory_console/inspect_sql.py index 7df2ba9..72118cc 100644 --- a/console/src/lean_memory_console/inspect_sql.py +++ b/console/src/lean_memory_console/inspect_sql.py @@ -314,5 +314,8 @@ def list_entities(db_path: Path, page: int = 1, page_size: int = 50) -> dict: "a6c7f41188a196929ff4e7c257a181c100af9f0f7091aa93d428a9a7c0eec8ef" ) EXPECTED_SANITIZER_FINGERPRINT = ( - "dfef8699dd9519f2ef6592be577d25fdb2ae761e298d7063ced3323a14e2cf77" + # Bumped for WP10a Task 6: Memory._maintenance_store() reuses the identical + # `_SAFE_NS.sub("_", namespace) or "default"` expression (memory.py:298), + # adding a third matching line; sanitizer semantics unchanged, mirror valid. + "973f203a76ab1b535ae8e81dcb830145c92bf481cf4c815b5fbca0044e5d044c" ) From d2229aeb13edd4b6f5a08e703338bfdc16a59435 Mon Sep 17 00:00:00 2001 From: Wuesteon Date: Thu, 16 Jul 2026 22:53:11 +0800 Subject: [PATCH 10/14] feat(maintain): lean-memory-maintain CLI + memory_clear lease-refusal MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 7 (spec §6.1, §7.1, §7.3). CLI (new maintain/cli.py + pyproject script): --root [--namespace] [--apply] [--auto-only] [--json]. Dry-run is the default (zero writes, no lease); --apply runs the auto band + stages proposals; --auto-only stages nothing. Reuses Memory.maintain() per namespace (dedicated 5000 ms store). Human report to its own stdout; --json for machines. --root falls back to $LM_DATA_ROOT. --auto-only threads down (default False) through run_transforms / runner.run / Memory.maintain, skipping the propose phases — all existing behavior/tests preserved. memory_clear now refuses (returns an explanatory string, does not raise) while a live fresh maintenance lease is held, via the shared runner.live_lease_is_fresh (reuses _heartbeat_is_fresh + _STALE_FLOOR_S — staleness rule not duplicated); a stale/absent lease clears as before. The runner stops cleanly at a batch boundary when its namespace file is unlinked mid-run (os.path.exists on store.path → internal _NamespaceCleared signal → RunReport aborted_file_gone; can't mask real errors). §7.3 residual sliver documented in the tool docstring. Tests (new test_maintenance_cli.py, 12): cross-process CLI --apply vs a live in-process writer interleave with no unhandled 'database is locked'; dry-run writes nothing; --auto-only stages nothing but merges; --json fields; missing-root nonzero; memory_clear refusal fresh vs stale; runner mid-run unlink stop. Full suite 263 passed, 1 skipped. Co-Authored-By: Claude Fable 5 --- pyproject.toml | 1 + src/lean_memory/maintain/__init__.py | 3 +- src/lean_memory/maintain/cli.py | 196 ++++++++++++ src/lean_memory/maintain/runner.py | 83 ++++- src/lean_memory/maintain/transforms.py | 61 ++-- src/lean_memory/mcp_server.py | 36 ++- src/lean_memory/memory.py | 7 +- tests/test_maintenance_cli.py | 401 +++++++++++++++++++++++++ 8 files changed, 752 insertions(+), 36 deletions(-) create mode 100644 src/lean_memory/maintain/cli.py create mode 100644 tests/test_maintenance_cli.py diff --git a/pyproject.toml b/pyproject.toml index 8d79861..98432e6 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -76,6 +76,7 @@ dev = [ [project.scripts] lean-memory-mcp = "lean_memory.mcp_server:main" +lean-memory-maintain = "lean_memory.maintain.cli:main" [project.urls] Homepage = "https://github.com/Wuesteon/lean-memory" diff --git a/src/lean_memory/maintain/__init__.py b/src/lean_memory/maintain/__init__.py index 5490000..774640f 100644 --- a/src/lean_memory/maintain/__init__.py +++ b/src/lean_memory/maintain/__init__.py @@ -17,7 +17,7 @@ Summarizer, default_summarizer, ) -from .runner import MaintenanceRunner, RunReport +from .runner import MaintenanceRunner, RunReport, live_lease_is_fresh from .transforms import ( Merge, StagedProposal, @@ -53,4 +53,5 @@ "TransformReport", "MaintenanceRunner", "RunReport", + "live_lease_is_fresh", ] diff --git a/src/lean_memory/maintain/cli.py b/src/lean_memory/maintain/cli.py new file mode 100644 index 0000000..9f0fdba --- /dev/null +++ b/src/lean_memory/maintain/cli.py @@ -0,0 +1,196 @@ +"""`lean-memory-maintain` — the primary sleep-time maintenance trigger (spec §6.1). + + lean-memory-maintain --root PATH [--namespace NS] [--apply] [--auto-only] [--json] + +Runs the offline maintenance job over one namespace, or every namespace under a +root, from its OWN process — so cross-process safety comes from the lease + +busy_timeout + short batches (§7), never from an in-process gateway. That is why +this lives in the CORE package beside `lean-memory-mcp`, with no console dependency +(§6.1). + +Semantics (spec §6.1, §7.1): + * `--root` is required, OR defaults to $LM_DATA_ROOT when that env var is set (the + spec's `--root $LM_DATA_ROOT` idiom). With neither, the CLI errors. + * DRY-RUN IS THE DEFAULT. Without `--apply`, every namespace is a report-only pass: + zero writes, no lease row, no proposals — it just computes the would-do report. + * `--apply` executes the auto band (dedup-exact, evict auto) AND stages proposals. + * `--auto-only` (only meaningful with `--apply`) runs the auto band and stages + NOTHING — the provably-safe transforms only. + * Without `--namespace`, every `*.db` file under the root is processed in turn. + * Each namespace opens a DEDICATED maintenance store at busy_timeout_ms=5000 via + `Memory.maintain()` (§7.1) — reused verbatim rather than re-driving the runner, so + the store construction, dim wiring, lease, and cleanup match the in-process path + exactly. + * Human-readable per-namespace report to stdout (this is the CLI's OWN process — its + stdout is free, unlike the MCP server's JSON-RPC channel). `--json` emits one + machine-readable object with per-namespace reports and stable keys. + * Exit 0 on success, including a below-threshold no-op and a "skipped: lease held" + line (a lease held by another run is a NORMAL outcome, not an error). Nonzero only + on a real error: an unusable root, or an unexpected exception during a run. +""" + +from __future__ import annotations + +import argparse +import json +import os +import sys +from pathlib import Path +from typing import Optional, Sequence + +from ..memory import Memory +from .config import MaintenanceConfig +from .runner import RunReport + + +def _resolve_root(arg_root: Optional[str]) -> Path: + """The data root: --root wins; else $LM_DATA_ROOT (the spec's `--root $LM_DATA_ROOT` + idiom, §6.1); else an error. Raises SystemExit(2) with a clear message on neither.""" + raw = arg_root if arg_root is not None else os.environ.get("LM_DATA_ROOT") + if not raw: + raise SystemExit( + "error: --root is required (or set LM_DATA_ROOT). " + "Point it at the directory holding the per-namespace .db files." + ) + root = Path(raw).expanduser() + if not root.is_dir(): + raise SystemExit(f"error: root is not a directory: {root}") + return root + + +def _discover_namespaces(root: Path) -> list[str]: + """Every namespace under `root`, derived from its `.db` file (BET 4: + one SQLite file per namespace). The filename stem IS the on-disk (sanitized) + namespace, so it round-trips through Memory's namespace→path mapping unchanged. + WAL/SHM sidecars (`*.db-wal`, `*.db-shm`) are NOT separate namespaces — glob only + the exact `*.db` suffix. Sorted for deterministic output/tests.""" + return sorted(p.stem for p in root.glob("*.db") if p.suffix == ".db") + + +def _report_to_dict(namespace: str, report: RunReport, *, apply: bool, auto_only: bool) -> dict: + """A stable, JSON-able summary of one namespace's RunReport (`--json` payload).""" + tr = report.transform_report + return { + "namespace": namespace, + "status": report.status, # 'ok' | 'skipped' + "mode": ("auto-only" if auto_only else "apply") if apply else "dry-run", + "skipped_reason": report.skipped_reason, + "below_threshold": report.below_threshold, + "aborted_file_gone": report.aborted_file_gone, + "run_id": report.run_id, + "config_hash": report.config_hash, + "merges": len(tr.merges) if tr else 0, + "demoted": len(tr.demoted_ids) if tr else 0, + "staged": len(tr.proposals) if tr else 0, + "dropped_proposals": tr.dropped_proposals if tr else 0, + "threshold_stats": report.threshold_stats, + } + + +def _human_line(d: dict) -> str: + """A one-line human summary of a namespace's report dict (stdout, non-JSON mode).""" + ns = d["namespace"] + if d["status"] == "skipped": + return f"{ns}: skipped ({d['skipped_reason']})" + if d["below_threshold"]: + return f"{ns}: no-op (below threshold)" + tail = "" + if d["aborted_file_gone"]: + tail = " [stopped: namespace cleared mid-run]" + return ( + f"{ns}: {d['mode']} " + f"merges={d['merges']} demoted={d['demoted']} " + f"staged={d['staged']} dropped={d['dropped_proposals']}{tail}" + ) + + +def _parse_args(argv: Optional[Sequence[str]]) -> argparse.Namespace: + p = argparse.ArgumentParser( + prog="lean-memory-maintain", + description="Run the sleep-time maintenance job (dry-run by default).", + ) + p.add_argument( + "--root", + default=None, + help="Data root holding per-namespace .db files (defaults to $LM_DATA_ROOT).", + ) + p.add_argument( + "--namespace", + default=None, + help="Maintain only this namespace (default: every namespace under the root).", + ) + p.add_argument( + "--apply", + action="store_true", + help="Execute the auto band AND stage proposals (default: dry-run, no writes).", + ) + p.add_argument( + "--auto-only", + action="store_true", + dest="auto_only", + help="With --apply: run ONLY the auto band; stage nothing.", + ) + p.add_argument( + "--json", + action="store_true", + dest="as_json", + help="Emit one machine-readable JSON object with per-namespace reports.", + ) + return p.parse_args(argv) + + +def main(argv: Optional[Sequence[str]] = None) -> int: + """Console-script / module entry point. `argv=None` reads sys.argv (testability). + + Returns the process exit code (0 success, nonzero error). `_resolve_root` raises + SystemExit(2) on a bad/missing root — argparse's own convention — which the console + script surfaces as a nonzero exit with the message on stderr. + """ + args = _parse_args(argv) + root = _resolve_root(args.root) # SystemExit(2) on bad/missing root + + if args.namespace is not None: + namespaces = [args.namespace] + else: + namespaces = _discover_namespaces(root) + + config = MaintenanceConfig() + mem = Memory(root=root) + results: list[dict] = [] + exit_code = 0 + try: + for ns in namespaces: + try: + report = mem.maintain( + ns, config=config, apply=args.apply, auto_only=args.auto_only, + ) + except Exception as exc: # a genuine run error — report it, keep going. + exit_code = 1 + results.append({ + "namespace": ns, + "status": "error", + "error": f"{type(exc).__name__}: {exc}", + }) + continue + results.append( + _report_to_dict(ns, report, apply=args.apply, auto_only=args.auto_only) + ) + finally: + mem.close() + + if args.as_json: + print(json.dumps({"root": str(root), "namespaces": results}, sort_keys=True)) + else: + if not results: + print(f"no namespaces found under {root}") + for d in results: + if d.get("status") == "error": + print(f"{d['namespace']}: error ({d['error']})", file=sys.stderr) + else: + print(_human_line(d)) + + return exit_code + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/lean_memory/maintain/runner.py b/src/lean_memory/maintain/runner.py index 6a3ab74..0a1aead 100644 --- a/src/lean_memory/maintain/runner.py +++ b/src/lean_memory/maintain/runner.py @@ -41,6 +41,7 @@ from __future__ import annotations import json +import os import time from dataclasses import dataclass, field from typing import Callable, Optional @@ -56,6 +57,19 @@ _STALE_FLOOR_S = 300.0 # 5 minutes +class _NamespaceCleared(Exception): + """Internal control-flow signal: the namespace file vanished mid-run (§7.3). + + Raised at a batch/heartbeat boundary when `os.path.exists(store.path)` is False — + memory_clear unlinked the DB while this run held (what was) a live lease. It is a + clean STOP, not an error: the run's already-committed batches are consistent; the + lease row died with the file, so there is nothing left to mark 'aborted' (§7.3). + Caught inside `run()`, which returns a normal RunReport with aborted_file_gone=True. + NEVER used for any other condition — only a genuinely missing file trips it, so it + can never mask a real error. + """ + + @dataclass class RunReport: """The outcome of one `MaintenanceRunner.run()` — a small, JSON-able record. @@ -77,6 +91,10 @@ class RunReport: cursor_id: Optional[str] = None config_hash: Optional[str] = None took_over_run_id: Optional[str] = None # id of a stale run this run aborted, if any + # True iff the run stopped early because the namespace file vanished mid-run + # (memory_clear landed at/after a batch boundary — §7.3). Status stays 'ok': the + # run did its committed batches cleanly, then stopped when its DB was unlinked. + aborted_file_gone: bool = False def _heartbeat_is_fresh(live_run: dict, now_s: float, stale_after_s: float) -> bool: @@ -92,6 +110,26 @@ def _heartbeat_is_fresh(live_run: dict, now_s: float, stale_after_s: float) -> b return age_s < stale_after_s +def live_lease_is_fresh(store: Store, namespace: str, now_ms: int) -> bool: + """True iff `namespace` has a LIVE maintenance lease with a FRESH heartbeat (§7.2). + + The single source of truth for "a maintenance run is actively holding this + namespace" — used by `memory_clear`'s refusal (§7.3). It reuses the runner's OWN + staleness rule: `get_live_run` finds the status='running' row, `_heartbeat_is_fresh` + against the 5-minute `_STALE_FLOOR_S` floor decides freshness. A stale or absent + lease returns False (the clear may proceed). The formula is NOT duplicated here. + + A cheap read: the store is opened read-only by the caller and this issues one + indexed SELECT. `now_ms` is injectable so callers/tests never depend on wall time. + """ + live = store.get_live_run(namespace) + if live is None: + return False + # A fresh CLI process has observed no batches, so the floor (5 min) applies — + # exactly the runner's own claim-time check for a lease it did not itself create. + return _heartbeat_is_fresh(live, now_ms / 1000.0, _STALE_FLOOR_S) + + def _proposal_fact_ids(proposal: dict) -> list[str]: """Parse the fact ids a pending proposal references, kind-aware (§4.4/§5 payloads). @@ -299,7 +337,7 @@ def _try_claim_lease(self, now_ms: int) -> tuple[Optional[str], Optional[str], O return result["run_id"], None, result.get("took_over") # ── dry run (§6.3 — report-only, zero writes, no lease) ── - def dry_run(self) -> RunReport: + def dry_run(self, *, auto_only: bool = False) -> RunReport: """Compute the full would-do report with ZERO writes and NO lease (§6.3). A dry-run mutates nothing — no proposals, no auto transforms, not even a lease @@ -308,6 +346,11 @@ def dry_run(self) -> RunReport: suppresses every store write and counts instead). The returned RunReport has status 'ok' and run_id=None (no ledger row exists). This is the `apply=False` default path for `Memory.maintain()` and the CLI/MCP tools. + + `auto_only` (default False) mirrors the run() switch: report only the auto-band + would-do (no propose phases). It has no independent effect on writes — a dry-run + writes nothing regardless — but keeps the reported counts honest for a + `--auto-only` preview. """ cfg_hash = self.config.config_hash() prior = self.store.last_finished_run(self.namespace) @@ -335,7 +378,7 @@ def dry_run(self) -> RunReport: report = run_transforms( self.store, self.config, self._now_ms(), run_id="dry-run", slots=slots, summarizer=self.summarizer, extra_exclude_ids=exclude_ids, - pending_signatures=pending_sigs, dry_run=True, + pending_signatures=pending_sigs, dry_run=True, auto_only=auto_only, ) stats["merges"] = len(report.merges) stats["staged"] = len(report.proposals) @@ -349,18 +392,25 @@ def dry_run(self) -> RunReport: # ── the run ── def run( - self, *, on_batch: Optional[Callable[[], None]] = None, dry_run: bool = False + self, + *, + on_batch: Optional[Callable[[], None]] = None, + dry_run: bool = False, + auto_only: bool = False, ) -> RunReport: """Execute one maintenance run end to end (lease → thresholds → transforms). `on_batch` is a test hook invoked at every batch boundary (after the runner's - own heartbeat) — used by the heartbeat/crash tests to observe cadence and to - simulate a mid-run crash. Production passes None. + own heartbeat AND the namespace-cleared check) — used by the heartbeat/crash + tests to observe cadence and to simulate a mid-run crash or a mid-run clear. + Production passes None. `dry_run=True` delegates to `dry_run()` — report-only, zero writes, no lease. + `auto_only=True` runs ONLY the auto band (dedup_exact + evict auto), staging no + proposals (spec §6.1 `--auto-only`). Default False preserves all behavior. """ if dry_run: - return self.dry_run() + return self.dry_run(auto_only=auto_only) cfg_hash = self.config.config_hash() claim_ms = self._now_ms() @@ -373,7 +423,15 @@ def run( # (via the batch hook installed below) — the §7.2 "every batch commit and at # least every 30 s" cadence. The propose phase is bounded by # proposal_budget_per_run, so it never runs long without a phase-boundary beat. + # + # At each boundary we also check the namespace file still exists (§7.3): if + # memory_clear unlinked it mid-run, keep committing to a ghost inode would + # silently lose work, so we STOP cleanly (raise _NamespaceCleared, caught below). + # The check is a cheap os.path.exists on the store's own path; it fires ONLY on a + # genuinely missing file, so it can never mask a real error. def _beat() -> None: + if not os.path.exists(self.store.path): + raise _NamespaceCleared(self.store.path) self.store.heartbeat_run(run_id, self._now_ms()) if on_batch is not None: on_batch() @@ -433,6 +491,7 @@ def _beat() -> None: summarizer=self.summarizer, extra_exclude_ids=exclude_ids, pending_signatures=pending_sigs, + auto_only=auto_only, ) finally: self._uninstall_batch_hook() @@ -444,6 +503,18 @@ def _beat() -> None: stats["dropped_proposals"] = report.dropped_proposals return self._finish_ok(run_id, stats, cursor_id, cfg_hash, took_over, below=False, report=report) + except _NamespaceCleared: + # memory_clear unlinked the DB mid-run (§7.3). This is a clean STOP, not an + # error: the run's already-committed batches are consistent, and the lease + # row died with the file — there is nothing left to mark 'aborted' (any + # write would just recreate a stray file). Report the clean stop; the + # already-committed transform work (if any) stands in `report`. + return RunReport( + status="ok", run_id=run_id, below_threshold=False, + threshold_stats=stats, transform_report=report, + cursor_id=cursor_id, config_hash=cfg_hash, + took_over_run_id=took_over, aborted_file_gone=True, + ) except BaseException: # A failure DURING our run: mark our own row aborted (consistent DB per # per-batch commits) and re-raise. The next runner takes over from state. diff --git a/src/lean_memory/maintain/transforms.py b/src/lean_memory/maintain/transforms.py index ec73802..e804243 100644 --- a/src/lean_memory/maintain/transforms.py +++ b/src/lean_memory/maintain/transforms.py @@ -494,6 +494,7 @@ def run_transforms( extra_exclude_ids: Optional[set[str]] = None, pending_signatures: Optional[dict[str, set[frozenset[str]]]] = None, dry_run: bool = False, + auto_only: bool = False, ) -> TransformReport: """Run all four transforms in the spec-mandated intra-run order (§4.4). @@ -504,6 +505,12 @@ def run_transforms( (`proposal_budget_per_run`) is shared across the three propose transforms; truncation is REPORTED in `dropped_proposals`, never silent. + `auto_only` (default False, preserving all existing behavior/tests): skip the + Phase-1 propose transforms entirely and run ONLY the auto band (dedup_exact, + evict auto-band). This is the `--auto-only` CLI switch (spec §6.1): apply the + provably-safe transforms, stage NOTHING. With no staged proposals, the auto + phase's exclusion set is just the prior-run pending ids (`extra_exclude_ids`). + `slots` is materialized once (the pre-transform snapshot of touched slots) and reused for both the near-dup proposals and the exact-dup autos. @@ -528,33 +535,35 @@ def run_transforms( report = TransformReport() # ── Phase 1: STAGE ALL PROPOSALS over the pre-transform snapshot ── - budget = config.proposal_budget_per_run - remaining = budget - - near, dropped = dedup_near( - store, config, now, slots, run_id=run_id, budget=remaining, - skip_signatures=pending_signatures.get("dedup_near"), dry_run=dry_run, - ) - report.proposals.extend(near) - report.dropped_proposals += dropped - remaining = budget - len(report.proposals) + # Skipped entirely under `auto_only` (--auto-only): stage NOTHING, run only autos. + if not auto_only: + budget = config.proposal_budget_per_run + remaining = budget + + near, dropped = dedup_near( + store, config, now, slots, run_id=run_id, budget=remaining, + skip_signatures=pending_signatures.get("dedup_near"), dry_run=dry_run, + ) + report.proposals.extend(near) + report.dropped_proposals += dropped + remaining = budget - len(report.proposals) - summ, dropped = summarize( - store, config, now, summarizer, run_id=run_id, budget=max(0, remaining), - skip_signatures=pending_signatures.get("summarize"), dry_run=dry_run, - ) - report.proposals.extend(summ) - report.dropped_proposals += dropped - remaining = budget - len(report.proposals) - - # A fact referenced by a prior-run pending proposal is never re-proposed for - # eviction (excluding its id covers both re-propose and auto-demote — §4.4). - ev_prop, dropped = evict_propose( - store, config, now, run_id=run_id, budget=max(0, remaining), - exclude_ids=extra_exclude_ids, dry_run=dry_run, - ) - report.proposals.extend(ev_prop) - report.dropped_proposals += dropped + summ, dropped = summarize( + store, config, now, summarizer, run_id=run_id, budget=max(0, remaining), + skip_signatures=pending_signatures.get("summarize"), dry_run=dry_run, + ) + report.proposals.extend(summ) + report.dropped_proposals += dropped + remaining = budget - len(report.proposals) + + # A fact referenced by a prior-run pending proposal is never re-proposed for + # eviction (excluding its id covers both re-propose and auto-demote — §4.4). + ev_prop, dropped = evict_propose( + store, config, now, run_id=run_id, budget=max(0, remaining), + exclude_ids=extra_exclude_ids, dry_run=dry_run, + ) + report.proposals.extend(ev_prop) + report.dropped_proposals += dropped # ── Phase 2: AUTO transforms, EXCLUDING every staged-proposal target ── # This run's staged ids UNION prior runs' still-pending referenced ids: neither a diff --git a/src/lean_memory/mcp_server.py b/src/lean_memory/mcp_server.py index fb743f5..960ba85 100644 --- a/src/lean_memory/mcp_server.py +++ b/src/lean_memory/mcp_server.py @@ -23,7 +23,9 @@ from mcp.server.fastmcp import FastMCP +from .maintain import live_lease_is_fresh from .memory import _SAFE_NS, Memory +from .types import now_ms def _data_root() -> Path: @@ -136,12 +138,42 @@ def memory_search(namespace: str, query: str, k: int = 5) -> str: @mcp.tool() def memory_clear(namespace: str) -> str: - """Delete all memory for a namespace by removing its SQLite file. Irreversible.""" + """Delete all memory for a namespace by removing its SQLite file. Irreversible. + + Refuses (returns an explanatory message, changing nothing) while a LIVE maintenance + lease with a fresh heartbeat is held for the namespace (spec §7.3): a POSIX unlink + cannot safely interrupt an in-flight maintenance run — the run's open handle would + keep committing to the unlinked (ghost) inode, silently losing that work. So clear + waits for the run to finish or its lease to go stale. A stale or absent lease clears + normally; the maintenance runner itself independently skips a namespace cleared + mid-run at its next batch boundary. + + Residual race (spec §7.3, known limitation): a clear that lands in the sliver + BETWEEN this lease-check and the unlink is not prevented — full cross-process file + locking is deliberately out of scope for v1. The two guards (this refusal + the + runner's batch-boundary skip) shrink the window; they do not close it. + """ + path = _namespace_path(namespace) + # No file ⇒ nothing to clear and no lease possible. Opening a store here would + # CREATE the file, so short-circuit (unlink below is a no-op on missing files). + if path.exists(): + # Cheap read: open a dedicated maintenance store on the file and ask the runner's + # own staleness rule whether a live lease is held (no raw SQL here — stdout must + # stay the JSON-RPC channel, and the store/runner own the query). + lease_store = _mem()._maintenance_store(namespace) + try: + if live_lease_is_fresh(lease_store, namespace, now_ms()): + return ( + f"refused: namespace '{namespace}' has a live maintenance run " + "holding it; clear again once maintenance finishes." + ) + finally: + lease_store.close() + # Release any cached open connection so the file handle is freed before unlink. store = _mem()._stores.pop(namespace, None) if store is not None: store.close() - path = _namespace_path(namespace) for p in (path, path.with_suffix(".db-wal"), path.with_suffix(".db-shm")): p.unlink(missing_ok=True) return f"cleared namespace '{namespace}'" diff --git a/src/lean_memory/memory.py b/src/lean_memory/memory.py index 92b1972..27f7967 100644 --- a/src/lean_memory/memory.py +++ b/src/lean_memory/memory.py @@ -316,6 +316,7 @@ def maintain( *, config: Optional[MaintenanceConfig] = None, apply: bool = False, + auto_only: bool = False, trigger: str = "cli", summarizer: Optional[Summarizer] = None, ) -> "RunReport": @@ -327,6 +328,10 @@ def maintain( full would-do report with zero writes and takes no lease (a dry-run writes nothing, so it needs no lease). `apply=True` claims the lease, runs the auto band, and stages proposals. + + `auto_only=True` (only meaningful with apply=True) runs ONLY the auto band and + stages NOTHING — the `--auto-only` CLI/auto-spawn switch (§6.1). Default False + preserves the full apply behavior (autos + proposals). """ store = self._maintenance_store(namespace) try: @@ -334,7 +339,7 @@ def maintain( store, namespace, config=config, trigger=trigger, summarizer=summarizer, ) - return runner.run(dry_run=not apply) + return runner.run(dry_run=not apply, auto_only=auto_only) finally: store.close() diff --git a/tests/test_maintenance_cli.py b/tests/test_maintenance_cli.py new file mode 100644 index 0000000..258e376 --- /dev/null +++ b/tests/test_maintenance_cli.py @@ -0,0 +1,401 @@ +"""Task 7 — `lean-memory-maintain` CLI + `memory_clear` lease-refusal (spec §6.1, §7.3). + +All offline (FakeEmbedder, deterministic stub summarizer) against real SqliteStores. +Pins: + + - CLI dry-run IS the default: no --apply ⇒ ZERO spine writes and NO lease row + (full-DB dump byte-identical, no maintenance_run row). + - --apply performs the auto band AND stages proposals; --apply --auto-only performs + the auto band but stages NOTHING (the --auto-only switch threads down to + run_transforms and skips the propose phases). + - --json emits one parseable object with per-namespace reports carrying stable keys. + - A missing root (no --root, no $LM_DATA_ROOT) exits nonzero with a message on stderr. + - Without --namespace, every *.db under the root is processed. + - CROSS-PROCESS (the headline): the CLI runs as a REAL subprocess doing --apply while + the parent process holds a live store writer doing bounded adds — they interleave + without an unhandled `database is locked` (busy_timeout 5000 + short batches absorb + the contention). Both complete; CLI exits 0 with a valid report. + - memory_clear refuses while a LIVE fresh maintenance lease is held (file survives); + a STALE lease lets the clear proceed (file removed). + - The runner stops cleanly at a batch boundary when the namespace file is unlinked + mid-run (no exception escapes; the report notes aborted_file_gone). + +Time is injected where staleness matters so no test sleeps on real wall-clock. +""" + +from __future__ import annotations + +import json +import subprocess +import sys +import threading + +import pytest + +from lean_memory.embed.fake import FakeEmbedder +from lean_memory.maintain import MaintenanceConfig, MaintenanceRunner, live_lease_is_fresh +from lean_memory.maintain.cli import main as cli_main +from lean_memory.store.sqlite_store import SqliteStore +from lean_memory.types import Entity, Episode, Fact + +NOW = 2_000_000_000_000 # fixed wall-clock (epoch ms) + + +# ── seeding helpers ──────────────────────────────────────────────────────────── +def _open(path): + emb = FakeEmbedder() + return SqliteStore(path, dim=emb.dim, coarse_dim=emb.coarse_dim), emb + + +def _seed_namespace(path, *, with_near_dup=True): + """Create a namespace DB with an exact-dup pair (auto-merge target) and, optionally, + a near-dup pair (propose target). Returns the loser id of the exact-dup pair so + tests can assert it was retired. A fresh DB has NO prior finished run, so the CLI's + first run is unconditionally over threshold (§6.6) even at the default config.""" + store, emb = _open(path) + ep = Episode(namespace="ns", raw="seed", t_ref=1_000) + store.add_episode(ep) + subj = store.upsert_entity(Entity(namespace="ns", name="user", type="person")) + + def add(text, *, predicate, valid_at, embed_text=None): + f = Fact( + namespace="ns", subject_id=subj.id, predicate=predicate, fact_text=text, + valid_at=valid_at, episode_id=ep.id, salience=5.0, + ) + full, coarse = emb.embed_with_coarse(embed_text if embed_text is not None else text) + store.add_fact(f, full, coarse) + return f + + # Exact-dup pair (same normalized value, differing case/whitespace) → AUTO merge. + survivor = add("user works at acme", predicate="works_at", valid_at=1_000) + loser = add("USER works at ACME", predicate="works_at", valid_at=2_000) + + if with_near_dup: + # Near-dup pair (distinct texts, identical embed vector ⇒ cosine 1.0 >= tau_near) + # → a DEDUP-NEAR *proposal* (never auto-applied). + add("likes jazz music", predicate="likes", valid_at=1_000, embed_text="MUSICVEC") + add("likes jazz tunes", predicate="likes", valid_at=2_000, embed_text="MUSICVEC") + + store.close() + return survivor.id, loser.id + + +def _fact_dump(path): + store, _ = _open(path) + try: + rows = store._db.execute( + "SELECT id, is_latest, valid_at, valid_to, superseded_by, tier, " + "access_count, last_access, record_kind FROM fact ORDER BY id" + ).fetchall() + return [tuple(r) for r in rows] + finally: + store.close() + + +def _run_count(path): + store, _ = _open(path) + try: + return store._db.execute("SELECT COUNT(*) AS n FROM maintenance_run").fetchone()["n"] + finally: + store.close() + + +def _proposal_count(path): + store, _ = _open(path) + try: + return store._db.execute( + "SELECT COUNT(*) AS n FROM maintenance_proposal" + ).fetchone()["n"] + finally: + store.close() + + +def _loser_row(path, loser_id): + store, _ = _open(path) + try: + return store._db.execute( + "SELECT is_latest, superseded_by FROM fact WHERE id=?", (loser_id,) + ).fetchone() + finally: + store.close() + + +# ── dry-run IS the default (zero writes, no lease) ───────────────────────────── +def test_dry_run_default_writes_nothing(tmp_path, capsys): + path = tmp_path / "ns.db" + _seed_namespace(path) + before = _fact_dump(path) + + rc = cli_main(["--root", str(tmp_path), "--namespace", "ns"]) # no --apply + + assert rc == 0 + # ZERO spine delta: the fact table is byte-identical. + assert _fact_dump(path) == before + # And NO lease row — a dry-run writes nothing, so it takes no lease. + assert _run_count(path) == 0 + assert _proposal_count(path) == 0 + out = capsys.readouterr().out + assert "ns:" in out + assert "dry-run" in out + + +# ── --apply performs autos AND stages proposals ──────────────────────────────── +def test_apply_merges_and_stages(tmp_path): + path = tmp_path / "ns.db" + _survivor, loser = _seed_namespace(path) + + rc = cli_main(["--root", str(tmp_path), "--namespace", "ns", "--apply"]) + + assert rc == 0 + # The exact-dup loser was auto-retired (verb (c)). + row = _loser_row(path, loser) + assert row["is_latest"] == 0 + assert row["superseded_by"] is not None + # And the near-dup pair was STAGED as a proposal. + assert _proposal_count(path) >= 1 + # A lease row exists and finished 'ok'. + assert _run_count(path) == 1 + + +# ── --apply --auto-only performs autos but stages NOTHING ────────────────────── +def test_apply_auto_only_stages_nothing_but_merges(tmp_path): + path = tmp_path / "ns.db" + _survivor, loser = _seed_namespace(path) + + rc = cli_main(["--root", str(tmp_path), "--namespace", "ns", "--apply", "--auto-only"]) + + assert rc == 0 + # Auto merge still happened. + row = _loser_row(path, loser) + assert row["is_latest"] == 0 + assert row["superseded_by"] is not None + # But NOTHING was staged — the propose phases were skipped. + assert _proposal_count(path) == 0 + + +# ── --json parses with stable per-namespace fields ───────────────────────────── +def test_json_output_parses(tmp_path, capsys): + _seed_namespace(tmp_path / "ns.db") + + rc = cli_main(["--root", str(tmp_path), "--namespace", "ns", "--apply", "--json"]) + + assert rc == 0 + payload = json.loads(capsys.readouterr().out) + assert payload["root"] == str(tmp_path) + assert isinstance(payload["namespaces"], list) + rep = payload["namespaces"][0] + for key in ( + "namespace", "status", "mode", "below_threshold", "aborted_file_gone", + "merges", "demoted", "staged", "dropped_proposals", "threshold_stats", + ): + assert key in rep, key + assert rep["namespace"] == "ns" + assert rep["status"] == "ok" + assert rep["mode"] == "apply" + + +# ── all namespaces under the root when --namespace is omitted ────────────────── +def test_all_namespaces_processed(tmp_path, capsys): + _seed_namespace(tmp_path / "alpha.db") + _seed_namespace(tmp_path / "beta.db") + + rc = cli_main(["--root", str(tmp_path), "--json"]) # dry-run, all namespaces + + assert rc == 0 + payload = json.loads(capsys.readouterr().out) + names = {r["namespace"] for r in payload["namespaces"]} + assert names == {"alpha", "beta"} + + +# ── root resolution errors ───────────────────────────────────────────────────── +def test_missing_root_errors_nonzero(tmp_path, monkeypatch, capsys): + monkeypatch.delenv("LM_DATA_ROOT", raising=False) + with pytest.raises(SystemExit) as exc: + cli_main([]) # no --root, no env + # SystemExit with a nonzero/message payload (argparse-style). + assert exc.value.code != 0 + # The message is the SystemExit payload (a string) — surfaced on stderr by the + # console-script wrapper. Assert it mentions the root requirement. + assert "root" in str(exc.value.code).lower() + + +def test_root_from_env(tmp_path, monkeypatch, capsys): + _seed_namespace(tmp_path / "ns.db") + monkeypatch.setenv("LM_DATA_ROOT", str(tmp_path)) + rc = cli_main(["--namespace", "ns", "--json"]) # --root omitted; env supplies it + assert rc == 0 + payload = json.loads(capsys.readouterr().out) + assert payload["root"] == str(tmp_path) + + +# ── CROSS-PROCESS: subprocess CLI --apply vs a live in-process writer ────────── +def test_cross_process_cli_vs_live_writer(tmp_path): + """The headline (§7.1): a REAL subprocess runs the CLI --apply while THIS process + holds a live store writer doing bounded adds. They interleave without an unhandled + `database is locked` — busy_timeout 5000 + short batches absorb the contention. + Both complete; the CLI exits 0 with a valid report.""" + from lean_memory import Memory + + path = tmp_path / "live.db" + _seed_namespace(path) + + # Launch the CLI --apply as a non-blocking subprocess, THEN drive live adds for as + # long as it runs (plus a bounded floor) so the two genuinely overlap on the one + # file. The whole Memory lifecycle (open → add → close) lives ON the writer thread — + # SQLite pins a connection to its creating thread. A `database is locked` on either + # side surfaces as an add error or a nonzero subprocess exit. + add_errors: list[BaseException] = [] + + proc = subprocess.Popen( + [sys.executable, "-m", "lean_memory.maintain.cli", + "--root", str(tmp_path), "--namespace", "live", "--apply", "--json"], + stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, + ) + + def _writer(): + mem = Memory(root=tmp_path) + try: + i = 0 + # Bounded: at least 20 adds, and keep going while the subprocess is alive + # (cap at 200 so a hung child can never make this unbounded). + while i < 200 and (i < 20 or proc.poll() is None): + mem.add("live", f"I use widget number {i}.") + i += 1 + except BaseException as exc: + add_errors.append(exc) + finally: + mem.close() + + t = threading.Thread(target=_writer) + t.start() + try: + out, err = proc.communicate(timeout=60) + finally: + t.join(timeout=30) + + # No unhandled lock error on either side. + assert not add_errors, add_errors + assert proc.returncode == 0, (out, err) + assert "database is locked" not in (out + err).lower() + # Valid JSON report from the subprocess. + payload = json.loads(out) + rep = payload["namespaces"][0] + assert rep["namespace"] == "live" + assert rep["status"] == "ok" + + +# ── memory_clear lease-refusal (spec §7.3) ───────────────────────────────────── +def _stamp_live_lease(path, heartbeat_ms): + """Insert a live (status='running') maintenance_run row with the given heartbeat.""" + store, _ = _open(path) + try: + run_id = store.create_run("ns", "cli", heartbeat_ms, MaintenanceConfig().config_hash()) + store.heartbeat_run(run_id, heartbeat_ms) + return run_id + finally: + store.close() + + +def test_live_lease_helper_freshness(tmp_path): + """The shared staleness helper: a fresh heartbeat is live; a >5min-old one is not.""" + path = tmp_path / "ns.db" + _seed_namespace(path, with_near_dup=False) + _stamp_live_lease(path, NOW) + + store, _ = _open(path) + try: + assert live_lease_is_fresh(store, "ns", NOW) is True + # 6 minutes later → stale (past the 5-minute floor). + assert live_lease_is_fresh(store, "ns", NOW + 6 * 60_000) is False + finally: + store.close() + + +@pytest.fixture +def server(tmp_path, monkeypatch): + """The MCP server module rooted at tmp with stub backends (mirrors test_mcp_server).""" + monkeypatch.setenv("LM_DATA_ROOT", str(tmp_path)) + import importlib + + import lean_memory.mcp_server as srv + + importlib.reload(srv) + from lean_memory import Memory + + srv._MEM = Memory(root=tmp_path) + yield srv + srv._MEM.close() + + +async def _call(server, name, args): + from mcp.shared.memory import create_connected_server_and_client_session + + async with create_connected_server_and_client_session(server.mcp._mcp_server) as session: + result = await session.call_tool(name, args) + return result.content[0].text + + +@pytest.fixture +def anyio_backend(): + return "asyncio" + + +@pytest.mark.anyio +async def test_memory_clear_refuses_under_live_lease(server, tmp_path): + from lean_memory.types import now_ms + + # Seed the namespace file the server will resolve for 'ns'. + path = server._namespace_path("ns") + _seed_namespace(path, with_near_dup=False) + # A live lease with a fresh (now) heartbeat. + _stamp_live_lease(path, now_ms()) + + out = await _call(server, "memory_clear", {"namespace": "ns"}) + + assert out.startswith("refused:") + assert path.exists() # the file is untouched + + +@pytest.mark.anyio +async def test_memory_clear_proceeds_under_stale_lease(server, tmp_path): + path = server._namespace_path("ns") + _seed_namespace(path, with_near_dup=False) + # A live-status row but a heartbeat 10 minutes in the PAST → stale → clear proceeds. + from lean_memory.types import now_ms + + _stamp_live_lease(path, now_ms() - 10 * 60_000) + + out = await _call(server, "memory_clear", {"namespace": "ns"}) + + assert out == "cleared namespace 'ns'" + assert not path.exists() + + +# ── runner stops cleanly when the file is unlinked mid-run (§7.3) ────────────── +def test_runner_stops_when_file_unlinked_mid_run(tmp_path): + """Unlink the namespace file at the first batch boundary (via on_batch); the runner + must STOP cleanly — no exception escapes, report notes aborted_file_gone, and the + already-committed batches stand.""" + path = tmp_path / "ns.db" + # Seed enough exact-dup slots that the auto phase runs at least one batch. + _seed_namespace(path) + + store, emb = _open(path) + cfg = MaintenanceConfig(min_new_facts=1) # trip threshold on the first run anyway + runner = MaintenanceRunner(store, "ns", config=cfg, trigger="cli") + + fired = {"n": 0} + + def _on_batch(): + fired["n"] += 1 + # Unlink the DB (and sidecars) at the first boundary — simulating memory_clear. + for p in (path, path.with_suffix(".db-wal"), path.with_suffix(".db-shm")): + p.unlink(missing_ok=True) + + # Must NOT raise — the runner catches the internal cleared-signal and reports it. + report = runner.run(on_batch=_on_batch) + store.close() + + assert fired["n"] >= 1 + assert report.status == "ok" + assert report.aborted_file_gone is True From 2d7b4d096600ae1bfe221b896e8d24d547cfd73c Mon Sep 17 00:00:00 2001 From: Wuesteon Date: Thu, 16 Jul 2026 23:00:07 +0800 Subject: [PATCH 11/14] docs(wp10a): EngineGateway maintenance methods are v1/WP10a Implementation surfaced the decisive fact the earlier boundary fix missed: both console MCP surfaces (observe_mcp.py stdio + routes/mcp.py HTTP) write only through EngineGateway, so the spec's v1 MCP tools cannot ship without the four gateway methods. Spec/plan/workpackets re-aligned; WP10b consumes. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_0198CpP3tf1SbtDGaRX2NWvJ --- .../2026-07-16-sleep-time-maintenance.md | 4 ++-- ...026-07-16-sleep-time-maintenance-design.md | 23 ++++++++++++------- docs/superpowers/workpackets.md | 12 ++++++---- 3 files changed, 24 insertions(+), 15 deletions(-) diff --git a/docs/superpowers/plans/2026-07-16-sleep-time-maintenance.md b/docs/superpowers/plans/2026-07-16-sleep-time-maintenance.md index 45f8e10..70bd322 100644 --- a/docs/superpowers/plans/2026-07-16-sleep-time-maintenance.md +++ b/docs/superpowers/plans/2026-07-16-sleep-time-maintenance.md @@ -203,10 +203,10 @@ Spec §6.1, §7.3. Spec §6.3–§6.5, §1.3.10. The v0.1.3 lesson lives here; be exact. **Files:** -- Modify: `src/lean_memory/mcp_server.py`, `console/src/lean_memory_console/observe_mcp.py`, `console/src/lean_memory_console/routes/mcp.py`, `plugin/.mcp.json` + `server.json` (reconcile), `tests/test_stdout_hygiene.py` +- Modify: `src/lean_memory/mcp_server.py`, `console/src/lean_memory_console/observe_mcp.py`, `console/src/lean_memory_console/routes/mcp.py`, `console/src/lean_memory_console/engine.py` (4 `EngineGateway` maintenance methods — boundary corrected from WP10b: both console MCP surfaces write only through the gateway, so the v1 tools need them), `plugin/.mcp.json` + `server.json` (reconcile), `tests/test_stdout_hygiene.py`, console tests - New: `plugin/commands/review-memory.md`, `tests/test_mcp_maintenance_tools.py` -- [ ] **Step 1:** the four tools — `memory_maintenance_run(namespace, apply=False)` (**dry-run default**, symmetric with CLI), `memory_maintenance_status` (must not trigger the lazy model build — reads the DB only; pin with a test), `memory_review_queue(namespace, kind=None, limit=20)` (grouped by entity, evidence inline), `memory_review_decide(namespace, proposal_id, decision, edited_text=None)` (approve|reject|edit|promote) — registered on **all three** surfaces: core stdio, console `observe_mcp.py` (what the plugin ships), console HTTP mount. In the same change, update the exact-set pin `console/tests/test_mcp_parity.py::test_wrapper_exposes_exactly_add_and_search` to the new six-tool set — it asserts set equality and goes red the moment the wrapper grows. +- [ ] **Step 1:** the four tools — `memory_maintenance_run(namespace, apply=False)` (**dry-run default**, symmetric with CLI), `memory_maintenance_status` (must not trigger the lazy model build — reads the DB only; pin with a test), `memory_review_queue(namespace, kind=None, limit=20)` (grouped by entity, evidence inline), `memory_review_decide(namespace, proposal_id, decision, edited_text=None)` (approve|reject|edit|promote) — registered on **all three** surfaces: core stdio, console `observe_mcp.py` (what the plugin ships), console HTTP mount. The console surfaces reach the engine ONLY through `EngineGateway` (spec §1.3.8), so this step also adds the four gateway methods (`maintain`, `review_queue`, `decide`, `promote`), each wrapping `retry_busy` + the per-namespace asyncio lock + the single worker thread exactly like `add`/`search` (spec §8). In the same change, update the exact-set pin `console/tests/test_mcp_parity.py::test_wrapper_exposes_exactly_add_and_search` to the new six-tool set — it asserts set equality and goes red the moment the wrapper grows. - [ ] **Step 2:** `@mcp.prompt() review-memory-maintenance` on the console stdio server + the plugin command file `plugin/commands/review-memory.md` (the client-portable path). Prompt text: fetch queue → present batched by entity/kind with evidence → collect explicit user verdicts → decide per item → summarize; **forbids the client agent deciding without an explicit user verdict**; batch verbs only on explicit user statements. - [ ] **Step 3:** auto-spawn (`LM_MAINT_AUTO=1`, default OFF), inside `_mem()` on first tool call: indexed staleness read → `Popen([...,'--apply','--auto-only'], stdin=DEVNULL, stdout=DEVNULL, stderr=, start_new_session=True, close_fds=True)`. fd 1 must never be inherited. - [ ] **Step 4:** extend the stdout-hygiene test to cover tool calls with `LM_MAINT_AUTO=1` (parent JSON-RPC stream byte-clean while the child runs); manifest reconciliation checked by the existing server-manifest test (extend `tests/test_server_manifest.py` if it doesn't already cover the tool list). diff --git a/docs/superpowers/specs/2026-07-16-sleep-time-maintenance-design.md b/docs/superpowers/specs/2026-07-16-sleep-time-maintenance-design.md index b912d21..c5f3fc9 100644 --- a/docs/superpowers/specs/2026-07-16-sleep-time-maintenance-design.md +++ b/docs/superpowers/specs/2026-07-16-sleep-time-maintenance-design.md @@ -786,13 +786,17 @@ the packet table — not gated on the six-week read. DEDUP-EXACT auto (decision 2026-07-16: auto from day one); EVICT auto-band + proposals; DEDUP-NEAR proposals; SUMMARIZE proposals (extractive stub); CLI + ledger/lease; `MaintenanceConfig`; MCP tools on all three surfaces + - prompt + plugin command; as-of grid test at the store predicate. Review - via MCP is fully usable without UI. + prompt + plugin command + the four `EngineGateway` methods (the console + MCP surfaces write **only** through the gateway, §1.3.8 — so the §6.3 v1 + tools cannot ship without them; boundary corrected during implementation); + as-of grid test at the store predicate. Review via MCP is fully usable + without UI. - **v1.1 (WP10b, ~3-4 days — decision 2026-07-16: starts right after WP10a merges, not gated on the demand read):** console Review page + - `EngineGateway` methods + `inspect_sql` proposal reads. (The fingerprint - update is **v1** work, per §5 — deferring it would merge WP10a with a red - console suite; rev 3 fixed this doc's earlier triple-assignment.) + `inspect_sql` proposal reads, consuming the WP10a gateway methods. (The + fingerprint update is likewise **v1** work, per §5 — deferring either + would break WP10a's own surfaces; rev 3 fixed this doc's earlier + triple-assignment.) - **v2 (design-first):** space reclamation (drop full-dim vectors for cold facts, keep coarse 256-dim; LSM-style reader gating; explicit `--reclaim`), Ollama summarizer default-on with `[llm]`, WP5-integrated @@ -810,7 +814,7 @@ the packet table — not gated on the six-week read. | `mcp_server.py` | 4 tools + opt-in auto-spawn (Popen primitives per §6.5) | | `console/.../observe_mcp.py`, `console/.../routes/mcp.py` | same 4 tools + prompt (the surfaces the plugin ships) | | `console/.../inspect_sql.py` | `EXPECTED_SCHEMA_FINGERPRINT` update — the §5 DDL trips it; same change as the migration | -| `console/.../engine.py` (v1.1) | 4 new `EngineGateway` methods | +| `console/.../engine.py` | 4 new `EngineGateway` methods (v1 — the console MCP surfaces write only through the gateway, §1.3.8) | | `plugin/` (`.mcp.json`, `commands/review-memory.md`), `server.json` | manifest reconciliation + command file | | `pyproject.toml` | `lean-memory-maintain` script | | `tests/test_maintenance_*.py` | §10 | @@ -998,8 +1002,11 @@ reproduced exactly. Confirmed and fixed in this rev 3: - **Packet-boundary contradictions:** the console fingerprint update was triple-assigned (WP10a per §5/plan; WP10b per the old §9.2/§9.3/ workpackets — the WP10b reading merges WP10a with a red console suite); - now unambiguously v1/WP10a. The `EngineGateway` methods row in §9.3 is - now marked (v1.1), matching §9.2/§8/plan/workpackets. + now unambiguously v1/WP10a. The `EngineGateway` methods were re-assigned + twice: rev 3 first marked them v1.1, then implementation (Task 8) surfaced + the decisive fact — both console MCP surfaces write **only** through + `EngineGateway` (§1.3.8), so the §6.3 v1 tools cannot ship without the + four methods. They are v1/WP10a; §9.2/§9.3/plan/workpackets now agree. - **Seam fix:** `supersede_fact` now returns the full closed-id set so the §4.3 staleness cascade provably sees duplicate-cascade-closed sources. - **§8.1** dropped a stray "auto-approve confidence band" lever that diff --git a/docs/superpowers/workpackets.md b/docs/superpowers/workpackets.md index 017d2b3..870ea70 100644 --- a/docs/superpowers/workpackets.md +++ b/docs/superpowers/workpackets.md @@ -427,7 +427,9 @@ through Claude Code via MCP tools + a shipped review prompt. EVICT auto-band + proposals; DEDUP-NEAR + SUMMARIZE proposals (extractive stub default, `[llm]` Ollama opt-in); tier filters + `include_cold`; `MaintenanceConfig`; lease/ledger/runner + CLI `lean-memory-maintain`; -4 MCP tools on all three MCP surfaces + review prompt + plugin command file; +4 MCP tools on all three MCP surfaces + review prompt + plugin command file ++ the 4 `EngineGateway` maintenance methods (the console MCP surfaces write +only through the gateway — boundary corrected during implementation); opt-in auto-spawn. **Scope (out):** console UI (WP10b); physical space reclamation (v2, design-first); deletion of any kind (WP5's problem). @@ -456,10 +458,10 @@ M (~3-4 days) · **Lane D + console files.** proposals grouped by entity with before/after evidence; verbs approve / keep / edit-then-approve / promote; batch approve per group. -**Files:** `console/src/lean_memory_console/engine.py` (4 new -`EngineGateway` methods), `console/.../inspect_sql.py` (proposal reads — -the schema fingerprint lands in WP10a with the migration, spec §5), -new review router, `ui/src/pages/Review.tsx`, `ui/src/App.tsx`, +**Files:** `console/.../inspect_sql.py` (proposal reads — +the schema fingerprint lands in WP10a with the migration, spec §5; the 4 +`EngineGateway` maintenance methods likewise land in WP10a, which this +packet consumes), new review router, `ui/src/pages/Review.tsx`, `ui/src/App.tsx`, `ui/src/components/Layout.tsx`, `ui/src/api.ts`, console + UI tests. **Acceptance criteria:** decisions round-trip through `EngineGateway` (never From dc3e320fe77abab1082d06fa85dda7f16ef1a4ba Mon Sep 17 00:00:00 2001 From: Wuesteon Date: Thu, 16 Jul 2026 23:15:38 +0800 Subject: [PATCH 12/14] feat(mcp): maintenance tools on all three surfaces, review prompt, auto-spawn MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The four sleep-time maintenance tools (§6.3) — memory_maintenance_run (dry-run by default, symmetric with the CLI), memory_maintenance_status (model-free ledger read), memory_review_queue, memory_review_decide — registered with identical names/signatures on all three MCP surfaces: core stdio (mcp_server.py), console stdio (observe_mcp.py), console HTTP mount (routes/mcp.py). Both console surfaces reach the engine only through EngineGateway's four new methods (maintain/review_queue/decide/promote), each wrapping retry_busy + the per-namespace asyncio lock + the single worker thread exactly like add/search (§8). Status is provably model-free: a new maintain/mcp_support.py owns the raw SQL for the ledger read + staleness check, so mcp_server.py never calls _mem() for status (pinned by a test that makes the builder raise). The review-workflow prompt (review-memory-maintenance) is registered on the console stdio server and mirrored in plugin/commands/review-memory.md; both forbid the client agent from deciding without an explicit user verdict (§6.4). Opt-in auto-spawn (LM_MAINT_AUTO=1, default OFF) fires once per process on the first tool call with the exact §6.5 Popen primitives — fd 1 is never inherited. Parity pin updated to the exact six-tool set on both console surfaces; server manifests reconciled and pinned against the servers' actual tool surface (the v0.1.3 lesson). Both suites green: core 281 passed + 1 skipped, console 138. Co-Authored-By: Claude Fable 5 --- console/src/lean_memory_console/engine.py | 100 ++++++ console/src/lean_memory_console/mcp_tools.py | 135 ++++++++ .../src/lean_memory_console/observe_mcp.py | 6 + console/src/lean_memory_console/routes/mcp.py | 5 + console/tests/test_engine.py | 105 +++++++ console/tests/test_mcp_parity.py | 31 +- console/tests/test_observe_mcp.py | 46 +++ console/tests/test_plugin_manifest.py | 15 +- plugin/commands/review-memory.md | 45 +++ src/lean_memory/maintain/mcp_support.py | 166 ++++++++++ src/lean_memory/mcp_server.py | 121 +++++++- tests/test_mcp_maintenance_tools.py | 288 ++++++++++++++++++ tests/test_server_manifest.py | 66 ++++ tests/test_stdout_hygiene.py | 169 ++++++++++ 14 files changed, 1289 insertions(+), 9 deletions(-) create mode 100644 console/src/lean_memory_console/mcp_tools.py create mode 100644 plugin/commands/review-memory.md create mode 100644 src/lean_memory/maintain/mcp_support.py create mode 100644 tests/test_mcp_maintenance_tools.py diff --git a/console/src/lean_memory_console/engine.py b/console/src/lean_memory_console/engine.py index f6178a7..8462397 100644 --- a/console/src/lean_memory_console/engine.py +++ b/console/src/lean_memory_console/engine.py @@ -17,6 +17,8 @@ from typing import Callable, TypeVar from lean_memory import Memory +from lean_memory.maintain import mcp_support +from lean_memory.maintain.cli import _report_to_dict as _maint_report_to_dict from .config import ConsoleConfig, is_reserved_namespace, ns_db_path from .events import EventLog @@ -24,6 +26,16 @@ T = TypeVar("T") +def _maintenance_report_to_dict(namespace: str, report, *, apply: bool) -> dict: + """The maintenance run summary, shaped identically to the core MCP surface. + + Reuses the core CLI's RunReport→dict projection so the console tool's return is + byte-for-byte the same shape as core `memory_maintenance_run` (tool-name and + return parity across all three surfaces, §6.3). auto_only is always False from the + gateway path — the console never runs the auto-only preview mode.""" + return _maint_report_to_dict(namespace, report, apply=apply, auto_only=False) + + def _build_memory(config: ConsoleConfig) -> Memory: """Build the console's Memory, opportunistically upgrading to real backends. @@ -269,6 +281,94 @@ async def search( ) return SearchResult(hits=hits, duration_ms=duration_ms) + # ── sleep-time maintenance (design spec §8, §6.3) ────────────────────────── + # Four public methods mirroring add/search EXACTLY: retry_busy inside the single + # worker thread, under the per-namespace asyncio lock. Both console MCP surfaces + # (observe_mcp.py stdio + routes/mcp.py HTTP) reach the engine ONLY through these. + async def maintain( + self, namespace: str, *, apply: bool = False + ) -> dict: + """Run one maintenance pass on `namespace` (§6.3). DRY-RUN by default. + + Lock consequence (§8): a console-invoked maintain() holds this namespace's + gateway lock for the whole run, so it blocks live add/search on that namespace + until the run returns. The underlying MaintenanceRunner works in SHORT per-batch + commits (it releases the SQLite write lock between batches), so this is + tolerable without any chunking logic here — we keep the wrapper simple and let + the runner's own batch cadence bound the stall. Returns the run summary dict. + """ + if is_reserved_namespace(namespace): + raise ValueError(f"reserved namespace rejected: {namespace!r}") + async with self._lock(namespace): + report = await self._run( + lambda: retry_busy( + lambda: self._memory.maintain( + namespace, apply=apply, trigger="console" + ) + ) + ) + return _maintenance_report_to_dict(namespace, report, apply=apply) + + async def maintenance_status(self, namespace: str) -> dict: + """The namespace's maintenance ledger — runs + pending proposals (§6.3). + + A pure ledger read via the model-free ``mcp_support.read_status`` against this + gateway's data root, shaped IDENTICALLY to the core server's status tool. No + lock and no worker thread: it opens its own read-only connection and never + touches the serving store, so it cannot contend with a live add/search.""" + return await self._run( + lambda: retry_busy( + lambda: mcp_support.read_status( + self._config.data_root, namespace + ) + ) + ) + + async def review_queue( + self, namespace: str, *, kind: str | None = None, limit: int = 20 + ) -> list: + """Pending proposals grouped by entity, with evidence (§6.3). Read-shaped but + routed through the write worker + lock because review_queue lazily EXPIRES + overdue proposals (a write) — it must not race a concurrent add on the file.""" + async with self._lock(namespace): + return await self._run( + lambda: retry_busy( + lambda: self._memory.review_queue( + namespace, kind=kind, limit=limit + ) + ) + ) + + async def decide( + self, + namespace: str, + proposal_id: str, + decision: str, + *, + edited_text: str | None = None, + ) -> dict: + """Decide a proposal: approve | reject | edit | promote (§6.3). Applies on + approve through the lifecycle, all under the per-namespace lock + retry_busy.""" + async with self._lock(namespace): + return await self._run( + lambda: retry_busy( + lambda: self._memory.decide( + proposal_id, decision, namespace=namespace, + edited_text=edited_text, decided_by="console", + ) + ) + ) + + async def promote(self, namespace: str, fact_id: str) -> dict: + """Explicitly promote a fact back to the hot tier (§4.4). Explicit-only — + there is no automatic promotion anywhere. Lock + retry_busy like the rest.""" + async with self._lock(namespace): + return await self._run( + lambda: retry_busy( + lambda: self._memory.promote(fact_id, namespace=namespace) + ) + ) + def close(self) -> None: # Close the engine on the same dedicated worker that opened its store # connections (sqlite3 forbids cross-thread use), then retire the pool. diff --git a/console/src/lean_memory_console/mcp_tools.py b/console/src/lean_memory_console/mcp_tools.py new file mode 100644 index 0000000..f4c6f32 --- /dev/null +++ b/console/src/lean_memory_console/mcp_tools.py @@ -0,0 +1,135 @@ +"""Shared MCP tool registration for the console's two surfaces (design spec §6.3). + +The console ships the SAME memory + maintenance tools on both its stdio server +(``observe_mcp.py`` — what the plugin runs) and its Docker HTTP mount +(``routes/mcp.py``). Registering them once here, against a passed-in FastMCP + the +gateway, keeps tool names, signatures, and return shapes IDENTICAL across the two — +and identical to the core stdio server (``lean_memory.mcp_server``), which is the §6.3 +requirement and the v0.1.3 manifest-parity lesson. + +Every tool reaches the engine ONLY through ``EngineGateway`` (spec §1.3.8): the four +maintenance methods (``maintain``/``review_queue``/``decide``/``promote``) each wrap +``retry_busy`` + the per-namespace asyncio lock + the single worker thread, exactly +like ``add``/``search``. + +The review-workflow PROMPT lives on the stdio server only (``register_review_prompt``) +— MCP prompt surfacing is a stdio-client capability; the plugin command file is the +portable path for everyone else (§6.4). +""" + +from __future__ import annotations + +import json +from typing import Any + +from mcp.server.fastmcp import FastMCP + +from .engine import EngineGateway + +# The prompt text: the client agent runs THIS workflow. It MUST NOT decide any +# proposal without an explicit user verdict, and batch verbs ("approve all exact +# dedups") map only to explicit user statements (§6.4). Kept as a module constant so +# the stdio prompt and any future surface share one canonical wording. +REVIEW_PROMPT_TEXT = """\ +You are helping the user review staged sleep-time memory-maintenance proposals for a \ +namespace. Maintenance has already run and STAGED judgment calls (near-duplicate \ +merges, summaries, evictions) as *proposals*; nothing has changed in memory yet. Your \ +job is to walk the user through them and record ONLY the decisions the user explicitly \ +makes. + +HARD RULE — you may NOT decide any proposal on your own. Every approve / reject / edit \ +/ promote MUST come from an explicit user verdict in this conversation. If the user has \ +not stated a verdict for an item, leave it pending. A batch verb (e.g. "approve all \ +exact dedups", "reject every eviction") is valid ONLY when the user says it in those \ +words — never infer a batch decision from a single example or from silence. Silence is \ +not consent; unreviewed proposals expire on their own. + +Workflow: +1. Call `memory_review_queue(namespace=..., limit=...)` to fetch the pending queue. It \ + comes grouped by subject entity, each proposal carrying its evidence payload. +2. Present the proposals to the user, batched by entity and kind, showing the evidence \ + (the before/after texts, cosine for near-dups, source facts for summaries, the \ + value signals for evictions). Keep it scannable. +3. Collect the user's EXPLICIT verdicts. Ask when a verdict is missing or ambiguous. Do \ + not proceed on an item the user has not ruled on. +4. For each item the user decided, call \ + `memory_review_decide(namespace=..., proposal_id=..., decision=..., edited_text=...)` \ + with decision in {approve, reject, edit, promote}. `edited_text` applies only to a \ + summarize proposal the user chose to reword before approving. +5. Summarize what was applied, what was rejected, and what remains pending (untouched, \ + still awaiting the user). +""" + + +def register_maintenance_tools(mcp: FastMCP, gateway: EngineGateway) -> None: + """Register the four §6.3 maintenance tools on `mcp`, routed through `gateway`. + + Identical names/signatures to the core stdio server: memory_maintenance_run + (dry-run default), memory_maintenance_status, memory_review_queue, + memory_review_decide. + """ + + @mcp.tool() + async def memory_maintenance_run( + namespace: str, apply: bool = False + ) -> dict[str, Any]: + """Run one sleep-time maintenance pass (§6.3). DRY-RUN by default (apply=False): + computes the would-do report with zero writes. apply=True runs the auto band + and stages proposals. Symmetric with the CLI. Returns the run summary.""" + return await gateway.maintain(namespace, apply=apply) + + @mcp.tool() + async def memory_maintenance_status(namespace: str) -> dict[str, Any]: + """Report the namespace's maintenance ledger — runs + pending proposals (§6.3). + + A pure model-free ledger read, shaped identically to the core server's status + tool ({namespace, runs, pending_proposals, last_run}). Use it to see whether + maintenance is due and how many proposals wait.""" + return await gateway.maintenance_status(namespace) + + @mcp.tool() + async def memory_review_queue( + namespace: str, kind: str | None = None, limit: int = 20 + ) -> dict[str, Any]: + """List pending maintenance proposals, grouped by entity, with evidence (§6.3). + + `kind` filters to 'dedup_near' | 'summarize' | 'evict'. Each proposal includes + its parsed evidence payload so the reviewer sees exactly what would change.""" + groups = await gateway.review_queue(namespace, kind=kind, limit=limit) + # Round-trip through JSON so numpy/None edge values serialize cleanly, matching + # the core server's json.dumps(..., default=str) shape. + return json.loads(json.dumps({"groups": groups}, default=str)) + + @mcp.tool() + async def memory_review_decide( + namespace: str, + proposal_id: str, + decision: str, + edited_text: str | None = None, + ) -> dict[str, Any]: + """Decide a maintenance proposal: approve | reject | edit | promote (§6.3). + + approve applies at decide-time with apply-time re-validation; reject leaves the + spine byte-identical; edit (summarize only) approves the human-edited text; + promote (evict only) rejects the eviction and lifts the fact to the hot tier.""" + result = await gateway.decide( + namespace, proposal_id, decision, edited_text=edited_text + ) + return json.loads(json.dumps(result, default=str)) + + +def register_review_prompt(mcp: FastMCP) -> None: + """Register the `review-memory-maintenance` prompt on a stdio FastMCP (§6.4). + + The prompt hands the client agent the review workflow but FORBIDS it from deciding + without an explicit user verdict. Stdio-only: prompt surfacing is a stdio-client + capability; the plugin command file is the portable path (§6.4).""" + + @mcp.prompt(name="review-memory-maintenance") + def review_memory_maintenance(namespace: str = "") -> str: + """Walk the user through staged memory-maintenance proposals and record only the + decisions the user explicitly makes.""" + header = ( + f"Namespace to review: {namespace}\n\n" if namespace else "" + ) + return header + REVIEW_PROMPT_TEXT diff --git a/console/src/lean_memory_console/observe_mcp.py b/console/src/lean_memory_console/observe_mcp.py index daec489..bb123fd 100644 --- a/console/src/lean_memory_console/observe_mcp.py +++ b/console/src/lean_memory_console/observe_mcp.py @@ -14,6 +14,7 @@ from .config import ConsoleConfig from .engine import EngineGateway from .events import EventLog +from .mcp_tools import register_maintenance_tools, register_review_prompt def build_mcp(gateway: EngineGateway) -> FastMCP: @@ -52,6 +53,11 @@ async def memory_search(namespace: str, query: str, k: int = 5) -> dict[str, Any ] } + # The four sleep-time maintenance tools, identical to the core + HTTP surfaces + # (§6.3), plus the stdio-only review-workflow prompt (§6.4). + register_maintenance_tools(mcp, gateway) + register_review_prompt(mcp) + return mcp diff --git a/console/src/lean_memory_console/routes/mcp.py b/console/src/lean_memory_console/routes/mcp.py index 8e76a1c..577e1d6 100644 --- a/console/src/lean_memory_console/routes/mcp.py +++ b/console/src/lean_memory_console/routes/mcp.py @@ -54,6 +54,7 @@ from ..config import ConsoleConfig from ..engine import EngineGateway +from ..mcp_tools import register_maintenance_tools # Loopback host/origin default allow-list for the inner MCP transport-security # check. Host entries use the SDK's exact + ``base:*`` port-wildcard matching; @@ -125,6 +126,10 @@ async def memory_search( ] } + # The same four maintenance tools as the stdio surfaces (§6.3). No review prompt + # here — MCP prompts are a stdio-client capability; HTTP clients use the tools. + register_maintenance_tools(mcp, gateway) + return mcp diff --git a/console/tests/test_engine.py b/console/tests/test_engine.py index 51aea8f..ce3d0e1 100644 --- a/console/tests/test_engine.py +++ b/console/tests/test_engine.py @@ -201,3 +201,108 @@ async def test_search_records_ui_origin(gateway, tmp_path): searches = log.list_events("proj", kind="search") log.close() assert searches["items"][0]["payload"]["origin"] == "ui" + + +# ── sleep-time maintenance gateway methods (spec §8, §6.3) ──────────────────── +def _stage_evict_proposal(root, namespace): + """Stage one evict proposal directly on the namespace file — the review path's + input. Dims match FakeEmbedder (768/256), the gateway's stub embedder.""" + import json + + import numpy as np + from lean_memory.store.sqlite_store import SqliteStore + from lean_memory.types import Entity, Episode, Fact, new_id, now_ms + + path = root / f"{namespace}.db" + store = SqliteStore(path, dim=768, coarse_dim=256) + try: + ent = store.upsert_entity(Entity(namespace=namespace, name="Zoe", type=None)) + ep = Episode(namespace=namespace, raw="seed", t_ref=now_ms(), source="user") + store.add_episode(ep) + fact = Fact( + id=new_id(), namespace=namespace, subject_id=ent.id, predicate="about", + object_literal="x", fact_text="Zoe once liked trivia.", valid_at=now_ms(), + episode_id=ep.id, confidence=1.0, salience=1.0, is_inference=0, + ingested_at=now_ms(), created_at=now_ms(), + ) + store.add_fact( + fact, np.zeros(768, dtype=np.float32), np.zeros(256, dtype=np.float32) + ) + run_id = store.create_run(namespace, "cli", now_ms(), "hash") + pid = store.stage_proposal( + run_id, namespace, "evict", + json.dumps({"fact_id": fact.id, "fact_text": fact.fact_text}), + now_ms(), now_ms() + 30 * 86_400_000, "stub", + ) + store.finish_run(run_id, "ok", now_ms(), None, fact.id) + return pid, fact.id + finally: + store.close() + + +@pytest.mark.anyio +async def test_gateway_maintain_dry_run_writes_nothing(gateway, tmp_path): + await gateway.add("proj", "I work at Acme.") + report = await gateway.maintain("proj", apply=False) + assert report["mode"] == "dry-run" + import sqlite3 + + con = sqlite3.connect(f"file:{tmp_path / 'proj.db'}?mode=ro", uri=True) + runs = con.execute("SELECT COUNT(*) FROM maintenance_run").fetchone()[0] + con.close() + assert runs == 0 + + +@pytest.mark.anyio +async def test_gateway_maintain_apply_records_run(gateway, tmp_path): + await gateway.add("proj", "I work at Acme.") + report = await gateway.maintain("proj", apply=True) + assert report["mode"] == "apply" + import sqlite3 + + con = sqlite3.connect(f"file:{tmp_path / 'proj.db'}?mode=ro", uri=True) + ok = con.execute( + "SELECT COUNT(*) FROM maintenance_run WHERE status='ok'" + ).fetchone()[0] + con.close() + assert ok == 1 + + +@pytest.mark.anyio +async def test_gateway_review_queue_and_decide_roundtrip(gateway, tmp_path): + pid, _fid = _stage_evict_proposal(tmp_path, "proj") + groups = await gateway.review_queue("proj") + ids = [p["id"] for g in groups for p in g["proposals"]] + assert pid in ids + result = await gateway.decide("proj", pid, "reject") + assert result["outcome"] == "rejected" + # Gone from the pending queue after the decision. + groups2 = await gateway.review_queue("proj") + assert all(p["id"] != pid for g in groups2 for p in g["proposals"]) + + +@pytest.mark.anyio +async def test_gateway_promote_roundtrips(gateway, tmp_path): + _pid, fid = _stage_evict_proposal(tmp_path, "proj") + result = await gateway.promote("proj", fid) + assert result["outcome"] == "promoted" + assert result["tier"] == "hot" + + +@pytest.mark.anyio +async def test_gateway_maintenance_status_shape(gateway, tmp_path): + """Status is a model-free ledger read shaped like the core server's.""" + _pid, _fid = _stage_evict_proposal(tmp_path, "proj") + status = await gateway.maintenance_status("proj") + assert status["namespace"] == "proj" + assert status["runs"] >= 1 + assert status["pending_proposals"] == 1 + assert status["last_run"]["status"] == "ok" + + +@pytest.mark.anyio +async def test_gateway_maintenance_status_empty_namespace(gateway): + status = await gateway.maintenance_status("never_written") + assert status["runs"] == 0 + assert status["pending_proposals"] == 0 + assert status["last_run"] is None diff --git a/console/tests/test_mcp_parity.py b/console/tests/test_mcp_parity.py index 1725f8a..8ac9a25 100644 --- a/console/tests/test_mcp_parity.py +++ b/console/tests/test_mcp_parity.py @@ -15,7 +15,20 @@ def _tool_params(mcp_obj): return out -def test_wrapper_exposes_exactly_add_and_search(tmp_path): +# The exact tool set the console stdio wrapper exposes (§6.3): the two memory tools +# plus the four sleep-time maintenance tools. Set equality goes RED the moment the +# wrapper grows or drops a tool — the pin that keeps all three surfaces in lockstep. +_EXPECTED_TOOLS = { + "memory_add", + "memory_search", + "memory_maintenance_run", + "memory_maintenance_status", + "memory_review_queue", + "memory_review_decide", +} + + +def test_wrapper_exposes_exactly_the_six_tools(tmp_path): cfg = ConsoleConfig(data_root=tmp_path, mode="local", models="stub") log = EventLog(tmp_path) gw = EngineGateway(cfg, log) @@ -23,7 +36,21 @@ def test_wrapper_exposes_exactly_add_and_search(tmp_path): names = set(_tool_params(wrapper).keys()) gw.close() log.close() - assert names == {"memory_add", "memory_search"} + assert names == _EXPECTED_TOOLS + + +def test_http_mount_exposes_the_same_tool_set(tmp_path): + """The Docker HTTP mount registers the identical six tools (§6.3 surface parity).""" + from lean_memory_console.routes.mcp import _build_http_mcp + + cfg = ConsoleConfig(data_root=tmp_path, mode="local", models="stub") + log = EventLog(tmp_path) + gw = EngineGateway(cfg, log) + http_mcp = _build_http_mcp(gw, cfg.mcp_allowed_hosts) + names = set(_tool_params(http_mcp).keys()) + gw.close() + log.close() + assert names == _EXPECTED_TOOLS def test_memory_clear_absent(tmp_path): diff --git a/console/tests/test_observe_mcp.py b/console/tests/test_observe_mcp.py index 640f2db..cab7c6e 100644 --- a/console/tests/test_observe_mcp.py +++ b/console/tests/test_observe_mcp.py @@ -64,3 +64,49 @@ async def test_reserved_namespace_rejected(wrapper): with pytest.raises(Exception) as excinfo: await mcp.call_tool("memory_add", {"namespace": "_events", "text": "x"}) assert "reserved" in str(excinfo.value).lower() + + +# ── maintenance tools + review prompt on the stdio wrapper (§6.3, §6.4) ──────── +@pytest.mark.anyio +async def test_maintenance_run_dry_by_default(wrapper): + mcp, _root = wrapper + await mcp.call_tool("memory_add", {"namespace": "proj", "text": "Frank owns a red car."}) + out = _unwrap(await mcp.call_tool("memory_maintenance_run", {"namespace": "proj"})) + assert out["mode"] == "dry-run" + + +@pytest.mark.anyio +async def test_review_queue_tool_returns_groups(wrapper): + mcp, _root = wrapper + out = _unwrap(await mcp.call_tool("memory_review_queue", {"namespace": "empty"})) + # No proposals staged for a fresh namespace → an empty groups list, not an error. + assert out["groups"] == [] + + +@pytest.mark.anyio +async def test_maintenance_status_tool_shape(wrapper): + mcp, _root = wrapper + out = _unwrap(await mcp.call_tool("memory_maintenance_status", {"namespace": "fresh"})) + # Model-free ledger read, same shape as the core server's status. + assert out["namespace"] == "fresh" + assert out["runs"] == 0 + assert out["pending_proposals"] == 0 + assert out["last_run"] is None + + +def test_review_prompt_is_registered(wrapper): + mcp, _root = wrapper + names = {p.name for p in mcp._prompt_manager.list_prompts()} + assert "review-memory-maintenance" in names + + +@pytest.mark.anyio +async def test_review_prompt_forbids_agent_deciding(wrapper): + mcp, _root = wrapper + rendered = await mcp.get_prompt("review-memory-maintenance", {"namespace": "proj"}) + text = " ".join( + m.content.text for m in rendered.messages if hasattr(m.content, "text") + ).lower() + # The hard rule: no agent-initiated decisions without an explicit user verdict. + assert "may not decide" in text or "explicit user verdict" in text + assert "silence is not consent" in text diff --git a/console/tests/test_plugin_manifest.py b/console/tests/test_plugin_manifest.py index 37b00bc..2931e96 100644 --- a/console/tests/test_plugin_manifest.py +++ b/console/tests/test_plugin_manifest.py @@ -45,12 +45,13 @@ def test_marketplace_points_at_local_plugin() -> None: assert p["description"] -def test_all_four_commands_exist_with_frontmatter() -> None: +def test_all_commands_exist_with_frontmatter() -> None: cmds = { "memory-ui.md": "/memory:ui", "memory-status.md": "/memory:status", "memory-server-up.md": "/memory:server-up", "memory-server-down.md": "/memory:server-down", + "review-memory.md": "/memory:review-memory", } for filename in cmds: path = PLUGIN / "commands" / filename @@ -61,6 +62,18 @@ def test_all_four_commands_exist_with_frontmatter() -> None: assert "description:" in text.split("---", 2)[1] +def test_review_memory_command_forbids_agent_decisions() -> None: + """The review command must carry the §6.4 guardrail: no agent-initiated decisions + without an explicit user verdict, and no batch verb inferred from silence.""" + text = (PLUGIN / "commands" / "review-memory.md").read_text().lower() + assert "may not decide" in text + assert "explicit user verdict" in text + assert "silence is not consent" in text + # It drives the two review tools by name. + assert "memory_review_queue" in text + assert "memory_review_decide" in text + + def test_server_commands_use_packaged_compose_path() -> None: up = (PLUGIN / "commands" / "memory-server-up.md").read_text() down = (PLUGIN / "commands" / "memory-server-down.md").read_text() diff --git a/plugin/commands/review-memory.md b/plugin/commands/review-memory.md new file mode 100644 index 0000000..819bb3e --- /dev/null +++ b/plugin/commands/review-memory.md @@ -0,0 +1,45 @@ +--- +description: Review staged sleep-time memory-maintenance proposals — you record only the decisions the user explicitly makes. +--- + +Maintenance runs offline and STAGES judgment calls (near-duplicate merges, +summaries, evictions) as *proposals*; nothing changes in memory until a human +approves. Walk the user through the queue and record only their explicit +verdicts. + +**Hard rule — you may NOT decide any proposal on your own.** Every +approve / reject / edit / promote MUST come from an explicit user verdict in +this conversation. If the user has not ruled on an item, leave it pending. A +batch verb ("approve all exact dedups", "reject every eviction") is valid ONLY +when the user says it in those words — never infer a batch decision from a +single example or from silence. Silence is not consent; unreviewed proposals +expire on their own. + +Ask the user which namespace to review if they have not said. Then: + +1. **Fetch the queue.** Call the `memory_review_queue` MCP tool + (`namespace=`, optional `kind` in `dedup_near|summarize|evict`, + `limit`). It returns proposals grouped by subject entity, each carrying its + evidence payload. + +2. **Present, batched by entity and kind, with evidence.** Show the before/after + texts, cosine for near-dups, source facts for summaries, and the value + signals for evictions. Keep it scannable so the user can rule quickly. + +3. **Collect explicit verdicts.** Ask when a verdict is missing or ambiguous. + Do not proceed on an item the user has not decided. + +4. **Decide per item the user ruled on.** Call `memory_review_decide` + (`namespace`, `proposal_id`, `decision`, optional `edited_text`) with + `decision` in `approve|reject|edit|promote`. `edited_text` applies only to a + summarize proposal the user chose to reword before approving. `promote` is + for lifting an eviction's fact back to the hot tier. + +5. **Summarize.** Report what was applied, what was rejected, and what remains + pending (untouched, still awaiting the user). + +Notes for the user: +- Approving is the only thing that changes stored memory. Rejecting leaves the + spine byte-identical; ignoring a proposal lets it expire on its own. +- Use **one namespace per project/session** — the same guidance as the other + memory commands. diff --git a/src/lean_memory/maintain/mcp_support.py b/src/lean_memory/maintain/mcp_support.py new file mode 100644 index 0000000..65fe138 --- /dev/null +++ b/src/lean_memory/maintain/mcp_support.py @@ -0,0 +1,166 @@ +"""Model-free helpers for the MCP maintenance surfaces (design spec §6.3, §6.5). + +Two things the MCP tools need that must NOT drag in the lazy model build: + + * ``read_status(root, namespace)`` — the ledger-only status read behind + ``memory_maintenance_status``. It opens a raw read-only SQLite connection to the + namespace file and counts runs + pending proposals; it never constructs a + ``Memory`` or a ``SqliteStore`` (both of which want an embedder and its dims). + Forcing the ~2 GB model download to answer "when did maintenance last run?" is + exactly the v0.1.3-class mistake; the status tool is pinned model-free by + ``tests/test_mcp_maintenance_tools.py``. + + * ``is_stale(root, namespace, config)`` + ``spawn_maintenance(...)`` — the opt-in + auto-spawn primitives behind ``LM_MAINT_AUTO`` (§6.5). ``is_stale`` reuses the same + cheap ledger read; ``spawn_maintenance`` fires the CLI detached with the EXACT + Popen primitives the spec pins — fd 1 (the JSON-RPC channel) is NEVER inherited. + +All raw SQL for these read paths lives HERE, keeping ``mcp_server.py`` SQL-free. +The namespace→file mapping mirrors ``Memory._namespace_path`` (BET 4: one file per +namespace); the sanitizer is the same ``_SAFE_NS`` regex. +""" + +from __future__ import annotations + +import os +import re +import sqlite3 +import subprocess +import sys +from pathlib import Path +from typing import Optional + +from .config import MS_PER_DAY, MaintenanceConfig + +# Mirror of memory._SAFE_NS / console config.SAFE_NS_RE — the on-disk namespace name. +_SAFE_NS = re.compile(r"[^A-Za-z0-9_.-]") + + +def namespace_path(root: Path, namespace: str) -> Path: + """The SQLite file backing a namespace under ``root`` (mirrors Memory's naming).""" + safe = _SAFE_NS.sub("_", namespace) or "default" + return root / f"{safe}.db" + + +def _ro_connect(path: Path) -> Optional[sqlite3.Connection]: + """Open a read-only connection to an EXISTING namespace file, or None. + + A namespace with no DB file yet (never written) is not an error for a status + read — it just has zero runs and zero proposals. `mode=ro` refuses to create the + file, and a missing file surfaces as OperationalError, which we map to None. + """ + if not path.exists(): + return None + try: + con = sqlite3.connect(f"file:{path}?mode=ro", uri=True) + except sqlite3.OperationalError: + return None + con.row_factory = sqlite3.Row + return con + + +def read_status(root: Path, namespace: str) -> dict: + """Ledger-only maintenance status for a namespace — NO model build (§6.3). + + Returns a JSON-friendly dict: + { + "namespace": , + "runs": , + "pending_proposals": , + "last_run": {id, status, started_at, finished_at, trigger} | None, + } + A never-written namespace (no file) or a v1-schema DB predating the ledger tables + reports zeros and last_run=None rather than raising — the honest empty answer. + """ + empty = { + "namespace": namespace, + "runs": 0, + "pending_proposals": 0, + "last_run": None, + } + con = _ro_connect(namespace_path(root, namespace)) + if con is None: + return empty + try: + # A pre-v2 DB has no ledger tables; treat a missing table as "no maintenance". + try: + runs = con.execute("SELECT COUNT(*) FROM maintenance_run").fetchone()[0] + pending = con.execute( + "SELECT COUNT(*) FROM maintenance_proposal WHERE status='pending'" + ).fetchone()[0] + last = con.execute( + "SELECT id, status, started_at, finished_at, trigger " + "FROM maintenance_run " + "ORDER BY COALESCE(finished_at, started_at) DESC, id DESC LIMIT 1" + ).fetchone() + except sqlite3.OperationalError: + return empty + return { + "namespace": namespace, + "runs": runs, + "pending_proposals": pending, + "last_run": dict(last) if last else None, + } + finally: + con.close() + + +def is_stale(root: Path, namespace: str, config: MaintenanceConfig) -> bool: + """A single cheap ledger read: is this namespace due for a maintenance run? (§6.5) + + Stale iff there is NO finished (status='ok') run ever, OR the last one finished + more than ``config.max_days_between_runs`` days ago. This is the same staleness + signal the runner's age threshold uses, read here without any model build so + auto-spawn can decide in microseconds on the first tool call. + """ + con = _ro_connect(namespace_path(root, namespace)) + if con is None: + return True # never written / no DB → a first run is due if there's anything + try: + try: + row = con.execute( + "SELECT finished_at FROM maintenance_run WHERE status='ok' " + "ORDER BY finished_at DESC, id DESC LIMIT 1" + ).fetchone() + except sqlite3.OperationalError: + return True # pre-v2 DB with no ledger table → never maintained + finally: + con.close() + if row is None or row["finished_at"] is None: + return True + import time + + now_ms = int(time.time() * 1000) + age_days = (now_ms - row["finished_at"]) / MS_PER_DAY + return age_days >= config.max_days_between_runs + + +def spawn_maintenance(root: Path, namespace: str) -> subprocess.Popen: + """Fire the maintenance CLI detached, apply+auto-only, with the EXACT §6.5 + Popen primitives — fd 1 (the JSON-RPC channel) is NEVER inherited. + + ``stdin=DEVNULL, stdout=DEVNULL`` (the v0.1.3 stdout-hygiene rule: the child must + not write to the parent's fd 1), ``stderr`` to a per-root log file if one can be + opened else DEVNULL, ``start_new_session=True`` (own session — reparented on parent + exit, no zombies since the parent never waits), ``close_fds=True`` (no inherited + descriptors). The parent NEVER waits on the returned handle. + """ + log_path = root / "maintenance-autospawn.log" + try: + stderr = open(log_path, "ab") # noqa: SIM115 — handle owned by the child + except OSError: + stderr = subprocess.DEVNULL + return subprocess.Popen( + [ + sys.executable, "-m", "lean_memory.maintain.cli", + "--root", str(root), + "--namespace", namespace, + "--apply", "--auto-only", + ], + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=stderr, + start_new_session=True, + close_fds=True, + env=os.environ.copy(), + ) diff --git a/src/lean_memory/mcp_server.py b/src/lean_memory/mcp_server.py index 960ba85..7999dda 100644 --- a/src/lean_memory/mcp_server.py +++ b/src/lean_memory/mcp_server.py @@ -16,14 +16,17 @@ from __future__ import annotations +import json import os import sys from pathlib import Path -from typing import Optional +from typing import Any, Optional from mcp.server.fastmcp import FastMCP -from .maintain import live_lease_is_fresh +from .maintain import live_lease_is_fresh, mcp_support +from .maintain.cli import _report_to_dict +from .maintain.config import MaintenanceConfig from .memory import _SAFE_NS, Memory from .types import now_ms @@ -93,10 +96,48 @@ def _build_memory(root: Path) -> Memory: # (tool calls are serialized on one event loop; no concurrent _mem() races). _MEM: Optional[Memory] = None +# Opt-in auto-spawn fires AT MOST ONCE per server process (§6.5). This flag flips on +# the first tool call regardless of whether a child was actually spawned, so a fresh +# (non-stale) namespace does not re-check on every subsequent call. +_AUTO_SPAWN_FIRED = False -def _mem() -> Memory: - """Return the module-level Memory, building it (once) on first use.""" + +def _maybe_auto_spawn(namespace: Optional[str] = None) -> None: + """Opt-in (`LM_MAINT_AUTO=1`, default OFF) background maintenance spawn (§6.5). + + Fires AT MOST ONCE per server process, on the FIRST tool call, for the namespace + that call touched. Reads the ledger cheaply (no model build) to decide staleness; + if stale, spawns `lean-memory-maintain --apply --auto-only` detached with fd 1 + NEVER inherited (mcp_support.spawn_maintenance owns the exact Popen primitives). + Never blocks, never waits on the child, never raises into the tool path — a spawn + failure must not break a memory_add. + """ + global _AUTO_SPAWN_FIRED + if _AUTO_SPAWN_FIRED: + return + _AUTO_SPAWN_FIRED = True # once-per-process, set before any work so it never retries + if os.environ.get("LM_MAINT_AUTO") != "1" or not namespace: + return + root = _data_root() + try: + if not root.is_dir(): + return + if mcp_support.is_stale(root, namespace, MaintenanceConfig()): + mcp_support.spawn_maintenance(root, namespace) + except Exception: + # Auto-spawn is best-effort background hygiene; it must never fail a tool call. + pass + + +def _mem(namespace: Optional[str] = None) -> Memory: + """Return the module-level Memory, building it (once) on first use. + + The first call also triggers the opt-in auto-spawn staleness check for `namespace` + (§6.5) — the lazy-build point is the only "first tool call" hook the stdio server + has, and every tool passes the namespace it is operating on so the spawn targets it. + """ global _MEM + _maybe_auto_spawn(namespace) if _MEM is None: _MEM = _build_memory(_data_root()) return _MEM @@ -111,7 +152,7 @@ def _namespace_path(namespace: str) -> Path: @mcp.tool() def memory_add(namespace: str, text: str) -> str: """Ingest text into the namespace's memory. Returns how many facts were written.""" - written = _mem().add(namespace, text) + written = _mem(namespace).add(namespace, text) n = len(written) return f"wrote {n} fact{'s' if n != 1 else ''}" @@ -119,7 +160,7 @@ def memory_add(namespace: str, text: str) -> str: @mcp.tool() def memory_search(namespace: str, query: str, k: int = 5) -> str: """Search a namespace's memory and return the top-k facts as a bulleted list.""" - hits = _mem().search(namespace, query, k=k) + hits = _mem(namespace).search(namespace, query, k=k) if not hits: return "No facts found." # De-duplicate by exact fact_text, keeping the highest-ranked instance and @@ -179,6 +220,74 @@ def memory_clear(namespace: str) -> str: return f"cleared namespace '{namespace}'" +# ── sleep-time maintenance tools (design spec §6.3) ────────────────────────── +# Identical tool names/signatures to the two console surfaces (observe_mcp.py, +# routes/mcp.py). memory_maintenance_run is DRY-RUN by default, symmetric with the +# CLI. memory_maintenance_status is provably model-free (it never calls _mem()). + + +@mcp.tool() +def memory_maintenance_run(namespace: str, apply: bool = False) -> dict[str, Any]: + """Run one sleep-time maintenance pass on a namespace (§6.3). + + DRY-RUN by default (apply=False): computes the full would-do report with ZERO + writes — no ledger row, no proposals. apply=True claims the lease, runs the + provably-safe auto band (exact-dup retirement + auto-band eviction) AND stages the + judgment-call proposals for human review. Symmetric with `lean-memory-maintain`. + + Returns the run summary: mode, staged/merged/demoted counts, and threshold stats. + """ + report = _mem(namespace).maintain(namespace, apply=apply, trigger="mcp") + return _report_to_dict(namespace, report, apply=apply, auto_only=False) + + +@mcp.tool() +def memory_maintenance_status(namespace: str) -> dict[str, Any]: + """Report a namespace's maintenance ledger — runs + pending proposals (§6.3). + + MODEL-FREE by contract: this reads the namespace DB directly and NEVER builds the + embedder/reranker (it does not call _mem()). Answering "when did maintenance last + run?" must never trigger the ~2 GB first-run model download. + """ + return mcp_support.read_status(_data_root(), namespace) + + +@mcp.tool() +def memory_review_queue( + namespace: str, kind: Optional[str] = None, limit: int = 20 +) -> str: + """List pending maintenance proposals, grouped by entity, with evidence (§6.3). + + Each group carries the subject entity and its proposals; each proposal includes its + parsed `payload` (the evidence) so a reviewer sees what would change. `kind` filters + to one of 'dedup_near' | 'summarize' | 'evict'. Overdue proposals lazily expire. + Returns a JSON string (the grouped list). + """ + groups = _mem(namespace).review_queue(namespace, kind=kind, limit=limit) + return json.dumps(groups, sort_keys=True, default=str) + + +@mcp.tool() +def memory_review_decide( + namespace: str, + proposal_id: str, + decision: str, + edited_text: Optional[str] = None, +) -> str: + """Decide a maintenance proposal: approve | reject | edit | promote (§6.3). + + approve applies the proposal's verbs at decide-time (with apply-time target + re-validation); reject leaves the spine byte-identical; edit (summarize only) + approves the human-edited text; promote (evict only) rejects the eviction and + lifts the fact back to the hot tier. Returns a JSON result string. + """ + result = _mem(namespace).decide( + proposal_id, decision, namespace=namespace, + edited_text=edited_text, decided_by="mcp", + ) + return json.dumps(result, sort_keys=True, default=str) + + def main() -> None: """Console-script / module entry point: serve over stdio.""" mcp.run() diff --git a/tests/test_mcp_maintenance_tools.py b/tests/test_mcp_maintenance_tools.py new file mode 100644 index 0000000..15337fb --- /dev/null +++ b/tests/test_mcp_maintenance_tools.py @@ -0,0 +1,288 @@ +"""The four sleep-time maintenance MCP tools on the CORE stdio server (spec §6.3). + +All offline (stub backends), rooted at a tmp dir with LM_DATA_ROOT set before the +module is imported (mirrors test_mcp_server.py). The tools round-trip end to end: + + * memory_maintenance_run — DRY-RUN by default (writes nothing), apply=True stages + * memory_maintenance_status— ledger-only, provably WITHOUT building the models + * memory_review_queue — pending proposals grouped by entity, with evidence + * memory_review_decide — approve|reject|edit|promote against staged proposals + +The status model-free pin is the load-bearing one: the v0.1.3 lesson is that the +first-run model build must never be forced where a cheap ledger read suffices. +""" + +from __future__ import annotations + +import json + +import pytest + +mcp = pytest.importorskip("mcp", reason="optional [mcp] extra not installed") + + +@pytest.fixture +def anyio_backend(): + return "asyncio" + + +@pytest.fixture +def server(tmp_path, monkeypatch): + """Fresh core server module rooted at a tmp dir with stub backends.""" + monkeypatch.setenv("LM_DATA_ROOT", str(tmp_path)) + monkeypatch.delenv("LM_MAINT_AUTO", raising=False) + import importlib + + import lean_memory.mcp_server as srv + + importlib.reload(srv) + from lean_memory import Memory + + srv._MEM = Memory(root=tmp_path) + yield srv + if srv._MEM is not None: # a test may null it to prove status is model-free + srv._MEM.close() + + +async def _call(server, name, args): + from mcp.shared.memory import create_connected_server_and_client_session + + async with create_connected_server_and_client_session( + server.mcp._mcp_server + ) as session: + result = await session.call_tool(name, args) + return result.content[0].text + + +# ── memory_maintenance_run ──────────────────────────────────────────────────── +@pytest.mark.anyio +async def test_maintenance_run_dry_by_default_writes_nothing(server, tmp_path): + """Dry-run default: no maintenance_run row, no proposals — symmetric with CLI.""" + await _call(server, "memory_add", {"namespace": "u1", "text": "I work at Acme."}) + out = await _call(server, "memory_maintenance_run", {"namespace": "u1"}) + data = json.loads(out) + assert data["mode"] == "dry-run" + # No ledger row was written by a dry run. + import sqlite3 + + db = tmp_path / "u1.db" + con = sqlite3.connect(f"file:{db}?mode=ro", uri=True) + runs = con.execute("SELECT COUNT(*) FROM maintenance_run").fetchone()[0] + props = con.execute("SELECT COUNT(*) FROM maintenance_proposal").fetchone()[0] + con.close() + assert runs == 0, "dry-run must write no ledger row" + assert props == 0, "dry-run must stage no proposals" + + +@pytest.mark.anyio +async def test_maintenance_run_apply_claims_lease(server, tmp_path): + """apply=True writes a finished ledger row (the auto band + staging ran).""" + await _call(server, "memory_add", {"namespace": "u2", "text": "I work at Acme."}) + out = await _call( + server, "memory_maintenance_run", {"namespace": "u2", "apply": True} + ) + data = json.loads(out) + assert data["mode"] == "apply" + import sqlite3 + + db = tmp_path / "u2.db" + con = sqlite3.connect(f"file:{db}?mode=ro", uri=True) + runs = con.execute( + "SELECT COUNT(*) FROM maintenance_run WHERE status='ok'" + ).fetchone()[0] + con.close() + assert runs == 1 + + +# ── memory_maintenance_status: MODEL-FREE (the pin) ─────────────────────────── +@pytest.mark.anyio +async def test_status_does_not_build_the_model(server, monkeypatch): + """status MUST read the ledger without forcing the lazy model build (§6.3). + + We make the model builder raise, drop the cached Memory, and assert status + still returns — proving it never called _mem(). + """ + # A prior add builds & caches Memory; drop it so _mem() would rebuild. + await _call(server, "memory_add", {"namespace": "u3", "text": "I work at Acme."}) + server._MEM = None + + def boom(root): + raise RuntimeError("status must not build the model") + + monkeypatch.setattr(server, "_build_memory", boom) + out = await _call(server, "memory_maintenance_status", {"namespace": "u3"}) + data = json.loads(out) + assert "runs" in data and "pending_proposals" in data + assert server._MEM is None, "status forced a model build" + + +@pytest.mark.anyio +async def test_status_reports_after_apply(server): + await _call(server, "memory_add", {"namespace": "u4", "text": "I work at Acme."}) + await _call(server, "memory_maintenance_run", {"namespace": "u4", "apply": True}) + out = await _call(server, "memory_maintenance_status", {"namespace": "u4"}) + data = json.loads(out) + assert data["runs"] >= 1 + assert data["last_run"] is not None + assert data["last_run"]["status"] == "ok" + + +@pytest.mark.anyio +async def test_status_unknown_namespace_is_empty(server): + out = await _call(server, "memory_maintenance_status", {"namespace": "ghost"}) + data = json.loads(out) + assert data["runs"] == 0 + assert data["pending_proposals"] == 0 + assert data["last_run"] is None + + +# ── review_queue + review_decide against staged proposals ───────────────────── +def _stage_evict_proposal(root, namespace): + """Directly stage one evict proposal via the store — the review path's input. + + Uses a dedicated maintenance store (the same class the tools use) so the row is + a real proposal the tools then read/decide. Returns (proposal_id, fact_id). + """ + from lean_memory.store.sqlite_store import SqliteStore + from lean_memory.types import Entity, Episode, Fact, new_id, now_ms + + # Dims MUST match FakeEmbedder (768/256) — the server opens its maintenance store + # for this same file with the FakeEmbedder's dims, and a mismatch would be refused. + path = root / f"{namespace}.db" + store = SqliteStore(path, dim=768, coarse_dim=256) + try: + ent = store.upsert_entity(Entity(namespace=namespace, name="Frank", type=None)) + ep = Episode(namespace=namespace, raw="seed", t_ref=now_ms(), source="user") + store.add_episode(ep) + fact = Fact( + id=new_id(), namespace=namespace, subject_id=ent.id, predicate="about", + object_literal="x", fact_text="Frank likes trivia.", valid_at=now_ms(), + episode_id=ep.id, confidence=1.0, salience=1.0, is_inference=0, + ingested_at=now_ms(), created_at=now_ms(), + ) + import numpy as np + + full = np.zeros(768, dtype=np.float32) + coarse = np.zeros(256, dtype=np.float32) + store.add_fact(fact, full, coarse) + run_id = store.create_run(namespace, "cli", now_ms(), "hash") + payload = json.dumps({"fact_id": fact.id, "fact_text": fact.fact_text}) + pid = store.stage_proposal( + run_id, namespace, "evict", payload, now_ms(), + now_ms() + 30 * 86_400_000, "stub", + ) + store.finish_run(run_id, "ok", now_ms(), None, fact.id) + return pid, fact.id + finally: + store.close() + + +@pytest.mark.anyio +async def test_review_queue_lists_staged_proposal(server, tmp_path): + pid, _fid = _stage_evict_proposal(tmp_path, "u5") + out = await _call(server, "memory_review_queue", {"namespace": "u5"}) + data = json.loads(out) + ids = [p["id"] for group in data for p in group["proposals"]] + assert pid in ids + # Evidence payload rides along parsed. + for group in data: + for p in group["proposals"]: + if p["id"] == pid: + assert p["payload"]["fact_id"] + + +@pytest.mark.anyio +async def test_review_queue_kind_filter(server, tmp_path): + _stage_evict_proposal(tmp_path, "u6") + out = await _call( + server, "memory_review_queue", {"namespace": "u6", "kind": "summarize"} + ) + data = json.loads(out) + assert data == [] # no summarize proposals staged + + +@pytest.mark.anyio +async def test_review_decide_reject(server, tmp_path): + pid, _fid = _stage_evict_proposal(tmp_path, "u7") + out = await _call( + server, "memory_review_decide", + {"namespace": "u7", "proposal_id": pid, "decision": "reject"}, + ) + data = json.loads(out) + assert data["outcome"] == "rejected" + # Now gone from the pending queue. + q = json.loads(await _call(server, "memory_review_queue", {"namespace": "u7"})) + assert all(p["id"] != pid for group in q for p in group["proposals"]) + + +@pytest.mark.anyio +async def test_review_decide_promote(server, tmp_path): + pid, fid = _stage_evict_proposal(tmp_path, "u8") + out = await _call( + server, "memory_review_decide", + {"namespace": "u8", "proposal_id": pid, "decision": "promote"}, + ) + data = json.loads(out) + # promote on an evict proposal rejects the proposal and promotes the fact. + assert data["outcome"] in ("promoted", "rejected_and_promoted", "rejected") + + +@pytest.mark.anyio +async def test_review_decide_edit_records_text(server, tmp_path): + """A summarize proposal edited-then-approved records the human text (§4.3).""" + from lean_memory.store.sqlite_store import SqliteStore + from lean_memory.types import Entity, Episode, Fact, new_id, now_ms + import numpy as np + + ns = "u9" + path = tmp_path / f"{ns}.db" + store = SqliteStore(path, dim=768, coarse_dim=256) # match FakeEmbedder dims + try: + ent = store.upsert_entity(Entity(namespace=ns, name="Ann", type=None)) + ep = Episode(namespace=ns, raw="seed", t_ref=now_ms(), source="user") + store.add_episode(ep) + srcs = [] + full = np.zeros(768, dtype=np.float32) + coarse = np.zeros(256, dtype=np.float32) + for i in range(2): + f = Fact( + id=new_id(), namespace=ns, subject_id=ent.id, predicate="likes", + object_literal=f"o{i}", fact_text=f"Ann likes thing {i}.", + valid_at=now_ms(), episode_id=ep.id, confidence=1.0, salience=1.0, + is_inference=0, ingested_at=now_ms(), created_at=now_ms(), + ) + store.add_fact(f, full, coarse) + srcs.append(f.id) + run_id = store.create_run(ns, "cli", now_ms(), "hash") + payload = json.dumps({ + "subject_id": ent.id, + "source_fact_ids": srcs, + "summary_text": "Ann likes several things.", + }) + pid = store.stage_proposal( + run_id, ns, "summarize", payload, now_ms(), + now_ms() + 30 * 86_400_000, "stub", + ) + store.finish_run(run_id, "ok", now_ms(), None, srcs[-1]) + finally: + store.close() + + out = await _call( + server, "memory_review_decide", + { + "namespace": ns, "proposal_id": pid, "decision": "edit", + "edited_text": "Ann has curated interests.", + }, + ) + data = json.loads(out) + assert data["outcome"] in ("applied", "expired") # applied on fresh targets + import sqlite3 + + con = sqlite3.connect(f"file:{path}?mode=ro", uri=True) + con.row_factory = sqlite3.Row + row = con.execute( + "SELECT edited_text, status FROM maintenance_proposal WHERE id=?", (pid,) + ).fetchone() + con.close() + if data["outcome"] == "applied": + assert row["edited_text"] == "Ann has curated interests." diff --git a/tests/test_server_manifest.py b/tests/test_server_manifest.py index 2318741..9b41f34 100644 --- a/tests/test_server_manifest.py +++ b/tests/test_server_manifest.py @@ -11,6 +11,8 @@ import re from pathlib import Path +import pytest + try: import tomllib except ImportError: # Python 3.10 — tomli ships via the [dev] extra @@ -18,6 +20,20 @@ _ROOT = Path(__file__).resolve().parents[1] +# The canonical tool surface the shipped MCP server exposes (§6.3): the two memory +# tools plus the four sleep-time maintenance tools. The v0.1.3 lesson is that a +# manifest must describe what the server actually exposes — so we pin the set here +# and reconcile both shipped stdio manifests (server.json → core lean-memory-mcp; +# plugin/.mcp.json → console lean-memory-console mcp) against a working install. +_EXPECTED_TOOLS = { + "memory_add", + "memory_search", + "memory_maintenance_run", + "memory_maintenance_status", + "memory_review_queue", + "memory_review_decide", +} + def _manifest() -> dict: return json.loads((_ROOT / "server.json").read_text()) @@ -54,3 +70,53 @@ def test_package_dunder_version_matches_pyproject(): pyproject = tomllib.loads((_ROOT / "pyproject.toml").read_text()) assert lean_memory.__version__ == pyproject["project"]["version"] + + +# ── manifest ⇄ server-surface reconciliation (§6.3, the v0.1.3 lesson) ───────── +def _core_tool_names() -> set[str]: + """The exact tool set the core stdio server (server.json's entry point) exposes.""" + pytest.importorskip("mcp", reason="optional [mcp] extra not installed") + import lean_memory.mcp_server as srv + + return {t.name for t in srv.mcp._tool_manager.list_tools()} + + +def test_core_server_exposes_the_maintenance_tools_plus_clear(): + """What server.json ships (lean-memory-mcp) exposes the pinned six-tool set PLUS + memory_clear (core-only; the console deliberately omits the deletion surface, §6). + The four maintenance tools must all be present with names identical to the console + surfaces — the reconciliation the v0.1.3 lesson demands.""" + names = _core_tool_names() + assert _EXPECTED_TOOLS <= names, f"missing maintenance/memory tools: {_EXPECTED_TOOLS - names}" + assert names == _EXPECTED_TOOLS | {"memory_clear"} + + +def test_server_json_entrypoint_is_the_core_stdio_script(): + """server.json's packageArgument names the core stdio console script, whose module + is the one the tool-surface pin above checks — the manifest ⇄ surface link.""" + manifest = _manifest() + pkg_args = manifest["packages"][0]["packageArguments"] + values = [a.get("value") for a in pkg_args] + assert "lean-memory-mcp" in values + pyproject = tomllib.loads((_ROOT / "pyproject.toml").read_text()) + scripts = pyproject["project"]["scripts"] + assert scripts["lean-memory-mcp"] == "lean_memory.mcp_server:main" + + +def test_maintain_script_is_declared_for_auto_spawn(): + """Auto-spawn (§6.5) invokes `python -m lean_memory.maintain.cli`; the same code is + exposed as the lean-memory-maintain console script. Pin its declaration so the + module path the spawn relies on cannot silently move.""" + pyproject = tomllib.loads((_ROOT / "pyproject.toml").read_text()) + scripts = pyproject["project"]["scripts"] + assert scripts["lean-memory-maintain"] == "lean_memory.maintain.cli:main" + + +def test_plugin_mcp_json_ships_the_console_stdio_server(): + """plugin/.mcp.json ships `lean-memory-console mcp` — the console stdio server whose + six-tool surface is pinned in console/tests/test_mcp_parity.py. Both shipped stdio + manifests thus resolve to a server exposing the identical pinned tool set (§6.3).""" + data = json.loads((_ROOT / "plugin" / ".mcp.json").read_text()) + entry = data["mcpServers"]["lean-memory"] + assert entry["command"] == "uvx" + assert entry["args"] == ["lean-memory-console", "mcp"] diff --git a/tests/test_stdout_hygiene.py b/tests/test_stdout_hygiene.py index 988234f..ec90cbe 100644 --- a/tests/test_stdout_hygiene.py +++ b/tests/test_stdout_hygiene.py @@ -11,9 +11,12 @@ no downloads) while exercising the real _ensure* code paths. """ +import subprocess import sys import types +import pytest + from lean_memory.extract.gliner_extractor import Gliner2Generator @@ -83,3 +86,169 @@ def test_reranker_lazy_load_keeps_stdout_clean(monkeypatch, capsys): r._ensure() out, _ = capsys.readouterr() assert out == "", f"model load leaked to stdout: {out!r}" + + +# ── auto-spawn stdout hygiene (LM_MAINT_AUTO=1, spec §6.5) ──────────────────── +# The opt-in auto-spawn fires the maintenance CLI as a DETACHED child on the first +# tool call. The v0.1.3 lesson is that fd 1 (the JSON-RPC channel) must never be +# inherited: the child's stdout is DEVNULL and the exact Popen primitives are pinned. + +mcp = pytest.importorskip("mcp", reason="optional [mcp] extra not installed") + + +@pytest.fixture +def anyio_backend(): + return "asyncio" + + +def _fresh_server(tmp_path, monkeypatch): + """Reload the core server rooted at tmp_path with stub backends and auto ON.""" + monkeypatch.setenv("LM_DATA_ROOT", str(tmp_path)) + monkeypatch.setenv("LM_MAINT_AUTO", "1") + import importlib + + import lean_memory.mcp_server as srv + + importlib.reload(srv) + from lean_memory import Memory + + srv._MEM = Memory(root=tmp_path) + srv._AUTO_SPAWN_FIRED = False # reload already reset it; be explicit + return srv + + +async def _call(srv, name, args): + from mcp.shared.memory import create_connected_server_and_client_session + + async with create_connected_server_and_client_session(srv.mcp._mcp_server) as s: + result = await s.call_tool(name, args) + return result.content[0].text + + +def test_autospawn_uses_exact_popen_primitives(tmp_path, monkeypatch): + """A stale namespace + LM_MAINT_AUTO=1 spawns with the EXACT §6.5 Popen kwargs. + + We seed a namespace DB (so there is something to maintain), monkeypatch + subprocess.Popen to capture kwargs WITHOUT launching, then fire a tool call and + assert fd 1 is never inherited (stdout=DEVNULL) and the detach primitives hold. + """ + from lean_memory import Memory + + seed = Memory(root=tmp_path) + seed.add("proj", "I work at Acme.") + seed.close() + + srv = _fresh_server(tmp_path, monkeypatch) + + captured: dict = {} + + class _FakePopen: + def __init__(self, argv, **kwargs): + captured["argv"] = argv + captured["kwargs"] = kwargs + + def poll(self): + return 0 + + monkeypatch.setattr(srv.mcp_support.subprocess, "Popen", _FakePopen) + + import asyncio + + asyncio.run(_call(srv, "memory_maintenance_status", {"namespace": "proj"})) + # status is model-free and does not go through _mem(); the FIRST tool call that + # touches _mem() triggers the spawn — drive one. + if "argv" not in captured: + asyncio.run(_call(srv, "memory_add", {"namespace": "proj", "text": "hi"})) + + assert "argv" in captured, "auto-spawn did not fire for a stale namespace" + argv, kwargs = captured["argv"], captured["kwargs"] + # The CLI invocation: apply + auto-only, module form under this interpreter. + assert argv[0] == sys.executable + assert argv[1:3] == ["-m", "lean_memory.maintain.cli"] + assert "--apply" in argv and "--auto-only" in argv + assert "proj" in argv + # fd 1 NEVER inherited — the load-bearing hygiene assertion. + assert kwargs["stdout"] == subprocess.DEVNULL + assert kwargs["stdin"] == subprocess.DEVNULL + assert kwargs["start_new_session"] is True + assert kwargs["close_fds"] is True + + +def test_autospawn_fires_at_most_once(tmp_path, monkeypatch): + """The spawn check runs once per process, even across many tool calls (§6.5).""" + from lean_memory import Memory + + seed = Memory(root=tmp_path) + seed.add("proj", "I work at Acme.") + seed.close() + + srv = _fresh_server(tmp_path, monkeypatch) + calls = {"n": 0} + + def _count_spawn(root, namespace): + calls["n"] += 1 + + monkeypatch.setattr(srv.mcp_support, "spawn_maintenance", _count_spawn) + + import asyncio + + asyncio.run(_call(srv, "memory_add", {"namespace": "proj", "text": "one"})) + asyncio.run(_call(srv, "memory_add", {"namespace": "proj", "text": "two"})) + asyncio.run(_call(srv, "memory_search", {"namespace": "proj", "query": "x"})) + # One namespace, one stale spawn, and the once-flag blocks re-checking after. + assert calls["n"] == 1 + + +def test_autospawn_off_by_default(tmp_path, monkeypatch): + """Without LM_MAINT_AUTO=1 (default), no child is ever spawned.""" + from lean_memory import Memory + + seed = Memory(root=tmp_path) + seed.add("proj", "I work at Acme.") + seed.close() + + monkeypatch.setenv("LM_DATA_ROOT", str(tmp_path)) + monkeypatch.delenv("LM_MAINT_AUTO", raising=False) + import importlib + + import lean_memory.mcp_server as srv + + importlib.reload(srv) + srv._MEM = Memory(root=tmp_path) + fired = {"n": 0} + monkeypatch.setattr( + srv.mcp_support, "spawn_maintenance", lambda r, n: fired.__setitem__("n", 1) + ) + + import asyncio + + asyncio.run(_call(srv, "memory_add", {"namespace": "proj", "text": "hi"})) + srv._MEM.close() + assert fired["n"] == 0 + + +def test_autospawn_real_child_runs_to_completion(tmp_path, monkeypatch): + """A REAL detached child runs to completion against a tmp root; parent fd 1 is + untouched. This exercises the actual Popen call end to end (not a monkeypatch), + proving the CLI module invocation is well-formed and the child exits cleanly. + """ + from lean_memory import Memory + + seed = Memory(root=tmp_path) + seed.add("proj", "I work at Acme.") + seed.close() + + proc = subprocess.Popen( + [ + sys.executable, "-m", "lean_memory.maintain.cli", + "--root", str(tmp_path), "--namespace", "proj", + "--apply", "--auto-only", + ], + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + start_new_session=True, + close_fds=True, + ) + _out, err = proc.communicate(timeout=60) + assert proc.returncode == 0, f"child failed: {err!r}" From ca524b1253a812a7c3ab7bf1a876c1a679eb68c8 Mon Sep 17 00:00:00 2001 From: Wuesteon Date: Thu, 16 Jul 2026 23:23:05 +0800 Subject: [PATCH 13/14] docs: sleep-time maintenance user docs + close-out MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit README "Sleep-time maintenance & review" section (user story, CLI, cron recipe, Claude Code review workflow, safety story); CHANGELOG Unreleased entry; ARCHITECTURE Phase 3 status table; WP10a packet row → implementation complete, awaiting review + merge (merge still gated on WP1 launch). Co-Authored-By: Claude Fable 5 --- ARCHITECTURE.md | 22 ++++++++++++++ CHANGELOG.md | 45 +++++++++++++++++++++++++++++ README.md | 51 +++++++++++++++++++++++++++++++++ docs/superpowers/workpackets.md | 2 +- 4 files changed, 119 insertions(+), 1 deletion(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index b31a878..691be52 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -40,6 +40,28 @@ Implementation status, design decisions, benchmark results, and known limitation | MCP server | ✅ | `memory_add` / `memory_search` / `memory_clear` via FastMCP (stdio transport) | | Terminal demo agent | ✅ | `examples/chat.py` — full add→retrieve→supersede loop with Claude | +### Phase 3 — Sleep-Time Maintenance + +Offline "sleep-time" job that cleans up stored memory between sessions, with a +staged human-review queue. Preserves the ADD-only spine and as-of semantics +(§3.1 visibility theorem incl. ingest commutation) — nothing is ever deleted. +Default-off, post-launch; no change to the first-run path. Design: +`docs/superpowers/specs/2026-07-16-sleep-time-maintenance-design.md`. + +| Component | Status | Notes | +|---|---|---| +| Store verbs + `batch()` + `busy_timeout` | ✅ | `retire_duplicate`/`set_tier`/`iter_*` + unit-of-work; engine-wide busy_timeout (dedicated maintenance store at 5000 ms) | +| Schema v2 (user_version-gated 1→2 migration) | ✅ | `record_kind`, `fact_derivation`, `maintenance_run`/`maintenance_proposal` ledger; v1 fixture + upgrade round-trip test | +| Two ingest hooks (commutation) | ✅ | duplicate-cascade + summary-staleness cascade; exact no-ops until maintenance has ever run | +| DEDUP-EXACT / EVICT auto-band | ✅ | the only auto-applied transforms — as-of-safe in isolation and under later ingest | +| DEDUP-NEAR / SUMMARIZE / EVICT proposals | ✅ | judgment calls staged for human review; extractive stub default, `[llm]` Ollama opt-in; unreviewed proposals expire (30d) | +| Runner + lease + `MaintenanceConfig` | ✅ | atomic lease, heartbeat, work thresholds, cursor, crash-resume; frozen-dataclass config hashed per run | +| Proposal lifecycle | ✅ | CAS decide, one-transaction apply with target re-validation, stale-target + timeout expiry, explicit-only promotion | +| Tier-filtered retrieval | ✅ | default latest-mode hides cold; `as_of` never filters tier; `include_cold=True` opts out | +| `lean-memory-maintain` CLI + cron recipe | ✅ | dry-run default; `--apply`/`--auto-only`/`--json`; core package, no console dependency | +| MCP tools ×4 + prompt + plugin command | ✅ | `memory_maintenance_run/_status`, `memory_review_queue/_decide` on all three MCP surfaces; `review-memory-maintenance` prompt + `/review-memory` command; opt-in auto-spawn (`LM_MAINT_AUTO=1`) | +| Console Review UI (WP10b) | ⬜ | next-morning click-through page — separate packet, starts after this merges | + ### Phase 2 — Next | Item | Status | diff --git a/CHANGELOG.md b/CHANGELOG.md index eb6538e..c3b1301 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,51 @@ All notable changes to lean-memory are documented here. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Added + +- **Sleep-time maintenance** — an offline job that cleans up stored memory + between sessions (dedupe, summarize-old, evict-low-value) while preserving the + ADD-only spine and as-of query semantics. Nothing is ever deleted: maintenance + only appends, retires via the existing `superseded_by` flip, or demotes to a + cold tier, so full history stays queryable at any past point in time — + bit-for-bit identical at the store visibility predicate, pinned by executable + tests. Design: + `docs/superpowers/specs/2026-07-16-sleep-time-maintenance-design.md`. + - **`lean-memory-maintain` CLI** (in the core package beside `lean-memory-mcp`) + — **dry-run by default**; `--apply` runs the auto band and stages proposals, + `--auto-only` runs only the provably-safe band, `--json` emits a + machine-readable report. `--root` defaults to `$LM_DATA_ROOT`; `--namespace` + scopes to one namespace. Ships a cron/launchd recipe in the README. + - **Two-tier autonomy** — only exact-duplicate retirement and a strict + eviction band auto-apply. Near-duplicate merges, summaries, and softer + evictions are staged as **proposals** a human reviews; an unreviewed proposal + **expires** after 30 days (default) rather than auto-applying — silence is + never consent. + - **Conversational review in Claude Code** — four MCP tools + (`memory_maintenance_run`, dry-run by default; `memory_maintenance_status`, + model-free; `memory_review_queue`; `memory_review_decide`) on the core + `lean-memory-mcp` server plus both console MCP surfaces, a + `review-memory-maintenance` MCP prompt, and a `/review-memory` plugin command + that walks the queue grouped by entity and records only explicit user + verdicts. + - **Tier-filtered retrieval** — default latest-mode search hides cold-demoted + facts; `as_of` queries never filter tier, and `search(..., include_cold=True)` + opts out. Promotion back to hot is explicit-only — reads never durably change + surfaces. + - **Schema v2** — `user_version`-gated 1→2 migration (v1-format files upgrade + in place, verified by a checked-in fixture) adding `record_kind`, + `fact_derivation` lineage, and the `maintenance_run` / `maintenance_proposal` + ledger tables. Two exact-no-op ingest hooks (duplicate-cascade, + summary-staleness cascade) keep offline transforms coherent under later + ingest; both are byte-identical no-ops until maintenance has ever run. + - **Concurrency & crash safety** — engine-wide `PRAGMA busy_timeout`, a + `batch()` unit-of-work, an atomic maintenance lease with heartbeat and + crash-resume, and `memory_clear` refusing while a live maintenance lease is + held. `LM_MAINT_AUTO=1` opts into a detached background auto-run on the first + tool call of a stale namespace (off by default). + ## [0.1.3] - 2026-07-12 Publish-readiness release: an independent multi-team review of v0.1.2 found diff --git a/README.md b/README.md index c70e1ec..f00587f 100644 --- a/README.md +++ b/README.md @@ -129,6 +129,57 @@ semantically meaningless for real use — install `[mcp,models,extract]`). > `[llm]` (a local Ollama model) upgrades that escalated tier to real > constrained typing. See ARCHITECTURE.md → Known Limitations. +## Sleep-time maintenance & review + +Memory accumulates cruft: the same fact restated a dozen ways, old records that +never come up, clusters begging to be summarized. lean-memory cleans it up the +way sleep consolidates memory — an **offline job you run off-hours** that dedupes, +summarizes, and demotes low-value records, then hands you the judgment calls to +click through the next morning, in the web console **or conversationally in +Claude Code**. + +**The CLI** (`lean-memory-maintain`) is the primary trigger. It is **dry-run by +default** — it reports what it *would* do and writes nothing: + +```bash +lean-memory-maintain --root ~/.lean_memory # dry-run: report only, zero writes +lean-memory-maintain --root ~/.lean_memory --apply # auto-apply safe transforms + stage the rest +lean-memory-maintain --root ~/.lean_memory --auto-only # with --apply: ONLY the provably-safe band, stage nothing +lean-memory-maintain --root ~/.lean_memory --json # one machine-readable object, stable keys +``` + +`--root` defaults to `$LM_DATA_ROOT`; add `--namespace NS` to run a single +namespace instead of every `*.db` under the root. **Overnight, on a schedule** — +one crontab line runs the safe band nightly at 3am and stages everything else +for you: + +```cron +0 3 * * * lean-memory-maintain --root ~/.lean_memory --apply >> ~/.lean_memory/maintain.log 2>&1 +``` + +**Next-morning review in Claude Code.** Judgment calls (near-duplicate merges, +summaries, evictions) are staged as *proposals* — nothing changes in stored +memory until you approve. Run the `/review-memory` plugin command (or invoke the +`review-memory-maintenance` MCP prompt) and Claude walks you through the queue, +grouped by entity with before/after evidence, recording only the verdicts you +give. Four MCP tools back it — `memory_maintenance_run` (dry-run by default, +like the CLI), `memory_maintenance_status`, `memory_review_queue`, and +`memory_review_decide` — available on the core `lean-memory-mcp` server and both +console MCP surfaces. Set `LM_MAINT_AUTO=1` to opt into a background auto-run +(safe band only) on the first tool call of a stale namespace; it is off by +default. + +**The safety story in one paragraph.** Nothing is ever deleted — maintenance +only appends, retires (the same `superseded_by` flip ordinary supersession +uses), or demotes to a cold tier, so your full history stays queryable as-of any +past point in time, bit-for-bit identical at the store predicate and pinned by +executable tests. Only two transforms auto-apply: exact-duplicate retirement and +a strict eviction band; everything judgmental is staged for a human, and an +unreviewed proposal **expires** after 30 days rather than auto-applying — +silence is never consent. Cold-demoted facts stay reachable via `as_of` queries +and `search(..., include_cold=True)`, and promotion back to the hot tier is +explicit-only, so a read never durably changes what your agent sees. + ## Real Model Quality The default backends are offline stubs — deterministic and dependency-free, but semantically meaningless. Swap in real models for production-quality retrieval: diff --git a/docs/superpowers/workpackets.md b/docs/superpowers/workpackets.md index 870ea70..6565ba1 100644 --- a/docs/superpowers/workpackets.md +++ b/docs/superpowers/workpackets.md @@ -40,7 +40,7 @@ on the same files). | WP7 Async API | `wp7-async` | A | WP4–WP6 | six-week read | open | | WP8 Integrations & distribution wave | `wp8-*` (per sub-packet) | C | WP0 | six-week read (spec §5) | open | | WP9 NLI middle tier (contingent) | `wp9-nli-resolver` | A | WP0 | **contingent** — see trigger | open | -| WP10a Sleep-time maintenance (engine + MCP review) | `wp10a-sleep-maintenance` | A | WP1 | — (conscious post-launch addition, recorded 2026-07-16) | **claimed** (2026-07-16: implementation started on `wp10a-sleep-maintenance` by user direction; **merge remains gated on WP1 launch**) | +| WP10a Sleep-time maintenance (engine + MCP review) | `wp10a-sleep-maintenance` | A | WP1 | — (conscious post-launch addition, recorded 2026-07-16) | **implementation COMPLETE** on `wp10a-sleep-maintenance` (2026-07-16: all 9 plan tasks done; core + console suites green) — awaiting final whole-branch review + merge; **merge remains gated on WP1 launch** | | WP10b Maintenance review UI | `wp10b-review-ui` | D | WP10a | — | open | | Memory UI | `worktree-memory-ui` | D | — | — | **MERGED** (2026-07-14: PR #2 → main 9d840b6; console 125 + core 141 green on merged main; lane D released) | From aae7ff79b48dcd3d5183a0caf6764fbf8e6c4d04 Mon Sep 17 00:00:00 2001 From: Wuesteon Date: Thu, 16 Jul 2026 23:31:22 +0800 Subject: [PATCH 14/14] docs(wp10a): record WP10b carry-in cleanups from the final branch review Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_0198CpP3tf1SbtDGaRX2NWvJ --- docs/superpowers/workpackets.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/superpowers/workpackets.md b/docs/superpowers/workpackets.md index 6565ba1..636cbf0 100644 --- a/docs/superpowers/workpackets.md +++ b/docs/superpowers/workpackets.md @@ -469,6 +469,15 @@ raw SQL writes); CAS "already decided elsewhere" surfaced in the UI; console suite green; spec §8.1 fatigue levers present (entity grouping, batch approve, budget cap). +**Carry-in cleanups from WP10a's final whole-branch review** (small; touch +files this packet already opens): move the proposal-budget check ahead of the +summarizer invocation in `maintain/transforms.py` (matters once Ollama is the +`[llm]` summarizer); add an exists-guard so CLI dry-run against an explicit +nonexistent `--namespace` doesn't create an empty `.db`; declare `path` +on the Store ABC; document the `memory_maintenance_run(apply=True)` +(auto band + stages) vs auto-spawn `--auto-only` (auto band only) asymmetry +in the tool docstring. + --- ## Anti-goals (decided, with evidence — do not open packets for these)