WP10a: sleep-time maintenance — offline dedup/summarize/evict with a human review queue

ARCHITECTURE.md

@@ -40,6 +40,28 @@ Implementation status, design decisions, benchmark results, and known limitation
 | MCP server | ✅ | `memory_add` / `memory_search` / `memory_clear` via FastMCP (stdio transport) |
 | Terminal demo agent | ✅ | `examples/chat.py` — full add→retrieve→supersede loop with Claude |
 
+### Phase 3 — Sleep-Time Maintenance
+
+Offline "sleep-time" job that cleans up stored memory between sessions, with a
+staged human-review queue. Preserves the ADD-only spine and as-of semantics
+(§3.1 visibility theorem incl. ingest commutation) — nothing is ever deleted.
+Default-off, post-launch; no change to the first-run path. Design:
+`docs/superpowers/specs/2026-07-16-sleep-time-maintenance-design.md`.
+
+| Component | Status | Notes |
+|---|---|---|
+| Store verbs + `batch()` + `busy_timeout` | ✅ | `retire_duplicate`/`set_tier`/`iter_*` + unit-of-work; engine-wide busy_timeout (dedicated maintenance store at 5000 ms) |
+| Schema v2 (user_version-gated 1→2 migration) | ✅ | `record_kind`, `fact_derivation`, `maintenance_run`/`maintenance_proposal` ledger; v1 fixture + upgrade round-trip test |
+| Two ingest hooks (commutation) | ✅ | duplicate-cascade + summary-staleness cascade; exact no-ops until maintenance has ever run |
+| DEDUP-EXACT / EVICT auto-band | ✅ | the only auto-applied transforms — as-of-safe in isolation and under later ingest |
+| DEDUP-NEAR / SUMMARIZE / EVICT proposals | ✅ | judgment calls staged for human review; extractive stub default, `[llm]` Ollama opt-in; unreviewed proposals expire (30d) |
+| Runner + lease + `MaintenanceConfig` | ✅ | atomic lease, heartbeat, work thresholds, cursor, crash-resume; frozen-dataclass config hashed per run |
+| Proposal lifecycle | ✅ | CAS decide, one-transaction apply with target re-validation, stale-target + timeout expiry, explicit-only promotion |
+| Tier-filtered retrieval | ✅ | default latest-mode hides cold; `as_of` never filters tier; `include_cold=True` opts out |
+| `lean-memory-maintain` CLI + cron recipe | ✅ | dry-run default; `--apply`/`--auto-only`/`--json`; core package, no console dependency |
+| MCP tools ×4 + prompt + plugin command | ✅ | `memory_maintenance_run/_status`, `memory_review_queue/_decide` on all three MCP surfaces; `review-memory-maintenance` prompt + `/review-memory` command; opt-in auto-spawn (`LM_MAINT_AUTO=1`) |
+| Console Review UI (WP10b) | ⬜ | next-morning click-through page — separate packet, starts after this merges |
+
 ### Phase 2 — Next
 
 | Item | Status |

CHANGELOG.md

@@ -4,6 +4,51 @@ All notable changes to lean-memory are documented here. The format is based on
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project
 adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Added
+
+- **Sleep-time maintenance** — an offline job that cleans up stored memory
+  between sessions (dedupe, summarize-old, evict-low-value) while preserving the
+  ADD-only spine and as-of query semantics. Nothing is ever deleted: maintenance
+  only appends, retires via the existing `superseded_by` flip, or demotes to a
+  cold tier, so full history stays queryable at any past point in time —
+  bit-for-bit identical at the store visibility predicate, pinned by executable
+  tests. Design:
+  `docs/superpowers/specs/2026-07-16-sleep-time-maintenance-design.md`.
+  - **`lean-memory-maintain` CLI** (in the core package beside `lean-memory-mcp`)
+    — **dry-run by default**; `--apply` runs the auto band and stages proposals,
+    `--auto-only` runs only the provably-safe band, `--json` emits a
+    machine-readable report. `--root` defaults to `$LM_DATA_ROOT`; `--namespace`
+    scopes to one namespace. Ships a cron/launchd recipe in the README.
+  - **Two-tier autonomy** — only exact-duplicate retirement and a strict
+    eviction band auto-apply. Near-duplicate merges, summaries, and softer
+    evictions are staged as **proposals** a human reviews; an unreviewed proposal
+    **expires** after 30 days (default) rather than auto-applying — silence is
+    never consent.
+  - **Conversational review in Claude Code** — four MCP tools
+    (`memory_maintenance_run`, dry-run by default; `memory_maintenance_status`,
+    model-free; `memory_review_queue`; `memory_review_decide`) on the core
+    `lean-memory-mcp` server plus both console MCP surfaces, a
+    `review-memory-maintenance` MCP prompt, and a `/review-memory` plugin command
+    that walks the queue grouped by entity and records only explicit user
+    verdicts.
+  - **Tier-filtered retrieval** — default latest-mode search hides cold-demoted
+    facts; `as_of` queries never filter tier, and `search(..., include_cold=True)`
+    opts out. Promotion back to hot is explicit-only — reads never durably change
+    surfaces.
+  - **Schema v2** — `user_version`-gated 1→2 migration (v1-format files upgrade
+    in place, verified by a checked-in fixture) adding `record_kind`,
+    `fact_derivation` lineage, and the `maintenance_run` / `maintenance_proposal`
+    ledger tables. Two exact-no-op ingest hooks (duplicate-cascade,
+    summary-staleness cascade) keep offline transforms coherent under later
+    ingest; both are byte-identical no-ops until maintenance has ever run.
+  - **Concurrency & crash safety** — engine-wide `PRAGMA busy_timeout`, a
+    `batch()` unit-of-work, an atomic maintenance lease with heartbeat and
+    crash-resume, and `memory_clear` refusing while a live maintenance lease is
+    held. `LM_MAINT_AUTO=1` opts into a detached background auto-run on the first
+    tool call of a stale namespace (off by default).
+
 ## [0.1.3] - 2026-07-12
 
 Publish-readiness release: an independent multi-team review of v0.1.2 found

README.md

@@ -129,6 +129,57 @@ semantically meaningless for real use — install `[mcp,models,extract]`).
 > `[llm]` (a local Ollama model) upgrades that escalated tier to real
 > constrained typing. See ARCHITECTURE.md → Known Limitations.
 
+## Sleep-time maintenance & review
+
+Memory accumulates cruft: the same fact restated a dozen ways, old records that
+never come up, clusters begging to be summarized. lean-memory cleans it up the
+way sleep consolidates memory — an **offline job you run off-hours** that dedupes,
+summarizes, and demotes low-value records, then hands you the judgment calls to
+click through the next morning, in the web console **or conversationally in
+Claude Code**.
+
+**The CLI** (`lean-memory-maintain`) is the primary trigger. It is **dry-run by
+default** — it reports what it *would* do and writes nothing:
+
+```bash
+lean-memory-maintain --root ~/.lean_memory              # dry-run: report only, zero writes
+lean-memory-maintain --root ~/.lean_memory --apply      # auto-apply safe transforms + stage the rest
+lean-memory-maintain --root ~/.lean_memory --auto-only   # with --apply: ONLY the provably-safe band, stage nothing
+lean-memory-maintain --root ~/.lean_memory --json        # one machine-readable object, stable keys
+```
+
+`--root` defaults to `$LM_DATA_ROOT`; add `--namespace NS` to run a single
+namespace instead of every `*.db` under the root. **Overnight, on a schedule** —
+one crontab line runs the safe band nightly at 3am and stages everything else
+for you:
+
+```cron
+0 3 * * *  lean-memory-maintain --root ~/.lean_memory --apply >> ~/.lean_memory/maintain.log 2>&1
+```
+
+**Next-morning review in Claude Code.** Judgment calls (near-duplicate merges,
+summaries, evictions) are staged as *proposals* — nothing changes in stored
+memory until you approve. Run the `/review-memory` plugin command (or invoke the
+`review-memory-maintenance` MCP prompt) and Claude walks you through the queue,
+grouped by entity with before/after evidence, recording only the verdicts you
+give. Four MCP tools back it — `memory_maintenance_run` (dry-run by default,
+like the CLI), `memory_maintenance_status`, `memory_review_queue`, and
+`memory_review_decide` — available on the core `lean-memory-mcp` server and both
+console MCP surfaces. Set `LM_MAINT_AUTO=1` to opt into a background auto-run
+(safe band only) on the first tool call of a stale namespace; it is off by
+default.
+
+**The safety story in one paragraph.** Nothing is ever deleted — maintenance
+only appends, retires (the same `superseded_by` flip ordinary supersession
+uses), or demotes to a cold tier, so your full history stays queryable as-of any
+past point in time, bit-for-bit identical at the store predicate and pinned by
+executable tests. Only two transforms auto-apply: exact-duplicate retirement and
+a strict eviction band; everything judgmental is staged for a human, and an
+unreviewed proposal **expires** after 30 days rather than auto-applying —
+silence is never consent. Cold-demoted facts stay reachable via `as_of` queries
+and `search(..., include_cold=True)`, and promotion back to the hot tier is
+explicit-only, so a read never durably changes what your agent sees.
+
 ## Real Model Quality
 
 The default backends are offline stubs — deterministic and dependency-free, but semantically meaningless. Swap in real models for production-quality retrieval:

console/src/lean_memory_console/engine.py

@@ -17,13 +17,25 @@
 from typing import Callable, TypeVar
 
 from lean_memory import Memory
+from lean_memory.maintain import mcp_support
+from lean_memory.maintain.cli import _report_to_dict as _maint_report_to_dict
 
 from .config import ConsoleConfig, is_reserved_namespace, ns_db_path
 from .events import EventLog
 
 T = TypeVar("T")
 
 
+def _maintenance_report_to_dict(namespace: str, report, *, apply: bool) -> dict:
+    """The maintenance run summary, shaped identically to the core MCP surface.
+
+    Reuses the core CLI's RunReport→dict projection so the console tool's return is
+    byte-for-byte the same shape as core `memory_maintenance_run` (tool-name and
+    return parity across all three surfaces, §6.3). auto_only is always False from the
+    gateway path — the console never runs the auto-only preview mode."""
+    return _maint_report_to_dict(namespace, report, apply=apply, auto_only=False)
+
+
 def _build_memory(config: ConsoleConfig) -> Memory:
     """Build the console's Memory, opportunistically upgrading to real backends.
 
@@ -269,6 +281,94 @@ async def search(
         )
         return SearchResult(hits=hits, duration_ms=duration_ms)
 
+    # ── sleep-time maintenance (design spec §8, §6.3) ──────────────────────────
+    # Four public methods mirroring add/search EXACTLY: retry_busy inside the single
+    # worker thread, under the per-namespace asyncio lock. Both console MCP surfaces
+    # (observe_mcp.py stdio + routes/mcp.py HTTP) reach the engine ONLY through these.
+    async def maintain(
+        self, namespace: str, *, apply: bool = False
+    ) -> dict:
+        """Run one maintenance pass on `namespace` (§6.3). DRY-RUN by default.
+
+        Lock consequence (§8): a console-invoked maintain() holds this namespace's
+        gateway lock for the whole run, so it blocks live add/search on that namespace
+        until the run returns. The underlying MaintenanceRunner works in SHORT per-batch
+        commits (it releases the SQLite write lock between batches), so this is
+        tolerable without any chunking logic here — we keep the wrapper simple and let
+        the runner's own batch cadence bound the stall. Returns the run summary dict.
+        """
+        if is_reserved_namespace(namespace):
+            raise ValueError(f"reserved namespace rejected: {namespace!r}")
+        async with self._lock(namespace):
+            report = await self._run(
+                lambda: retry_busy(
+                    lambda: self._memory.maintain(
+                        namespace, apply=apply, trigger="console"
+                    )
+                )
+            )
+        return _maintenance_report_to_dict(namespace, report, apply=apply)
+
+    async def maintenance_status(self, namespace: str) -> dict:
+        """The namespace's maintenance ledger — runs + pending proposals (§6.3).
+
+        A pure ledger read via the model-free ``mcp_support.read_status`` against this
+        gateway's data root, shaped IDENTICALLY to the core server's status tool. No
+        lock and no worker thread: it opens its own read-only connection and never
+        touches the serving store, so it cannot contend with a live add/search."""
+        return await self._run(
+            lambda: retry_busy(
+                lambda: mcp_support.read_status(
+                    self._config.data_root, namespace
+                )
+            )
+        )
+
+    async def review_queue(
+        self, namespace: str, *, kind: str | None = None, limit: int = 20
+    ) -> list:
+        """Pending proposals grouped by entity, with evidence (§6.3). Read-shaped but
+        routed through the write worker + lock because review_queue lazily EXPIRES
+        overdue proposals (a write) — it must not race a concurrent add on the file."""
+        async with self._lock(namespace):
+            return await self._run(
+                lambda: retry_busy(
+                    lambda: self._memory.review_queue(
+                        namespace, kind=kind, limit=limit
+                    )
+                )
+            )
+
+    async def decide(
+        self,
+        namespace: str,
+        proposal_id: str,
+        decision: str,
+        *,
+        edited_text: str | None = None,
+    ) -> dict:
+        """Decide a proposal: approve | reject | edit | promote (§6.3). Applies on
+        approve through the lifecycle, all under the per-namespace lock + retry_busy."""
+        async with self._lock(namespace):
+            return await self._run(
+                lambda: retry_busy(
+                    lambda: self._memory.decide(
+                        proposal_id, decision, namespace=namespace,
+                        edited_text=edited_text, decided_by="console",
+                    )
+                )
+            )
+
+    async def promote(self, namespace: str, fact_id: str) -> dict:
+        """Explicitly promote a fact back to the hot tier (§4.4). Explicit-only —
+        there is no automatic promotion anywhere. Lock + retry_busy like the rest."""
+        async with self._lock(namespace):
+            return await self._run(
+                lambda: retry_busy(
+                    lambda: self._memory.promote(fact_id, namespace=namespace)
+                )
+            )
+
     def close(self) -> None:
         # Close the engine on the same dedicated worker that opened its store
         # connections (sqlite3 forbids cross-thread use), then retire the pool.

console/src/lean_memory_console/inspect_sql.py

@@ -311,8 +311,11 @@ def list_entities(db_path: Path, page: int = 1, page_size: int = 50) -> dict:
 # Filled once from the first run's printed digests (Step 5), then a test pins
 # equality so engine drift turns the suite red.
 EXPECTED_SCHEMA_FINGERPRINT = (
-    "c12a9560c065a6cc9be19b91b71b4ebee1b74eceff477040073f85943c43742f"
+    "a6c7f41188a196929ff4e7c257a181c100af9f0f7091aa93d428a9a7c0eec8ef"
 )
 EXPECTED_SANITIZER_FINGERPRINT = (
-    "dfef8699dd9519f2ef6592be577d25fdb2ae761e298d7063ced3323a14e2cf77"
+    # Bumped for WP10a Task 6: Memory._maintenance_store() reuses the identical
+    # `_SAFE_NS.sub("_", namespace) or "default"` expression (memory.py:298),
+    # adding a third matching line; sanitizer semantics unchanged, mirror valid.
+    "973f203a76ab1b535ae8e81dcb830145c92bf481cf4c815b5fbca0044e5d044c"
 )