WP10a: sleep-time maintenance — offline dedup/summarize/evict with a human review queue#3
Merged
Merged
Conversation
… claim packet Spec carries a two-round adversarial verification record (46 findings vs rev 1; independent 15-finding round vs rev 2 — the transitive duplicate-chain resurrection blocker is fixed in rev 3). Registers WP10a/ WP10b in the roadmap, amends the consolidation anti-goal to hosted-only, and claims WP10a (merge stays gated on WP1 launch). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0198CpP3tf1SbtDGaRX2NWvJ
Foundation for the sleep-time maintenance job (design spec §4.0, §7.1). - busy_timeout: SqliteStore gains busy_timeout_ms=1500 (kw-only); _connect issues PRAGMA busy_timeout. Maintenance opens stores with 5000 ms. - batch(): BEGIN IMMEDIATE unit-of-work with commit suppression routed through a single _commit() helper (all existing mutators converted), one COMMIT at exit, ROLLBACK on exception, non-re-entrant. execute() only. - retire_duplicate(loser, survivor): two-surface is_latest flip + superseded_by, valid_to untouched. Chain invariant maintained two ways — resolve survivor to live canonical (depth 1) and re-point existing open losers of the loser — so a transitive B->A->D retirement can't resurrect B. Rejects survivor==loser. - set_tier / get_embedding (float32 round-trip, no re-embed) / iter_latest_facts (id high-water) / iter_slots_touched_since (new-facts- since-cursor -> DISTINCT slots; catches dups on quiet slots). Store ABC updated with the six new abstract methods. Tests: tests/test_store_maintenance_verbs.py (15, offline) — two-surface sync, batch atomicity, get_embedding round-trip, cursor gap, transitive chain invariant. Full suite 155 passed, 1 skipped; four pinned files untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First persisted-format change. Restructures _init_schema into a user_version-gated migration: the always-run SCHEMA_SQL blob (all IF-NOT-EXISTS) now carries the three maintenance tables (fact_derivation, maintenance_run, maintenance_proposal) + indexes, while the non-idempotent `ALTER TABLE fact ADD COLUMN record_kind` lives ONLY in the `if user_version < 2:` branch, stamping 2 after and never lowering a newer stamp. Threads record_kind through the Fact dataclass, add_fact's INSERT, and _row_to_fact. Adds pure ledger/proposal CRUD to the Store ABC + SqliteStore (create_run/heartbeat_run/finish_run/get_live_run, stage_proposal/ get_proposal/list_proposals) — no decide/apply logic. The maintenance_run INSERT is the atomic lease claim via the ux_run_live partial-unique index. Ships the WP6 migration obligation: a checked-in tiny v1-format fixture DB (built by a committed deterministic script) + a migration regression test that opens it, upgrades once, and REOPENS cleanly — pinning the ALTER-idempotence trap with a real second open. Updates the console EXPECTED_SCHEMA_FINGERPRINT for the new CREATE TABLE lines. Core: 169 passed, 1 skipped. Console: 125 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Restore the ingest-commutation condition (spec §3.1.3), fixing the two empirically-demonstrated rev-1 wrong answers (resurrection, stale summary; §14). Hook 1 — duplicate-cascade in supersede_fact (§4.0): after closing `old`, close its OPEN retired duplicates (superseded_by=old, valid_to NULL) at the same V and RETURN [old_id] + cascade-closed ids. Single level suffices — Task 1's chain invariant keeps every open duplicate pointing directly at the live survivor. One transaction. supersede_fact return type None → list[str] (ABC updated). Hook 2 — summary-staleness cascade in _apply_supersession (§4.3): feed the FULL closed set returned by every supersede_fact call (explicit targets PLUS duplicate-cascade-closed rows — the load-bearing seam) into find_summaries_derived_from; each still-latest derived summary flips is_latest=0, valid_to=new.valid_at, invalidated_by=new.id (+vec mirror). memory.py stays free of raw SQL. New store methods (schema-v2 derivation surface): add_derivation, find_summaries_derived_from, invalidate_summary — all double-guarded on is_latest=1 so a re-fire is a no-op. Both hooks are exact no-ops until maintenance has produced rows — first-run path byte-unchanged (§10.10 pinned). Tests: new tests/test_ingest_commutation.py (9) — resurrection, transitive B→A→D, stale summary (vacuity-guarded), cascade-closed-duplicate seam, no-op pin. Suite: 178 passed, 1 skipped; four named pins green untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New src/lean_memory/maintain/ package (design spec §3.6, §4.1–§4.4): - MaintenanceConfig: frozen dataclass, all §3.6/§6.6 defaults; evict_threshold a documented conservative/tunable 0.15 (spec pins none); config_hash() = sha256 of canonical JSON. - score.value(fact, now): §4.4 standing value; recency anchor VERBATIM the retriever's (last_access or valid_at), DECAY_LAMBDA imported (never re-derived) — backfills score stale exactly as the ranker de-ranks them. - Summarizer seam: ExtractiveStubSummarizer (deterministic, pinned) + OllamaSummarizer behind an [llm] import guard; LM_FORCE_STUBS honored. - Four transforms as pure Store-ABC functions: dedup_exact (AUTO — value- preserving normalization only, argmin(valid_at)/min-id survivor, cluster usage-stats merge, per-slot batch()), dedup_near + summarize + evict_propose (PROPOSE — zero spine writes), evict_auto (strict band). run_transforms enforces the §4.4 intra-run ordering: stage all proposals over the pre- transform snapshot first, then autos excluding staged targets; shared proposal budget with reported (never silent) truncation. Store: add merge_usage_stats (ABC + SqliteStore) for the DEDUP-EXACT survivor stats merge — plain UPDATE, batch-aware. Tests (offline, +34): test_maintenance_transforms.py (config/score/summarizer/ normalization/dedup/evict/ordering/budget/zero-spine-delta) and test_maintenance_asof_grid.py (§10.1 store-predicate invariance over a T grid under autos+staging; §10.8 ranking-delta pin; no inverted intervals). Full suite: 212 passed, 1 skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MaintenanceRunner drives one maintenance run on one namespace store: atomic lease claim (ux_run_live) with fresh-skip / stale-takeover / lost-race paths; OR-combined work thresholds (facts, salience, age) with a cheap below-threshold no-op; advance-before-write cursor high-water and the iter_slots_touched_since cursor-gap fix; cross-run exclusion of facts under review plus an identical-pending dedupe guard for crash idempotence; heartbeats at phase and batch boundaries; takeover never rolls back. run_transforms gains extra_exclude_ids + pending_signatures (threaded to dedup_near/summarize as skip_signatures) — the runner's cross-run coupling, no logic duplicated. 16 new tests (real subprocess lease race, cursor-gap dedup, crash-resume convergence, heartbeat cadence); suite 228 passed, 1 skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three review findings on the maintenance runner: 1. Lease claim is now ONE transaction (spec §7.2): check → optional stale-abort → INSERT claim are wrapped in store.batch() (single BEGIN IMMEDIATE / COMMIT), IntegrityError caught OUTSIDE the batch after __exit__ rolls back. A failed claim now rolls back the stale-abort UPDATE too (prior run stays 'running', not stranded 'aborted') — pinned by a new atomicity test. 2. A below-threshold no-op no longer advances the cursor: it carries the last work-doing run's cursor forward, so a duplicate landing on a quiet slot stays in the next real run's iter_slots_touched_since window and is eventually deduped — pinned by a new regression test. 3. Added a last_finished_run(namespace) read verb to the Store ABC + SqliteStore; the runner uses it instead of reaching into store._db. Suite: 230 passed, 1 skipped (18 runner tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Proposal decide+apply lifecycle (maintain/lifecycle.py, new): approve|reject|edit| promote with the §5 CAS, one-batch approve-and-apply (CAS -> re-validate targets -> per-kind verbs -> applied_at), stale-target expiry that rolls the approve back so only the expiry commits (spine untouched), lazy timeout expiry, and model/embedding work computed before the batch window (§7.1). Store gains cas_decide_proposal / mark_proposal_applied / expire_proposal. Memory façade: maintain (dedicated 5000ms store, dry-run default = zero writes/no lease), review_queue (grouped by entity, lazy timeout expiry), decide, promote, and search(include_cold=). Runner + run_transforms + all five transforms gain a dry_run flag (default False; Task 4/5 behavior byte-identical). Retrieval: dense/sparse tier='hot' filter applied ONLY in default latest-mode (as_of never filters tier; include_cold opts out); find_latest_in_slot deliberately un-filtered so ingest supersession still sees cold facts. Tests: proposal lifecycle (CAS double-decide across two handles, re-apply -> already-applied, timeout + stale-target expiry with full-DB-hash spine invariance, edited-approve human provenance + source=user re-score, summarize apply end-to-end + old-summary supersession, evict+promote round-trip, dim-mismatch refusal, the §10.1 apply-path as-of grid re-run + a backdating fault-injection guard) and the as_of x include_cold x tier matrix. Full offline suite: 251 passed, 1 skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0198CpP3tf1SbtDGaRX2NWvJ
Memory._maintenance_store() reuses the identical namespace-sanitize expression, adding a third line matching the tripwire's pattern. Verified semantics unchanged (same _SAFE_NS regex, same 'or "default"' fallback), so the console config.py mirror stays valid — this is the tripwire's designed verify-then-bump workflow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0198CpP3tf1SbtDGaRX2NWvJ
Task 7 (spec §6.1, §7.1, §7.3). CLI (new maintain/cli.py + pyproject script): --root [--namespace] [--apply] [--auto-only] [--json]. Dry-run is the default (zero writes, no lease); --apply runs the auto band + stages proposals; --auto-only stages nothing. Reuses Memory.maintain() per namespace (dedicated 5000 ms store). Human report to its own stdout; --json for machines. --root falls back to $LM_DATA_ROOT. --auto-only threads down (default False) through run_transforms / runner.run / Memory.maintain, skipping the propose phases — all existing behavior/tests preserved. memory_clear now refuses (returns an explanatory string, does not raise) while a live fresh maintenance lease is held, via the shared runner.live_lease_is_fresh (reuses _heartbeat_is_fresh + _STALE_FLOOR_S — staleness rule not duplicated); a stale/absent lease clears as before. The runner stops cleanly at a batch boundary when its namespace file is unlinked mid-run (os.path.exists on store.path → internal _NamespaceCleared signal → RunReport aborted_file_gone; can't mask real errors). §7.3 residual sliver documented in the tool docstring. Tests (new test_maintenance_cli.py, 12): cross-process CLI --apply vs a live in-process writer interleave with no unhandled 'database is locked'; dry-run writes nothing; --auto-only stages nothing but merges; --json fields; missing-root nonzero; memory_clear refusal fresh vs stale; runner mid-run unlink stop. Full suite 263 passed, 1 skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Implementation surfaced the decisive fact the earlier boundary fix missed: both console MCP surfaces (observe_mcp.py stdio + routes/mcp.py HTTP) write only through EngineGateway, so the spec's v1 MCP tools cannot ship without the four gateway methods. Spec/plan/workpackets re-aligned; WP10b consumes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0198CpP3tf1SbtDGaRX2NWvJ
…to-spawn The four sleep-time maintenance tools (§6.3) — memory_maintenance_run (dry-run by default, symmetric with the CLI), memory_maintenance_status (model-free ledger read), memory_review_queue, memory_review_decide — registered with identical names/signatures on all three MCP surfaces: core stdio (mcp_server.py), console stdio (observe_mcp.py), console HTTP mount (routes/mcp.py). Both console surfaces reach the engine only through EngineGateway's four new methods (maintain/review_queue/decide/promote), each wrapping retry_busy + the per-namespace asyncio lock + the single worker thread exactly like add/search (§8). Status is provably model-free: a new maintain/mcp_support.py owns the raw SQL for the ledger read + staleness check, so mcp_server.py never calls _mem() for status (pinned by a test that makes the builder raise). The review-workflow prompt (review-memory-maintenance) is registered on the console stdio server and mirrored in plugin/commands/review-memory.md; both forbid the client agent from deciding without an explicit user verdict (§6.4). Opt-in auto-spawn (LM_MAINT_AUTO=1, default OFF) fires once per process on the first tool call with the exact §6.5 Popen primitives — fd 1 is never inherited. Parity pin updated to the exact six-tool set on both console surfaces; server manifests reconciled and pinned against the servers' actual tool surface (the v0.1.3 lesson). Both suites green: core 281 passed + 1 skipped, console 138. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
README "Sleep-time maintenance & review" section (user story, CLI, cron recipe, Claude Code review workflow, safety story); CHANGELOG Unreleased entry; ARCHITECTURE Phase 3 status table; WP10a packet row → implementation complete, awaiting review + merge (merge still gated on WP1 launch). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0198CpP3tf1SbtDGaRX2NWvJ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
An offline "sleep-time" maintenance job that runs between sessions and cleans up stored memory — deduplicating exact restatements, staging summaries of old fact clusters, and demoting low-value facts to a cold tier — while preserving the ADD-only spine and as-of query semantics. Everything judgmental lands in a human review queue, approved either in the console (WP10b) or conversationally through Claude Code via MCP tools + a shipped review prompt.
Design:
docs/superpowers/specs/2026-07-16-sleep-time-maintenance-design.md(rev 3, two-round adversarial verification record in §14). Plan:docs/superpowers/plans/2026-07-16-sleep-time-maintenance.md(9 tasks, all done).Highlights
batch()unit-of-work,busy_timeout,retire_duplicate(transitive chain invariant),set_tier, ledger/proposal CRUD; schema v2 via auser_version-gated migration with a checked-in v1 fixture + reopen test.BEGIN IMMEDIATEtransaction), heartbeat + stale takeover, work thresholds, cursor that survives no-op runs, crash-resume convergence.tier='hot'filter in default latest-mode only —as_ofqueries never filter tier;include_cold=Trueopt-out; byte-identical when nothing is cold.lean-memory-maintainCLI (dry-run default) + cron recipe;memory_clearlease-refusal; 4 MCP tools with identical signatures on all three servers (core stdio, console stdio the plugin ships, console HTTP) via 4 newEngineGatewaymethods;review-memory-maintenanceprompt +plugin/commands/review-memory.md(both forbid agent-initiated decisions); opt-in auto-spawn (LM_MAINT_AUTO=1, exact Popen primitives, fd 1 never inherited).Verification
test_spine,test_asof_sparse,test_functional_slot_supersession,test_search_now) andbench/untouched.Merge note
Per the packet decision in
docs/superpowers/workpackets.md, WP10a is post-launch work: merge is gated on WP1 (v0.1.3 tag + launch execution). The feature is default-off and touches nothing on the first-run path (pinned), so merging earlier is safe if the gate is consciously waived.🤖 Generated with Claude Code
https://claude.ai/code/session_0198CpP3tf1SbtDGaRX2NWvJ