diff --git a/PI-DREAM.md b/PI-DREAM.md new file mode 100644 index 0000000..2fd5217 --- /dev/null +++ b/PI-DREAM.md @@ -0,0 +1,89 @@ +# pi-dream — memory consolidation for pi-memory + +Local working notes for the `feat/pi-dream-consolidation` branch (PR: https://github.com/jayzeng/pi-memory/pull/34). + +> This file is local-only documentation. The upstream-facing docs live in `README.md`. + +## What it does + +After many sessions, `MEMORY.md` accumulates near-duplicate entries and older entries superseded by newer ones about the same topic. Bloat is injected into every session start — wasted context tokens + stale-recall risk. + +pi-dream detects both patterns and removes redundant older copies through the standard recovery-record pipeline (fully reversible). + +## Detection + +| Pattern | Method | Default threshold | +|---|---|---| +| Near-duplicates | Jaccard similarity over word tokens (≥3 chars) | ≥ 0.75 | +| Superseded | Similar older/newer pair on same topic | ≥ 0.6 similarity AND ≥ 7 days age gap | + +Entry unit = timestamped block (`` until next meta comment). Trailing unstamped lines after the last stamp belong to that entry (same semantics as `forgetBlocks`). + +## Commands & tools + +### `/pi-dream [auto|report|apply]` + +Drives the agent via `pi.sendUserMessage`. Modes: + +| Mode | Behavior | +|---|---| +| *(none)* = **auto** | Report → if ALL findings are pure duplicates (zero content loss) → applies immediately, shows recovery ID. If superseded entries found → stops, presents findings, asks first | +| `report` | Read-only full findings with previews | +| `apply` | Apply everything found, show removed entries + recovery ID | +| bad arg | Usage hint | + +### `memory_dream` tool (agent-invocable) + +``` +memory_dream {} # report mode (default) +memory_dream { mode: "report" } # read-only findings +memory_dream { mode: "apply" } # consolidate, returns recovery ID +memory_dream { duplicateSimilarity: 0.8 } # stricter dup threshold +memory_dream { supersedeSimilarity: 0.5 } # looser supersede detection +``` + +Returns: findings with previews / removed count + `recoveryId`. + +### Undo + +``` +memory_restore { recoveryId: "" } +``` + +Recovery records live in `~/.pi/agent/memory/recovery/.json` before any file mutation. + +## Files + +| Path | Role | +|---|---| +| `index.ts` | All logic: `parseMemoryBlocks`, `dreamSimilarity`, `dreamAnalyze`, `dreamDropIndices`, `dreamApply` (exported pure functions) + tool/command registration (~line 670 analysis, ~line 2355 tool) | +| `test/unit.test.ts` | `describe("pi-dream consolidation")` — 10 tests | + +Pipeline on apply: `dreamAnalyze` → `dreamDropIndices` → `dreamApply` → `writeRecoveryRecord("long_term")` → write file → `snapshotDirty = true` → `scheduleQmdUpdate()`. + +## Install (this machine) + +`~/.pi/agent/settings.json`: + +```json +"git:github.com/KrissTos/pi-memory@feat/pi-dream-consolidation" +``` + +Note: branch separator is `@`, not `#`. After merge upstream, switch back to `"npm:pi-memory"` and run `pi update --extensions`. + +## Dev workflow + +```bash +cd ~/Projects/pi-memory-pr +bun test test/unit.test.ts # 192 tests +bunx tsc --noEmit # typecheck +bunx biome check index.ts # lint/format (pre-commit hook enforces all three) +git push origin feat/pi-dream-consolidation # auto-updates PR #34 +``` + +Pre-commit hooks (`.githooks/`) run tests + lint on every commit — commit fails if format drifts; fix with `bunx biome check --write index.ts`. + +## Roadmap ideas + +- Weekly scheduled run via `pi-schedule-prompt`: recurring prompt "Run memory_dream report; apply if pure duplicates only" (validate judgment on real runs first) +- Phase 2: semantic contradiction merging (needs LLM call, e.g. exit-summary-style model access) diff --git a/README.md b/README.md index 819156e..a6b658c 100644 --- a/README.md +++ b/README.md @@ -41,8 +41,8 @@ pi install npm:pi-memory pi install ./pi-memory ``` -That's it — the six core tools (`memory_write`, `memory_forget`, `memory_restore`, -`memory_read`, `scratchpad`, `memory_status`) work immediately with no other setup. +That's it — the core tools (`memory_write`, `memory_forget`, `memory_restore`, +`memory_dream`, `memory_read`, `scratchpad`, `memory_status`) work immediately with no other setup. Search is opt-in below. ### Optional: enable search with qmd @@ -80,6 +80,7 @@ Without qmd, the core tools still work fully — only `memory_search` and select | `memory_write` | Write to MEMORY.md (long-term) or daily log | | `memory_forget` | Delete matching entries and create a durable recovery record | | `memory_restore` | Restore a deletion using the recovery ID returned by `memory_forget` | +| `memory_dream` | Consolidate MEMORY.md: report/apply near-duplicate and superseded entries (pi-dream) | | `memory_read` | Read any memory file or list daily logs | | `scratchpad` | Add/done/undo/clear/list checklist items | | `memory_search` | Search across all memory files (requires qmd) | @@ -175,6 +176,7 @@ This ensures in-progress context survives compaction and is visible in the next - **Persistence**: Memory files are plain markdown on disk — readable, editable, and git-friendly. - **Recoverable deletion**: `memory_forget` stores complete deleted entries under `recovery/` before changing memory and returns a recovery ID that `memory_restore` can use. Recovery JSON is outside qmd's `**/*.md` index. +- **pi-dream consolidation** (`memory_dream` tool, `/pi-dream` command): detects near-duplicate entries (Jaccard similarity over word tokens) and older entries superseded by newer ones about the same topic. `mode='report'` analyzes without touching files; `mode='apply'` removes redundant older entries through the same recovery-record pipeline as `memory_forget`, so consolidation is undoable. Thresholds are tunable via `duplicateSimilarity` / `supersedeSimilarity` parameters. - **Tool response previews**: Write/scratchpad tools return size-capped previews instead of full file contents. - **qmd auto-setup**: On first session start with qmd available, the extension creates the collection and path contexts automatically. - **qmd re-indexing**: After every write, a debounced `qmd update` runs in the background (fire-and-forget, non-blocking) unless disabled via `PI_MEMORY_QMD_UPDATE`. diff --git a/index.ts b/index.ts index 3ad496d..c521b64 100644 --- a/index.ts +++ b/index.ts @@ -724,6 +724,187 @@ export function forgetBlocks(content: string, match: string): { content: string; }; } +// --------------------------------------------------------------------------- +// pi-dream: memory consolidation analysis +// --------------------------------------------------------------------------- + +export interface DreamBlock { + /** Timestamp meta comment line ("" for unstamped paragraph blocks). */ + meta: string; + body: string; + timestamp: Date | null; +} + +export interface DreamAnalysisOptions { + /** Jaccard similarity at/below which two entries are considered duplicates. Default 0.75. */ + duplicateSimilarity?: number; + /** Jaccard similarity at which an older entry counts as superseded by a newer one. Default 0.6. */ + supersedeSimilarity?: number; + /** Minimum age gap in days between a superseded entry and its newer replacement. Default 7. */ + supersedeMinAgeDays?: number; +} + +export interface DreamAnalysis { + blocks: DreamBlock[]; + /** Groups of block indices (length > 1) whose text is near-identical. */ + duplicateGroups: number[][]; + /** Pairs where the older entry is considered superseded by the newer one. */ + superseded: Array<{ olderIndex: number; newerIndex: number }>; +} + +const DREAM_TIMESTAMP_REGEX = + /^`; + + test("parseMemoryBlocks splits stamped entries and unstamped paragraphs", () => { + const content = [ + "# Heading", + "", + stamp("2026-01-01 10:00:00"), + "first entry body", + "", + stamp("2026-02-01 10:00:00"), + "second entry body line one", + "second entry body line two", + "", + "unstamped paragraph one", + "", + "unstamped paragraph two", + ].join("\n"); + const blocks = parseMemoryBlocks(content); + expect(blocks.length).toBe(3); // "# Heading" + 2 stamped entries (trailing unstamped lines belong to last entry) + expect(blocks[1].meta).toContain("2026-01-01"); + expect(blocks[2].body).toContain("second entry body line two"); + expect(blocks[blocks.length - 1].body).toContain("unstamped paragraph two"); + }); + + test("dreamSimilarity is high for near-duplicates and low for unrelated text", () => { + const a = "User prefers bun test over vitest for the memory package"; + const b = "user prefers bun test over vitest for the memory package!"; + const c = "Deploy pipeline runs on Fridays via GitHub Actions"; + expect(dreamSimilarity(a, b)).toBeGreaterThan(0.9); + expect(dreamSimilarity(a, c)).toBeLessThan(0.2); + }); + + test("dreamAnalyze groups near-identical entries as duplicates", () => { + const body = "API key rotation happens monthly using rotate-keys script"; + const content = [ + stamp("2026-01-01 10:00:00"), + body, + "", + stamp("2026-03-01 10:00:00"), + `${body} (unchanged)`, + ].join("\n"); + const analysis = dreamAnalyze(content); + expect(analysis.duplicateGroups.length).toBe(1); + expect(dreamDropIndices(analysis)).toEqual([0]); // older copy dropped + }); + + test("dreamAnalyze flags older entries superseded by newer ones", () => { + const content = [ + stamp("2026-01-01 10:00:00"), + "deployment target is staging.example.com with manual approval step before release", + "", + stamp("2026-06-01 10:00:00"), + "deployment target is staging.example.com with automated approval gate before release", + ].join("\n"); + const analysis = dreamAnalyze(content); + expect(analysis.duplicateGroups.length).toBe(0); + expect(analysis.superseded).toEqual([{ olderIndex: 0, newerIndex: 1 }]); + }); + + test("supersede detection requires minimum age gap", () => { + const content = [ + stamp("2026-06-01 10:00:00"), + "database host is db-primary.internal port 5432 with connection pool of twenty", + "", + stamp("2026-06-03 10:00:00"), + "database host is db-replica.internal port 5432 with connection pool of twenty", + ].join("\n"); + const analysis = dreamAnalyze(content, { supersedeSimilarity: 0.5 }); + // Only 2 days apart — below default min age gap; also below duplicate threshold. + expect(analysis.superseded.length).toBe(0); + }); + + test("dreamApply removes older copies, keeps newest, returns full removed blocks", () => { + const body = "release checklist lives in docs/release.md and is updated quarterly"; + const content = [stamp("2026-01-01 10:00:00"), body, "", stamp("2026-05-01 10:00:00"), body].join("\n"); + const result = dreamApply(content); + expect(result.removed.length).toBe(1); + expect(result.removed[0]).toContain("2026-01-01"); + expect(result.keptContent).toContain("2026-05-01"); + expect(result.keptContent).not.toContain("2026-01-01"); + }); + + let dreamTools: Record; + + beforeEach(() => { + setupTmpDir(); + const mockPi = createMockPi(); + dreamTools = mockPi.tools; + registerExtension(mockPi.pi as any); + }); + + test("memory_dream registers with correct name", () => { + expect(dreamTools.memory_dream).toBeDefined(); + }); + + test("memory_dream report mode leaves file untouched", async () => { + fs.writeFileSync( + path.join(tmpDir, "MEMORY.md"), + `${stamp("2026-01-01 10:00:00")}\ndup body shared words here\n\n${stamp("2026-02-01 10:00:00")}\ndup body shared words here\n`, + "utf-8", + ); + const result = await dreamTools.memory_dream.execute("c1", { mode: "report" }, null, null, {}); + expect(result.content[0].text).toContain("pi-dream report"); + expect(fs.readFileSync(path.join(tmpDir, "MEMORY.md"), "utf-8")).toContain("dup body shared words"); // unchanged + }); + + test("memory_dream apply mode removes duplicates with recovery id", async () => { + fs.writeFileSync( + path.join(tmpDir, "MEMORY.md"), + `${stamp("2026-01-01 10:00:00")}\ndup body shared words here\n\n${stamp("2026-02-01 10:00:00")}\ndup body shared words here\n`, + "utf-8", + ); + const result = await dreamTools.memory_dream.execute("c1", { mode: "apply" }, null, null, {}); + expect(result.content[0].text).toContain("Removed 1 redundant entry"); + expect(result.details.recoveryId).toBeDefined(); + const remaining = fs.readFileSync(path.join(tmpDir, "MEMORY.md"), "utf-8"); + expect(remaining).toContain("2026-02-01"); + expect(remaining).not.toContain("2026-01-01"); + + // Recovery record must restore the removed entry. + const restored = await dreamTools.memory_restore.execute( + "c2", + { recoveryId: result.details.recoveryId }, + null, + null, + {}, + ); + expect(fs.readFileSync(path.join(tmpDir, "MEMORY.md"), "utf-8")).toContain("2026-01-01"); + void restored; + }); + + test("memory_dream reports healthy memory when nothing removable", async () => { + fs.writeFileSync(path.join(tmpDir, "MEMORY.md"), "one unique fact about bun test runner config\n", "utf-8"); + const result = await dreamTools.memory_dream.execute("c1", {}, null, null, {}); + expect(result.content[0].text).toContain("Memory looks healthy"); + }); +});