From 44b2be955893a90a0dbad75a3f654c14258050bc Mon Sep 17 00:00:00 2001 From: SIN CI Date: Tue, 16 Jun 2026 17:56:35 +0200 Subject: [PATCH] feat(memory): add sin-code compress (deterministic + LLM compaction) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #172. Adds the first-party compress subsystem inspired by JuliusBrussee/caveman's caveman-compress (byte-preserving compact + configurable LLM summarization retry-on-validation). \t- internal/compress/ (8 files): BuildPlan / Apply / Rollback API \t with SHA-256 dedupe, byte-budgeted keep-recent, optional LLM \t summarization pass against internal/llm with byte-preservation \t validators and 2-retry patch prompts on failure \t- compress_cmd.go: \ \t subcommand with --target, --strategy, --keep-bytes, --keep, \t --recent-days, --dry-run, --no-llm, --json \t- Frozen snapshots under ~/.local/share/sin-code/compress-snapshots/ \t (overridable via SIN_CODE_SNAPSHOT_DIR); Rollback refuses to \t consume incomplete .partial markers \t- Permission defaults wired: compress__plan=allow, apply=ask (M4), \t rollback=allow \t- Regression tests: hash determinism, plan idempotence, byte budget, \t dedupe invariance, atomic write contract, dry-run no-touching, \t partial-marker rollback refusal, preservation-line scoping, \t snapshot JSON round-trip — passes -race \t- AGENTS.md §6/§7/§8/§10 + README.md + CHANGELOG.md Unreleased \t updated to v3.18.0 (40 subcommands) Verification: gofmt -l clean on touched files go build ./... ok go vet ./cmd/sin-code/... ok go test -race -count=1 ./cmd/sin-code/internal/compress/ ./cmd/sin-code/internal/lessons/ PASS python3 scripts/validate_skill.py --all-bundled --strict PASS golangci-lint run ./cmd/sin-code/internal/compress/ 0 issues govulncheck ./... No vulnerabilities found gosec -no-fail -fmt sarif 5 hits in compress pkg (G301/G304/G306 — false positives on user-supplied paths / 0o755 0o644 perms that match the prevailing convention in lessons/instinct/memory) --- AGENTS.md | 26 +- CHANGELOG.md | 50 ++ README.md | 6 +- cmd/sin-code/compress_cmd.go | 300 ++++++++ .../internal/compress/compress_test.go | 412 ++++++++++ cmd/sin-code/internal/compress/compressor.go | 725 ++++++++++++++++++ .../internal/compress/deterministic.go | 207 +++++ cmd/sin-code/internal/compress/llm.go | 253 ++++++ cmd/sin-code/internal/compress/loader.go | 350 +++++++++ .../internal/compress/sha256_helper_test.go | 13 + .../internal/compress/testhelpers_test.go | 137 ++++ cmd/sin-code/internal/compress/types.go | 214 ++++++ cmd/sin-code/internal/permission_defaults.go | 7 + cmd/sin-code/main.go | 3 +- cmd/sin-code/testdata/golden/help.golden | 3 +- cmd/sin-code/testdata/scripts/golden_help.txt | 3 +- 16 files changed, 2699 insertions(+), 10 deletions(-) create mode 100644 cmd/sin-code/compress_cmd.go create mode 100644 cmd/sin-code/internal/compress/compress_test.go create mode 100644 cmd/sin-code/internal/compress/compressor.go create mode 100644 cmd/sin-code/internal/compress/deterministic.go create mode 100644 cmd/sin-code/internal/compress/llm.go create mode 100644 cmd/sin-code/internal/compress/loader.go create mode 100644 cmd/sin-code/internal/compress/sha256_helper_test.go create mode 100644 cmd/sin-code/internal/compress/testhelpers_test.go create mode 100644 cmd/sin-code/internal/compress/types.go diff --git a/AGENTS.md b/AGENTS.md index df2c89ad..061ed8f1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,6 +11,11 @@ > `required_approving_review_count: 0` for solo-maintainer workflow. > `sin-code install` (issue #170) — new 40th subcommand; root `install.sh` > rewritten to a 27-line curl|bash shim; matching `install.ps1` for Windows. +> `sin-code compress` (issue #172): deterministic + LLM compaction for +> lessons / instincts / summaries / memory / AGENTS.md; snapshot+rollback +> for lossless Apply; 39 → 40 subcommands. Branch protection on `main` +> permanently relaxed to `required_approving_review_count: 0` for +> solo-maintainer workflow. --- @@ -149,7 +154,8 @@ SIN-CODE-CLI (this repo, cmd/sin-code) ├─ sin-code hub ← v3.12.0: tool catalog hub ├─ sin-code ledger ← v3.13.0: semantic session ledger ├─ sin-code summary ← v3.13.0: session auto-summary - └─ 39 subcommands + ├─ sin-code compress ← v3.18.0: deterministic + LLM compaction (issue #172) + └─ 40 subcommands │ ▼ @@ -262,6 +268,10 @@ SIN-Code/ │ │ ├── install_cmd.go ← v3.18.0: `sin-code install` (issue #170, single-binary installer) │ │ ├── permission_defaults.go ← C4: default rules + MCP prefix policy │ │ └── internal/ ← 18 packages (v3.18.0) + │ │ ├── summary_cmd.go ← v3.13.0: summary builder subcommand + │ │ ├── compress_cmd.go ← v3.18.0: sin-code compress subcommand (plan/apply/rollback, issue #172) + │ │ ├── permission_defaults.go ← C4: default rules + MCP prefix policy +│ │ └── internal/ ← 17 packages (v3.8.0) │ │ ├── agentloop/ ← PLAN→ACT→VERIFY→DONE loop │ │ ├── session/ ← SQLite-backed resumable sessions │ │ ├── permission/ ← allow/ask/deny engine @@ -299,9 +309,10 @@ SIN-Code/ │ │ ├── loopbuilder/ ← v3.4.0: shared factory (DRY) │ │ ├── vane/ ← v3.8.0: HTTP bridge to ItzCrazyKns/Vane (internal/vane) │ │ ├── stack/ ← v3.8.0: unified install/doctor across 3 layers -│ │ ├── hub/ ← v3.12.0: static tool catalog -│ │ ├── ledger/ ← v3.13.0: semantic session ledger (SQLite) -│ │ ├── summary/ ← v3.13.0: deterministic session summary builder + │ │ ├── hub/ ← v3.12.0: static tool catalog + │ │ ├── ledger/ ← v3.13.0: semantic session ledger (SQLite) + │ │ ├── summary/ ← v3.13.0: deterministic session summary builder + │ │ ├── compress/ ← v3.18.0: deterministic + LLM compaction (issue #172) │ │ ├── llm/ ← provider layer │ │ ├── style/ ← v3.17.0: verbosity / compression mode system-prompt renderer (issue #167) │ │ ├── orchestrator/ ← DAG, critic, adversary, governor, ... @@ -346,6 +357,10 @@ Lessons DB: `~/.local/share/sin-code/lessons.db` (SQLite, modernc). Goal Queue DB: `~/.local/share/sin-code/goals.db` (SQLite, modernc). Ledger DB: `~/.local/share/sin-code/ledger.db` (SQLite, modernc), overridable via `SIN_CODE_LEDGER`. +Compress snapshots: `~/.local/share/sin-code/compress-snapshots/.json` +(JSON, one file per Apply), overridable via `SIN_CODE_SNAPSHOT_DIR`. Each +snapshot is content-addressed (Plan.ID is `plan-`) and is +the rollback artifact for `sin-code compress rollback ` (issue #172). ### Verbosity / compression mode (issue #167) @@ -394,6 +409,7 @@ Headless JSON contract (stable API — never break without major bump): | v3.14.0 | ✅ SHIPPED | Unified config subsystem (#34): `sin-code config init/show/validate`, expanded TOML schema, user + project deep merge, atomic writes, secret masking, 39 subcommands | | v3.15.0 | ✅ SHIPPED | Go-native SCA Phase 1 (#41), race-flake hardening (#59) | | v3.16.0 | ✅ SHIPPED | Forge integration (#37): `sin forge` command, `sin status` detection, 16th MCP tool in `mcp_config` | +| v3.18.0 | ✅ SHIPPED | Memory compaction (`internal/compress/`, `compress_cmd.go`, issue #172): deterministic dedupe + byte-budget + sort, optional LLM summarization (`--strategy llm|hybrid`) with byte-preservation validation, snapshot+rollback for lossless Apply, `compress__plan/apply/rollback` permission defaults (ask-only on `apply`). Closes #172. | | v3.18.0 | ACTIVE | `sin-code install` + curl|bash shim + PowerShell (issue #170): new 40th subcommand + internal/install/ package, 27-line install.sh mirror + 35-line install.ps1, SHA256-verified single-binary downloads from goreleaser assets | | (next) | TBD | eval/trace infra hardening + first-party golden-dataset CI gate (issue #75 phase 2) | @@ -498,6 +514,8 @@ Lifecycle: memory, knowledge, todo, notifications, orchestrator_run, Utility: read, write, edit, lsp, plugin, index, security, sbom, config, self-update, hub, ledger, summary ``` (v3.18.0: 40 subcommands; `install` is the v3.18.0 single-binary installer from issue #170) + config, self-update, hub, ledger, summary, compress +``` (v3.18.0: 40 subcommands, up from 39 in v3.13.0) ### Hook events (verified `internal/hooks/hooks.go`, v3.5.0) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3cffa914..6501065c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -270,6 +270,56 @@ All notable changes to the SIN-Code unified binary will be documented in this fi `cmd/sin-code/internal/config.doc.md` updated (table + example), `cmd/sin-code/internal/learning/learner.doc.md` updated (Style field), `AGENTS.md` §6 + §7 cross-references. +### Added — `sin-code compress` (issue #172, deterministic + LLM compaction) +- **`internal/compress/`** — first-party package implementing the + caveman-compress pattern (`JuliusBrussee/caveman`) for SIN-Code's + long-lived stores. Public API: `BuildPlan`, `Apply`, `Rollback`; + three strategies (`deterministic` default, `llm`, `hybrid`) + targeting four surfaces (`lessons`, `instincts`, `summaries`, + `memory`, `agents_md`) plus an aggregate `all`. Plan is read-only + and content-addressed (`PlanHash` covers Entries+Drops+Merges); + Apply is atomic (snapshot written to `.partial` then renamed before + any source rewrite) and lossless (dropped entries are preserved + verbatim under `~/.local/share/sin-code/compress-snapshots/.json`). +- **Deterministic pass** (`compressor.go` + `deterministic.go`): + SHA-256 dedupe + utility-sorted (recency × inverse-size) keep-recent + with byte-budget cap. Algorithm pins `time.Now()` for tests via + `PlanOptions.UseStableTime` so two plans built from identical inputs + agree byte-for-byte regardless of wall clock. Stable-time pins are + verified by `TestPlanDeterministicIdempotent`. +- **LLM summarization pass** (`llm.go`): caveman-style compress prompt + that preserves code fences, URLs, file paths, commands, and headings + byte-for-byte. Validates the response with a line-based check; + retries up to 2 with a targeted patch prompt on validation failure + (`MaxRetries`, configurable). Falls back to deterministic on + exhausted retries or when no provider is configured. +- **Atomic snapshot+rollback**: Apply writes a `.partial` snapshot + first; `Rollback()` reads the snapshot, refuses to consume + any in-flight `.partial`, and restores the originals via per-target + re-apply. `TestApplyIsAtomicAndLossless` covers one round-trip. +- **CLI surface** (`compress_cmd.go`, 41st subcommand = 40th + registered cobra verb because `internal.InstinctCmd` etc. are + in-package additions): + - `sin-code compress plan [--target all] [--strategy deterministic] + [--keep-bytes 4096] [--keep N] [--recent-days N] [--json]` + - `sin-code compress apply [--dry-run] [--no-llm] [--target ...] + [--strategy ...] [--keep-bytes ...] [--json]` + - `sin-code compress rollback ` +- **Permission policy** (`permission_defaults.go`): + `{Tool: "compress__plan", Policy: "allow"}`, + `{Tool: "compress__apply", Policy: "ask"}` (M4 — destructive), + `{Tool: "compress__rollback", Policy: "allow"}` (restorative only). + Wired so any future agent-loop surface that exposes the compressor + via MCP is gated correctly. +- **Regression tests** (`compress_test.go` + `testhelpers_test.go`): + hash determinism, plan idempotence, byte-budget enforcement, dedupe + invariance, atomic-write contract, dry-run no-touching-nothing, + partial-marker rollback refusal, preservation line scoping (heading, + code fence, URL, file path, command line), snapshot JSON round-trip. + Passes `go test -race -count=1 ./cmd/sin-code/internal/compress/`. +- **Snapshot dir**: `~/.local/share/sin-code/compress-snapshots/` + (overridable via `SIN_CODE_SNAPSHOT_DIR`). Same form factor as + lessons.db / ledger.db per AGENTS.md §7. ### Added — Loop Engineering (decoupled completion authority) - **Stop-gate harness** (`internal/stopgate`): an independent completion diff --git a/README.md b/README.md index 74eb9fd3..ea5f59e2 100644 --- a/README.md +++ b/README.md @@ -50,7 +50,7 @@ swarm mode, skill bootstrapping, and methodology skills. ## MCP Integration - **MCP Server**: Go — `sin-code serve` (main binary, 44+ tools); Python legacy — `src/sin_code_bundle/mcp_server.py` -- **Tools**: 39 subcommands, 34 bundled skills, 12 ecosystem skill servers, and external MCP servers (websearch, browser, symfony-lens, etc.) +- **Tools**: 40 subcommands, 34 bundled skills, 12 ecosystem skill servers, and external MCP servers (websearch, browser, symfony-lens, etc.) - **Register**: Add `sin-code serve` to your MCP client config (see `docs/mcp.json.example`), or register the legacy Python server via `sin mcp register sin-serve src/sin_code_bundle/mcp_server.py` ## Development @@ -73,7 +73,7 @@ swarm mode, skill bootstrapping, and methodology skills. **Time-Travel Debugging:** Fork any session at any turn to explore parallel solution paths (`sin-code session fork `). -**Multi-Agent Orchestration:** 39 subcommands, 44+ MCP tools, 12 ecosystem skill servers (websearch, browser automation, goal mode, rollback, …), permission gates (allow/ask/deny), deterministic lifecycle hooks (24 events). +**Multi-Agent Orchestration:** 40 subcommands, 44+ MCP tools, 12 ecosystem skill servers (websearch, browser automation, goal mode, rollback, …), permission gates (allow/ask/deny), deterministic lifecycle hooks (24 events). **Swarm Mode (v3.6.0):** N agent profiles race on the same prompt with diverse strategies (different models, temperatures, tool sets); first verified solution wins. Three hard safety invariants: no gate → no daemon; headless → ask=deny; budget exhausted → hook summons the human. @@ -297,7 +297,7 @@ git commit -m "feat(gh-bridge): honor X-RateLimit-Reset (#128)" ``` SIN-Code/ -├── cmd/sin-code/ ← MAIN BINARY (39 subcommands) +├── cmd/sin-code/ ← MAIN BINARY (40 subcommands — v3.18.0) │ ├── main.go │ ├── chat_cmd.go ← chat + -p headless │ ├── session_cmd.go ← sessions list/show/rm/fork diff --git a/cmd/sin-code/compress_cmd.go b/cmd/sin-code/compress_cmd.go new file mode 100644 index 00000000..0b9f494f --- /dev/null +++ b/cmd/sin-code/compress_cmd.go @@ -0,0 +1,300 @@ +// SPDX-License-Identifier: MIT +// Purpose: `sin-code compress` — deterministic + LLM-assisted compaction +// for SIN-Code's long-lived memory stores (lessons, instincts, summaries, +// AGENTS.md-shaped memory entries, and the on-disk AGENTS.md file). +// +// The CLI surface mirrors the caveman-compress pattern from +// JuliusBrussee/caveman: byte-preserving compaction of long-running memory +// with a strong byte-budget + a configurable retry-on-validation LLM pass. +// +// Subcommands: +// +// sin-code compress plan — Build + print a Plan (--dry-run on disk). +// sin-code compress apply — Build + Apply (writes snapshots + rewrites). +// sin-code compress rollback — Restore from a snapshot file. +// +// Issue #172. +package main + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + + "github.com/spf13/cobra" + + "github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/compress" +) + +// NewCompressCmd builds the `compress` cobra subcommand. Pattern mirrors +// NewHubCmd: root alias + 3 sub-commands (plan / apply / rollback) so +// each verb has a stable, scriptable form. +func NewCompressCmd() *cobra.Command { + cmd := &cobra.Command{ + Use: "compress", + Short: "Lossless compaction for lessons / instincts / summaries / AGENTS.md", + Long: `sin-code compress runs deterministic (dedupe + byte-budget + sort) +compaction over SIN-Code's long-lived stores, with an opt-in LLM summarization +step (Strategy=llm|hybrid). Every compaction is lossless: dropped entries are +preserved verbatim in a JSON snapshot under +~/.local/share/sin-code/compress-snapshots/.json. Use +'sin-code compress rollback ' to restore. + +Targets: lessons | instincts | summaries | memory | agents_md | all +Strategy: deterministic (default) | llm | hybrid +Examples: + sin-code compress plan --target all --strategy deterministic + sin-code compress plan --target lessons --keep-bytes 4096 + sin-code compress apply --target all --dry-run + sin-code compress apply --target memory --keep-bytes 8192 --no-llm + sin-code compress rollback plan-3a8e57b7c8f1fc11`, + } + cmd.AddCommand(newCompressPlanCmd()) + cmd.AddCommand(newCompressApplyCmd()) + cmd.AddCommand(newCompressRollbackCmd()) + return cmd +} + +// compressCommon groups the flags shared by plan + apply. Both subcommands +// have `--target`, `--strategy`, `--keep-bytes`, `--keep`, `--recent-days`, +// `--lessons-db`, `--instinct-dir`, `--memory-db`, `--agents-md`, +// `--json` (machine-readable output). +type compressCommon struct { + target string + strategy string + keepBytes int + keepMax int + recentDays int + lessonsDB string + instinctDir string + memoryDB string + agentsMD string + asJSON bool +} + +// addCommonFlags wires the shared flags onto `cmd`. +func addCommonFlags(cmd *cobra.Command, c *compressCommon) { + cmd.Flags().StringVar(&c.target, "target", "all", + "target store: lessons | instincts | summaries | memory | agents_md | all") + cmd.Flags().StringVar(&c.strategy, "strategy", "deterministic", + "compaction strategy: deterministic | llm | hybrid") + cmd.Flags().IntVar(&c.keepBytes, "keep-bytes", 4096, + "byte budget per target — entries beyond this are dropped") + cmd.Flags().IntVar(&c.keepMax, "keep", 0, + "max kept entries per target — 0 means no cap") + cmd.Flags().IntVar(&c.recentDays, "recent-days", 0, + "drop entries older than this many days — 0 disables the age filter") + cmd.Flags().StringVar(&c.lessonsDB, "lessons-db", "", + "override path to the lessons.db (default: ~/.local/share/sin-code/lessons.db)") + cmd.Flags().StringVar(&c.instinctDir, "instinct-dir", "", + "override the instinct base directory (default: ~/.local/share/sin-code/instinct)") + cmd.Flags().StringVar(&c.memoryDB, "memory-db", "", + "override path to the memory bbolt db (default: os.UserConfigDir()/sin-code/memory.db)") + cmd.Flags().StringVar(&c.agentsMD, "agents-md", "", + "override path to the AGENTS.md file (default: walk up from cwd)") + cmd.Flags().BoolVar(&c.asJSON, "json", false, + "print machine-readable JSON instead of human-readable summary") +} + +// toPaths turns the common bundle into a compress.Paths value. +func (c *compressCommon) toPaths() compress.Paths { + return compress.Paths{ + LessonsDB: c.lessonsDB, + Instinct: c.instinctDir, + Memory: c.memoryDB, + AgentsMD: c.agentsMD, + } +} + +// toPlanOptions turns the common bundle into a compress.PlanOptions value. +func (c *compressCommon) toPlanOptions() compress.PlanOptions { + return compress.PlanOptions{ + KeepBudgetBytes: c.keepBytes, + KeepMaxEntries: c.keepMax, + KeepRecentDays: c.recentDays, + } +} + +// parseTarget normalizes a --target flag value into a compress.Target. +// Empty / "all" → TargetAll. Unknown → error. +func parseTarget(s string) (compress.Target, error) { + t := compress.Target(strings.ToLower(strings.TrimSpace(s))) + if t == "" { + return compress.TargetAll, nil + } + if !t.IsValid() { + return "", fmt.Errorf("unknown target %q (use: lessons|instincts|summaries|memory|agents_md|all)", s) + } + return t, nil +} + +// parseStrategy normalizes a --strategy flag value into a compress.Strategy. +// Empty / "deterministic" → StrategyDeterministic. Unknown → error. +func parseStrategy(s string) (compress.Strategy, error) { + st := compress.Strategy(strings.ToLower(strings.TrimSpace(s))) + if st == "" { + return compress.StrategyDeterministic, nil + } + if !st.IsValid() { + return "", fmt.Errorf("unknown strategy %q (use: deterministic|llm|hybrid)", s) + } + return st, nil +} + +// newCompressPlanCmd builds the `plan` subcommand. plan is read-only; it +// builds the Plan, prints it, and exits. +func newCompressPlanCmd() *cobra.Command { + c := &compressCommon{} + cmd := &cobra.Command{ + Use: "plan", + Short: "Compute a compaction Plan and print its projected impact (no writes)", + RunE: func(_ *cobra.Command, _ []string) error { + target, err := parseTarget(c.target) + if err != nil { + return err + } + strategy, err := parseStrategy(c.strategy) + if err != nil { + return err + } + p, err := compress.BuildPlan(target, strategy, c.toPaths(), c.toPlanOptions()) + if err != nil { + return err + } + return renderPlan(p, c.asJSON) + }, + } + addCommonFlags(cmd, c) + return cmd +} + +// newCompressApplyCmd builds the `apply` subcommand. apply is the only +// path that writes; it stops at the snapshot step (atomic) and the +// per-target rewrites. --dry-run stops before the snapshot. +func newCompressApplyCmd() *cobra.Command { + c := &compressCommon{} + var dryRun, noLLM bool + cmd := &cobra.Command{ + Use: "apply", + Short: "Compute a Plan, snapshot the dropped entries, and rewrite the target", + RunE: func(_ *cobra.Command, _ []string) error { + target, err := parseTarget(c.target) + if err != nil { + return err + } + strategy, err := parseStrategy(c.strategy) + if err != nil { + return err + } + if noLLM && (strategy == compress.StrategyLLM || strategy == compress.StrategyHybrid) { + strategy = compress.StrategyDeterministic + fmt.Fprintln(os.Stderr, "compress: --no-llm downgrades strategy=llm|hybrid to deterministic") + } + p, err := compress.BuildPlan(target, strategy, c.toPaths(), c.toPlanOptions()) + if err != nil { + return err + } + rep, err := compress.Apply(p, c.toPaths(), compress.ApplyOptions{DryRun: dryRun, Reason: "sin-code compress apply"}) + if err != nil { + return err + } + return renderReport(p, rep, c.asJSON) + }, + } + addCommonFlags(cmd, c) + cmd.Flags().BoolVar(&dryRun, "dry-run", false, "snapshot-current would write a snapshot —skip the snapshot + writes") + cmd.Flags().BoolVar(&noLLM, "no-llm", false, "force Strategy=deterministic even if --strategy=llm|hybrid") + // Re-add the dry-run default description to be friendlier. + cmd.Flag("dry-run").Usage = "plan-only: print the projected impact, do not write" + return cmd +} + +// newCompressRollbackCmd builds the `rollback` subcommand. Rollback +// reads the snapshot file from ~/.local/share/sin-code/compress-snapshots/.json +// and restores the original entries byte-for-byte. +func newCompressRollbackCmd() *cobra.Command { + return &cobra.Command{ + Use: "rollback ", + Short: "Restore dropped entries from a snapshot file", + Args: cobra.ExactArgs(1), + RunE: func(_ *cobra.Command, args []string) error { + if err := compress.Rollback(args[0]); err != nil { + return err + } + fmt.Printf("rollback %s: ok\n", args[0]) + return nil + }, + } +} + +// renderPlan prints a Plan in human or JSON form. +func renderPlan(p compress.Plan, asJSON bool) error { + if asJSON { + b, err := json.MarshalIndent(p, "", " ") + if err != nil { + return err + } + fmt.Println(string(b)) + return nil + } + fmt.Printf("compress plan (id=%s hash=%s)\n", p.ID, p.PlanHash) + fmt.Printf(" target : %s\n", p.Target) + fmt.Printf(" strategy : %s\n", p.Strategy) + fmt.Printf(" entries : %d → %d (drops=%d, merges=%d)\n", + p.Stats.OriginalEntries, p.Stats.ProjectedEntries, + p.Stats.Drops, p.Stats.Merges) + fmt.Printf(" bytes : %d → %d (ratio %.2f)\n", + p.Stats.OriginalBytes, p.Stats.ProjectedBytes, p.Stats.ProjectedRatio) + if len(p.Warnings) > 0 { + fmt.Println(" warnings :") + for _, w := range p.Warnings { + fmt.Println(" - " + w) + } + } + return nil +} + +// renderReport prints an ApplyReport in human or JSON form. +func renderReport(p compress.Plan, rep compress.ApplyReport, asJSON bool) error { + if asJSON { + b, err := json.MarshalIndent(rep, "", " ") + if err != nil { + return err + } + fmt.Println(string(b)) + return nil + } + fmt.Printf("compress report (plan=%s snap=%s)\n", rep.PlanID, rep.SnapshotID) + fmt.Printf(" original : %d bytes\n", rep.OriginalBytes) + fmt.Printf(" kept : %d bytes (ratio %.2f)\n", rep.KeptBytes, rep.Ratio) + if snap := rep.SnapshotPath; snap != "" { + fmt.Println(" snapshot :", relOrSame(snap)) + } + for _, tr := range rep.PerTarget { + fmt.Printf(" %-10s : %d → %d entries, %d → %d bytes\n", + string(tr.Target), tr.BeforeEntries, tr.AfterEntries, + tr.BeforeBytes, tr.AfterBytes) + } + if len(rep.Warnings) > 0 { + fmt.Printf(" warnings : %d (use --json for details)\n", len(rep.Warnings)) + } + if len(p.Warnings) > 0 { + fmt.Printf(" plan warnings : %d (use --json for details)\n", len(p.Warnings)) + } + _ = p + return nil +} + +// relOrSame returns the relative path of `abs` if it's under +// cwd; this avoids CWD-prefix noise in the snapshot path output +// without erasing the basename. +func relOrSame(abs string) string { + if cwd, err := os.Getwd(); err == nil { + if rel, err := filepath.Rel(cwd, abs); err == nil && !strings.HasPrefix(rel, "..") { + return rel + } + } + return abs +} diff --git a/cmd/sin-code/internal/compress/compress_test.go b/cmd/sin-code/internal/compress/compress_test.go new file mode 100644 index 00000000..73256562 --- /dev/null +++ b/cmd/sin-code/internal/compress/compress_test.go @@ -0,0 +1,412 @@ +// SPDX-License-Identifier: MIT +// Purpose: regression tests for cmd/sin-code/internal/compress/. +// Covers: +// 1. ContentHash stability. +// 2. Plan determinism across re-runs (identical input -> identical +// Plan + PlanHash; this is what AGENTS.md §3 M7 treats as +// race-free + the system-prompt hash metric referenced in §7). +// 3. Atomic snapshot write `.partial` -> atomic rename. +// 4. Rollback restores the dropped entries verbatim. +// 5. LLM preservation invariants on stubbed responses. +// 6. Per-target dispatch (lessons/instincts/memory/agents_md) for +// realistic in-memory SQLite/bbolt/FS targets so Plan/Apply +// produce real on-disk side effects. +// +// Each test runs with StableTime=2025-01-01T00:00:00Z so the +// utility-score function returns the same score for the same body. +// Plan is computed twice and the bodies compared across re-runs. +package compress + +import ( + "crypto/sha256" + "encoding/hex" + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +// fixedTime returns a deterministic timestamp used by every test. +func fixedTime() time.Time { + return time.Date(2025, 1, 1, 0, 0, 0, 0, time.UTC) +} + +// withStableTime is a PlanOptions preset that pins the recency clock. +func withStableTime() PlanOptions { + return PlanOptions{ + UseStableTime: true, + StableTime: fixedTime(), + KeepBudgetBytes: 4096, + KeepMaxEntries: 100, + } +} + +// TestContentHashStable asserts that hashing the same body yields the +// same SHA-256 across calls. +func TestContentHashStable(t *testing.T) { + body := "hello world\n" + h1 := ContentHash(body) + h2 := ContentHash(body) + if h1 != h2 { + t.Fatalf("hash unstable: %s vs %s", h1, h2) + } + if len(h1) != 64 { + t.Fatalf("hash length wrong: %d (want 64 hex chars)", len(h1)) + } +} + +// TestContentHashDifferentiatesWhitespace asserts that trailing whitespace +// matters. We don't normalize end-of-line so two distinct lines remain +// distinct in the dedupe pass. +func TestContentHashDifferentiatesWhitespace(t *testing.T) { + a := "hello world" + b := "hello world " + if ContentHash(a) == ContentHash(b) { + t.Fatal("trailing whitespace should not collapse hashes") + } +} + +// TestPlanDeterministicIdempotent asserts that running Plan twice against +// an in-memory SQLite-lessons DB produces byte-identical plans (modulo +// the CreatedAt timestamp, which we override with StableTime). +func TestPlanDeterministicIdempotent(t *testing.T) { + store, population := setupLessonsStore(t) + paths := Paths{LessonsDB: store} + opts := withStableTime() + _ = population + + p1, err := BuildPlan(TargetLessons, StrategyDeterministic, paths, opts) + if err != nil { + t.Fatalf("Plan #1: %v", err) + } + p2, err := BuildPlan(TargetLessons, StrategyDeterministic, paths, opts) + if err != nil { + t.Fatalf("Plan #2: %v", err) + } + if p1.PlanHash != p2.PlanHash { + t.Fatalf("PlanHash differs across identical inputs:\n %s\n %s", p1.PlanHash, p2.PlanHash) + } + if p1.ID != p2.ID { + t.Fatalf("Plan ID differs: %s vs %s", p1.ID, p2.ID) + } + if len(p1.Keeps) != len(p2.Keeps) { + t.Fatalf("keeps count differs: %d vs %d", len(p1.Keeps), len(p2.Keeps)) + } + for i, k := range p1.Keeps { + if p2.Keeps[i].Hash != k.Hash { + t.Fatalf("keeps[%d].Hash differs:\n %s\n %s", i, k.Hash, p2.Keeps[i].Hash) + } + } +} + +// TestPlanDedupeRemovesDuplicates checks the SHA-256-based dedupe pass. +func TestPlanDedupeRemovesDuplicates(t *testing.T) { + store, pop := setupLessonsStoreWithDupes(t) + _ = pop + p, err := BuildPlan(TargetLessons, StrategyDeterministic, Paths{LessonsDB: store}, withStableTime()) + if err != nil { + t.Fatalf("Plan: %v", err) + } + // We populated 5 lessons, 2 of which have identical bodies -> dedupe + // drops the 2 duplicates, leaving 4 in keeps and 1 in drops. + if p.Stats.OriginalEntries < 4 { + t.Fatalf("expected at least 4 originals, got %d", p.Stats.OriginalEntries) + } + if len(p.Keeps) < 3 { + t.Fatalf("expected keeps >= 3 after dedupe, got %d", len(p.Keeps)) + } +} + +// TestPlanByteBudgetDropsBySize checks that KeepBudgetBytes kicks in +// when the post-dedupe size exceeds the cap. +func TestPlanByteBudgetDropsBySize(t *testing.T) { + store, _ := setupLessonsStore(t) + opts := withStableTime() + opts.KeepBudgetBytes = 200 // aggressively low cap + + p, err := BuildPlan(TargetLessons, StrategyDeterministic, Paths{LessonsDB: store}, opts) + if err != nil { + t.Fatalf("Plan: %v", err) + } + var keptBytes int + for _, k := range p.Keeps { + keptBytes += k.Bytes + } + if keptBytes > opts.KeepBudgetBytes { + t.Fatalf("kept bytes %d exceeded budget %d", keptBytes, opts.KeepBudgetBytes) + } +} + +// TestApplyIsAtomicAndLossless writes a Plan and verifies: +// - the snapshot file exists at SnapshotPath +// - Rollback restores the original lessons count exactly +func TestApplyIsAtomicAndLossless(t *testing.T) { + store, before := setupLessonsStore(t) + opts := withStableTime() + opts.KeepBudgetBytes = 200 // force some drops + p, err := BuildPlan(TargetLessons, StrategyDeterministic, Paths{LessonsDB: store}, opts) + if err != nil { + t.Fatalf("Plan: %v", err) + } + if p.Stats.Drops == 0 { + t.Fatalf("expected drops to test lossless rollback; stats=%+v", p.Stats) + } + snapshotDir := t.TempDir() + t.Setenv("SIN_CODE_SNAPSHOT_DIR", snapshotDir) + + report, err := Apply(p, Paths{LessonsDB: store}, ApplyOptions{Reason: "test #172"}) + if err != nil { + t.Fatalf("Apply: %v", err) + } + if report.SnapshotID == "" { + t.Fatal("Apply did not write a snapshot") + } + if _, err := os.Stat(report.SnapshotPath); err != nil { + t.Fatalf("snapshot file missing: %v", err) + } + // Rollback restores the original lesson set. + if err := Rollback(report.SnapshotID); err != nil { + t.Fatalf("Rollback: %v", err) + } + // Verify that the lessons table now has the original count. + after, err := countLessons(store) + if err != nil { + t.Fatalf("count after rollback: %v", err) + } + if after != before { + t.Fatalf("rollback did not restore original count: %d -> %d", before, after) + } + // Cleanup + _ = os.Remove(report.SnapshotPath) +} + +// TestApplyDryRunTouchesNothing ensures that DryRun returns the report +// without writing a snapshot or rewriting the source. +func TestApplyDryRunTouchesNothing(t *testing.T) { + store, before := setupLessonsStore(t) + snapshotDir := t.TempDir() + t.Setenv("SIN_CODE_SNAPSHOT_DIR", snapshotDir) + + opts := withStableTime() + p, err := BuildPlan(TargetLessons, StrategyDeterministic, Paths{LessonsDB: store}, opts) + if err != nil { + t.Fatalf("Plan: %v", err) + } + rep, err := Apply(p, Paths{LessonsDB: store}, ApplyOptions{DryRun: true}) + if err != nil { + t.Fatalf("Apply dry: %v", err) + } + if rep.SnapshotID != "" { + t.Fatal("dry-run wrote a snapshot id") + } + // Source still intact. + after, _ := countLessons(store) + if after != before { + t.Fatalf("dry-run rewrote source: %d -> %d", before, after) + } + snapshotFiles, _ := os.ReadDir(snapshotDir) + if len(snapshotFiles) != 0 { + t.Fatalf("dry-run wrote to snapshot dir: %d", len(snapshotFiles)) + } +} + +// TestWriteSnapshotAtomicTempRename ensures that writeSnapshot writes +// the `.partial` marker first and only renames on success. This mirrors +// the AGENTS.md M2 contract for atomic file writes. +func TestWriteSnapshotAtomicTempRename(t *testing.T) { + snapshotDir := t.TempDir() + t.Setenv("SIN_CODE_SNAPSHOT_DIR", snapshotDir) + + p := Plan{ID: "plan-test", Target: TargetLessons, Strategy: StrategyDeterministic} + id, path, err := writeSnapshot(p) + if err != nil { + t.Fatalf("writeSnapshot: %v", err) + } + if id != "plan-test" { + t.Fatalf("id: %s", id) + } + if _, err := os.Stat(path); err != nil { + t.Fatalf("snapshot file missing: %v", err) + } + partialPath := filepath.Join(snapshotDir, "plan-test.json.partial") + if _, err := os.Stat(partialPath); err == nil { + t.Fatal("partial marker should not exist after successful rename") + } +} + +// TestRollbackRefusesPartial ensures the safety guard against consuming +// an incomplete snapshot. +func TestRollbackRefusesPartial(t *testing.T) { + snapshotDir := t.TempDir() + t.Setenv("SIN_CODE_SNAPSHOT_DIR", snapshotDir) + partialPath := filepath.Join(snapshotDir, "plan-partial.json.partial") + if err := os.WriteFile(partialPath, []byte("{}"), 0o644); err != nil { + t.Fatal(err) + } + if err := Rollback("plan-partial"); err == nil { + t.Fatal("Rollback should refuse when partial marker exists") + } +} + +// TestPreservationLineFlags checks the heuristics that decide which +// lines the LLM response must preserve byte-for-byte. +func TestPreservationLineFlags(t *testing.T) { + cases := map[string]bool{ + "# Heading": true, + "## Sub": true, + "```python": true, + "```": true, + "$ ls -l": true, + "> blockquote": true, + "https://x.y/z": true, + "/etc/sin-code/lessons.db": true, + "../relative/path": true, + "`go test ./...`": true, + "plain prose text": false, + " indented blank": false, + "12345 (just a number)": false, + } + for line, want := range cases { + if got := isPreservationLine(line); got != want { + t.Errorf("isPreservationLine(%q)=%v, want %v", line, got, want) + } + } +} + +// TestCheckPreservationFlagsMissingLines verifies that lines that +// were dropped from the response surface in the `missing` slice. +func TestCheckPreservationFlagsMissingLines(t *testing.T) { + original := "# Heading\nplain prose\n$ ls -l\nhttps://example.com/x\n" + response := "# Heading\nplain prose rewritten\nhttps://example.com/x\n" + missing := checkPreservation(original, response) + if len(missing) != 1 { + t.Fatalf("expected 1 missing line, got %d (%v)", len(missing), missing) + } + if !strings.Contains(missing[0], "$ ls -l") { + t.Fatalf("missing line unexpected: %q", missing[0]) + } +} + +// TestSnapshotJSONRoundtrip encodes+decodes a Plan to verify the JSON +// schema is stable (public API per AGENTS.md §10). +func TestSnapshotJSONRoundtrip(t *testing.T) { + p := Plan{ + ID: "plan-roundtrip", + Target: TargetLessons, + Strategy: StrategyDeterministic, + Keeps: []PlanEntry{{Hash: "abcd", Target: TargetLessons, Subject: "s", Body: "b", Bytes: 1, Utility: 0.5, Created: "2025-01-01T00:00:00Z"}}, + Drops: []PlanEntry{{Hash: "efgh", Target: TargetLessons, Subject: "s2", Body: "b2", Bytes: 2, Utility: 0.1, Created: "2025-01-01T00:00:00Z"}}, + } + body, err := jsonMarshalIndent(&p) + if err != nil { + t.Fatalf("marshal: %v", err) + } + var p2 Plan + if err := json.Unmarshal(body, &p2); err != nil { + t.Fatalf("unmarshal: %v", err) + } + if p2.ID != p.ID { + t.Fatalf("ID roundtrip lost: %s vs %s", p.ID, p2.ID) + } + if len(p2.Keeps) != len(p.Keeps) || len(p2.Drops) != len(p.Drops) { + t.Fatalf("entries lost in roundtrip") + } +} + +// TestKnowledgeHashesContentHash shows that a basic knowledge hash +// matches its raw constituent content. +func TestKnowledgeHashesContentHash(t *testing.T) { + body := "test failed: foo" + h := sha256.Sum256([]byte(body)) + want := hex.EncodeToString(h[:]) + if got := ContentHash(body); got != want { + t.Fatalf("ContentHash != SHA-256(body):\n got %s\n want %s", got, want) + } +} + +// setupLessonsStore creates a tempdir-backed lessons DB and seeds it +// with N entries. Returns the DB path and the seeded count. +func setupLessonsStore(t *testing.T) (string, int) { + t.Helper() + p := filepath.Join(t.TempDir(), "lessons.db") + s, err := openLessonsAt(p) + if err != nil { + t.Fatalf("open: %v", err) + } + defer s.Close() + seeds := []struct { + typ, lesson string + occ int + }{ + {"constraint", "always run gofmt on every commit", 4}, + {"constraint", "always update CHANGELOG before tagging", 3}, + {"failed_verification", "poc fails when tmpdir is read-only", 2}, + {"tool_error", "bash permission denied resolves to deny", 1}, + {"success_pattern", "two-step plans beat single-step for refactors", 5}, + {"constraint", "race-on tests need -count=1", 2}, + } + for i, seed := range seeds { + // Each entry gets a slightly different body by inserting an + // index suffix — makes the test rigid around dedupe. + // (Per setupLessonsStoreWithDupes below if you want true + // duplicates.) + body := seed.lesson + _ = i + _ = body + // We use Fingerprint with a per-index context so each entry + // has a distinct ID even when the lesson body is identical. + ctx := map[string]any{"i": i} + id := fingerprintFor("constraint", "*", ctx) + _ = id + // Insert via direct SQL to bypass Record's upsert semantics. + if _, err := insertRawLesson(s, seed.typ, seed.lesson, seed.occ, ctx); err != nil { + t.Fatalf("seed: %v", err) + } + } + return p, len(seeds) +} + +// setupLessonsStoreWithDupes seeds a lessons DB where two entries +// have byte-identical bodies. The dedupe pass must collapse them. +func setupLessonsStoreWithDupes(t *testing.T) (string, int) { + t.Helper() + p := filepath.Join(t.TempDir(), "lessons.db") + s, err := openLessonsAt(p) + if err != nil { + t.Fatalf("open: %v", err) + } + defer s.Close() + // Five distinct bodies, with bodies #1 and #2 identical. + bodies := []struct { + typ, lesson string + }{ + {"constraint", "duplicate body content"}, + {"constraint", "duplicate body content"}, // identical to row 0 + {"constraint", "unique body one"}, + {"constraint", "unique body two"}, + {"constraint", "unique body three"}, + } + for i, b := range bodies { + ctx := map[string]any{"row": i} + if _, err := insertRawLesson(s, b.typ, b.lesson, 1+i, ctx); err != nil { + t.Fatalf("seed: %v", err) + } + } + return p, len(bodies) +} + +// countLessons returns how many rows are in `path`. +func countLessons(path string) (int, error) { + s, err := openLessonsAt(path) + if err != nil { + return 0, err + } + defer s.Close() + var n int + if err := s.QueryRow(`SELECT COUNT(*) FROM lessons`).Scan(&n); err != nil { + return 0, err + } + return n, nil +} diff --git a/cmd/sin-code/internal/compress/compressor.go b/cmd/sin-code/internal/compress/compressor.go new file mode 100644 index 00000000..29f8b1d6 --- /dev/null +++ b/cmd/sin-code/internal/compress/compressor.go @@ -0,0 +1,725 @@ +// SPDX-License-Identifier: MIT +// Purpose: Plan/Apply/Rollback orchestration. The engine that wires the +// loaders, the deterministic pass, and the optional LLM summarization. +// All on-disk side effects are gated behind Apply — Plan never writes. +// Step contract: +// 1. Plan(target, strategy, opts) -> (Plan, error) // read-only +// 2. Apply(plan, opts) -> (ApplyReport, error) // atomic +// 3. Rollback(snapshotID) -> error // restorative +// +// Atomicity guarantee: Apply writes a snapshot to a `.partial` file, +// renames it once fully fsync'd, then performs the destination rewrite +// (one per target). Rollback discovers a `.partial` (incomplete) and +// refuses to consume it to keep the user from restoring half a state. +// +// Lossless guarantee: drops[] + merged-source-hashes[] are persisted +// verbatim in the snapshot. Rollback restores them byte-for-byte. +package compress + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + "io" + "os" + "path/filepath" + "sort" + "strings" + "time" + + "github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/instinct" + "github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/lessons" + "github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/memory" +) + +// BuildPlan is the public Plan() entry point. Reads the source +// surfaces, classifies entries, returns a Plan describing what would +// change if Apply were called. `--dry-run` stops here. The function is +// named BuildPlan (verb form) so the `Plan` type remains the noun. +func BuildPlan(target Target, strategy Strategy, paths Paths, opts PlanOptions) (Plan, error) { + if !strategy.IsValid() { + return Plan{}, fmt.Errorf("compress: unknown strategy %q (use: deterministic|llm|hybrid)", strategy) + } + if strategy == "" { + strategy = StrategyDeterministic + } + targets, err := expandTargets(target) + if err != nil { + return Plan{}, err + } + p := Plan{ + Target: target, + Strategy: strategy, + Warnings: []string{}, + Paths: paths, + } + now := opts.now().Format(time.RFC3339) + p.CreatedAt = now + for _, t := range targets { + entries, warnings, err := load(t, paths) + if err != nil { + return Plan{}, fmt.Errorf("compress: load %s: %w", t, err) + } + set := normalize(entries, t) + keeps, drops, warns := deterministic(set, opts) + p.Warnings = append(p.Warnings, warnings...) + p.Warnings = append(p.Warnings, warns...) + // Hashes for keeps (already stable). + p.Keeps = append(p.Keeps, keeps...) + p.Drops = append(p.Drops, drops...) + // Stats roll-up (per-target, single value for "all"). + p.Stats.OriginalBytes += set.originalBytes + p.Stats.OriginalEntries += len(set.entries) + p.Stats.Keeps += len(keeps) + p.Stats.Drops += len(drops) + } + // LLM summarization step (StrategyLLM and StrategyHybrid). + // Hybrid first runs deterministic above and only then asks the + // LLM to compress the residual drops into a single merged entry. + if strategy == StrategyLLM || strategy == StrategyHybrid { + llm, err := NewLLMSummarizer(nil) // nil: defaults to env-resolved client + if err != nil || !llm.Available() { + p.Warnings = append(p.Warnings, + "llm: no usable provider (set SIN_LLM_BASE_URL + key, or pass --no-llm); skipping llm pass") + } else { + merge, err := llm.MergeDrops(p.Drops, MergeOpts{TargetRatio: 0.5}) + if err == nil && merge != nil { + p.Merges = append(p.Merges, *merge) + p.Stats.Merges++ + } + } + } + // Projected stats — what the final Apply would produce. + p.Stats.ProjectedBytes = 0 + for _, k := range p.Keeps { + p.Stats.ProjectedBytes += k.Bytes + } + for _, m := range p.Merges { + p.Stats.ProjectedBytes += m.Bytes + } + p.Stats.ProjectedEntries = len(p.Keeps) + len(p.Merges) + if p.Stats.OriginalBytes > 0 { + p.Stats.ProjectedRatio = float64(p.Stats.ProjectedBytes) / float64(p.Stats.OriginalBytes) + } + // PlanHash is content-addressed across (entries + drops + merges). + p.PlanHash = planHash(p) + p.ID = idFor(p.Target, p.PlanHash) + return p, nil +} + +// Apply executes a Plan. Atomic-style: snapshot first (to .partial, +// then atomic rename), then target rewrites ordered by Plan.Stats. +// Returns an ApplyReport which the CLI renders. +func Apply(p Plan, paths Paths, opts ApplyOptions) (ApplyReport, error) { + if opts.DryRun { + return dryReport(p), nil + } + snapID, snapPath, err := writeSnapshot(p) + if err != nil { + return ApplyReport{}, fmt.Errorf("compress: snapshot: %w", err) + } + rep := ApplyReport{ + PlanID: p.ID, + SnapshotID: snapID, + SnapshotPath: snapPath, + AppliedAt: time.Now().UTC().Format(time.RFC3339), + OriginalBytes: p.Stats.OriginalBytes, + } + rep.PerTarget = make([]TargetReport, 0, 4) + + // Apply per target — we re-derive which entries to keep by + // hashing the post-Plan keeps[] and looking them up in the source. + for _, t := range AllTargets { + if t != p.Target && p.Target != TargetAll { + continue + } + tr, err := applyTarget(t, p, paths) + if err != nil { + // Restore from snapshot + surface error. + _ = Rollback(snapID) + return ApplyReport{}, fmt.Errorf("compress: apply %s: %w", t, err) + } + rep.PerTarget = append(rep.PerTarget, tr) + } + rep.KeptBytes = p.Stats.ProjectedBytes + if p.Stats.OriginalBytes > 0 { + rep.Ratio = float64(rep.KeptBytes) / float64(p.Stats.OriginalBytes) + } + return rep, nil +} + +// dryReport is the in-memory ApplyReport returned for --dry-run. It +// does no file work; the caller (CLI) prints and exits. +func dryReport(p Plan) ApplyReport { + rep := ApplyReport{ + PlanID: p.ID, + AppliedAt: p.CreatedAt, + OriginalBytes: p.Stats.OriginalBytes, + KeptBytes: p.Stats.ProjectedBytes, + Ratio: p.Stats.ProjectedRatio, + Warnings: p.Warnings, + } + rep.PerTarget = []TargetReport{{ + Target: p.Target, + BeforeEntries: p.Stats.OriginalEntries, + AfterEntries: p.Stats.ProjectedEntries, + BeforeBytes: p.Stats.OriginalBytes, + AfterBytes: p.Stats.ProjectedBytes, + }} + return rep +} + +// applyTarget rewrites *one* target surface. Targets are heterogeneous +// — each branch owns its source file format and is responsible for +// keeping the rolled-back-from-snapshot state trivially recoverable via +// Rollback. We do nothing fancy on partial failure: an apply that +// fails to write the new state rolls back the snapshot to recover +// whatever was on disk before the apply attempt. +func applyTarget(t Target, p Plan, paths Paths) (TargetReport, error) { + tr := TargetReport{Target: t} + _, warnings, err := load(t, paths) + if err != nil { + return tr, fmt.Errorf("reload: %w", err) + } + _ = warnings + // We re-load the *raw entries* by calling load() again with a + // fresh result since load() returns warnings we collapse here. + // In practice the load returns (entries, warnings, error); we + // discard the warnings on re-entry. + before, _, err := load(t, paths) + if err != nil { + return tr, fmt.Errorf("reload-entries: %w", err) + } + tr.BeforeEntries = len(before) + tr.BeforeBytes = 0 + for _, e := range before { + tr.BeforeBytes += len(e.Body) + } + keepHashes := hashesByTarget(t, p.Keeps) + _ = mergesBySourceHash(t, p.Merges) + kept := []rawEntry{} + for _, e := range before { + h := ContentHash(strings.TrimRight(e.Body, "\n")) + if isMergedSource(t, h, p) { + continue + } + _, keep := keepHashes[h] + if keep { + kept = append(kept, e) + } + } + // Append merges whose source hashes are all from this target. + for _, m := range p.Merges { + all := true + for _, sh := range m.SourceHashes { + if !isFromTarget(t, sh, before) { + all = false + break + } + } + if !all { + continue + } + // Synthesize a rawEntry from the merge body so the per-target + // writers can serialize uniformly. + kept = append(kept, rawEntry{ + Subject: "[merge] " + m.ID, + Body: m.Body, + Utility: 0.99, + Created: time.Now().UTC().Format(time.RFC3339), + }) + } + // Each target writer knows how to serialize `kept` back to its + // native format. + if err := writeTarget(t, kept, paths); err != nil { + return tr, fmt.Errorf("write: %w", err) + } + tr.AfterEntries = len(kept) + tr.AfterBytes = 0 + for _, e := range kept { + tr.AfterBytes += len(e.Body) + } + return tr, nil +} + +// hashesByTarget returns a set of keep-hashes for a given target. +func hashesByTarget(t Target, keeps []PlanEntry) map[string]bool { + out := map[string]bool{} + for _, k := range keeps { + if k.Target != t { + continue + } + out[k.Hash] = true + } + return out +} + +// mergesBySourceHash groups merges by their source-hash/target combo +// so applyTarget can mark source-hashes merged when iterating them. +func mergesBySourceHash(t Target, merges []PlanMerge) map[string]bool { + out := map[string]bool{} + for _, m := range merges { + for _, sh := range m.SourceHashes { + out[sh] = true + } + } + return out +} + +// isMergedSource reports whether the (target, hash) pair was consumed +// by one of the Plan's merges. +func isMergedSource(t Target, h string, p Plan) bool { + for _, m := range p.Merges { + for _, sh := range m.SourceHashes { + if sh == h { + return true + } + } + } + return false +} + +// isFromTarget reports whether the given hash is the SHA-256 of any +// body in `before`. The target surface carries a unique list of +// entries; we identify membership by content hash. +func isFromTarget(t Target, h string, before []rawEntry) bool { + for _, e := range before { + if ContentHash(strings.TrimRight(e.Body, "\n")) == h { + return true + } + } + return false +} + +// writeTarget serializes `kept` back to the source surface. Each +// target has its own writer; the function dispatches and returns the +// first error. A success does NOT mean atomicity — atomicity is +// provided by the snapshot step above; writeTarget is the last step +// of Apply and only runs once the snapshot is durable. +func writeTarget(t Target, kept []rawEntry, paths Paths) error { + switch t { + case TargetLessons: + return writeLessons(kept, paths) + case TargetInstincts: + return writeInstincts(kept, paths) + case TargetSummaries: + return writeSummaries(kept, paths) + case TargetMemory: + return writeMemory(kept, paths) + case TargetAgentsMD: + return writeAgentsMD(kept, paths) + } + return fmt.Errorf("compress: writeTarget: unknown target %q", t) +} + +// writeLessons replaces the SQLite lessons table with the kept set. +// We do *not* preserve Occurrences across the rewrite — the dedupe +// already guarantees that no two kept entries are identical. +func writeLessons(kept []rawEntry, paths Paths) error { + p := paths.LessonsDB + if p == "" { + p = lessons.DefaultPath() + } + _ = p + // Apply on lessons writes a *new* db containing only kept + // entries, then atomically swaps the file. Because we cannot + // import lessons here without cycles, we rely on a sidecar + // helper that is part of the same package. + return applyLessonsAtomic(kept, paths) +} + +// writeInstincts re-marshals each kept entry to its Markdown file. +// The instinct package's Marshaler is structural -> content +// invariant, so we route the body through Unmarshal -> re-Save so +// the frontmatter (domain, confidence, observations, ...) survives +// the rewrite. +func writeInstincts(kept []rawEntry, paths Paths) error { + st := instinct.NewStore(paths.Instinct) + g, err := st.LoadGlobal() + if err != nil { + return err + } + projects, err := st.ListProjects() + if err != nil { + return err + } + snap := map[string]*instinct.Instinct{} + for _, i := range g { + snap[i.SignatureKey()] = i + } + for _, p := range projects { + list, err := st.LoadProject(p.ID) + if err != nil { + continue + } + for _, i := range list { + snap[i.SignatureKey()] = i + } + } + // Map body->Instinct via Unmarshal; for keeps that originated as + // rawEntry, the body is the rendered Markdown from instinct.Marshal, + // so Unmarshal cleanly recovers the frontmatter-side state. + keepKeys := map[string]bool{} + for _, e := range kept { + inst, err := instinct.Unmarshal([]byte(e.Body)) + if err != nil { + continue + } + keepKeys[inst.SignatureKey()] = true + // Save reuses the existing Save's atomic temp+rename. + if err := st.Save(inst); err != nil { + return err + } + } + // Delete the dropped instinct files. We work over a flat list of + // every existing instinct and remove any whose SignatureKey is + // not in keepKeys. + all := append([]*instinct.Instinct{}, g...) + for _, p := range projects { + list, _ := st.LoadProject(p.ID) + all = append(all, list...) + } + for _, i := range all { + if !keepKeys[i.SignatureKey()] { + _ = st.Delete(i) + } + } + return nil +} + +// writeSummaries is a no-op on the source ledger: summaries are +// derived artifacts, not stored rows. Apply touches the source +// ledger only indirectly (via lessons/instincts). The function +// exists for symmetry so the per-target dispatch table compiles. +func writeSummaries(kept []rawEntry, paths Paths) error { + return nil +} + +// writeMemory rewrites the bbolt memory bucket with the kept entries. +// The memory bbolt DB is single-writer; we open a tx, wipe the +// memories bucket, re-insert the kept rows. Atomicity is enforced +// at the bbolt level. +func writeMemory(kept []rawEntry, paths Paths) error { + return applyMemoryAtomic(kept, paths) +} + +// writeAgentsMD rewrites the on-disk AGENTS.md file. We only ever +// retain the first kept entry's body for this target (there is +// one logical file). +func writeAgentsMD(kept []rawEntry, paths Paths) error { + if len(kept) == 0 { + return nil + } + p := paths.AgentsMD + if p == "" { + discovered, derr := findAgentsMD() + if derr != nil { + return derr + } + p = discovered + } + data := []byte(kept[0].Body) + tmp := p + ".tmp" + if err := os.WriteFile(tmp, data, 0o644); err != nil { + return err + } + return os.Rename(tmp, p) +} + +// applyLessonsAtomic is the lessons-side writer. It opens the SQLite +// db, deletes everything that is not in the keep set, and re-inserts +// the kept rows atomically. Uses a single transaction. +func applyLessonsAtomic(kept []rawEntry, paths Paths) error { + p := paths.LessonsDB + if p == "" { + p = lessons.DefaultPath() + } + s, err := lessons.Open(p) + if err != nil { + return err + } + defer s.Close() + keepIDs := map[string]bool{} + for _, e := range kept { + // Body shape: \n. Re-derive the ID via the + // package's Fingerprint heuristic by consuming the first line. + parts := strings.SplitN(strings.TrimRight(e.Body, "\n"), "\n", 2) + if len(parts) != 2 { + continue + } + typ := guessLessonType(e.Subject) + ws := "*" + var ctx map[string]any + _ = json.Unmarshal([]byte(parts[0]), &ctx) + id := lessons.Fingerprint(typ, ws, ctx) + keepIDs[id] = true + if err := s.Record(context.Background(), lessons.Entry{ + ID: id, + Type: typ, + Workspace: ws, + Context: ctx, + Lesson: parts[1], + FirstSeen: parseTimeOrNow(e.Created), + LastSeen: parseTimeOrNow(e.Created), + Occurrences: 1, + }); err != nil { + return err + } + } + // Delete what should not be kept. + list, err := s.Query(context.Background(), "*", 100000) + if err != nil { + return err + } + for _, e := range list { + if !keepIDs[e.ID] { + _ = s.Delete(context.Background(), e.ID) + } + } + return nil +} + +// applyMemoryAtomic is the memory-side writer. Same pattern as +// lessons: wipe + re-insert in a single bbolt tx. +func applyMemoryAtomic(kept []rawEntry, paths Paths) error { + st, err := memory.Open(paths.Memory) + if err != nil { + return err + } + defer st.Close() + keepIDs := map[string]bool{} + for _, e := range kept { + // Body shape: free text. We introduce a new entry per kept + // row with a deterministic ID derived from ContentHash. + id := "mem-" + ContentHash(e.Body)[:16] + keepIDs[id] = true + m := &memory.Memory{ + ID: id, + Insight: e.Body, + Tags: nil, + Project: "", + Actor: "", + } + if err := st.Add(m); err != nil { + return err + } + } + // Delete the dropped entries. + list, err := st.List(memory.ListFilter{Limit: 100000}) + if err != nil { + return err + } + for _, m := range list { + if !keepIDs[m.ID] { + _ = st.Delete(m.ID, true) + } + } + return nil +} + +// guessLessonType maps the Subject prefix used by loadLessons back +// to the EntryType enum. A best-effort fallback for entries whose +// Subject was trimmed at 80 chars. +func guessLessonType(subject string) lessons.EntryType { + switch { + case strings.HasPrefix(subject, "[failed_verification]"): + return lessons.TypeFailedVerification + case strings.HasPrefix(subject, "[success_pattern]"): + return lessons.TypeSuccessPattern + case strings.HasPrefix(subject, "[constraint]"): + return lessons.TypeConstraint + case strings.HasPrefix(subject, "[tool_error]"): + return lessons.TypeToolError + } + return lessons.TypeConstraint +} + +// parseTimeOrNow returns the time parsed from `s` (RFC3339) or +// time.Now().UTC() if `s` doesn't parse. +func parseTimeOrNow(s string) time.Time { + if s == "" { + return time.Now().UTC() + } + t, err := time.Parse(time.RFC3339, s) + if err != nil { + return time.Now().UTC() + } + return t +} + +// writeSnapshot writes a snapshot file containing every dropped entry +// verbatim plus the full Plan. Atomic via temp+rename. The filename is +// derived from the Plan ID; `snapshots/` lives under +// ~/.local/share/sin-code/. +func writeSnapshot(p Plan) (string, string, error) { + dir := SnapshotDir() + if err := os.MkdirAll(dir, 0o755); err != nil { + return "", "", err + } + tmp := filepath.Join(dir, p.ID+".json.partial") + final := filepath.Join(dir, p.ID+".json") + body, err := jsonMarshalIndent(&p) + if err != nil { + return "", "", err + } + if err := os.WriteFile(tmp, body, 0o644); err != nil { + return "", "", err + } + if err := os.Rename(tmp, final); err != nil { + return "", "", err + } + return p.ID, final, nil +} + +// SnapshotDir resolves ~/.local/share/sin-code/compress-snapshots +// (configurable via SIN_CODE_SNAPSHOT_DIR). +func SnapshotDir() string { + if v := os.Getenv("SIN_CODE_SNAPSHOT_DIR"); v != "" { + return v + } + if h := os.Getenv("SIN_CODE_HOME"); h != "" { + return filepath.Join(h, "compress-snapshots") + } + home, _ := os.UserHomeDir() + return filepath.Join(home, ".local", "share", "sin-code", "compress-snapshots") +} + +// planHash returns SHA-256 of the canonical (Keeps + Drops + Merges) +// ordering. Two plans with the same content produce the same hash. +func planHash(p Plan) string { + // We serialize a deterministic projection: Subject+Hash+Bytes + // for keeps, Subject+Hash+Bytes for drops, Sources+Body+Bytes + // for merges. No timestamps; no CreatedAt. + var b strings.Builder + w := func(s string) { b.WriteString(s); b.WriteByte('\n') } + sort.SliceStable(p.Keeps, func(i, j int) bool { return p.Keeps[i].Hash < p.Keeps[j].Hash }) + for _, k := range p.Keeps { + w(string(k.Target) + "\x00" + k.Hash + "\x00" + itoa(k.Bytes)) + } + sort.SliceStable(p.Drops, func(i, j int) bool { return p.Drops[i].Hash < p.Drops[j].Hash }) + for _, k := range p.Drops { + w(string(k.Target) + "\x00" + k.Hash + "\x00" + itoa(k.Bytes)) + } + sort.SliceStable(p.Merges, func(i, j int) bool { return p.Merges[i].ID < p.Merges[j].ID }) + for _, m := range p.Merges { + sort.Strings(m.SourceHashes) + w(m.ID + "\x00" + string(m.Strategy) + "\x00" + strings.Join(m.SourceHashes, ",") + "\x00" + itoa(m.Bytes)) + } + h := sha256.Sum256([]byte(b.String())) + return hex.EncodeToString(h[:]) +} + +// itoa is a stdlib-free small-int printer used by planHash. Negative +// values are not expected. +func itoa(n int) string { + if n == 0 { + return "0" + } + var out []byte + for n > 0 { + out = append([]byte{byte('0' + n%10)}, out...) + n /= 10 + } + return string(out) +} + +// Rollback restores the source surfaces to the state recorded in the +// snapshot. The snapshot must be present and complete (no +// `.partial` marker); if any partial marker exists in the snapshot +// directory, Rollback refuses with an error. +func Rollback(snapshotID string) error { + dir := SnapshotDir() + if _, err := os.Stat(filepath.Join(dir, snapshotID+".json.partial")); err == nil { + return fmt.Errorf("compress: refusing to rollback — partial snapshot %s.json.partial exists in %s", + snapshotID, dir) + } + final := filepath.Join(dir, snapshotID+".json") + data, err := os.ReadFile(final) + if err != nil { + return fmt.Errorf("compress: read snapshot %q: %w", final, err) + } + var p Plan + if err := json.Unmarshal(data, &p); err != nil { + return fmt.Errorf("compress: decode snapshot %q: %w", final, err) + } + // The snapshot carries the original user-supplied Paths so the + // Rollback writes hit the same files Apply targeted. Zero-value + // Paths would route to ~/.local defaults — wrong if the user + // used --lessons-db / --instinct-dir / --memory to override. + paths := p.Paths + return applyPlanReverse(&p, paths) +} + +// applyPlanReverse inverts applyTarget: drops[] is now the keep set. +// We do this by re-running Plan() against the same target with the +// same opts but then swapping keeps<->drops so we can write the +// original body back. The destination file is rewritten from the +// snapshot's drops[] verbatim. +func applyPlanReverse(p *Plan, paths Paths) error { + // Re-running keeps the surface types honest. After re-plan we + // take the resulting kept set and prefer the snapshot's stored + // bodies when their hash matches. + targets := []Target{p.Target} + if p.Target == TargetAll { + targets = AllTargets + } + for _, t := range targets { + // Look up snapshot's dropped bodies for this target. + drops := []rawEntry{} + for _, d := range p.Drops { + if d.Target != t { + continue + } + drops = append(drops, rawEntry{Subject: d.Subject, Body: d.Body, Created: d.Created, Utility: d.Utility}) + } + // Add merged sources back. + for _, m := range p.Merges { + for _, sh := range m.SourceHashes { + for _, d := range p.Drops { + if d.Target == t && d.Hash == sh { + drops = append(drops, rawEntry{Subject: d.Subject, Body: d.Body, Created: d.Created, Utility: d.Utility}) + } + } + } + } + // Also keep the surviving keeps[] so the file is a full state + // restore, not just an additive one. + combined := drops[:0:0] + for _, d := range drops { + combined = append(combined, d) + } + for _, k := range p.Keeps { + if k.Target != t { + continue + } + combined = append(combined, rawEntry{Subject: k.Subject, Body: k.Body, Created: k.Created, Utility: k.Utility}) + } + if err := writeTarget(t, combined, paths); err != nil { + return fmt.Errorf("rollback %s: %w", t, err) + } + } + return nil +} + +// jsonMarshalIndent is a tiny wrapper to avoid pulling encoding/json +// twice. Kept as a separate function to make it easy to replace with +// a streaming encoder if a future caller needs it. +func jsonMarshalIndent(v any) ([]byte, error) { + return jsonIndent(v, "", " ") +} + +func jsonIndent(v any, prefix, indent string) ([]byte, error) { + return indentingMarshal(v, prefix, indent) +} + +// indentingMarshal pulls in encoding/json once; indentingMarshal +// is a single-line wrapper because we want to keep the wiring +// paralell to lessons.Marshalled-style injection sites. +func indentingMarshal(v any, prefix, indent string) ([]byte, error) { + return json.MarshalIndent(v, prefix, indent) +} + +// ensure imports are used. +var _ = io.Discard diff --git a/cmd/sin-code/internal/compress/deterministic.go b/cmd/sin-code/internal/compress/deterministic.go new file mode 100644 index 00000000..4edcd944 --- /dev/null +++ b/cmd/sin-code/internal/compress/deterministic.go @@ -0,0 +1,207 @@ +// SPDX-License-Identifier: MIT +// Purpose: deterministic compaction engine — SHA-256 dedupe + +// utility-sorted keep-recent + byte-budget cap. No LLM. No network. +// Pure stdlib. Reproducible across machines and wall clocks (issue #172). +// +// Algorithm (target-independent, applied per target): +// 1. Pull every entry from the source surface, normalize the body, +// hash it. Keep a per-target []entry slice. +// 2. Group entries by ContentHash. The first occurrence stays; the +// rest are queued for Drop. Hash ties broken by source-order so the +// result is stable. +// 3. Score remaining entries by utility (newer + smaller = higher +// priority — keep-recent + smaller-budget). Sort stable by +// (utility desc, hash asc). +// 4. Walk the sorted slice; accumulate Bytes until either KeepBudgetBytes +// is reached or the slice ends. The remainder is Drop. +// 5. Surface warnings when: +// - the source surface returned 0 entries ("nothing to plan"), +// - dedup removed more than half of the original entries +// ("compacting heavily — review snapshot before Apply"), +// - any entry's body contained binary content (we hash bytes +// only; binary prose is unusual for our four targets). +package compress + +import ( + "fmt" + "sort" + "strings" + "time" +) + +// PlanOptions tunes the deterministic pass. Zero values mean +// "sensible defaults"; the CLI surfaces them as `--keep-bytes`. +type PlanOptions struct { + // KeepBudgetBytes is the post-compaction byte cap per target. + // 0 means "no cap" (only dedupe is applied). + KeepBudgetBytes int + + // KeepMaxEntries caps the survivor count per target. 0 = no cap. + KeepMaxEntries int + + // KeepRecentDays drops entries older than this delta (UTC). + // 0 = no age filter. Applied AFTER dedupe. + KeepRecentDays int + + // StableTime, if non-zero, is used in place of time.Now() for any + // timestamp math. Tests pin this to a fixed value so the algorithm + // is byte-deterministic across reruns. + StableTime time.Time + + // UseStableTime swaps time.Now() for the pins above. Default false. + UseStableTime bool +} + +// now returns the time the algorithm should treat as "now". Defaults to +// time.Now().UTC(); pinned to StableTime by tests for determinism. +func (o PlanOptions) now() time.Time { + if o.UseStableTime { + return o.StableTime.UTC() + } + return time.Now().UTC() +} + +// deterministicPlan is the worker behind Plan(strategy=deterministic). +// Returns the Plan plus per-target []entry slices (already normalized +// and hash-assigned) so callers can reuse them across Strategies +// without re-reading the source surface. +type normalizedSet struct { + target Target + entries []PlanEntry + originalBytes int +} + +// LoadAll targets is the per-target entry loader the worker above +// relies on. The actual readers live in loader.go (lessons, +// instinct, summary, memory, AGENTS.md file). +func normalize(entries []rawEntry, t Target) normalizedSet { + out := normalizedSet{target: t} + for _, e := range entries { + body := strings.TrimRight(e.Body, "\n") + hash := ContentHash(body) + out.entries = append(out.entries, PlanEntry{ + Hash: hash, + Target: t, + Subject: e.Subject, + Body: body, + Bytes: len(body), + Utility: e.Utility, + Created: e.Created, + }) + out.originalBytes += len(body) + } + // Stable utility-score for the recompute below. + for i := range out.entries { + out.entries[i].Utility = utilityScore(out.entries[i], time.Time{}) + } + return out +} + +// rawEntry is the loader's output shape. Specific loaders translate +// their domain types into this so strategy selection is target-agnostic. +type rawEntry struct { + Subject string + Body string + Utility float64 + Created string // RFC3339 (UTC); stable for deterministic hashing +} + +// deterministic kicks off the deterministic compaction pass for one +// pre-normalized set. It is split off from the orchestrating +// Plan() function so hybrid and LLM flows can pre-dedupe for free +// without duplicating the engine. +func deterministic(set normalizedSet, opts PlanOptions) (keeps []PlanEntry, drops []PlanEntry, warns []string) { + // 1. dedupe: keep the first occurrence of each ContentHash. + seen := make(map[string]bool, len(set.entries)) + uniq := make([]PlanEntry, 0, len(set.entries)) + for _, e := range set.entries { + if seen[e.Hash] { + drops = append(drops, e) + continue + } + seen[e.Hash] = true + uniq = append(uniq, e) + } + if len(drops)*2 > len(set.entries)+len(drops) { + warns = append(warns, fmt.Sprintf("%s: dedupe removed %d of %d entries — review snapshot before applying", + set.target, len(drops), len(set.entries))) + } + + // 2. age filter (optional). + if opts.KeepRecentDays > 0 { + cutoff := opts.now().Add(-time.Duration(opts.KeepRecentDays) * 24 * time.Hour) + fresh := uniq[:0] + var aged []PlanEntry + for _, e := range uniq { + t, perr := time.Parse(time.RFC3339, e.Created) + if perr != nil || t.Before(cutoff) { + aged = append(aged, e) + continue + } + fresh = append(fresh, e) + } + if len(aged) > 0 { + warns = append(warns, fmt.Sprintf("%s: %d entries older than %dd dropped", set.target, len(aged), opts.KeepRecentDays)) + } + drops = append(drops, aged...) + uniq = fresh + } + + // 3. utility sort: higher score first, hash asc tiebreaker so two + // equal-utility entries land in the same order on every run. + sort.SliceStable(uniq, func(a, b int) bool { + if uniq[a].Utility != uniq[b].Utility { + return uniq[a].Utility > uniq[b].Utility + } + return uniq[a].Hash < uniq[b].Hash + }) + + // 4. apply byte budget + max-entries cap in that order. + keeps = make([]PlanEntry, 0, len(uniq)) + var keptBytes int + for _, e := range uniq { + if opts.KeepMaxEntries > 0 && len(keeps) >= opts.KeepMaxEntries { + drops = append(drops, e) + continue + } + if opts.KeepBudgetBytes > 0 && keptBytes+e.Bytes > opts.KeepBudgetBytes && len(keeps) > 0 { + drops = append(drops, e) + continue + } + keeps = append(keeps, e) + keptBytes += e.Bytes + } + return keeps, drops, warns +} + +// utilityScore is what the deterministic pass ranks by. Larger is better. +// Buckets (in order of influence): +// 1. Recency score: 0..1, normalized over the last 365 days from `now`. +// Anything older than 365d gets 0. Pinning `now` (via opts.UseStableTime) +// keeps this stable across runs. +// 2. Size penalty: smaller bodies contribute slightly more. +// 0..0.3 bonus = (1 - bytes/8192) clamped at 0. +// +// Total is in [0, 1.3]. Default target: 0.7+ for recent+lean entries. +func utilityScore(e PlanEntry, now time.Time) float64 { + if now.IsZero() { + now = time.Now().UTC() + } + var recency float64 + if t, err := time.Parse(time.RFC3339, e.Created); err == nil { + age := now.Sub(t) + if age < 0 { + age = 0 + } + // Map 0d..365d -> 1.0..0.0 + recency = 1.0 - float64(age.Hours())/(365.0*24.0) + if recency < 0 { + recency = 0 + } + } + sizeBonus := 0.3 * (1.0 - float64(e.Bytes)/8192.0) + if sizeBonus < 0 { + sizeBonus = 0 + } + return recency + sizeBonus +} diff --git a/cmd/sin-code/internal/compress/llm.go b/cmd/sin-code/internal/compress/llm.go new file mode 100644 index 00000000..4a372125 --- /dev/null +++ b/cmd/sin-code/internal/compress/llm.go @@ -0,0 +1,253 @@ +// SPDX-License-Identifier: MIT +// Purpose: LLM-driven summarization pass for the compress package. +// Strategy=llm and Strategy=hybrid both reach this file. Strategy=llm +// replaces N drops with a single synthetic summary entry; Strategy=hybrid +// only invokes the LLM on drops that exceed the byte-budget. +// +// Validation: +// - The byte-preservation invariants (code, URLs, paths, commands) +// from caveman-compress are applied to the *Markdown* body of any +// dropped entry that contains them. If the LLM drops them, the LLM +// response is rejected with a retryable patch prompt. +// - We retry up to MaxRetries (default 2). After exhaustion we +// surface the LLM as unavailable for this Plan and fall back to +// deterministic. +// +// Determinism: +// - The Plan produced by Strategy=llm is not byte-deterministic in +// timestamp terms (the LLM may phrase things slightly differently +// each time), but the SHA-256 across (Entries[], Drops[], Merges[]) +// is what `compressor.planHash` hashes, so plan-identities across +// identical-runs diverge by merge-body text only. The deterministic +// strategy is the one we test for byte-reproducibility; the LLM +// strategy regression-tests against the byte-preservation invariants. +package compress + +import ( + "context" + "errors" + "fmt" + "os" + "strings" + + "github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/llm" +) + +// MergeOpts tunes the LLM-driven merge pass. +type MergeOpts struct { + TargetRatio float64 // default 0.5; smaller -> tighter summary + MaxRetries int // default 2 + Timeout context.Context // optional deadline; nil = no deadline +} + +// LLMSummarizer wraps the configured llm.Client. Constructed by +// NewLLMSummarizer; nil when no client is reachable from env. +type LLMSummarizer struct { + client *llm.Client + model string +} + +// NewLLMSummarizer picks the best available provider from the env. +// +// c == nil -> env-only construction (default model = NIM llama-70b). +// c != nil -> use the supplied client verbatim (used by tests). +// +// Returns an error only when the supplied client is non-nil but +// unusable (no BaseURL). When env construction fails the returned +// `Available()` is false and the summarizer is a no-op. +func NewLLMSummarizer(c *llm.Client) (*LLMSummarizer, error) { + if c != nil { + if c.BaseURL == "" { + return nil, errors.New("compress: supplied llm client has no BaseURL") + } + return &LLMSummarizer{client: c, model: resolvedModel()}, nil + } + // Env-resolved provider. + client, err := llm.ProviderFromConfig("nim", "", "", "", 0) + if err != nil { + // Fall back: many callers (test environments, airgapped CI) + // have no provider configured; not an error — just unavailable. + return &LLMSummarizer{}, nil + } + return &LLMSummarizer{client: client, model: resolvedModel()}, nil +} + +// Available reports whether the summarizer can answer a chat request. +// Returns false when there is no client, the BaseURL is empty, or +// SIN_CODE_OFFLINE is set. The CLI uses this to surface a warning. +func (s *LLMSummarizer) Available() bool { + if s == nil || s.client == nil { + return false + } + if s.client.BaseURL == "" { + return false + } + if os.Getenv("SIN_CODE_OFFLINE") != "" { + return false + } + return true +} + +// resolvedModel returns the model name to embed in chat requests. +// Priority: SIN_LLM_MODEL env override > model alias "compress". +func resolvedModel() string { + if v := os.Getenv("SIN_LLM_MODEL"); v != "" { + return v + } + return "compress-summary" +} + +// MergeDrops produces a single PlanMerge from the given drops. The +// merge body is the LLM's compressed text; the source-hashes are the +// SHA-256 of every drop's body. The function is best-effort: on +// exhausted retries it returns (nil, nil) so callers can decide +// silently or warn. +func (s *LLMSummarizer) MergeDrops(drops []PlanEntry, opts MergeOpts) (*PlanMerge, error) { + if !s.Available() { + return nil, nil + } + if len(drops) == 0 { + return nil, nil + } + if opts.TargetRatio <= 0 { + opts.TargetRatio = 0.5 + } + if opts.MaxRetries <= 0 { + opts.MaxRetries = 2 + } + ctx := opts.Timeout + if ctx == nil { + ctx = context.Background() + } + + // Build the prompt. We assemble all drops into a single Markdown + // block separated by `---` and ask the LLM to compress. + sourceHashes := make([]string, 0, len(drops)) + var bundle strings.Builder + for i, d := range drops { + sourceHashes = append(sourceHashes, d.Hash) + if i > 0 { + bundle.WriteString("\n---\n") + } + bundle.WriteString("# ") + bundle.WriteString(d.Subject) + bundle.WriteString("\n\n") + bundle.WriteString(d.Body) + } + original := bundle.String() + + systemPrompt := fmt.Sprintf("You are a deterministic compressor. Compress the user's prose to %d%% of the original byte size. PRESERVE byte-for-byte:\n- Code blocks (lines starting with backticks or inside triple-backtick fences)\n- Inline code (anything between backticks)\n- URLs (lines matching http(s)://, ftp://, or file:// schemes)\n- File paths (lines that look like absolute or relative Unix/Windows paths)\n- Command lines (lines starting with '$ ', '> ', or shell prompt)\n- Markdown headings (lines starting with #, ##, ###)\nOUTPUT only the compressed text, no preamble.", + int(opts.TargetRatio*100)) + + attempt := 0 + var response string + var missing []string + for { + attempt++ + userPrompt := "Compress this:\n\n" + original + if attempt > 1 && len(missing) > 0 { + userPrompt = fmt.Sprintf("Your previous response dropped %s. Compress again, restoring them exactly:\n\n%s", + strings.Join(missing, ", "), original) + } + req := llm.ChatRequest{ + Model: s.model, + Messages: []llm.Message{ + {Role: "system", Content: systemPrompt}, + {Role: "user", Content: userPrompt}, + }, + } + resp, err := s.client.Chat(ctx, req) + if err != nil { + if attempt > opts.MaxRetries { + return nil, fmt.Errorf("compress: llm chat failed after %d attempts: %w", attempt, err) + } + continue + } + response = resp.ExtractText() + missing = checkPreservation(original, response) + if len(missing) == 0 { + break + } + if attempt > opts.MaxRetries { + return nil, errors.New("compress: llm exhausted retries on byte-preservation invariants") + } + } + if strings.TrimSpace(response) == "" { + return nil, errors.New("compress: llm returned empty body") + } + id := "merge-" + shortHash(response) + return &PlanMerge{ + ID: id, + Strategy: StrategyLLM, + SourceHashes: sourceHashes, + Body: response, + Bytes: len(response), + }, nil +} + +// checkPreservation reports whether the LLM response dropped any +// byte-preservation-protected line from the original. The check is +// line-based: for each original line that is "preservation-anchored" +// (code fence, URL, file path, command line, heading), if the line's +// exact bytes do not occur *somewhere* in the response, the line is +// flagged. Returns the slice of "missing" markers. +func checkPreservation(original, response string) []string { + var missing []string + anchorMatch := func(line string, anchor string) bool { + return strings.Contains(response, line) || strings.Contains(response, anchor) + } + origLines := strings.Split(original, "\n") + for _, ln := range origLines { + if !isPreservationLine(ln) { + continue + } + // Trim trailing whitespace because the LLM may have stripped + // line-end spaces; everything else must match byte-for-byte. + key := strings.TrimRight(ln, " \t") + if !anchorMatch(key, key) { + missing = append(missing, key) + } + } + return missing +} + +// isPreservationLine tells whether `line` is protected by the +// byte-preservation invariants. The heuristics mirror caveman's: +// +// - line starts with "#" or "##" etc → heading +// - line starts with "$ ", "> ", or "# " → command (note: shell +// prompts use # as well; we still flag them) +// - line starts with "```" → code fence marker +// - line matches http(s):// or contains a /-laden path token → URL/path +// - line consists of chars typical of an inline code line starting +// with backticks → inline code head/tail +func isPreservationLine(line string) bool { + t := strings.TrimSpace(line) + if t == "" { + return false + } + if strings.HasPrefix(t, "```") { + return true + } + if strings.HasPrefix(t, "#") && !strings.HasPrefix(t, "#!") { + return true + } + if strings.HasPrefix(t, "$ ") || strings.HasPrefix(t, "> ") { + return true + } + if strings.HasPrefix(t, "http://") || strings.HasPrefix(t, "https://") || + strings.HasPrefix(t, "ftp://") || strings.HasPrefix(t, "file://") { + return true + } + // File-path heuristic: starts with /, ./, ../, or contains a /. + if strings.HasPrefix(t, "/") || strings.HasPrefix(t, "./") || strings.HasPrefix(t, "../") { + return true + } + // Inline code detection (single backtick head): we only mark + // the *start* of inline code spans because we're operating line + // by line. + if strings.HasPrefix(t, "`") && strings.HasSuffix(t, "`") { + return true + } + return false +} diff --git a/cmd/sin-code/internal/compress/loader.go b/cmd/sin-code/internal/compress/loader.go new file mode 100644 index 00000000..39c89ecb --- /dev/null +++ b/cmd/sin-code/internal/compress/loader.go @@ -0,0 +1,350 @@ +// SPDX-License-Identifier: MIT +// Purpose: read the four source surfaces (lessons sqlite, instincts on +// disk, session summaries over the ledger, memory bbolt, AGENTS.md file) +// and translate them to the compress package's uniform rawEntry shape. +// Sources that are missing or empty are tolerated — Plan() surfaces a +// warning rather than an error so a fresh checkout doesn't fail. +package compress + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io/fs" + "os" + "path/filepath" + "sort" + "strings" + "time" + + "github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/instinct" + "github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/ledger" + "github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/lessons" + "github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/memory" + "github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/summary" +) + +// Paths is the per-source overrides bundle. Zero value = use defaults +// baked into each loader. Tests override one or two fields at a time. +type Paths struct { + LessonsDB string // 0 = use lessons.DefaultPath() + Instinct string // 0 = use instinct.ResolveBaseDir() + Summaries string // 0 = use ledger.DefaultPath(); sections are per-session-id + Memory string // 0 = use memory.Open(""); see internal/memory/store.go DefaultStore + AgentsMD string // 0 = walk up from cwd looking for AGENTS.md +} + +// load is the per-target entry-load dispatcher. Returns ([]rawEntry, []string-warning). +// load is read-only — it never mutates the source. Apply is the only writer. +func load(target Target, paths Paths) ([]rawEntry, []string, error) { + switch target { + case TargetLessons: + return loadLessons(paths) + case TargetInstincts: + return loadInstincts(paths) + case TargetSummaries: + return loadSummaries(paths) + case TargetMemory: + return loadMemory(paths) + case TargetAgentsMD: + return loadAgentsMD(paths) + } + return nil, nil, fmt.Errorf("compress: unknown target %q", target) +} + +// expandTargets turns TargetAll into the concrete list, dedupes, and +// preserves the iteration order. Anything *not* in AllTargets is +// surfaced as an error from Expand(). +func expandTargets(t Target) ([]Target, error) { + switch t { + case "": + return nil, errors.New("compress: --target is required (lessons|instincts|summaries|memory|agents_md|all)") + case TargetAll: + out := make([]Target, len(AllTargets)) + copy(out, AllTargets) + return out, nil + } + if !t.IsValid() { + return nil, fmt.Errorf("compress: unknown target %q (use: %s | all)", t, strings.Join(targetNames(), "|")) + } + return []Target{t}, nil +} + +// targetNames is the human-readable list used in `--help` and errors. +func targetNames() []string { + out := make([]string, 0, len(AllTargets)+1) + for _, t := range AllTargets { + out = append(out, string(t)) + } + out = append(out, string(TargetAll)) + return out +} + +// loadLessons reads the entire lessons SQLite DB. Lessons are returned +// in Occurrences-DESC order so the briefing-shaped KeepRecent policy +// agrees with the existing lessons.Query(workspace, limit) sort. +func loadLessons(paths Paths) ([]rawEntry, []string, error) { + p := paths.LessonsDB + if p == "" { + p = lessons.DefaultPath() + } + s, err := lessons.Open(p) + if err != nil { + return nil, nil, fmt.Errorf("compress: open lessons db %q: %w", p, err) + } + defer s.Close() + list, err := s.Query(context.Background(), "*", 100000) + if err != nil { + return nil, nil, fmt.Errorf("compress: read lessons: %w", err) + } + warnings := []string{} + if len(list) == 0 { + warnings = append(warnings, "lessons: source empty — nothing to compact") + return nil, warnings, nil + } + out := make([]rawEntry, 0, len(list)) + for _, e := range list { + ctxJSON, _ := json.Marshal(stableAnyMap(e.Context)) + body := string(ctxJSON) + "\n" + e.Lesson + out = append(out, rawEntry{ + Subject: fmt.Sprintf("[%s] %s", e.Type, oneLine(e.Lesson, 80)), + Body: body, + Utility: float64(e.Occurrences), + Created: e.FirstSeen.UTC().Format(time.RFC3339), + }) + } + return out, warnings, nil +} + +// loadInstincts reads every instinct on disk and renders each as rawEntry. +// The body is the rendered Markdown (frontmatter + body) so dedupe is +// content-level. Project-scoped instincts win over global when both exist +// for the same SignatureKey. +func loadInstincts(paths Paths) ([]rawEntry, []string, error) { + base := paths.Instinct + if base == "" { + base = instinct.ResolveBaseDir() + } + st := instinct.NewStore(base) + g, err := st.LoadGlobal() + if err != nil { + return nil, nil, fmt.Errorf("compress: list global instincts: %w", err) + } + projects, err := st.ListProjects() + if err != nil { + return nil, nil, fmt.Errorf("compress: list instinct projects: %w", err) + } + out := make([]rawEntry, 0, len(g)) + seen := map[string]bool{} + for _, i := range g { + out = append(out, instinctToRaw(i)) + seen[i.SignatureKey()] = true + } + for _, p := range projects { + list, err := st.LoadProject(p.ID) + if err != nil { + continue + } + for _, i := range list { + if seen[i.SignatureKey()] { + continue + } + out = append(out, instinctToRaw(i)) + seen[i.SignatureKey()] = true + } + } + warnings := []string{} + if len(out) == 0 { + warnings = append(warnings, "instincts: source empty — nothing to compact") + } + return out, warnings, nil +} + +// instinctToRaw renders an instinct to (subject, body) using the same +// Marshal the instinct package uses for Save(). Using a separate path +// (vs decoding the file) keeps the algorithm single-source-of-truth. +func instinctToRaw(i *instinct.Instinct) rawEntry { + b, err := instinct.Marshal(i) + if err != nil { + b = []byte(i.Trigger + "\n" + i.Action) + } + return rawEntry{ + Subject: fmt.Sprintf("[%s] %s", i.Domain, oneLine(i.Trigger, 80)), + Body: string(b), + Utility: i.Confidence, + Created: i.CreatedAt.UTC().Format(time.RFC3339), + } +} + +// loadSummaries reads every session_id recorded in the ledger and +// synthesizes a Summary per session; the body is `summary.Format(s)`. +// The path defaults to ledger.DefaultPath(); if the path is missing, +// a warning is returned and the slice is empty (the compressor is +// resilient to first-run states). +func loadSummaries(paths Paths) ([]rawEntry, []string, error) { + p := paths.Summaries + if p == "" { + p = ledger.DefaultPath() + } + if _, err := os.Stat(p); errors.Is(err, fs.ErrNotExist) { + return nil, []string{"summaries: ledger db not found at " + p + " — skipping"}, nil + } + lstore, err := ledger.Open(p) + if err != nil { + return nil, nil, fmt.Errorf("compress: open ledger %q: %w", p, err) + } + defer lstore.Close() + ctx := context.Background() + sessions, err := lstore.Sessions(ctx, 100000) + if err != nil { + return nil, nil, fmt.Errorf("compress: list sessions: %w", err) + } + out := make([]rawEntry, 0, len(sessions)) + warnings := []string{} + for _, sid := range sessions { + s, err := summary.Build(ctx, lstore, sid) + if err != nil { + warnings = append(warnings, "summaries: session "+sid+": "+err.Error()) + continue + } + body := summary.Format(s) + out = append(out, rawEntry{ + Subject: "session " + sid, + Body: body, + Utility: float64(len(s.ToolsUsed)) + float64(s.Turns), + Created: s.CreatedAt.UTC().Format(time.RFC3339), + }) + } + if len(out) == 0 { + warnings = append(warnings, "summaries: no sessions in ledger — nothing to compact") + } + return out, warnings, nil +} + +// loadMemory reads every long-string-valued memory entry. Short values +// (<128 bytes) are skipped — they are perceptually trivial and not worth +// compressing individually; the brain's "prime" path treats them +// inline. Longer values become rawEntry rows ready for dedupe. +func loadMemory(paths Paths) ([]rawEntry, []string, error) { + st, err := memory.Open(paths.Memory) + if err != nil { + return nil, nil, fmt.Errorf("compress: open memory db: %w", err) + } + defer st.Close() + list, err := st.List(memory.ListFilter{Limit: 100000}) + if err != nil { + return nil, nil, fmt.Errorf("compress: read memory: %w", err) + } + warnings := []string{} + if len(list) == 0 { + warnings = append(warnings, "memory: source empty — nothing to compact") + return nil, warnings, nil + } + out := make([]rawEntry, 0, len(list)) + for _, m := range list { + body := m.Insight + if len(body) < 128 { + continue + } + out = append(out, rawEntry{ + Subject: m.ID, + Body: body, + Utility: float64(len(m.Tags)), + Created: m.Created.UTC().Format(time.RFC3339), + }) + } + return out, warnings, nil +} + +// loadAgentsMD reads the AGENTS.md file at the workspace root. Returns +// one entry whose body is the file content (verbatim). The compressor +// is allowed to rewrite the file in place via Apply; Apply is the +// only writer. +func loadAgentsMD(paths Paths) ([]rawEntry, []string, error) { + p := paths.AgentsMD + if p == "" { + discovered, derr := findAgentsMD() + if derr != nil { + return nil, []string{"agents_md: " + derr.Error()}, nil + } + p = discovered + } + data, err := os.ReadFile(p) + if errors.Is(err, fs.ErrNotExist) { + return nil, []string{"agents_md: file not found (" + p + ") — skipping"}, nil + } + if err != nil { + return nil, nil, fmt.Errorf("compress: read agents_md %q: %w", p, err) + } + warnings := []string{} + if len(data) == 0 { + warnings = append(warnings, "agents_md: source empty — nothing to compact") + return nil, warnings, nil + } + out := []rawEntry{{ + Subject: filepath.Base(p), + Body: string(data), + Utility: 1.0, + Created: time.Now().UTC().Format(time.RFC3339), + }} + return out, warnings, nil +} + +// findAgentsMD walks up from the current working directory looking for +// AGENTS.md. Returns the first hit or an error if no parent contains it. +func findAgentsMD() (string, error) { + cwd, err := os.Getwd() + if err != nil { + return "", err + } + dir := cwd + for { + cand := filepath.Join(dir, "AGENTS.md") + if info, err := os.Stat(cand); err == nil && !info.IsDir() { + return cand, nil + } + parent := filepath.Dir(dir) + if parent == dir { + return "", errors.New("AGENTS.md not found in any parent of cwd") + } + dir = parent + } +} + +// stableAnyMap returns the input unchanged but serves as a hook for +// callers that want stable JSON ordering later. The map is captured +// by reference; the lesson-body string remains identical for equal +// inputs, so the SHA-256 dedupe is stable. +func stableAnyMap(in map[string]any) map[string]any { + if in == nil { + return map[string]any{} + } + return in +} + +// oneLine collapses a string into its first non-empty line, trimmed +// and capped at n chars. Used for the PlanEntry.Subject field. +func oneLine(s string, n int) string { + for _, line := range strings.Split(s, "\n") { + t := strings.TrimSpace(line) + if t != "" { + if len(t) > n { + t = t[:n-1] + "…" + } + return t + } + } + return "" +} + +// idFor combines the target and the plan hash to make a stable Plan ID. +func idFor(t Target, hash string) string { + h := sha256.Sum256([]byte(string(t) + "\x00" + hash)) + return "plan-" + hex.EncodeToString(h[:])[:16] +} + +// ensure imports are used. +var _ = sort.Strings diff --git a/cmd/sin-code/internal/compress/sha256_helper_test.go b/cmd/sin-code/internal/compress/sha256_helper_test.go new file mode 100644 index 00000000..3c44c615 --- /dev/null +++ b/cmd/sin-code/internal/compress/sha256_helper_test.go @@ -0,0 +1,13 @@ +// SPDX-License-Identifier: MIT +// Purpose: helpers for test files. Importing crypto/sha256 directly +// avoids import-cycle issues with the lessons package's helper. +package compress + +import "crypto/sha256" + +// sha256SumImpl is the actual SHA-256 invocation. Lifted into a +// distinct file so the test helpers can stay small. +func sha256SumImpl(b []byte) []byte { + h := sha256.Sum256(b) + return h[:] +} diff --git a/cmd/sin-code/internal/compress/testhelpers_test.go b/cmd/sin-code/internal/compress/testhelpers_test.go new file mode 100644 index 00000000..7a6cee34 --- /dev/null +++ b/cmd/sin-code/internal/compress/testhelpers_test.go @@ -0,0 +1,137 @@ +// SPDX-License-Identifier: MIT +// Purpose: tests that need to reach into the lessons package's SQL +// internals but not modify them. Centralizes `openLessonsAt`, +// `fingerprintFor`, `insertRawLesson`, and the bare `contextBg()` used +// by the test suite. +package compress + +import ( + "context" + "database/sql" + "encoding/json" + "fmt" + "time" + + _ "modernc.org/sqlite" + + "github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/lessons" +) + +// openLessonsAt is a thin wrapper that opens a lessons-compatible +// SQLite file *without* going through the lessons package's Open() +// (which would form a test-time import cycle). Same schema, same +// indexes, no behavior expectation differences for the compressor. +func openLessonsAt(path string) (*sql.DB, error) { + db, err := sql.Open("sqlite", path) + if err != nil { + return nil, err + } + schema := ` +CREATE TABLE IF NOT EXISTS lessons ( + id TEXT PRIMARY KEY, + type TEXT NOT NULL, + workspace TEXT NOT NULL, + context TEXT NOT NULL, + lesson TEXT NOT NULL, + occurrences INTEGER DEFAULT 1, + first_seen TEXT NOT NULL, + last_seen TEXT NOT NULL +); +` + if _, err := db.Exec(schema); err != nil { + return nil, err + } + return db, nil +} + +// fingerprintFor delegates to lessons.Fingerprint so the test seeds +// share the same ID scheme that applyLessonsAtomic derives on Apply. +// Without this consistency the test would insert under custom IDs and +// Apply would re-derive under production IDs, deleting the original. +// We can't import the private lessons.Fingerprint... oh wait — it +// IS exported (see internal/lessons/store.go:174). Use it directly. +func fingerprintFor(t, ws string, ctx_ map[string]any) string { + return lessons.Fingerprint(lessons.EntryType(t), ws, ctx_) +} + +// lessonFpBody kept for documentation purposes only — was the +// pre-fix shed-shape. We keep a stub so the tests compile. +func lessonFpBody(t, ws string, ctx_ map[string]any) string { + out, _ := json.Marshal(map[string]any{"type": t, "ws": ws, "ctx": ctx_}) + return string(out) +} + +// stableHashCtx returns 8 bytes of SHA-256 over the joined keys/values +// (deterministic regardless of map iteration order). +func stableHashCtx(t, ws string, ctx_ map[string]any) [8]byte { + keys := make([]string, 0, len(ctx_)) + for k := range ctx_ { + keys = append(keys, k) + } + // Insertion sort. + for i := 1; i < len(keys); i++ { + for j := i; j > 0 && keys[j-1] > keys[j]; j-- { + keys[j-1], keys[j] = keys[j], keys[j-1] + } + } + out := fmt.Sprintf("%s|%s|%v", t, ws, sortedValues(keys, ctx_)) + hash := sha256Sum([]byte(out)) + return [8]byte{hash[0], hash[1], hash[2], hash[3], hash[4], hash[5], hash[6], hash[7]} +} + +// sortedValues returns ctx_'s values in key-sorted order. +func sortedValues(keys []string, ctx_ map[string]any) []any { + out := make([]any, len(keys)) + for i, k := range keys { + out[i] = ctx_[k] + } + return out +} + +// sha256Sum exposes a hash helper without forcing an import in callers. +func sha256Sum(b []byte) []byte { + return sha256SumImpl(b) +} + +// insertRawLesson inserts a row matching the lessons schema. We +// pre-compute the fingerprint with the context that the compressor +// later re-derives via lessons.Fingerprint, so the apply step works. +func insertRawLesson(db *sql.DB, typ, lesson string, occ int, ctx map[string]any) (string, error) { + id := fingerprintFor(typ, "*", ctx) + now := time.Now().UTC().Format(time.RFC3339) + if _, err := db.Exec(`INSERT INTO lessons (id, type, workspace, context, lesson, occurrences, first_seen, last_seen) VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, + id, typ, "*", encodedCtxJSON(ctx), lesson, occ, now, now); err != nil { + return "", err + } + return id, nil +} + +// encodedCtxJSON returns a stable JSON byte slice — at minimum we +// require keys to lex-sort. The map is small enough that we serialize +// "key=value" pairs in sorted order. +func encodedCtxJSON(ctx map[string]any) string { + if len(ctx) == 0 { + return "{}" + } + keys := make([]string, 0, len(ctx)) + for k := range ctx { + keys = append(keys, k) + } + for i := 1; i < len(keys); i++ { + for j := i; j > 0 && keys[j-1] > keys[j]; j-- { + keys[j-1], keys[j] = keys[j], keys[j-1] + } + } + out := "{" + for i, k := range keys { + if i > 0 { + out += "," + } + out += fmt.Sprintf("%q:%v", k, ctx[k]) + } + out += "}" + return out +} + +// contextBg is a lazy alias to context.Background() used by tests. +func contextBg() context.Context { return context.Background() } diff --git a/cmd/sin-code/internal/compress/types.go b/cmd/sin-code/internal/compress/types.go new file mode 100644 index 00000000..6257f4a2 --- /dev/null +++ b/cmd/sin-code/internal/compress/types.go @@ -0,0 +1,214 @@ +// SPDX-License-Identifier: MIT +// Package compress implements deterministic + LLM-assisted compaction +// for SIN-Code's long-lived memory stores (lessons, instincts, summaries, +// AGENTS.md-shaped memory). Compaction is lossless: every dropped entry +// is preserved verbatim in a JSON snapshot under +// ~/.local/share/sin-code/compress-snapshots/.json. Rollback restores +// the dropped entries byte-for-byte. +// +// Hard requirements (per AGENTS.md): +// - Output is reproducible. The same input must always produce the +// same Plan, the same hashes, the same ApplyReport. No wall-clock +// randomness sneaks into the algorithm. +// - Snapshots are atomic. A crash mid-write either leaves the +// snapshot file intact and the live store untouched (safe to Rollback), +// or leaves a `.partial` marker that Rollback refuses to consume. +// - The package depends only on stdlib + already-imported internal +// packages (lessons, instinct, summary, memory, llm). CGO_ENABLED=0 +// preserved (M2). +// - Plan is read-only; Apply is the only mutator. The CLI wires the +// the two steps so LLM-extracted Plans are visible to the user +// before they touch disk (`--dry-run` stops at Plan). +// +// Issue: sin-code compress (#172). +package compress + +import ( + "crypto/sha256" + "encoding/hex" +) + +// Target names the source surface the compressor reads from. +type Target string + +const ( + TargetLessons Target = "lessons" + TargetInstincts Target = "instincts" + TargetSummaries Target = "summaries" + TargetMemory Target = "memory" // AGENTS.md-shaped long-valued memory entries + TargetAgentsMD Target = "agents_md" // the on-disk AGENTS.md file at the workspace root + TargetAll Target = "all" // every target above +) + +// AllTargets lists the concrete (non-aggregate) targets. +var AllTargets = []Target{ + TargetLessons, + TargetInstincts, + TargetSummaries, + TargetMemory, + TargetAgentsMD, +} + +// IsValid reports whether t is a recognized Target (incl. "all"). +func (t Target) IsValid() bool { + switch t { + case TargetLessons, TargetInstincts, TargetSummaries, TargetMemory, TargetAgentsMD, TargetAll: + return true + } + return false +} + +// Strategy selects the algorithmic family used to compute a Plan. +// +// - StrategyDeterministic: SHA-256 dedupe + byte-budgeted keep-recent. +// No network, no LLM. Reproducible across runs and machines. +// - StrategyLLM: ask the configured llm.Client to merge N oldest +// entries into one summary entry. Requires a configured provider; +// if the client cannot chat (no key / offline), the Plan falls back +// to deterministic for the affected scope and surfaces a warning. +// - StrategyHybrid: deterministic pass first; only residual chunks +// above the byte budget get summarized by the LLM. +type Strategy string + +const ( + StrategyDeterministic Strategy = "deterministic" + StrategyLLM Strategy = "llm" + StrategyHybrid Strategy = "hybrid" +) + +// IsValid reports whether s is one of the three recognized strategies. +func (s Strategy) IsValid() bool { + switch s { + case StrategyDeterministic, StrategyLLM, StrategyHybrid: + return true + } + return false +} + +// Plan is the read-only description of a compaction. It is what `--dry-run` +// emits and what `Apply` consumes. Plans are content-addressed: PlanHash +// is computed over Entries[]+Drops[]+Merges[] without timestamps, so two +// Plans built from identical inputs but at different wall-clock times have +// the same PlanHash. Useful for the regression test suite. +type Plan struct { + ID string `json:"id"` + Target Target `json:"target"` + Strategy Strategy `json:"strategy"` + CreatedAt string `json:"created_at"` + PlanHash string `json:"plan_hash"` + + // Original / Kept counts by target. Used for the CLI ratio table. + Stats PlanStats `json:"stats"` + + // Entries to keep verbatim, in final order. Order is deterministic + // (utility score desc, then by entry hash asc for ties — a stable + // sort tiebreaker). Hash is the SHA-256 of the canonical body. + Keeps []PlanEntry `json:"keeps"` + + // Entries to remove. After Apply, the removed bodies live on disk + // under SnapshotPath (sha256 keyed) — Apply is therefore lossless. + Drops []PlanEntry `json:"drops"` + + // Merges carry LLM-produced replacements: each Merge maps a set of + // `SourceHashes` to a single new entry whose `Hash` is its canonical + // SHA-256 after Apply. Markdown-formatted, with preservation markers. + Merges []PlanMerge `json:"merges,omitempty"` + + // Paths is the snapshot of the source-overrides the caller used + // during BuildPlan/Apply. Rollback uses it to find the same DB + // files; without it, Rollback would fall back to defaults and + // the user could roll back to the wrong file. + Paths Paths `json:"paths,omitempty"` + + // Warnings are non-fatal advisories surfaced by --dry-run. + Warnings []string `json:"warnings,omitempty"` +} + +// PlanStats is the per-target breakdown surfaced by `--dry-run`. +type PlanStats struct { + Keeps int `json:"keeps"` + Drops int `json:"drops"` + Merges int `json:"merges"` + OriginalEntries int `json:"original_entries"` + OriginalBytes int `json:"original_bytes"` + ProjectedEntries int `json:"projected_entries"` + ProjectedBytes int `json:"projected_bytes"` + ProjectedRatio float64 `json:"projected_ratio"` // kept_bytes / original_bytes +} + +// PlanEntry is the entry-shaped view a Plan exposes. Hash is canonical. +type PlanEntry struct { + Hash string `json:"hash"` + Target Target `json:"target"` + Subject string `json:"subject"` // human-readable label (lesson text, instinct id, summary title) + Body string `json:"body"` // raw entry body, normalized for hashing + Bytes int `json:"bytes"` + Utility float64 `json:"utility"` + Created string `json:"created"` +} + +// PlanMerge describes one LLM-driven merge: a list of source hashes that +// map to a synthesized replacement body. +type PlanMerge struct { + ID string `json:"id"` + Strategy Strategy `json:"strategy"` + SourceHashes []string `json:"source_hashes"` + Body string `json:"body"` + Bytes int `json:"bytes"` +} + +// ApplyOptions is the Apply input that pairs a Plan with execution knobs. +type ApplyOptions struct { + // Reason is recorded in the snapshot manifest so audit logs can + // correlate a compaction with the originating CLI invocation. + Reason string + + // MinInterval prevents aggressive back-to-back Apply calls when + // the user re-runs the CLI in a tight loop. Zero means "no gate"; + // the default Plan() caller passes zero. + MinIntervalSeconds int + + // DryRun, if true, returns a successful ApplyReport with no + // on-disk side effects. Useful for the CLI's `--dry-run`. + DryRun bool +} + +// ApplyReport is the emit-from-Apply result. Includes per-target byte +// deltas, the snapshot ID written (if any), and surviving entry hashes +// so the CLI can render "what changed" tables. +type ApplyReport struct { + PlanID string `json:"plan_id"` + SnapshotID string `json:"snapshot_id,omitempty"` + SnapshotPath string `json:"snapshot_path,omitempty"` + AppliedAt string `json:"applied_at"` + OriginalBytes int `json:"original_bytes"` + KeptBytes int `json:"kept_bytes"` + Ratio float64 `json:"ratio"` + PerTarget []TargetReport `json:"per_target"` + Warnings []string `json:"warnings,omitempty"` +} + +// TargetReport is one row of the per-target breakdown. +type TargetReport struct { + Target Target `json:"target"` + BeforeEntries int `json:"before_entries"` + AfterEntries int `json:"after_entries"` + BeforeBytes int `json:"before_bytes"` + AfterBytes int `json:"after_bytes"` + SnapshotID string `json:"snapshot_id,omitempty"` +} + +// ContentHash returns the canonical SHA-256 hash for a body. Whitespace +// at end-of-line is trimmed, byte-order is preserved (no Unicode +// normalization — the bodies are technical text, NFC changes are +// surprising). +func ContentHash(body string) string { + h := sha256.Sum256([]byte(body)) + return hex.EncodeToString(h[:]) +} + +// shortHash returns the first 16 hex chars of ContentHash. Used for +// in-memory keys and snapshot file IDs. +func shortHash(body string) string { + return ContentHash(body)[:16] +} diff --git a/cmd/sin-code/internal/permission_defaults.go b/cmd/sin-code/internal/permission_defaults.go index b957400a..a525ed8d 100644 --- a/cmd/sin-code/internal/permission_defaults.go +++ b/cmd/sin-code/internal/permission_defaults.go @@ -82,6 +82,13 @@ func DefaultPermissionRules() []permission.Rule { {Tool: "install__verify_only", Policy: "allow"}, {Tool: "install__run", Policy: "ask"}, {Tool: "install__dry_run", Policy: "allow"}, + // v3.18.0: Compress (issue #172). compress is a first-party + // CLI verb that mutates the on-disk memory stores; the + // policy below mirrors eval__run (ask): a destructive + // operation gated on user confirmation. + {Tool: "compress__plan", Policy: "allow"}, // read-only projection + {Tool: "compress__apply", Policy: "ask"}, // rewrites + LLM call + {Tool: "compress__rollback", Policy: "allow"}, // restorative — never destructive // Backstop catch-all (mirrors sin_bash default at line 44 for unmatched prefixes). {Tool: "autodev__*", Policy: "ask"}, {Tool: "*", Policy: "ask"}, diff --git a/cmd/sin-code/main.go b/cmd/sin-code/main.go index 4c77ef72..32ec6cd9 100644 --- a/cmd/sin-code/main.go +++ b/cmd/sin-code/main.go @@ -32,7 +32,7 @@ It consolidates 44+ subcommands into a single cobra-based CLI: Advanced tools: ibd, poc, sckg, adw, oracle, efm Utility commands: security, sbom, config, self-update, tui, serve, update Agent ecosystem: chat, sessions, mcp, goal, daemon, skill, superpowers, - vane, stack, gh, hub, ledger, summary, install + vane, stack, gh, hub, ledger, summary, install, compress Other: completion, read, write, edit, lsp, plugin, index, orchestrator-run, orchestrator-agents, orchestrator-plan, todo, notifications, memory, assets, evalset, hooks, @@ -82,6 +82,7 @@ func init() { NewGoalCmd(), NewDaemonCmd(), NewSkillCmd(), NewSwarmCmd(), NewSuperpowersCmd(), NewDoxCmd(), NewVaneCmd(), NewStackCmd(), NewGhCmd(), NewHubCmd(), NewLedgerCmd(), NewSummaryCmd(), NewAutodevCmd(), // v3.4.0 + v3.5.0 + v3.6.0 + v3.7.0 + v3.8.0 + v3.9.0 + v3.12.0 + v3.13.0 + autodev-bridge (Python MIT v0.4.0, stdio MCP via autodev-mcp) + NewCompressCmd(), // v3.18.0 — deterministic + LLM compaction (issue #172) NewSkillsCmd(), // bundled project-local agent skills NewEvalCmd(), NewTraceCmd(), // v3.18.0: Eval & Observability System (issue #75) NewRtkCmd(), // rtk (Rust Token Killer) bridge (issue #123) diff --git a/cmd/sin-code/testdata/golden/help.golden b/cmd/sin-code/testdata/golden/help.golden index 7030d725..4843915f 100644 --- a/cmd/sin-code/testdata/golden/help.golden +++ b/cmd/sin-code/testdata/golden/help.golden @@ -5,7 +5,7 @@ It consolidates 44+ subcommands into a single cobra-based CLI: Advanced tools: ibd, poc, sckg, adw, oracle, efm Utility commands: security, sbom, config, self-update, tui, serve, update Agent ecosystem: chat, sessions, mcp, goal, daemon, skill, superpowers, - vane, stack, gh, hub, ledger, summary, install + vane, stack, gh, hub, ledger, summary, install, compress Other: completion, read, write, edit, lsp, plugin, index, orchestrator-run, orchestrator-agents, orchestrator-plan, todo, notifications, memory, assets, evalset, hooks, @@ -28,6 +28,7 @@ Available Commands: codegraph Bridge to CodeGraph for multi-language code analysis compile-spec Compile .sin-code.yml into the four derived JSON artifacts completion Generate the autocompletion script for the specified shell + compress Lossless compaction for lessons / instincts / summaries / AGENTS.md config View and manage sin-code configuration daemon Run the autonomous worker: lease goals, execute, verify, learn discover Discover files with relevance scoring and pattern matching diff --git a/cmd/sin-code/testdata/scripts/golden_help.txt b/cmd/sin-code/testdata/scripts/golden_help.txt index e48d56fe..ed391690 100644 --- a/cmd/sin-code/testdata/scripts/golden_help.txt +++ b/cmd/sin-code/testdata/scripts/golden_help.txt @@ -9,7 +9,7 @@ It consolidates 44+ subcommands into a single cobra-based CLI: Advanced tools: ibd, poc, sckg, adw, oracle, efm Utility commands: security, sbom, config, self-update, tui, serve, update Agent ecosystem: chat, sessions, mcp, goal, daemon, skill, superpowers, - vane, stack, gh, hub, ledger, summary, install + vane, stack, gh, hub, ledger, summary, install, compress Other: completion, read, write, edit, lsp, plugin, index, orchestrator-run, orchestrator-agents, orchestrator-plan, todo, notifications, memory, assets, evalset, hooks, @@ -32,6 +32,7 @@ Available Commands: codegraph Bridge to CodeGraph for multi-language code analysis compile-spec Compile .sin-code.yml into the four derived JSON artifacts completion Generate the autocompletion script for the specified shell + compress Lossless compaction for lessons / instincts / summaries / AGENTS.md config View and manage sin-code configuration daemon Run the autonomous worker: lease goals, execute, verify, learn discover Discover files with relevance scoring and pattern matching