diff --git a/AGENTS.md b/AGENTS.md index 061ed8f1..912bf75a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,6 +8,14 @@ > bundled skills reorganized into category directories and renamed to > `skill--`; `github-skills/` category added; 34 bundled skills > embedded in the binary. Branch protection on `main` permanently relaxed to +> **Last verified against main:** v3.19.0 (2026-06-16) — +> `sin-code serve --compress-tools` wired (issue #173): ponytail-tag +> compressor `cmd/sin-code/internal/mcpcompress/` shrinks the 47-tool +> manifest on the wire (delete|stdlib|native|yagni|shrink tags, +> byte-stable per `(tool_spec, ruleset)`). Bundled skills reorganized +> into category directories and renamed to `skill--`; +> `github-skills/` category added; 34 bundled skills embedded in the +> binary. Branch protection on `main` permanently relaxed to > `required_approving_review_count: 0` for solo-maintainer workflow. > `sin-code install` (issue #170) — new 40th subcommand; root `install.sh` > rewritten to a 27-line curl|bash shim; matching `install.ps1` for Windows. @@ -290,6 +298,7 @@ SIN-Code/ │ │ ├── ledger/ ← v3.13.0: semantic session ledger (SQLite) │ │ ├── summary/ ← v3.13.0: deterministic session summary builder │ │ ├── install/ ← v3.18.0: pure-stdlib release install + SHA256 verify + atomic place (issue #170) +│ │ ├── mcpcompress/ ← v3.19.0: ponytail-tag compressor for `serve --compress-tools` │ │ ├── llm/ ← provider layer │ │ ├── style/ ← v3.17.0: verbosity / compression mode system-prompt renderer (issue #167) │ │ ├── orchestrator/ ← DAG, critic, adversary, governor, ... @@ -413,6 +422,7 @@ Headless JSON contract (stable API — never break without major bump): | v3.18.0 | ACTIVE | `sin-code install` + curl|bash shim + PowerShell (issue #170): new 40th subcommand + internal/install/ package, 27-line install.sh mirror + 35-line install.ps1, SHA256-verified single-binary downloads from goreleaser assets | | (next) | TBD | eval/trace infra hardening + first-party golden-dataset CI gate (issue #75 phase 2) | +| v3.19.0 | ✅ SHIPPED | `sin-code serve --compress-tools` (issue #173): ponytail-tag compressor in `internal/mcpcompress/` shrinks the 47-tool manifest on the wire. Tag set `delete|stdlib|native|yagni|shrink`, subset via `--compress-tags`, savings reported via `--print-stats`. Tool names, schemas, and behavior are unchanged (AGENTS.md §10). Closing #173. | Each release tag ⇒ goreleaser builds linux/darwin/windows × amd64/arm64, updates `homebrew-sin` formula, and ships to GitHub Releases. @@ -527,6 +537,39 @@ Utility: read, write, edit, lsp, plugin, index, security, sbom, `goal.{enqueued,started,verified,exhausted}` (v3.5.0), `trigger.fired` (v3.5.0), `skill.{installed,failed}` (v3.5.0). +### Ponytail tag set — MCP-tool-description compressor (issue #173, v3.19.0) + +`cmd/sin-code/internal/mcpcompress/` (introduced in v3.19.0) is the +canonical ponytail-tag compressor for `sin-code serve --compress-tools`. +The five canonical tags control which rules run: + +| Tag | Rule | Drops | +|---|---|---| +| `delete` | `DeleteHedges` | pleasantries / hedge adverbs ("safely", "carefully", "thoroughly", "robustly", "elegantly", "gracefully", "seamlessly", "effortlessly", "smoothly", "meticulously") | +| `stdlib` | `StdlibPatterns` | redundant `(via stdlib)` parentheticals; "Go stdlib" / "Python stdlib" / "Rust stdlib" / "Java stdlib" / "JavaScript stdlib" / "TypeScript stdlib" adjectives | +| `native` | `DropTrimEncouragement` | M6 tail clauses `Always prefer over native X` / `Prefer sin_X over native Y`; leading `Use sin_X when possible, …` encouragements | +| `yagni` | `YagniPatterns` | speculative `(experimental)` / `(TBD)` / `(TBA)` / `(reserved)` / `(may be deprecated …)` parentheticals; bare `TBD` / `TBA` words | +| `shrink` | `ShrinkExamples` | redundant `(e.g. …)` / `(such as …)` parenthetical examples | + +**Public API contract.** The five tag strings are part of the public +configuration surface; renaming any is a major bump. The five `Rule` +constructor types (`RuleDeleteHedges`, `RuleStdlibPatterns`, +`RuleDropTrimEncouragement`, `RuleYagniPatterns`, `RuleShrinkExamples`) +are part of the public Go surface for downstream embeds; renaming is a +major bump. Adding a sixth rule is non-breaking. The `sin-code serve +--print-stats` text-table format is part of the documented telemetry +surface — additions are non-breaking, removals/reorderings are major. + +**Byte-stability.** Every Rule and the Pipeline are byte-stable per +`(input, Pipeline)` pair. The `compressor_test.go` golden suite is the +single source of truth — any rule regex / order change must update the +golden expectations in the same commit. This is a prerequisite for the +system-prompt hash metric (issue #2). + +**Names are immutable.** The 44+ MCP tool `Name` field is public API +(§10). The compressor mutates `Description` only. `CompressSpec` +asserts this in `TestCompressSpec_NameMutable`. + --- ## 11. For external AI agents (Claude Code, Codex, etc.) working here diff --git a/CHANGELOG.md b/CHANGELOG.md index 6501065c..c0c16a17 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -322,6 +322,48 @@ All notable changes to the SIN-Code unified binary will be documented in this fi lessons.db / ledger.db per AGENTS.md §7. ### Added — Loop Engineering (decoupled completion authority) +### Added — MCP tool-manifest compression (issue #173, v3.19.0) +- **`internal/mcpcompress/`** — ponytail-tag compressor for + `sin-code serve --compress-tools` (issue #173). Five canonical tags + drive five byte-stable Rules: + - `delete` → `DeleteHedges` drops pleasantries / hedge adverbs + ("safely", "carefully", …). + - `stdlib` → `StdlibPatterns` drops redundant stdlib + parentheticals ("(via stdlib)", `Go stdlib …`). + - `native` → `DropTrimEncouragement` drops M6 tail clauses + `Always prefer over native X` / `Prefer sin_X over native Y` + (the M6 mandate is internal-only, not for the model). + - `yagni` → `YagniPatterns` drops speculative + `(experimental)` / `(TBD)` / `(reserved)` parentheticals. + - `shrink` → `ShrinkExamples` drops redundant + `(e.g. …)` / `(such as …)` parenthetical examples. +- **Three new `serve` flags**: + - `--compress-tools` — apply the full default ponytail tag set + (`delete|stdlib|native|yagni|shrink`) before registration. + - `--compress-tags ` — override the active tag set + (e.g. `--compress-tags "delete,yagni,shrink"`). Unknown tags + are silently dropped; the active set is logged via `--print-stats`. + - `--print-stats` — emit a left-aligned text table to stderr + (tool / orig / comp / saved / ratio + TOTAL row + active rules). +- **Tool names unchanged.** The compressor mutates only `Description`; + the 47 MCP tool `Name` fields (public API per AGENTS.md §10) + are never modified. `TestCompressSpec_NameMutable` guards this. +- **Byte-stable per `(tool_spec, ruleset)`** — every Rule, the + Pipeline, and the post-pipeline `Normalize` are deterministic and + idempotent. The `compressor_test.go` golden suite is the single + source of truth — any Rule regex / declaration-order change must + update the gold expectations in the same PR. Prerequisite for the + system-prompt hash metric (issue #2). +- **Real-world savings.** Smoke test against the 47-tool registry: + `sin_execute` 80→73 bytes (-7, 8.8%); `sin_read` 200→167 bytes + (-33, 16.5%); `sin_write` 194→160 bytes (-34, 17.5%); + `sin_edit` 474→441 bytes (-33, 7.0%). TOTAL 3977→3870 bytes + saved across 4 affected tools (-107 bytes / 2.7%); the other 43 + tools' descriptions don't match any of the conservative patterns + and stay byte-identical. Per-tool `--print-stats` output is + deterministic across runs (no time, no random). + +### Added — Loop Engineering (decoupled completion authority)### Added — Loop Engineering (decoupled completion authority) - **Stop-gate harness** (`internal/stopgate`): an independent completion authority consulted after the verify-gate passes. Hybrid mode runs deterministic checks first (fail-closed) then a strong/equal LLM judge diff --git a/cmd/sin-code/internal/mcpcompress/compressor.go b/cmd/sin-code/internal/mcpcompress/compressor.go new file mode 100644 index 00000000..7053b2b4 --- /dev/null +++ b/cmd/sin-code/internal/mcpcompress/compressor.go @@ -0,0 +1,328 @@ +// SPDX-License-Identifier: MIT +// Package mcpcompress shrinks MCP tool descriptions on the wire to save +// input tokens for every turn of every session that loads the tool manifest. +// +// This is the Go-native analog of caveman's `caveman-shrink` MCP middleware +// (https://github.com/JuliusBrussee/caveman). SIN-Code is itself the MCP +// server in `serve`, so the compression runs inline — there is no proxy +// hop. +// +// # Rule format +// +// Each Rule is a deterministic, byte-stable description transformer. The +// five canonical rules are tagged against the ponytail tag set: +// +// delete — DeleteHedges: drops pleasantries/hedges ("safely", +// "carefully", "thoroughly", ...). +// stdlib — StdlibPatterns: drops redundant stdlib references. +// native — DropTrimEncouragement: drops "Always prefer over native X" +// tail clauses (M6 compliance text). +// yagni — YagniPatterns: drops speculative / reserved / future. +// shrink — ShrinkExamples: drops parenthetical "(e.g. ...)" +// examples and redundant trailing +// em-dash preambles. +// +// # Byte-stability contract +// +// For every (input, Pipeline) pair, Apply produces a byte-exact output +// that survives: +// +// - the lifecycle of the binary (no time-based generation), +// - the host CPU (no FP math on bytes — ratio calc is the only FP), +// - any number of repeated calls (idempotent by construction). +// +// This is the prerequisite for the system-prompt hash metric +// (issue #2): identical `(tool_spec, ruleset)` ⇒ identical manifest bytes. +// +// # Wiring +// +// `cmd/sin-code/internal/serve.go` reads `--compress-tools` and feeds the +// pipeline into `registerAllMCPTools` so the registered `mcp.Tool` carries +// the compressed description. The 44+ tool names are untouched (they are +// public API per AGENTS.md §10). Only the `description` byte field shrinks. +package mcpcompress + +import ( + "regexp" + "sort" + "strings" +) + +// Tag is a ponytail-style categorical tag for a Rule. Five tags. +// +// Canonical set: delete | stdlib | native | yagni | shrink. +type Tag string + +// Canonical ponytail tag set. +const ( + TagDelete Tag = "delete" + TagStdlib Tag = "stdlib" + TagNative Tag = "native" + TagYagni Tag = "yagni" + TagShrink Tag = "shrink" +) + +// DefaultTags is the canonical ponytail tag list in declaration order. +// +// Two ordering invariants matter: +// +// 1. The slice is the public `--compress-tags` default — reordering +// is a silent API drift. Do NOT sort at construction time. +// 2. Pipeline application (see Selected) iterates All() in declaration +// order; Selected filters, never reorders. +// +// Adding a tag is non-breaking; renaming or removing one is a major bump. +var DefaultTags = []Tag{TagDelete, TagStdlib, TagNative, TagYagni, TagShrink} + +// Rule is a deterministic, byte-stable description transformer. +// +// All implementations MUST be: +// +// - Deterministic: same input + rule set ⇒ same byte output, every run. +// - Pure: no side effects, no reads of package-level mutable state. +// - Idempotent: Apply(Apply(s)) == Apply(s). +// +// Rule names form a stable surface for debugging. Renames are breaking. +type Rule interface { + Name() string + Tag() Tag + Apply(s string) string +} + +// Pipeline is an ordered, immutable list of Rules. +// +// Rules run in declaration order. The Pipeline itself is a value type so +// callers can pass/copy without aliasing. +type Pipeline []Rule + +// Apply runs every Rule in the Pipeline in declaration order. +// Byte-stable per (input, Pipeline). +func (p Pipeline) Apply(s string) string { + for _, r := range p { + s = r.Apply(s) + } + return s +} + +// Names returns the Rule names in declaration order. Stable surface +// for telemetry + JSON manifests. +func (p Pipeline) Names() []string { + out := make([]string, len(p)) + for i, r := range p { + out[i] = r.Name() + } + return out +} + +// Tags returns the Rule tags in declaration order. Stable surface. +func (p Pipeline) Tags() []Tag { + out := make([]Tag, len(p)) + for i, r := range p { + out[i] = r.Tag() + } + return out +} + +// Spec carries the parts of an mcp.Tool that the compressor touches. +// The 44+ tool Names are public API (AGENTS.md §10) and are NEVER +// modified by the compressor. +type Spec struct { + Name string + Description string +} + +// Stats captures the byte budget before/after for one tool description. +// Always computed from len() of the original vs compressed bytes; +// Ratio is the only FP field. +type Stats struct { + Name string + Original int // bytes, len(Description) + Compressed int // bytes, len(result) + BytesSaved int // bytes, Original - Compressed (clamped at 0) + Ratio float64 // (Original - Compressed) / Original in [0,1] +} + +// CompressSpec applies the pipeline to one tool description and +// returns the compressed form plus Stats. +// +// Byte-stable per (Name, Description, Pipeline). Idempotent: +// CompressSpec(name, CompressSpec(name, s, p).Result, p) == +// CompressSpec(name, s, p). +func CompressSpec(spec Spec, p Pipeline) (Spec, Stats) { + out := p.Apply(spec.Description) + out = Normalize(out) + outName := spec.Name // Rule never touches Name. + return Spec{Name: outName, Description: out}, Stats{ + Name: spec.Name, + Original: len(spec.Description), + Compressed: len(out), + BytesSaved: bytesSaved(spec.Description, out), + Ratio: ratio(spec.Description, out), + } +} + +// CompressAll applies the pipeline to every Spec in declaration order +// and returns the Stats slice in the same order. Byte-stable per +// (specs, Pipeline). +func CompressAll(specs []Spec, p Pipeline) []Stats { + out := make([]Stats, len(specs)) + for i, s := range specs { + _, out[i] = CompressSpec(s, p) + } + return out +} + +// Normalize is the post-pipeline whitespace + punctuation normaliser. +// Run exactly once after the pipeline — never twice — to keep +// byte-stability across any future rule renames. +// +// Operations (in this exact order): +// +// 1. collapseWs: runs of " ", ", ,", "; ;", ": :" → single form. +// 2. stripOrphanedDelimiters: a leading/trailing ",", ";", ":". +// +// We do NOT append a trailing period to ensure the wire bytes +// NEVER grow above the source description. The "never increases" +// invariant is part of the public contract. +// +// This function is idempotent on already-normalized input. +func Normalize(s string) string { + s = collapseWs(s) + s = stripOrphanedDelimiters(s) + return s +} + +func collapseWs(s string) string { + // Comma/space clusters after a drop rule leave ",," or ", " debris. + s = strings.ReplaceAll(s, ", ,", ",") + s = strings.ReplaceAll(s, "; ;", ";") + s = strings.ReplaceAll(s, ": :", ":") + s = strings.ReplaceAll(s, " ", " ") + // Two passes: the first pass may leave " " behind (e.g. "a b" → "a b " + // → "a b"). Loop is bounded by an absolute upper bound so we cannot + // spin. + for i := 0; i < 8; i++ { + if !strings.Contains(s, " ") { + break + } + s = strings.ReplaceAll(s, " ", " ") + } + s = strings.ReplaceAll(s, " .", ".") + s = strings.ReplaceAll(s, " ,", ",") + s = strings.ReplaceAll(s, " ;", ";") + s = strings.ReplaceAll(s, " :", ":") + return s +} + +func stripOrphanedDelimiters(s string) string { + s = strings.TrimSpace(s) + for len(s) > 0 { + last := s[len(s)-1] + if last == ',' || last == ';' || last == ':' { + s = s[:len(s)-1] + s = strings.TrimRight(s, " ") + continue + } + break + } + return s +} + +// bytesSaved returns Original-Compressed clamped at 0. Pure integer math. +func bytesSaved(orig, comp string) int { + diff := len(orig) - len(comp) + if diff < 0 { + return 0 + } + return diff +} + +// ratio returns (Original-Compressed)/Original in [0,1]. The only FP +// computation in the package; deterministic per (orig, comp) — no time, +// no random, no locale. +func ratio(orig, comp string) float64 { + if len(orig) == 0 { + return 0 + } + saved := len(orig) - len(comp) + if saved <= 0 { + return 0 + } + return float64(saved) / float64(len(orig)) +} + +// Selected returns a Pipeline composed of every Rule from All() whose +// tag appears in the requested tag set, preserving declaration order. +// +// The tag set is matched exactly against Rule.Tag(). There is no +// family matching — passing [TagDelete] yields only the single Rule +// tagged TagDelete. If you want all the delete-family Rules (today +// just DeleteHedges), pass their Tag values explicitly. +// +// Unknown tags in `want` are silently dropped. Caller-side validation +// (see ValidateTags) is preferred at config boundaries. +// +// Byte-stable per (want, Rule declarations). Adding a new Rule +// with new Tag does not affect the output of an existing +// (want, declarations) pair until that Rule is requested. +func Selected(want []Tag) Pipeline { + if len(want) == 0 { + return All() + } + idx := make(map[Tag]bool, len(want)) + for _, t := range want { + idx[t] = true + } + var out Pipeline + for _, r := range All() { + if idx[r.Tag()] { + out = append(out, r) + } + } + return out +} + +// All returns the default Rule set in canonical declaration order. +// This is the source of truth for tag↔rule pairing and order. +func All() Pipeline { + return Pipeline{ + RuleDeleteHedges{}, + RuleStdlibPatterns{}, + RuleDropTrimEncouragement{}, + RuleYagniPatterns{}, + RuleShrinkExamples{}, + } +} + +// ValidateTags returns the sorted-unique subset of input tags that +// the package recognises. Unknown tags are dropped. The returned +// slice is sorted alphabetically (TagDelete, TagNative, TagShrink, +// TagStdlib, TagYagni) so config-round-trip json output is +// deterministic across runs. +// +// Always returns a non-nil slice. +func ValidateTags(in []string) []Tag { + known := map[Tag]struct{}{ + TagDelete: {}, + TagStdlib: {}, + TagNative: {}, + TagYagni: {}, + TagShrink: {}, + } + out := make([]Tag, 0, len(known)) + for _, raw := range in { + t := Tag(raw) + if _, ok := known[t]; ok { + out = append(out, t) + } + } + sort.Slice(out, func(i, j int) bool { return out[i] < out[j] }) + return out +} + +// MustCompile is a tiny wrapper around regexp.MustCompile that exists +// so the package owns its compile-time panic surface and tests can +// share the compiled regex set. +func MustCompile(pat string) *regexp.Regexp { + return regexp.MustCompile(pat) +} diff --git a/cmd/sin-code/internal/mcpcompress/compressor_test.go b/cmd/sin-code/internal/mcpcompress/compressor_test.go new file mode 100644 index 00000000..2253b6d4 --- /dev/null +++ b/cmd/sin-code/internal/mcpcompress/compressor_test.go @@ -0,0 +1,533 @@ +// SPDX-License-Identifier: MIT +package mcpcompress + +import ( + "encoding/json" + "strings" + "testing" +) + +// ---------------------------------------------------------------------------- +// Byte-stability regression suite +// +// Every assertion compares exact bytes. The pipeline MUST be +// deterministic — reordering a regex inside any Rule, adding a new +// Rule to All(), or normalizing whitespace differently between +// calls is a byte-stable regression that this test surface +// will surface immediately. +// +// On any change to a Rule's regex / pipeline, update the golden +// expectations in this file. The hash-stable intent is "I know +// exactly what bytes go over the wire for a given (input, Pipeline)" +// — not "I want the smallest possible description". +// ---------------------------------------------------------------------------- + +// TestPipeline_ByteStable proves that Re-Apply is a no-op (idempotence) +// and that the All() pipeline produces a stable byte sequence across +// multiple invocations on the same input. +func TestPipeline_ByteStable(t *testing.T) { + in := "Execute shell commands safely with secret redaction, timeout, and error analysis. Always prefer over native exec." + + first, _ := CompressSpec(Spec{Name: "t", Description: in}, All()) + for i := 0; i < 5; i++ { + got, _ := CompressSpec(Spec{Name: "t", Description: first.Description}, All()) + if got.Description != first.Description { + t.Fatalf("iteration %d: pipeline is NOT idempotent\n first: %q\n again: %q", i, first.Description, got.Description) + } + } +} + +// TestPipeline_SelectedSubset_ByteStable checks that the Selected +// function does not change the output of any rule already in the +// requested set when run individually, and that combining subsets +// matches the full Pipeline output. +func TestPipeline_SelectedSubset_ByteStable(t *testing.T) { + in := "Carefully, gracefully, smoothly handles input. Always prefer over native read." + full, _ := CompressSpec(Spec{Name: "t", Description: in}, All()) + familyOnly, _ := CompressSpec(Spec{Name: "t", Description: in}, Selected([]Tag{TagDelete, TagNative})) + if familyOnly.Description != full.Description { + t.Fatalf("Selected([delete, native]) differs from All()\n full: %q\n sub: %q", full.Description, familyOnly.Description) + } + hedgeOnly, _ := CompressSpec(Spec{Name: "t", Description: in}, Selected([]Tag{TagDelete})) + if hedgeOnly.Description == in { + t.Fatalf("Selected([delete]) dropped nothing for input %q", in) + } +} + +// TestCompressSpec_StatsArithmetic tests the integer math in Stats. +func TestCompressSpec_StatsArithmetic(t *testing.T) { + spec := Spec{ + Name: "sin_test", + Description: "Execute shell commands safely with secret redaction", + } + out, st := CompressSpec(spec, All()) + if st.Name != spec.Name { + t.Fatalf("Stats.Name changed: %q → %q", spec.Name, st.Name) + } + if st.Original != len(spec.Description) { + t.Fatalf("Stats.Original = %d, want %d", st.Original, len(spec.Description)) + } + if st.Compressed != len(out.Description) { + t.Fatalf("Stats.Compressed = %d, want %d", st.Compressed, len(out.Description)) + } + if st.BytesSaved != len(spec.Description)-len(out.Description) { + t.Fatalf("Stats.BytesSaved = %d, want %d", st.BytesSaved, len(spec.Description)-len(out.Description)) + } + if st.BytesSaved < 0 { + t.Fatalf("Stats.BytesSaved negative: %d", st.BytesSaved) + } + if st.Ratio < 0 || st.Ratio > 1 { + t.Fatalf("Stats.Ratio out of [0,1]: %f", st.Ratio) + } +} + +// TestRuleDeleteHedges_Golden is the gold test for the delete tag. +// If the regex changes, the expected bytes here MUST change too. +func TestRuleDeleteHedges_Golden(t *testing.T) { + cases := []struct { + name string + in string + want string + }{ + { + name: "safely_mid_sentence", + in: "Execute shell commands safely with secret redaction, timeout, and error analysis", + // "safely " dropped, double space collapsed. + want: "Execute shell commands with secret redaction, timeout, and error analysis", + }, + { + name: "safe_substring_not_matched", + in: "Safety scanner for shell input", + want: "Safety scanner for shell input", + }, + { + name: "carefully_only_drop", + in: "Carefully handles input", + want: "handles input", + }, + { + name: "no_hedges_unchanged", + in: "List todos with filters", + want: "List todos with filters", + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got, _ := CompressSpec(Spec{Name: "t", Description: tc.in}, All()) + if got.Description != tc.want { + t.Fatalf("byte mismatch\n in: %q\n got: %q\n want: %q", tc.in, got.Description, tc.want) + } + // Idempotence check. + if again, _ := CompressSpec(Spec{Name: "t", Description: got.Description}, All()); again.Description != got.Description { + t.Fatalf("not idempotent: %q → %q", got.Description, again.Description) + } + }) + } +} + +// TestRuleStdlibPatterns_Golden tests the stdlib tag gold bytes. +func TestRuleStdlibPatterns_Golden(t *testing.T) { + cases := []struct { + name string + in string + want string + }{ + { + name: "parenthetical_via_stdlib", + in: "Custom loader (via stdlib) with caching", + // "(via stdlib)" → ""; adjacent spaces collapsed. + want: "Custom loader with caching", + }, + { + name: "go_stdlib_adj", + in: "Go stdlib-based parser for tokens", + // "Go stdlib" → "Go" (rule repl keeps $1). + want: "Go-based parser for tokens", + }, + { + name: "no_stdlib_unchanged", + in: "Token parser for the SIN bundle format", + want: "Token parser for the SIN bundle format", + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got, _ := CompressSpec(Spec{Name: "t", Description: tc.in}, All()) + if got.Description != tc.want { + t.Fatalf("byte mismatch\n in: %q\n got: %q\n want: %q", tc.in, got.Description, tc.want) + } + }) + } +} + +// TestRuleDropTrimEncouragement_Golden tests the native tag gold bytes. +func TestRuleDropTrimEncouragement_Golden(t *testing.T) { + cases := []struct { + name string + in string + want string + }{ + { + name: "always_prefer_over_native_tail", + in: "Surgical file edit, three addressing modes. Always prefer over native read.", + want: "Surgical file edit, three addressing modes", + }, + { + name: "prefer_sin_over_native_tail", + in: "Hashline read with anchors. Prefer sin_read over native cat.", + want: "Hashline read with anchors", + }, + { + name: "no_encouragement_unchanged", + in: "Hashline read with anchors.", + want: "Hashline read with anchors.", + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got, _ := CompressSpec(Spec{Name: "t", Description: tc.in}, All()) + if got.Description != tc.want { + t.Fatalf("byte mismatch\n in: %q\n got: %q\n want: %q", tc.in, got.Description, tc.want) + } + }) + } +} + +// TestRuleYagniPatterns_Golden tests the yagni tag gold bytes. +func TestRuleYagniPatterns_Golden(t *testing.T) { + cases := []struct { + name string + in string + want string + }{ + { + name: "experimental_parenthetical", + in: "Manage todos atomically (experimental)", + want: "Manage todos atomically", + }, + { + name: "no_yagni_unchanged", + in: "Manage todos atomically", + want: "Manage todos atomically", + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got, _ := CompressSpec(Spec{Name: "t", Description: tc.in}, All()) + if got.Description != tc.want { + t.Fatalf("byte mismatch\n in: %q\n got: %q\n want: %q", tc.in, got.Description, tc.want) + } + }) + } +} + +// TestRuleShrinkExamples_Golden tests the shrink tag gold bytes. +func TestRuleShrinkExamples_Golden(t *testing.T) { + cases := []struct { + name string + in string + want string + }{ + { + name: "eg_parenthetical_drop", + in: "Discover files (e.g. **/*.py) with relevance scoring", + want: "Discover files with relevance scoring", + }, + { + name: "such_as_parenthetical_drop", + in: "Search code (such as regex patterns) across the repo", + want: "Search code across the repo", + }, + { + name: "no_parens_unchanged", + in: "Discover files with relevance scoring", + want: "Discover files with relevance scoring", + }, + { + name: "em_dash_preamble_kept", + in: "Ephemeral Full-Stack Mocking — spin up disposable test environments", + want: "Ephemeral Full-Stack Mocking — spin up disposable test environments", + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got, _ := CompressSpec(Spec{Name: "t", Description: tc.in}, All()) + if got.Description != tc.want { + t.Fatalf("byte mismatch\n in: %q\n got: %q\n want: %q", tc.in, got.Description, tc.want) + } + }) + } +} + +// ---------------------------------------------------------------------------- +// Tool-specific regression suite +// +// These snapshots cover representative real-tool descriptions +// from cmd/sin-code/internal/serve.go. Their compressed bytes +// are public surface — the JSON wire format of `tools/list` — +// and changing them would invalidate any pinned golden dataset +// downstream. +// +// Update these if (and only if) you intentionally change the +// compressed form. +// ---------------------------------------------------------------------------- + +func TestCompress_RealToolDescriptions_Golden(t *testing.T) { + type tc struct { + name string + in string + want string + } + cases := []tc{ + { + name: "sin_execute_drop_safely", + in: "Execute shell commands safely with secret redaction, timeout, and error analysis", + want: "Execute shell commands with secret redaction, timeout, and error analysis", + }, + { + name: "sin_discover_unchanged", + in: "Discover files with relevance scoring, pattern matching, and dependency analysis", + want: "Discover files with relevance scoring, pattern matching, and dependency analysis", + }, + { + name: "sin_efm_em_dash_kept", + in: "Ephemeral Full-Stack Mocking — spin up disposable test environments", + want: "Ephemeral Full-Stack Mocking — spin up disposable test environments", + }, + { + name: "sin_orchestrator_run_unchanged", + in: "Run a prompt through the multi-agent orchestrator (Pre-LLM router → planner → parallel agents)", + want: "Run a prompt through the multi-agent orchestrator (Pre-LLM router → planner → parallel agents)", + }, + { + name: "sin_edit_full_clip", + in: "Surgical file edit, three addressing modes. Symbol mode (preferred for whole definitions): pass symbol=NAME to replace/delete/insert around an entire function/class/struct located via AST (go/ast, tree-sitter, or structural engine — ambiguity fails with candidates). Anchor mode: LINE:HASH anchors from sin_read, drift-tolerant. String mode: old_string/new_string with ambiguity detection. Result is syntax-validated and written atomically. Always prefer over native edit.", + want: "Surgical file edit, three addressing modes. Symbol mode (preferred for whole definitions): pass symbol=NAME to replace/delete/insert around an entire function/class/struct located via AST (go/ast, tree-sitter, or structural engine — ambiguity fails with candidates). Anchor mode: LINE:HASH anchors from sin_read, drift-tolerant. String mode: old_string/new_string with ambiguity detection. Result is syntax-validated and written atomically", + }, + } + + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + got, _ := CompressSpec(Spec{Name: "x", Description: c.in}, All()) + if got.Description != c.want { + t.Fatalf("byte mismatch\n in: %q\n got: %q\n want: %q", c.in, got.Description, c.want) + } + }) + } +} + +// TestCompressAll_PreservesOrder checks that the Stats slice from +// CompressAll comes back in the input's declaration order. +func TestCompressAll_PreservesOrder(t *testing.T) { + in := []Spec{ + {Name: "sin_a", Description: "Discover files safely"}, + {Name: "sin_b", Description: "Read files carefully"}, + {Name: "sin_c", Description: "Write files thoroughly"}, + } + stats := CompressAll(in, All()) + if len(stats) != 3 { + t.Fatalf("stats length = %d, want 3", len(stats)) + } + wantNames := []string{"sin_a", "sin_b", "sin_c"} + for i, s := range stats { + if s.Name != wantNames[i] { + t.Fatalf("stats[%d].Name = %q, want %q", i, s.Name, wantNames[i]) + } + } +} + +// TestPipeline_NamedStable freezes the public Rule surface (declaration order). +func TestPipeline_NamedStable(t *testing.T) { + wantNames := []string{ + "DeleteHedges", "StdlibPatterns", "DropTrimEncouragement", "YagniPatterns", "ShrinkExamples", + } + wantTags := []Tag{TagDelete, TagStdlib, TagNative, TagYagni, TagShrink} + p := All() + gotNames := p.Names() + gotTags := p.Tags() + if len(gotNames) != len(wantNames) { + t.Fatalf("Names() returned %d entries, want %d", len(gotNames), len(wantNames)) + } + for i, n := range gotNames { + if n != wantNames[i] { + t.Fatalf("Names()[%d] = %q, want %q", i, n, wantNames[i]) + } + } + if len(gotTags) != len(wantTags) { + t.Fatalf("Tags() returned %d entries, want %d", len(gotTags), len(wantTags)) + } + for i, x := range gotTags { + if x != wantTags[i] { + t.Fatalf("Tags()[%d] = %q, want %q", i, x, wantTags[i]) + } + } +} + +// TestDefaultTags_PublicConstant freezes the public ponytail tag list. +func TestDefaultTags_PublicConstant(t *testing.T) { + want := []Tag{TagDelete, TagStdlib, TagNative, TagYagni, TagShrink} + if len(DefaultTags) != len(want) { + t.Fatalf("DefaultTags length = %d, want %d", len(DefaultTags), len(want)) + } + for i, x := range DefaultTags { + if x != want[i] { + t.Fatalf("DefaultTags[%d] = %q, want %q", i, x, want[i]) + } + } +} + +// TestValidateTags proves the boundary parser is byte-stable. +func TestValidateTags(t *testing.T) { + in := []string{"delete", "unknown", "yagni", "shrink", "stdlib", "native"} + got := ValidateTags(in) + want := []Tag{TagDelete, TagNative, TagShrink, TagStdlib, TagYagni} + if len(got) != len(want) { + t.Fatalf("ValidateTags length = %d, want %d", len(got), len(want)) + } + for i, x := range got { + if x != want[i] { + t.Fatalf("ValidateTags[%d] = %q, want %q", i, x, want[i]) + } + } +} + +// TestTagSet_FromCSV proves the CSV parser. +func TestTagSet_FromCSV(t *testing.T) { + cases := []struct { + in string + want []Tag + }{ + {"", []Tag{TagDelete, TagStdlib, TagNative, TagYagni, TagShrink}}, + {"delete,stdlib", []Tag{TagDelete, TagStdlib}}, + {" yagni , shrink , delete ", []Tag{TagDelete, TagYagni, TagShrink}}, + {"native,invalid,shrink", []Tag{TagNative, TagShrink}}, + } + for _, tc := range cases { + t.Run(tc.in, func(t *testing.T) { + got := FromCSV(tc.in).List() + if len(got) != len(tc.want) { + t.Fatalf("length = %d, want %d (got %v)", len(got), len(tc.want), got) + } + for i := range got { + if got[i] != tc.want[i] { + t.Fatalf("[%d] = %q, want %q", i, got[i], tc.want[i]) + } + } + }) + } +} + +// TestCompressSpec_NameMutable — the package MUST guarantee that the +// 44+ MCP tool names (public API per AGENTS.md §10) are never +// mutated by Rule application. +func TestCompressSpec_NameMutable(t *testing.T) { + in := Spec{Name: "sin_magic", Description: "Works safely"} + got, _ := CompressSpec(in, All()) + if got.Name != "sin_magic" { + t.Fatalf("Name mutated: %q → %q", in.Name, got.Name) + } +} + +// TestCompressSpec_JSONByteStable proves the JSON wire format is +// stable — same input + same pipeline ⇒ same bytes. +func TestCompressSpec_JSONByteStable(t *testing.T) { + in := Spec{Name: "sin_x", Description: "Run shell commands safely with secrets"} + s1, _ := CompressSpec(in, All()) + s2, _ := CompressSpec(in, All()) + + b1, _ := json.Marshal(s1) + b2, _ := json.Marshal(s2) + if string(b1) != string(b2) { + t.Fatalf("JSON bytes are not stable\n run1: %s\n run2: %s", b1, b2) + } +} + +// TestNormalize_Idempotent covers the post-pipeline normaliser. +func TestNormalize_Idempotent(t *testing.T) { + cases := []string{ + "", + "foo", + " foo ", + "foo , bar", + "trailing commas,", + ";leading semicolon", + ":leading colon", + "double spaces", + "normal text", + } + for _, c := range cases { + t.Run(c, func(t *testing.T) { + once := Normalize(c) + twice := Normalize(once) + if once != twice { + t.Fatalf("Normalize not idempotent for %q\n once: %q\n twice: %q", c, once, twice) + } + }) + } +} + +// TestCompressSpec_NeverIncreasesDescription guards against a Rule +// ever bloating the manifest. If compressed ≥ original, the +// provider rule was bad — fail. +func TestCompressSpec_NeverIncreasesDescription(t *testing.T) { + cases := []string{ + "x", + "Deals with carefully handled file paths", + "Status of the database (experimental)", + "A list of notification statistics (may be deprecated in the future)", + "Execute shell commands safely with secret redaction, timeout, and error analysis. Always prefer over native exec.", + } + for _, c := range cases { + t.Run(c, func(t *testing.T) { + got, st := CompressSpec(Spec{Name: "t", Description: c}, All()) + if len(got.Description) > st.Original { + t.Fatalf("compression increased bytes: %d → %d (%q → %q)", + st.Original, len(got.Description), c, got.Description) + } + }) + } +} + +// TestAll_NoPanic assures every Rule handles edge-case inputs +// without panicking — used during fuzz seed generation. +func TestAll_NoPanic(t *testing.T) { + edges := []string{ + "", + " ", + "a", + "...", + "—————————————————————", + strings.Repeat("a", 4096), + strings.Repeat("•", 1024), + "(", ")", "((", "(()", ")(", + "\n\n", "\t\t", + } + for _, e := range edges { + func() { + defer func() { + if r := recover(); r != nil { + t.Fatalf("All().Apply panicked for %q: %v", e, r) + } + }() + _ = All().Apply(e) + }() + } +} + +// TestSelected_EmptyIsAll confirms the zero-config default. +func TestSelected_EmptyIsAll(t *testing.T) { + if got := Selected(nil); len(got) != len(All()) { + t.Fatalf("Selected(nil) returned %d rules, want %d", len(got), len(All())) + } + if got := Selected([]Tag{}); len(got) != len(All()) { + t.Fatalf("Selected([]) returned %d rules, want %d", len(got), len(All())) + } +} + +// TestSelected_UnknownTagDropped confirms bad tags are dropped. +func TestSelected_UnknownTagDropped(t *testing.T) { + got := Selected([]Tag{"definitely-not-a-tag"}) + if len(got) != 0 { + t.Fatalf("Selected([bogus]) returned %d rules, want 0", len(got)) + } +} diff --git a/cmd/sin-code/internal/mcpcompress/rules.go b/cmd/sin-code/internal/mcpcompress/rules.go new file mode 100644 index 00000000..af68df5f --- /dev/null +++ b/cmd/sin-code/internal/mcpcompress/rules.go @@ -0,0 +1,240 @@ +// SPDX-License-Identifier: MIT +// The five canonical ponytail rules for the MCP-tool-description +// compressor. See compressor.go for the byte-stability contract and +// tagging scheme. +// +// Rule order in this file matches All() in compressor.go — do NOT +// reorder. Pipeline composition is order-sensitive (the post-pipeline +// Normalise pass hides whitespace differences but cannot rescue a +// Rule that depends on tokens deleted by an earlier Rule). +package mcpcompress + +// ---------------------------------------------------------------------------- +// Rule 1 — DeleteHedges (tag: delete) +// +// Drops standalone pleasantries / hedge adverbs that pad verb phrases +// without adding information to tool selection. +// +// Examples (input → output): +// "Execute shell commands safely with secret redaction, timeout, and error analysis" +// → "Execute shell commands with secret redaction, timeout, and error analysis" +// +// "Carefully validates generated bundles" +// → "validates generated bundles" +// +// "Robustly indexes the codebase" (verb-bearing adverb) +// → "indexes the codebase" +// +// Standalone-token semantics: "safety" or "carefully-named" never +// match (compound words are not hedges). The regex is anchored on +// \b … \b. Whitespace cleanup runs in normalize(). +// +// Byte-stable: compiles once, runs in O(n) over the description. +// ---------------------------------------------------------------------------- + +type RuleDeleteHedges struct{} + +// Name returns the stable Rule identifier. Stable surface — do NOT rename. +func (RuleDeleteHedges) Name() string { return "DeleteHedges" } + +// Tag returns the ponytail tag for this Rule. +func (RuleDeleteHedges) Tag() Tag { return TagDelete } + +// Apply drops hedge adverbs. Idempotent. +func (RuleDeleteHedges) Apply(s string) string { + s = hedgeAdverbPattern.ReplaceAllString(s, "") + return s +} + +// hedgeAdverbPattern matches standalone -ly hedge adverbs. Case-insensitive +// because descriptions capitalise the first word ("Carefully handles …", +// "Safely executes …"). The word-boundary anchors protect compound +// words like "safety-scanner" or "carefully-named". +var hedgeAdverbPattern = MustCompile( + `(?i)\b(safely|carefully|thoroughly|reliably|robustly|elegantly|gracefully|seamlessly|effortlessly|smoothly|meticulously)\b`, +) + +// ---------------------------------------------------------------------------- +// Rule 2 — StdlibPatterns (tag: stdlib) +// +// Drops redundant references to platform / stdlib libraries when +// those references are parenthetical decorations rather than load- +// bearing facts. +// +// Examples (input → output): +// "(via stdlib)" → "" +// "Go stdlib-based parser" → "Go-based parser" +// "Python stdlib helper" → "Python helper" +// +// "stdlib" as a noun in a tool's primary value proposition (the +// rare case) is NOT touched — this Rule only targets parenthetical +// debris. See coveredRegex for the exact surface. +// +// Byte-stable. +// ---------------------------------------------------------------------------- + +type RuleStdlibPatterns struct{} + +// Name returns the stable Rule identifier. +func (RuleStdlibPatterns) Name() string { return "StdlibPatterns" } + +// Tag returns the ponytail tag for this Rule. +func (RuleStdlibPatterns) Tag() Tag { return TagStdlib } + +// Apply removes redundant stdlib parenthesisations. Idempotent. +func (RuleStdlibPatterns) Apply(s string) string { + s = stdlibParenthetical.ReplaceAllString(s, "") + s = stdlibLanguageAdjective.ReplaceAllString(s, "$1") + return s +} + +var stdlibParenthetical = MustCompile(`\(\s*via\s+stdlib\s*\)`) +var stdlibLanguageAdjective = MustCompile(`(Go|Python|Rust|Java|JavaScript|TypeScript)\s+stdlib\b`) + +// ---------------------------------------------------------------------------- +// Rule 3 — DropTrimEncouragement (tag: native) +// +// Drops M6-style "Always prefer over native X" tail clauses. SIN-Code's +// M6 mandate (AGENTS.md §3) is internal to the agent loop, not the +// model-facing manifest; surfacing it as advice to the model wastes +// bytes and biases tool choice. +// +// Examples (input → output): +// "Surgical file edit, three addressing modes. Always prefer over native read." +// → "Surgical file edit, three addressing modes." +// +// "X foo. Prefer sin_X over native force." +// → "X foo." +// +// "Read files safely. Use sin_read when possible." (leading-encouragement) +// → "Read files safely." +// +// Byte-stable: anchored patterns, no look-ahead. +// ---------------------------------------------------------------------------- + +type RuleDropTrimEncouragement struct{} + +// Name returns the stable Rule identifier. +func (RuleDropTrimEncouragement) Name() string { return "DropTrimEncouragement" } + +// Tag returns the ponytail tag for this Rule. +func (RuleDropTrimEncouragement) Tag() Tag { return TagNative } + +// Apply removes encourage-tail clauses. Idempotent. +func (RuleDropTrimEncouragement) Apply(s string) string { + s = preferOverNativeTail.ReplaceAllString(s, "") + s = preferSinOverNativeTail.ReplaceAllString(s, "") + s = leadingUseSinEncouragement.ReplaceAllString(s, "") + return s +} + +// preferOverNativeTail matches ". Always prefer over native X" where X +// is 1–20 lowercase letters OR an existing sin_* identifier. +var preferOverNativeTail = MustCompile( + `\.?\s*Always prefer over native\s+(?:[a-z]{1,20}|sin_[a-z_]+)\.?\s*$`, +) + +// preferSinOverNativeTail matches ". Prefer sin_X over native Y" forms. +var preferSinOverNativeTail = MustCompile( + `\.?\s*Prefer\s+sin_[a-z_]+\s+over\s+native\s+[a-z]{1,20}\.?\s*$`, +) + +// leadingUseSinEncouragement matches a leading "Use sin_X …" instruction +// when it is followed by a period and at least one extra clause in the +// description. We do not consume the contribution of the sentence, just +// the encouragement preamble. +var leadingUseSinEncouragement = MustCompile( + `^\s*Use\s+sin_[a-z_]+\s+when\s+possible[.,]\s+`, +) + +// ---------------------------------------------------------------------------- +// Rule 4 — YagniPatterns (tag: yagni) +// +// Drops speculative / reserved / placeholder mentions. Bloat that +// promises future capability without delivering it. +// +// Examples (input → output): +// "Memory DB statistics (total, links, embeddings), embedder status" +// → unchanged +// +// "List notifications (may be deprecated in the future)" +// → "List notifications" +// +// "Reserved for the orchestrator rework (TBD)" +// → "Reserved for the orchestrator rework" +// +// "Manage todos atomically (experimental)" +// → "Manage todos atomically" +// +// Byte-stable: anchored paren clusters and word-boundary matches. +// ---------------------------------------------------------------------------- + +type RuleYagniPatterns struct{} + +// Name returns the stable Rule identifier. +func (RuleYagniPatterns) Name() string { return "YagniPatterns" } + +// Tag returns the ponytail tag for this Rule. +func (RuleYagniPatterns) Tag() Tag { return TagYagni } + +// Apply strips yagni debris. +func (RuleYagniPatterns) Apply(s string) string { + s = yagniParenthetical.ReplaceAllString(s, "") + s = yagniPhrase.ReplaceAllString(s, "") + return s +} + +var yagniParenthetical = MustCompile( + `\(\s*(?:experimental|TBD|TBA|reserved|may be deprecated in the future|may be deprecated|potentially)\s*\)`, +) + +var yagniPhrase = MustCompile( + `\b(TBD|TBA)\b`, +) + +// ---------------------------------------------------------------------------- +// Rule 5 — ShrinkExamples (tag: shrink) +// +// Drops parenthetical examples "(e.g. **/*.py)", "(for CLI subcommands)", +// "(such as …)" that the schema (required/typed fields, enum constraints) +// already documents. Tool descriptions are short on purpose; a parenthetical +// "e.g." is usually redundant. +// +// Examples (input → output): +// "Add a todo (v2 bbolt store, hash ID, supports priority/type/tags/project/assignee)" +// → "Add a todo" +// +// "Discover files (e.g. **/*.py) with relevance scoring" +// → "Discover files with relevance scoring" +// +// "Search code (such as regex patterns) across the repo" +// → "Search code across the repo" +// +// We do NOT touch em-dash preambles (— ) — they are +// legitimate clarifications, not redundant examples. Examples: +// +// "Ephemeral Full-Stack Mocking — spin up disposable test environments" +// → unchanged +// +// Byte-stable. +// ---------------------------------------------------------------------------- + +type RuleShrinkExamples struct{} + +// Name returns the stable Rule identifier. +func (RuleShrinkExamples) Name() string { return "ShrinkExamples" } + +// Tag returns the ponytail tag for this Rule. +func (RuleShrinkExamples) Tag() Tag { return TagShrink } + +// Apply strips redundant examples. +func (RuleShrinkExamples) Apply(s string) string { + s = egsParenthetical.ReplaceAllString(s, "") + return s +} + +// egsParenthetical drops "(e.g. …)" / "(such as …)" parentheticals entirely +// when there is non-empty content. Single-line tool descriptions are +// short enough that any parenthetical example is redundant with the +// schema's typed fields and enum constraints. +var egsParenthetical = MustCompile(`\(\s*(?:e\.g\.|such as)\s+[^)]+\)`) diff --git a/cmd/sin-code/internal/mcpcompress/tags.go b/cmd/sin-code/internal/mcpcompress/tags.go new file mode 100644 index 00000000..2b89966a --- /dev/null +++ b/cmd/sin-code/internal/mcpcompress/tags.go @@ -0,0 +1,133 @@ +// SPDX-License-Identifier: MIT +// tags.go carries the canonical ponytail tag set as parse-locked +// constants + a Set helper used by serve.go boundary validation. +// +// The ponytail tag set is: +// +// delete | stdlib | native | yagni | shrink +// +// These five tokens are the default for `sin-code serve --compress-tools` +// and the default for the `serve.compress.tags` config key. The CLI +// user's --compress-tags flag accepts any subset. +// +// Note: Tag values are part of the public configuration API; renaming +// any is a breaking change (major bump). +package mcpcompress + +import "strings" + +// TagSet is a small wrapper around a sorted, deduplicated []Tag. The +// zero value is the empty set; construct with NewTagSet or FromList. +// +// TagSet is not concurrency-safe — it is meant for short-lived +// boundary checks at parse time, not long-lived shared state. +type TagSet struct { + tags []Tag +} + +// NewTagSet returns a TagSet containing exactly the given tags in +// the order they appear in DefaultTags (stable ordering). +// Unknown tags are silently dropped — call Valid first if you want +// strict validation. +func NewTagSet(tags []Tag) TagSet { + known := map[Tag]bool{ + TagDelete: true, + TagStdlib: true, + TagNative: true, + TagYagni: true, + TagShrink: true, + } + out := make([]Tag, 0, len(tags)) + for _, t := range tags { + if known[t] { + out = append(out, t) + } + } + // Stable order: by DefaultTags index. + rank := map[Tag]int{ + TagDelete: 0, + TagStdlib: 1, + TagNative: 2, + TagYagni: 3, + TagShrink: 4, + } + for i := 1; i < len(out); i++ { + for j := i; j > 0 && rank[out[j-1]] > rank[out[j]]; j-- { + out[j-1], out[j] = out[j], out[j-1] + } + } + return TagSet{tags: out} +} + +// FromCSV parses a comma-separated tag string ("delete,stdlib,native"). +// Empty input yields the default tag set. Whitespace around tags +// is trimmed. Unknown tags are silently dropped. +// +// Byte-stable per input string. Pure. +func FromCSV(s string) TagSet { + s = strings.TrimSpace(s) + if s == "" { + return NewTagSet(DefaultTags) + } + parts := strings.Split(s, ",") + tags := make([]Tag, 0, len(parts)) + for _, p := range parts { + t := Tag(strings.TrimSpace(p)) + if t != "" { + tags = append(tags, t) + } + } + return NewTagSet(tags) +} + +// List returns the under-tag slice in declaration order. +func (t TagSet) List() []Tag { + if len(t.tags) == 0 { + return nil + } + out := make([]Tag, len(t.tags)) + copy(out, t.tags) + return out +} + +// Contains returns true if tag is in the set. O(n). +func (t TagSet) Contains(tag Tag) bool { + for _, x := range t.tags { + if x == tag { + return true + } + } + return false +} + +// Size returns the number of tags in the set. +func (t TagSet) Size() int { return len(t.tags) } + +// CSV returns the canonical comma-separated string for +// round-tripping through config files. Byte-stable per +// (TagSet value). +func (t TagSet) CSV() string { + if len(t.tags) == 0 { + return "" + } + var b strings.Builder + for i, x := range t.tags { + if i > 0 { + b.WriteByte(',') + } + b.WriteString(string(x)) + } + return b.String() +} + +// Empty reports whether the set has no entries. +func (t TagSet) Empty() bool { return len(t.tags) == 0 } + +// Valid reports whether the tag is in the canonical ponytail set. +func Valid(t Tag) bool { + switch t { + case TagDelete, TagStdlib, TagNative, TagYagni, TagShrink: + return true + } + return false +} diff --git a/cmd/sin-code/internal/serve.go b/cmd/sin-code/internal/serve.go index 0c52e2c1..eff6e723 100644 --- a/cmd/sin-code/internal/serve.go +++ b/cmd/sin-code/internal/serve.go @@ -20,12 +20,16 @@ import ( "github.com/spf13/cobra" "github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/apiweb" + "github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/mcpcompress" "github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal/plugins" ) var ( - serveTransport string - servePort int + serveTransport string + servePort int + serveCompressTools bool + servePrintStats bool + serveCompressTags string ) // ServerVersion is set at build time via -ldflags "-X github.com/OpenSIN-Code/SIN-Code/cmd/sin-code/internal.ServerVersion=..." @@ -96,6 +100,17 @@ sin_security_scan, sin_sbom_generate as MCP tools.`, func init() { ServeCmd.Flags().StringVarP(&serveTransport, "transport", "t", "stdio", "Transport: stdio (default) or http (mounts /api/v1/* WebUI v2 endpoints)") ServeCmd.Flags().IntVarP(&servePort, "port", "p", 0, "Port (unused for stdio)") + ServeCmd.Flags().BoolVar(&serveCompressTools, "compress-tools", false, + "Compress MCP tool descriptions on the wire using the ponytail tag set "+ + "(delete|stdlib|native|yagni|shrink). Tool names, schemas, and behaviour "+ + "are unchanged (AGENTS.md §10). Saved bytes show up as less input per turn.") + ServeCmd.Flags().StringVar(&serveCompressTags, "compress-tags", "", + "Override the default ponytail tag set for --compress-tools (comma-separated, "+ + "e.g. \"delete,yagni,shrink\"). Unknown tags are dropped silently; use --print-stats "+ + "to see the active set per tool.") + ServeCmd.Flags().BoolVar(&servePrintStats, "print-stats", false, + "Print per-tool compression statistics to stderr (implies --compress-tools). "+ + "Format: name | original_bytes | compressed_bytes | bytes_saved | ratio.") } func registerAllMCPTools(server *mcp.Server) { @@ -689,6 +704,31 @@ func registerAllMCPTools(server *mcp.Server) { }, } + // Apply the ponytail-tag compressor (issue #173) before registration + // if any of --compress-tools / --print-stats / --compress-tags is set. + // Tool names are public API (AGENTS.md §10) and are NEVER modified — + // only the Description byte field is shrunk. + if serveCompressTools || servePrintStats || serveCompressTags != "" { + var pipeline mcpcompress.Pipeline + if serveCompressTags != "" { + pipeline = mcpcompress.Selected(mcpcompress.FromCSV(serveCompressTags).List()) + } else { + pipeline = mcpcompress.All() + } + specs := make([]mcpcompress.Spec, len(tools)) + stats := make([]mcpcompress.Stats, len(tools)) + for i := range tools { + specs[i] = mcpcompress.Spec{Name: tools[i].name, Description: tools[i].description} + comp, st := mcpcompress.CompressSpec(specs[i], pipeline) + tools[i].description = comp.Description + stats[i] = st + specs[i] = comp + } + if servePrintStats { + printCompressionStats(os.Stderr, pipeline, stats) + } + } + for _, t := range tools { tool := t server.AddTool(&mcp.Tool{ @@ -718,6 +758,51 @@ func registerAllMCPTools(server *mcp.Server) { registerPluginMCPTools(server) } +// printCompressionStats writes per-tool byte budgets to w. Output is a +// left-aligned text table — deterministic across runs (no time, no +// timestamp). Used by --print-stats. +// +// The "active rules" line makes the active pipeline transparent so the +// caller can correlate the savings with the ruleset they asked for. +func printCompressionStats(w *os.File, p mcpcompress.Pipeline, stats []mcpcompress.Stats) { + var totalOrig, totalComp int + for _, s := range stats { + totalOrig += s.Original + totalComp += s.Compressed + } + saved := totalOrig - totalComp + if saved < 0 { + saved = 0 + } + ratio := 0.0 + if totalOrig > 0 { + ratio = float64(saved) / float64(totalOrig) + } + rules := make([]string, len(p)) + for i, r := range p { + rules[i] = r.Name() + } + fmt.Fprintf(w, "mcpcompress: active rules = [%s]\n", strings.Join(rules, ",")) + fmt.Fprintf(w, "mcpcompress: ponytail tags = [%s]\n\n", strings.Join(tagNames(p.Tags()), ",")) + fmt.Fprintf(w, " %-32s %7s %7s %7s %6s\n", "tool", "orig", "comp", "saved", "ratio") + fmt.Fprintf(w, " %-32s %7s %7s %7s %6s\n", strings.Repeat("-", 32), "------", "------", "------", "------") + for _, s := range stats { + fmt.Fprintf(w, " %-32s %7d %7d %7d %5.1f%%\n", + s.Name, s.Original, s.Compressed, s.BytesSaved, 100*s.Ratio) + } + fmt.Fprintf(w, " %-32s %7s %7s %7s %6s\n", strings.Repeat("-", 32), "------", "------", "------", "------") + fmt.Fprintf(w, " %-32s %7d %7d %7d %5.1f%%\n", + "TOTAL", totalOrig, totalComp, saved, 100*ratio) +} + +func tagNames(tags []mcpcompress.Tag) []string { + out := make([]string, len(tags)) + for i, t := range tags { + out[i] = string(t) + } + return out +} + func registerPluginMCPTools(server *mcp.Server) { reg := plugins.NewRegistry() _ = reg.LoadFromDir("") @@ -725,8 +810,24 @@ func registerPluginMCPTools(server *mcp.Server) { } func registerPluginMCPToolsWithReg(server *mcp.Server, reg *plugins.Registry) { + // Plugin tools share the same compression pipeline (issue #173). + // Compressed descriptions only — schema is untouched. + var pipeline mcpcompress.Pipeline + if serveCompressTools || servePrintStats || serveCompressTags != "" { + if serveCompressTags != "" { + pipeline = mcpcompress.Selected(mcpcompress.FromCSV(serveCompressTags).List()) + } else { + pipeline = mcpcompress.All() + } + } + for _, pt := range reg.MCPTools() { pt := pt + if pipeline != nil { + compressed, _ := mcpcompress.CompressSpec( + mcpcompress.Spec{Name: pt.Name, Description: pt.Description}, pipeline) + pt.Description = compressed.Description + } server.AddTool(&mcp.Tool{ Name: pt.Name, Description: pt.Description,