[v3.20 rag] RAG over instinct store: semantic search for top-N selection

[Delqhi]

RAG over Instinct Store

The Instinct subsystem currently injects all active instincts into the system prompt via RenderSystemBlock. At 50+ active instincts, this becomes expensive. RAG (retrieval-augmented generation) selects the top-N most relevant per turn.

Why

A model that sees 50 instincts in the system prompt dilutes attention away from the actually-relevant 2-3. RAG solves this with a cheap retrieval step. The "active" set becomes "active AND relevant" — the model sees a smaller, more focused block.

What ships (RAG-v0 — the only scope for now)

Vector storage

Reuse internal/memory/Store (bbolt-based, already in the binary, CGO-free) to store a side-by-side vector index. Each Instinct gets a 384-dim embedding (sentence-transformers via ONNX, no Python dependency).

Embedding provider

Pluggable via an interface:

type Embedder interface {
    Embed(ctx context.Context, text string) ([]float32, error)
}

The default implementation calls a local ONNX runtime (github.com/yalue/onnxruntime_go or similar). The fallback calls an HTTP API (your existing model client) for embeddings.

Retrieval

On every RenderSystemBlock(active, max) call:

1. Embed the current task (or the latest user prompt) → 384-dim vector
2. Cosine-similarity against all active instinct embeddings
3. Return the top-5 most similar
4. Render the system block with only those 5

The full active set is still on disk and inspectable via sin instinct status and sin instinct show <id>.

Acceptance criteria

• sin instinct show <id> shows the embedding-dimension
• sin instinct search "<query>" returns the top-5 matches
• RenderSystemBlock(active, 5) returns at most 5 instincts, ranked by similarity
• Embedding generation is async — the agent loop never blocks on it
• Test coverage ≥ 80%

What does NOT ship (deferred)

• GOAP Planner (v1, ~4 weeks) — Goals to Action Plans, decomposed, with preconditions. PRP-Engine covers the v0 case.
• Federation (v2, ~3 months) — multiple SIN-Code instances sharing instinct sets via CRDT. Not for this issue, not for this year.

Mandates

• M2 (single binary): the ONNX runtime must be CGO_ENABLED=0 compatible. The github.com/yalue/onnxruntime_go library is; verify.
• M5 (module path): new code in cmd/sin-code/internal/rag/
• M7 (race-free): the embedding goroutine is a worker pool with bounded concurrency

Related

• Issue ## Ultimative SOTA Implementierung für SIN-Code: RAG, GOAP Planner & Federation #124 — the original "RAG, GOAP, Federation" SOTA vision (closed as a spec; this is the RAG-v0 implementation)
• internal/instinct/Manager.Active() — the current "all active" query
• internal/instinct/RenderSystemBlock() — the function being optimized
• internal/memory/Store — the bbolt storage being reused

Estimated scope

~1 week, 1 PR. RAG-v0 only. GOAP and Federation are separate future issues.

[Delqhi]

Closed: shipped in PR #210 (feat(rag): RAG over instinct store, issue #160 RAG-v0). The internal/rag/ package + sin instinct search CLI are live on main. ONNX stub is in place; the real ONNX wiring is a follow-up (v1.1) per the issue body's deferred list.