Repository navigation
Weak-match labels, the miss ledger, co-activation edges and shadow ranking - #17
Merged
Merged
Conversation
Two memories opened in the same stretch of work are probably about the same work, and nothing in the store noticed. `record_opened` now pairs the opened memory with every other live memory opened in the last 30 minutes and strengthens a derived `co_recalled` edge between them (Hebbian, rate 0.15 toward 1.0, one fire per pair per UTC day, 30-day half-life applied lazily on read, at most 32 edges per node, weakest evicted first). Phase 1 deliberately does not touch ranking: an unmeasured scoring change is how the per-file-cap experiment lost 0.070 nDCG. The edges are written so there is something real to measure. - co-opened, not co-shown: `shown` fires for every hit of every recall - memory<->memory only; pairs with an authored edge are skipped - canonical orientation (src = lower rowid), so one pair is one row - `link` (CLI and MCP) refuses rel=co_recalled: hand-authoring one forges usage and permanently masks the real signal for that pair - excluded from sync and export: derived, per-machine, regenerable - `edges_of` skips a rel this binary doesn't know instead of failing the whole query, so a newer store can't break an older `get` - `recall --linked` shows authored edges, not derived ones; the full record lists "often recalled with"; dashboard counts them on their own row
A ranked list always has a #1, so the fused score cannot tell "the answer" from "the least-bad thing in the store". Each recall hit is now labelled by evidence: `strong` when it contains the query's words AND both the lexical and semantic legs rank it in their own top 10 (or, with no semantic leg, covers every content word; or names a symbol exactly), otherwise `weak` — one word on the compact line (CLI and MCP), `"strength"` in --json. Calibrated against the labelled retrieval corpus with the real model (eval::strength, k=8): strong 23/42 relevant, weak 3/142, 0/6 unanswerable probes passed as answered. Measured and rejected: a middle `likely` tier (0/8 relevant), semantic rank alone as evidence (let 6/6 probes through), half-the-words coverage (relevant-but-weak 3 -> 10), and 5-char prefix stemming (11 more irrelevant strongs, 0 relevant). `mimir misses`: a recall page with no strong hit is recorded — query text, count, last asked — most asked first; a later strong answer to the same question resolves it, and a miss after that reopens it. `mimir report` gains "unanswered queries". Explicit recalls only, never the inject hook. Per-machine (not synced, not exported), never stores a query the secrets scanner flags, ages out with event_retention_days, `[learn] log_misses = false` to disable. Schema v13: recall_miss.
The retrieval eval proves a ranking change doesn't break the questions someone wrote down; it can't say whether the change helps the questions agents actually ask. Shadow mode runs a candidate beside the live ranking on every explicit recall (CLI and MCP), keeps its order only when it differs, and scores both against what the agent opened. - attribution: an `opened` is a verdict on the most recent page that showed that node, within 10 minutes — not on every page that listed it - scoreboard (`mimir shadow`, --json, --days): MRR of the opened hit, wins/losses/ties, exact two-sided sign test, and a verdict that needs 20 decided pages before it says anything - states plainly that the comparison favours live (agents open what the live order put on top): a lead is evidence, a deficit is not harm - never changes recall output; nothing promotes itself - first candidate, co-activation phase 2: score * (1 + 0.5 * w) for a memory with a visible co_recalled edge (w >= 0.5) to one opened in the last 30 minutes - its own table, not rows in recall_event, which several readers count whole; pruned with the recall_event retention - `[learn] shadow_ranking = false` turns it off Schema v14: shadow_rank.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three stacked commits. Each later one adds a migration and touches the same recall code, so they ship together (schema v12 → v14).
1. Co-activation edges, phase 1 (
09f7a67)Two memories opened within 30 minutes strengthen a derived
co_recallededge (Hebbian, one fire per pair per day, 30-day half-life, ≤32 per node). Recorded only, ranking untouched. Not synced or exported;linkrefuses to author one.2. Weak-match labels +
mimir misses(9df1682, schema v13)A ranked list always has a #1, so the score can't tell "the answer" from "the least-bad thing in the store". Each recall hit is now
strong(contains the query's words AND both legs rank it top-10) orweak— one word on the compact line,"strength"in--json.Calibrated on the labelled corpus with the real model (
eval::strength, k=8): weak hits relevant 3/142, strong 23/42, 0/6 unanswerable probes passed. Measured and rejected (numbers instrength.rs): a middlelikelytier (0/8), semantic rank alone as evidence (6/6 probes through), half-the-words coverage, prefix stemming.A page with no strong hit is a miss:
mimir misseslists them most-asked first, auto-resolves on a later strong answer, andmimir reportcounts open ones. Explicit recalls only (never the inject hook), per-machine, never stores a query the secrets scanner flags.[learn] log_misses = falsedisables.3. Shadow ranking —
mimir shadow(ab00744, schema v14)A candidate ranking runs beside the live one on every explicit recall; differing orders are kept and scored against what the agent opened from that page (MRR, wins/losses, exact sign test, no verdict under 20 decided pages). Recall output never changes and nothing promotes itself. First candidate is co-activation phase 2. The report states that the comparison favours live.
[learn] shadow_ranking = falsedisables.Verification
cargo test --workspace: 480 passed, 0 failed; clippy-D warningsandcargo fmt --checkclean locallyHERMETIC_BASELINEunchanged (no live ranking change)Upgrade note
Schema v14 is one-way: older binaries and running
mimir mcpservers refuse the store (SchemaTooNew). Reinstall, then restart MCP clients.