feat(rating): evidence-anchored ratings — write the receipt before the class

.claude/skills/rate-session-memories/SKILL.md

@@ -14,28 +14,31 @@ Call `memory.list_session_exposures` (no arguments — the server resolves the
 current session from env). Read the returned list. This is the ONLY valid set
 of ids to rate. The list in the RATE_MEMORIES directive may have been truncated.
 
-## STEP 2 — Classify each id
+## STEP 2 — Evidence first, then classify
 
-For each `(kind, id)` pair returned by the list, assign exactly ONE class:
+For each `(kind, id)`: FIRST try to write ONE line of evidence — what the
+memory changed in this session, or a quote of where you used it. Then:
 
-- **cited** — you quoted or directly referenced the memory in a reply.
-- **shaped** — the memory guided a decision but wasn't cited verbatim.
-- **ignored** — you saw it but it didn't affect this session. (Default.)
-- **misled** — it caused a wrong direction or wasted effort.
-- **overlooked** — the memory was relevant and you should have applied
-  it, but you didn't, until the user explicitly pointed you back to it.
+- Evidence line written → choose `cited` (quoted/directly referenced),
+  `shaped` (guided a decision), `misled` (sent you wrong), or `overlooked`
+  (user had to point you back to it). Include the evidence line in the
+  rating.
+- No evidence line possible → the class is `ignored`. Full stop. Do not
+  reverse the order; choosing a class first and rationalising evidence
+  after is how ratings drift.
 
 Rules:
 - Quote the id in your reasoning so you can't drift.
 - Do not invent ids. Do not skip ids.
-- If genuinely uncertain between two classes, prefer the lower one
-  (shaped > cited, ignored > shaped). `misled` is never a fallback.
+- If genuinely uncertain between two EVIDENCED classes, prefer the weaker
+  claim (shaped over cited). `misled` is never a fallback. Once an
+  evidence line exists, `ignored` is no longer available — uncertainty
+  about class never reverses the evidence decision.
 - `overlooked` is never a fallback. Use it ONLY when the user explicitly
   pointed you back to a memory you already had and had not applied —
   that user intervention is the observable anchor. Test for it first,
   separately from the cited/shaped/ignored axis. No anchor event → not
   `overlooked`.
-- Default is `ignored`. "Shaped" requires evidence you can point to.
 
 ## STEP 3 — Submit ALL ratings in ONE call
 
@@ -45,7 +48,8 @@ Call `memory.apply_session_ratings` with this exact shape (no `session_id`
 ```json
 {
   "ratings": [
-    {"kind": "reflection", "id": "r-abc...", "class": "cited"},
+    {"kind": "reflection", "id": "r-abc...", "class": "cited",
+     "evidence": "Used its junit-xml flag verbatim in the pytest invocation."},
     {"kind": "semantic",   "id": "s-def...", "class": "ignored"}
   ]
 }

better_memory/db/migrations/0016_rating_evidence.sql

@@ -0,0 +1,13 @@
+-- Migration 0016: evidence line on rated exposures.
+--
+-- Rating variance is the noise floor under every ranking signal: identical
+-- memory sets were rated `shaped` in one session and `ignored` in another
+-- (2026-07 A/B runs). Non-ignored ratings now carry a one-line evidence
+-- statement (what the memory changed, or a quote), enforced by
+-- MemoryRatingService and surfaced in the UI drawers. Audit-only: no
+-- scoring reads this column.
+--
+-- Distinct from reflections.evidence_count, which counts synthesis source
+-- observations and has nothing to do with rating evidence.
+
+ALTER TABLE session_memory_exposure ADD COLUMN evidence TEXT;

better_memory/hooks/session_close.py

@@ -224,12 +224,12 @@ def _emit_rating_directive_if_unrated(session_id: str) -> bool:
             + ("\n".join(refl_lines) if refl_lines else "  (none)")
             + f"\n\nSemantic memories ({len(sem_lines)}):\n"
             + ("\n".join(sem_lines) if sem_lines else "  (none)")
-            + "\n\nFor each id, classify as one of:\n"
-            "  cited / shaped / ignored / misled / overlooked "
-            "(default: ignored)\n\n"
-            "Most exposures default to `ignored` — only flag the few "
-            "that actually shaped the session or misled you. Invoke "
-            "the skill `rate-session-memories`."
+            + "\n\n"
+            "For each id: FIRST write one line of evidence (what the memory "
+            "changed, or a quote) - if you cannot, the class is `ignored`.\n"
+            "Classes: cited / shaped / ignored / misled / overlooked.\n"
+            "Non-ignored ratings without an evidence line are rejected. "
+            "Invoke the skill `rate-session-memories`."
         )
         encoded = directive.encode("utf-8")
         if len(encoded) > CAP_BYTES:

better_memory/mcp/handlers/sessions.py

@@ -143,13 +143,15 @@ async def credit(self, args: dict[str, Any]) -> list[TextContent]:
                 kind=args["kind"],
                 id=args["id"],
                 classification=args["class"],
+                evidence=args.get("evidence"),
             )
         else:
             payload = self._memory_rating.credit_one(
                 session_id=sid,
                 kind=args["kind"],
                 id=args["id"],
                 classification=args["class"],
+                evidence=args.get("evidence"),
             )
         return [TextContent(type="text", text=json.dumps(payload))]
 

better_memory/mcp/tools.py

@@ -477,7 +477,10 @@ def tool_definitions(
                 "(resolved server-side from CLAUDE_SESSION_ID). Called "
                 "at session end by the rate-session-memories skill. "
                 "Raises if CLAUDE_SESSION_ID is unset — call only inside "
-                "an active Claude session."
+                "an active Claude session. Non-ignored classes require "
+                "`evidence` (one line: what the memory changed, or a "
+                "quote); write the evidence line FIRST — no evidence "
+                "means the class is `ignored`."
             ),
             inputSchema={
                 "type": "object",
@@ -500,6 +503,15 @@ def tool_definitions(
                                         "misled", "overlooked",
                                     ]
                                 },
+                                # Optional here at the wire-schema level;
+                                # MemoryRatingService.apply_session_ratings
+                                # enforces the real contract (required +
+                                # non-empty for every non-ignored class,
+                                # <=EVIDENCE_MAX_CHARS).
+                                "evidence": {
+                                    "type": "string",
+                                    "maxLength": 500,
+                                },
                             },
                         },
                     },
@@ -516,16 +528,20 @@ def tool_definitions(
                 "class must be 'cited', 'shaped', 'misled', or "
                 "'overlooked' — NOT 'ignored'. Use 'overlooked' when the "
                 "user pointed you back to a memory you already had but "
-                "had not applied."
+                "had not applied. Always include `evidence`: one line — "
+                "what the memory changed, or a quote. If you cannot "
+                "write one, the memory was `ignored`; do not call "
+                "credit."
             ),
             inputSchema={
                 "type": "object",
                 "additionalProperties": False,
-                "required": ["kind", "id", "class"],
+                "required": ["kind", "id", "class", "evidence"],
                 "properties": {
                     "kind": {"enum": ["reflection", "semantic"]},
                     "id": {"type": "string"},
                     "class": {"enum": ["cited", "shaped", "misled", "overlooked"]},
+                    "evidence": {"type": "string", "maxLength": 500},
                 },
             },
         ),

better_memory/services/memory_rating.py

@@ -14,7 +14,7 @@
 import sqlite3
 from collections.abc import Callable
 from datetime import datetime
-from typing import Literal, TypedDict
+from typing import Any, Literal, TypedDict
 
 from better_memory._common import default_clock
 
@@ -61,6 +61,28 @@ class ApplySessionRatingsResult(TypedDict):
 _VALID_CLASSES: set[str] = {"cited", "shaped", "ignored", "misled", "overlooked"}
 _CREDIT_CLASSES: set[str] = {"cited", "shaped", "misled", "overlooked"}
 
+EVIDENCE_MAX_CHARS = 500
+
+
+def _validate_evidence(cls: str, evidence: object, *, where: str) -> str | None:
+    """Trim + enforce the evidence contract for one rating.
+
+    Non-ignored classes require a non-empty line; `ignored` may carry one.
+    Returns the trimmed value (or None). Raises ValueError with the
+    caller-supplied position prefix on violation.
+    """
+    trimmed = evidence.strip() if isinstance(evidence, str) else None
+    if cls != "ignored" and not trimmed:
+        raise ValueError(
+            f"{where}: class {cls!r} requires a non-empty evidence line "
+            "(what the memory changed, or a quote); if there is nothing "
+            "to point at, the class is 'ignored'")
+    if trimmed and len(trimmed) > EVIDENCE_MAX_CHARS:
+        raise ValueError(
+            f"{where}: evidence exceeds {EVIDENCE_MAX_CHARS} chars "
+            f"({len(trimmed)})")
+    return trimmed or None
+
 
 class MemoryRatingService:
     """Writes useful_count / times_misled on reflections + semantic memories,
@@ -84,12 +106,20 @@ def credit_one(
         kind: str,
         id: str,
         classification: str,
+        evidence: str | None = None,
     ) -> ApplyOutcome:
         """Apply one rating for (session_id, kind, id).
 
         Validation (ValueError before any DB write):
         - kind must be 'reflection' or 'semantic'.
         - classification must be 'cited', 'shaped', 'misled', or 'overlooked' (NOT 'ignored').
+        - evidence must be a non-empty (post-strip), <=EVIDENCE_MAX_CHARS
+          line; all credit classes are non-ignored, so this is effectively
+          required. `evidence` defaults to None only as a runtime-safety
+          compat shim for callers that have not yet been updated to pass
+          it (see MCP `memory.credit` handler) — None fails validation
+          with a clear ValueError instead of a TypeError, so an un-updated
+          caller gets a clean error response rather than a crash.
 
         Skip outcomes (no exception, no write, returned via dict):
         - 'not_exposed' — no matching exposure row for this session.
@@ -123,13 +153,17 @@ def credit_one(
                 f"Invalid classification: {classification!r}. "
                 f"Expected one of {_CREDIT_CLASSES}"
             )
+        trimmed_evidence = _validate_evidence(
+            classification, evidence, where="credit"
+        )
 
         now = self._clock().isoformat()
         self._conn.execute("SAVEPOINT memory_credit")
         try:
             outcome = self._apply_one(
                 session_id=session_id, kind=kind, memory_id=id,
                 classification=classification, now=now,
+                evidence=trimmed_evidence,
             )
         except BaseException:
             self._conn.execute("ROLLBACK TO SAVEPOINT memory_credit")
@@ -149,6 +183,7 @@ def _apply_one(
         memory_id: str,
         classification: str,
         now: str,
+        evidence: str | None = None,
     ) -> ApplyOutcome:
         """Inside-savepoint per-row apply. Returns the same dict shape as
         credit_one. Shared by credit_one and apply_session_ratings (Task 3).
@@ -223,10 +258,10 @@ def _apply_one(
         # 4. Stamp the exposure row.
         self._conn.execute(
             "UPDATE session_memory_exposure "
-            "SET rated_at = ?, classification = ? "
+            "SET rated_at = ?, classification = ?, evidence = ? "
             "WHERE session_id = ? AND memory_kind = ? AND memory_id = ?"
             "  AND rated_at IS NULL",
-            (now, classification, session_id, kind, memory_id),
+            (now, classification, evidence, session_id, kind, memory_id),
         )
 
         return {"applied": classification, "skipped": None}
@@ -236,7 +271,7 @@ def apply_session_ratings(
         self,
         *,
         session_id: str,
-        ratings: list[dict[str, str]],
+        ratings: list[dict[str, Any]],
     ) -> ApplySessionRatingsResult:
         """Atomic batch update at session end.
 
@@ -246,6 +281,10 @@ def apply_session_ratings(
         - each entry must have kind in {'reflection', 'semantic'},
           class in {'cited', 'shaped', 'ignored', 'misled', 'overlooked'},
           and a string id.
+        - each entry may carry an optional "evidence" string. Non-ignored
+          classes require a non-empty (post-strip), <=EVIDENCE_MAX_CHARS
+          evidence line; 'ignored' may carry one or omit it. The trimmed
+          value is stored on session_memory_exposure.evidence.
         - no duplicate (kind, id) pairs in one batch.
 
         Inside the SAVEPOINT, each entry runs through _apply_one. Skip
@@ -286,6 +325,9 @@ class in {'cited', 'shaped', 'ignored', 'misled', 'overlooked'},
                     f"ratings[{i}].class: invalid {cls!r}; "
                     f"expected one of {_VALID_CLASSES}"
                 )
+            r["_evidence"] = _validate_evidence(
+                cls, r.get("evidence"), where=f"ratings[{i}]"
+            )
             if not isinstance(rid, str) or not rid:
                 raise ValueError(f"ratings[{i}].id: must be non-empty string")
             key = (kind, rid)
@@ -314,6 +356,7 @@ class in {'cited', 'shaped', 'ignored', 'misled', 'overlooked'},
                     memory_id=r["id"],
                     classification=r["class"],
                     now=now,
+                    evidence=r["_evidence"],
                 )
                 applied_class = outcome["applied"]
                 skipped_reason = outcome["skipped"]

better_memory/services/relevant.py

@@ -297,7 +297,8 @@ def retrieve_relevant(
 )
 _BLOCK_FOOTER = (
     "If any entry above materially helps or misleads this task, credit it now: "
-    "memory_credit(kind, id, 'cited'|'shaped'|'misled').\n"
+    "memory_credit(kind, id, class, evidence) - include a one-line evidence "
+    "statement.\n"
     "</project-memory>"
 )
 

better_memory/storage/agentcore.py

@@ -1359,14 +1359,21 @@ def credit_one(
         kind: str,
         id: str,
         classification: str,
+        evidence: str | None = None,
     ) -> dict[str, Any]:
         """Apply one classification → counter increment on a record.
 
         Counter mapping (spec Rating model section):
             cited / shaped → useful_count
             ignored        → ignored_count
             misled         → times_misled
-            overlooked     → overlooked_count"""
+            overlooked     → overlooked_count
+
+        `evidence` is accepted for StorageBackend protocol parity only.
+        Agentcore mode has no exposure table (see list_session_exposures)
+        and therefore nowhere to store a rating-evidence line; it is not
+        validated or persisted here — sqlite is the only backend with an
+        evidence contract (MemoryRatingService)."""
         counter_key = _RATING_TO_COUNTER.get(classification)
         if counter_key is None:
             raise ValueError(

better_memory/storage/protocol.py

@@ -289,8 +289,17 @@ def credit_one(
         kind: str,
         id: str,
         classification: str,
+        evidence: str | None = None,
     ) -> Any:
-        """Apply a single rating for an exposed memory. Returns ApplyOutcome."""
+        """Apply a single rating for an exposed memory. Returns ApplyOutcome.
+
+        `evidence` is a one-line string (what the memory changed, or a
+        quote); non-ignored classes require a non-empty value —
+        SqliteBackend forwards it to MemoryRatingService, which validates
+        and stores it. AgentCoreBackend accepts the parameter for
+        signature parity but has no evidence storage (no exposure table)
+        and does not validate or persist it — see AgentCoreBackend.credit_one.
+        """
         ...
 
     # ----- Synthesis (sqlite-only — guarded by supports_synthesis) -----

better_memory/storage/sqlite.py

@@ -327,9 +327,11 @@ def credit_one(
         kind: str,
         id: str,
         classification: str,
+        evidence: str | None = None,
     ) -> Any:
         return self._memory_rating.credit_one(
             session_id=session_id, kind=kind, id=id, classification=classification,
+            evidence=evidence,
         )
 
     # ----- Synthesis -----