harden: work through both code-review batches (provenance pinning, consolidation, guard, CI, docs)

.github/workflows/ci.yml

@@ -15,10 +15,38 @@ jobs:
         with:
           python-version: '3.12'
       - name: Install dependencies
-        run: pip install cryptography cbor2
+        # numpy is needed to import the recall/daemon modules; fastembed loads its
+        # model lazily, so the import smoke below exercises that path without the
+        # heavyweight (and flaky) model download.
+        run: pip install cryptography cbor2 numpy
       - name: Byte-compile engine
         run: python -m compileall -q engine
+      - name: Import smoke (every engine module loads)
+        run: |
+          python - <<'PY'
+          import sys, pkgutil
+          sys.path.insert(0, "engine")
+          for m in pkgutil.iter_modules(["engine"]):
+              if m.name.startswith("test_"):
+                  continue
+              __import__(m.name)
+          print("all engine modules import OK")
+          PY
       - name: Provenance tests (credentials, transparency log, inclusion receipts)
         run: python engine/test_provenance.py
       - name: Soft-binding tests (text fingerprint)
         run: python engine/test_softbinding.py
+      - name: Engine tests (note IO, recall filter, consolidation budget, guard)
+        run: python engine/test_cogmem.py
+
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v6
+        with:
+          python-version: '3.12'
+      - name: Install ruff
+        run: pip install ruff
+      - name: Lint
+        run: ruff check .

README.md

@@ -11,13 +11,13 @@
 [![W3C Verifiable Credentials](https://img.shields.io/badge/W3C-Verifiable%20Credentials-005a9c.svg)](https://www.w3.org/TR/vc-data-model-2.0/)
 [![SCITT](https://img.shields.io/badge/IETF-SCITT--style%20log-005a9c.svg)](https://datatracker.ietf.org/wg/scitt/about/)
 [![License: Apache-2.0](https://img.shields.io/badge/License-Apache--2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
-[![local-first](https://img.shields.io/badge/local--first-no%20data%20leaves%20your%20machine-111827.svg)](#)
+[![local-first recall](https://img.shields.io/badge/local--first-recall%20runs%20on%20your%20machine-111827.svg)](#privacy)
 
 **A self-improving, verifiable memory layer for AI coding agents.**
 
 cogmem learns how you work across sessions so your agent gets more accurate and more autonomous over time: it stops repeating mistakes, keeps a live model of each project, and surfaces the right lesson at the right moment. Every memory is cryptographically signed and tamper-evident, so a poisoned or altered memory can be detected and rejected before it ever steers the agent.
 
-> Developed by [WritersLogic](https://github.com/writerslogic) — local-first intelligence, no data leaving your machine.
+> Developed by [WritersLogic](https://github.com/writerslogic) — local-first recall; your memory and identity key stay on your machine (see [Privacy](#privacy)).
 
 [![cogmem MCP server](https://glama.ai/mcp/servers/writerslogic/cogmem/badges/card.svg)](https://glama.ai/mcp/servers/writerslogic/cogmem)
 
@@ -231,7 +231,9 @@ This proves the whole chain: agent identity (`cawg.ica.credential_valid`) bound
 
 ## Privacy
 
-cogmem is local-first by design. Memories, embeddings, and the identity key live on your machine; semantic recall runs on a local model (fastembed). Nothing is sent anywhere.
+cogmem is local-first by design. Memories, embeddings, and the identity key live on your machine, and **semantic recall is fully local** — the embedding and reranker models (fastembed) run on-device, so querying your memory never leaves the machine.
+
+The **learning pipeline is not local**: acquisition, consolidation, the feedback judge, and the project/user-model synthesis send the relevant session transcript to the Anthropic API (`ANTHROPIC_API_KEY`). That is how rules are extracted and scored. If you need fully-offline operation, run with `--no-hooks` (recall still works) until a local-model extraction path lands. In short: **recall is local; learning calls the API.**
 
 ## Part of the Agent-Provenance Stack
 

THREAT-MODEL.md

@@ -75,7 +75,7 @@ but has no test. UNTESTED rows are the follow-up test backlog (collected in the
 | # | Threat | Mitigation (code) | Evidence (test) |
 |---|---|---|---|
 | T1 | **Memory poisoning** — edit a rule's statement after signing to change agent behavior. | `verify_credential` recomputes the `eddsa-jcs-2022` proof over canonical VC; `verify_vault` additionally requires `credentialSubject.statement == rule body`, so an unsigned edit is a tamper; `provenance_enforce` excludes tampered/unsigned rules from recall. | `test_tampered_credential_rejected` (statement swap → false). **Partial:** `verify_vault`'s body-vs-credential match and the `provenance_enforce` recall exclusion are **UNTESTED** at this layer. |
-| T2 | **Credential forgery from another key** — re-issue a memory under a foreign DID. | `verify_credential` resolves the issuer DID to its key and verifies the proof under it; the kid is the key, so swapping the issuer breaks the signature. | `test_credential_from_other_key_rejected`. |
+| T2 | **Credential forgery from another key** — re-issue a memory under a foreign DID. | `verify_credential` verifies the proof under the issuer's key **and** pins the issuer to the trusted agent DID (`_issuer_trusted`, anchored TOFU in `$COGMEM_HOME/trust.json`, outside `vault/`). Swapping the issuer but keeping the old signature breaks the signature; *re-signing* validly under a foreign key is now also rejected as an untrusted issuer. `verify_log`/`verify_sth` pin identically, so a whole chain re-signed under a foreign key fails. | `test_credential_from_other_key_rejected`, `test_self_signed_credential_from_untrusted_issuer_rejected`, `test_forged_log_under_untrusted_key_rejected`, `test_trust_established_on_first_key_use`. |
 | T3 | **Log-chain break** — alter or delete a logged lifecycle event. | `verify_log` walks `prevHash` links (`_last_entry_hash` over the full prior line) and verifies each entry's Ed25519 signature over `_entry_signing_input`; reports `broken_at`. | `test_log_tamper_detected` (breaks at position 1), `test_log_chain_intact`. |
 | T4 | **Forged log entry** — append an entry without the agent key. | Each entry is signed over its canonical fields; `verify_log` rejects a bad signature. | `test_forged_entry_rejected`. |
 | T5 | **Forged inclusion proof** — claim membership for a memory not in the committed tree (tampered leaf, path, or root). | `verify_receipt` checks the STH signature (`verify_sth`), binds `treeSize`, and runs RFC 6962 §2.1.1 inclusion verification (`_verify_inclusion`) with leaf prefix `0x00` / node prefix `0x01`. | `test_inclusion_receipt_verifies`, `test_receipt_single_entry`, `test_receipt_tampered_leaf_rejected`, `test_receipt_tampered_path_rejected`, `test_receipt_forged_root_rejected`. **Gap:** second-preimage via leaf/node domain-prefix confusion, `treeSize`-vs-actual-leaves mismatch, and empty-tree edge cases are **UNTESTED**. |
@@ -104,13 +104,19 @@ current MVP proves.
    The roadmap item "external transparency service" (PROVENANCE.md) would partially address this; it
    is **not implemented**.
 
-3. **Self-asserted, self-signed trust anchors.** The COSE statement kid *is* the public key
-   (self-asserted); the STH is signed by the agent's own key — there is **no external transparency
-   anchor** (PROVENANCE.md roadmap item #1). An inclusion receipt proves "this memory is in *a* tree
-   the agent signed," not "in an independent, witnessed log." The ICA/C2PA claim-signer story
-   similarly leans on dev/self-signed certs in the WritersProof producer path (brief H10): **CAWG ICA
-   validity is independent of X.509 claim-signer trust**, so an ICA can read "valid" while the claim
-   signer is attacker-controlled. cogmem does not close that gap; the consuming verifier must.
+3. **Self-asserted, self-signed trust anchors.** `verify_credential`/`verify_log`/`verify_sth` now
+   pin the issuer to a TOFU-anchored agent DID (`$COGMEM_HOME/trust.json`), so a memory or chain
+   re-signed under a *foreign* key is rejected (T2). This is the meaningful gain: an attacker who can
+   write only `vault/` content — the poison/sync threat — can no longer forge a self-consistent chain
+   under their own key. **Residual, unchanged:** (a) the anchor lives under `$COGMEM_HOME`, so an
+   attacker with write access to the *whole* home (including `trust.json` and `agent.key`) re-anchors
+   and wins — collapses to T6; OS-keychain custody of the key + anchor is the roadmap hardening. (b)
+   The STH is still signed by the agent's own key — there is **no external transparency anchor**
+   (PROVENANCE.md roadmap item #1), so an inclusion receipt proves "this memory is in *a* tree the
+   agent signed," not "in an independent, witnessed log." The ICA/C2PA claim-signer story similarly
+   leans on dev/self-signed certs in the WritersProof producer path (brief H10): **CAWG ICA validity
+   is independent of X.509 claim-signer trust**, so an ICA can read "valid" while the claim signer is
+   attacker-controlled. cogmem does not close that gap; the consuming verifier must.
 
 4. **Agent-path soft binding is a non-durable SHA-256 placeholder.** The text/soft binding tying the
    agent provenance to document content is, on the agent path, a SHA-256 reference — not a keyed,

cogmem

@@ -2,6 +2,7 @@
 # cogmem — single entry point for the cognitive memory system.
 #   cogmem review [list|promote <id>|reject <id>]   approve/decline Layer-A rules
 #   cogmem status                                    health summary
+#   cogmem doctor                                    end-to-end learning-loop health
 #   cogmem recall "<text>" | note "<text>"           query / record a memory
 #   cogmem guard "<command>"                          check a command against tripwires
 #   cogmem mcp                                        run the MCP server (stdio)
@@ -19,6 +20,7 @@ cmd="${1:-status}"
 shift 2>/dev/null || true
 
 case "$cmd" in
+  doctor)      exec "$PY" "$ENGINE/metrics.py" doctor ;;
   review)      exec "$PY" "$ENGINE/review.py" "$@" ;;
   model)       exec "$PY" "$ENGINE/usermodel.py" "$@" ;;
   progress)    exec "$PY" "$ENGINE/narrative.py" "$@" ;;
@@ -49,6 +51,6 @@ case "$cmd" in
     [[ "$PEND" -gt 0 ]] && echo "-> $PEND rule(s) await approval: cogmem review list"
     ;;
   *)
-    echo "Usage: cogmem [status|recall|note|guard|review|capture|mcp|consolidate|index|verify]" >&2
+    echo "Usage: cogmem [status|doctor|recall|note|guard|review|capture|mcp|consolidate|index|verify]" >&2
     exit 1 ;;
 esac

engine/acquire.py

@@ -29,57 +29,27 @@
 import os
 import re
 import sys
-import urllib.request
-import urllib.error
 from datetime import datetime, timezone
 from pathlib import Path
 
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import config
+from common import api_call, parse_json_block
+
 logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
 log = logging.getLogger("cogmem.acquire")
 
 COGMEM = Path(os.environ.get("COGMEM_HOME", Path.home() / ".claude" / "cogmem"))
 CANDIDATES_DIR = COGMEM / "vault" / "candidates"
 
-API_URL = "https://api.anthropic.com/v1/messages"
-DETECT_MODEL = "claude-haiku-4-5-20251001"   # cheap: runs every session
-EXTRACT_MODEL = "claude-sonnet-4-6"          # strong: runs only on signal
+DETECT_MODEL = config.model("detect")    # cheap: runs every session
+EXTRACT_MODEL = config.model("extract")  # strong: runs only on signal
 
 # Cap transcript size sent to the model. Most signal lives in user turns
 # (corrections, preferences) and assistant text; we drop thinking/tool noise.
 MAX_TRANSCRIPT_CHARS = 60_000
 
 
-def api_call(model: str, prompt: str, max_tokens: int) -> str | None:
-    key = os.environ.get("ANTHROPIC_API_KEY")
-    if not key:
-        log.error("ANTHROPIC_API_KEY not set")
-        return None
-    body = json.dumps({
-        "model": model,
-        "max_tokens": max_tokens,
-        "messages": [{"role": "user", "content": prompt}],
-    }).encode()
-    req = urllib.request.Request(
-        API_URL,
-        data=body,
-        headers={
-            "x-api-key": key,
-            "anthropic-version": "2023-06-01",
-            "content-type": "application/json",
-        },
-        method="POST",
-    )
-    try:
-        with urllib.request.urlopen(req, timeout=60) as resp:
-            data = json.loads(resp.read())
-        return data["content"][0]["text"]
-    except urllib.error.HTTPError as e:
-        log.error("API HTTP %s: %s", e.code, e.read()[:200])
-    except Exception as e:  # noqa: BLE001 — hook must never crash the session
-        log.error("API call failed: %s", e)
-    return None
-
-
 def extract_conversation(transcript_path: Path) -> str:
     """Flatten a Claude Code transcript to role-tagged text, dropping
     thinking blocks and tool noise (low signal, high token cost)."""
@@ -157,17 +127,6 @@ def slugify(text: str, maxlen: int = 50) -> str:
     return s[:maxlen].rstrip("-") or "rule"
 
 
-def parse_json_block(text: str) -> dict | None:
-    text = text.strip()
-    if text.startswith("```"):
-        text = text.split("\n", 1)[1].rsplit("```", 1)[0].strip()
-    try:
-        return json.loads(text)
-    except json.JSONDecodeError:
-        log.warning("Could not parse extractor output: %s", text[:200])
-        return None
-
-
 def write_candidate(rule: dict, session_id: str, now: str) -> Path | None:
     rule_text = (rule.get("rule") or "").strip()
     if not rule_text:

engine/artifacts.py

@@ -19,13 +19,14 @@
 
 sys.path.insert(0, str(Path(__file__).resolve().parent))
 from common import VAULT, api_call, parse_json_block, write_note
+import config
 from acquire import slugify
 
 logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
 log = logging.getLogger("cogmem.artifacts")
 
 CANDIDATES = VAULT / "candidates"
-MODEL = "claude-sonnet-4-6"
+MODEL = config.model("artifacts")
 MAX_COMMITS = 150
 
 

engine/config.py

@@ -9,7 +9,26 @@
 from pathlib import Path
 
 CONFIG = Path(__file__).resolve().parent / "config.json"
-DEFAULTS = {"recall_floor": 0.62, "recall_gap": 6.0, "provenance_enforce": False}
+
+# LLM ids per pipeline role and the local recall models, in one place so upgrades
+# don't need code edits and a config.json can override any of them.
+_MODELS = {
+    "detect": "claude-haiku-4-5-20251001",      # cheap signal check, every session
+    "extract": "claude-sonnet-4-6",             # rule extraction, only on signal
+    "judge": "claude-haiku-4-5-20251001",       # feedback verdicts
+    "consolidate": "claude-sonnet-4-6",         # dedup classification
+    "selfmodel": "claude-sonnet-4-6",
+    "projectstate": "claude-sonnet-4-6",
+    "usermodel": "claude-sonnet-4-6",
+    "narrative": "claude-sonnet-4-6",
+    "artifacts": "claude-sonnet-4-6",
+    "eval_gen": "claude-haiku-4-5-20251001",
+}
+_EMBED_MODEL = "BAAI/bge-small-en-v1.5"
+_RERANK_MODEL = "Xenova/ms-marco-MiniLM-L-6-v2"
+
+DEFAULTS = {"recall_floor": 0.62, "recall_gap": 6.0, "provenance_enforce": False,
+            "models": _MODELS, "embed_model": _EMBED_MODEL, "rerank_model": _RERANK_MODEL}
 
 
 def load() -> dict:
@@ -21,5 +40,19 @@ def load() -> dict:
     return dict(DEFAULTS)
 
 
+def model(role: str) -> str:
+    """LLM id for a pipeline role; a partial "models" map in config.json overrides
+    individual roles, falling back to the built-in default for any role it omits."""
+    return load().get("models", {}).get(role, _MODELS[role])
+
+
+def embed_model() -> str:
+    return load().get("embed_model", _EMBED_MODEL)
+
+
+def rerank_model() -> str:
+    return load().get("rerank_model", _RERANK_MODEL)
+
+
 def save(cfg: dict) -> None:
     CONFIG.write_text(json.dumps(cfg, indent=2))