trusty-memory: unembedded drawers cannot be detected per-palace or attributed by id — doctor check and verify-by-id both missing

[bobmatnyc]

Defect

#4906 fixed repair (retry + backfill + a failure ledger) but explicitly left detection out. Two gaps remain, confirmed still open 2026-08-06:

1. No doctor check on the vector_count/drawer_count gap. PR #4975's own body: "Follow-up not in this PR: A trusty-memory doctor check on the count gap ... Left out to keep this PR to the data-integrity fix."
2. No way to verify a specific set of drawer ids is embedded. palace_reembed (dry-run, shipped in #4975) returns the full still_missing_ids set, but there is no call that takes a caller's own id list and answers "are these embedded." A migration/deletion workflow has to fetch the entire missing-set and diff it itself.

memory_recall returning content is not a substitute — it can hit lexically, so a drawer can be present, durable, and permanently unfindable while recall still "passes."

CORRECTION (2026-08-06) — the count-gap premise below is disproven, do not build it as stated

#5005 found HnswStore's vector-id allocator aliases across processes (crates/trusty-memory/.../hnsw_store.rs:280,337-346): upsert has no uniqueness check, so a second live HnswStore over the same file re-issues ids already in use, overwriting a live drawer's vector with another's. Measured on palace trusty-tools: vector_id 988 is shared by 5 uuids; only 1 of the 5 is retrievable. VECTOR_KEYS count vs distinct mapped vector_id count showed a gap of 4 with 0 reported missing by palace_reembed.

This proves the drawer_count − vector_count gap is not the right signal, in both directions:

• Palace cto read 80/71, gap 9, missing 9 — correct only by coincidence (all-unembedded, no aliasing).
• Palace trusty-tools read gap 4, missing 0 (id aliasing — nothing is unembedded, the gap comes from id collisions, not absence).
• Orphan vectors can mask a real hole in the other direction (already documented at embed_repair.rs:16-22).
• The two surfaces filter drawer_count differently (console_metrics.rs:181 vs embed_repair.rs:135-141) and read different tables for vector_count — console is VECTORS.len() − DELETED_VECTORS.len() (hnsw_store.rs:462), reembed is the VECTOR_KEYS row count (hnsw_store.rs:493).

The module docs already said equal counts don't prove coverage. The converse is now also proven: unequal counts don't prove a hole, and a zero gap doesn't prove no aliasing.

Resolution item 1 below is superseded. The doctor check must not compare drawer_count to vector_count. Use palace_reembed dry-run's missing count PLUS a third number, the distinct-mapped-vector_id count against VECTOR_KEYS row count — the count-gap alone keeps missing the aliasing class entirely.

Also add to scope: console_metrics cannot serve as a health sweep at all. MAX_PALACES_IN_REPORT = 20 (console_metrics.rs:32) caps it, and an uncached palace reports 0/0 (console_metrics.rs:203-210), which reads as healthy. A live sweep found palace localLLM fully unembedded — 15 drawers, zero vectors — invisible to it.

Evidence

• 2026-08-06, palace trusty-tools: drawer_count 1301, vector_count 1297 — gap 4 (down from the 39 measured in #4906 on 2026-08-05).
• Live probe today: asked to enumerate the 4 missing drawers. Could not, and said why: "no tool in my current surface diffs drawer IDs against vector IDs." Root cause: the installed trusty-memory binary (built 2026-08-05 19:33) predates PR #4975's merge (2026-08-05 23:27), which shipped that exact capability — a binary-lag instance of #4751, not proof the capability is absent from the codebase. It still demonstrates gap 2 above: even on a current binary, the only path to the id list is a full-palace dry-run dump, not a targeted "verify these ids" call.
• 2026-08-06 incident (fully reverted, see #4834#issuecomment-5204668311): a static-memory migration deleted 72 files after passing a content-recall check. That gate cannot detect a dropped embed. Drawer 1509bc35-df9b-40d7-a50c-f1f4ae80fa64 was independently recorded on 2026-08-05 as written-in-batch and unretrievable, and its source file was among the 72 deleted. Nothing was lost; the near-miss is why detection needs to exist before the next deletion-bearing workflow.

Resolution

1. doctor check comparing vector_count to drawer_count per palace — warn or fail with the count; non-zero is never silent. Superseded — see CORRECTION above. The check must use palace_reembed dry-run's missing count plus the distinct-mapped-id vs VECTOR_KEYS-row-count comparison, not the drawer/vector count gap.
2. A verification call that takes a set of drawer ids and asserts all are embedded, so a migration/deletion workflow gates on a direct answer instead of diffing a full missing-set dump or trusting recall as a proxy.
3. Interim rule until both ship: any deletion-bearing workflow treats a non-zero missing count from palace_reembed dry-run, OR a nonzero distinct-id-vs-VECTOR_KEYS gap, as an unconditional block — not the drawer_count/vector_count gap, which is neither necessary nor sufficient (see CORRECTION).

Confidence: High — both gaps confirmed by reading the merged code (crates/trusty-memory/src/tools/palace_ops.rs, PR #4975 body) and reproduced live today.

Cross-links: #4906 (repair, closed), #4834 (blocked migration/deletion workflow), #4836 (recall-as-proxy is unsound for the same reason), #4751 (binary-lag root cause of today's probe failure), #5005 (id-aliasing defect that disproves the count-gap premise).

🤖🤖🤖 Generated with trusty-mpm — https://github.com/bobmatnyc/trusty-tools

[bobmatnyc]

Correction: the drawer_count − vector_count gap this ticket proposes checking is not the right detector — filed and evidenced in #5005.

HnswStore's vector-id allocator (hnsw_store.rs:280, AtomicU64 seeded per-process, never reserved in redb) aliases across processes, and upsert (hnsw_store.rs:337-346) has no uniqueness check — it silently overwrites a live drawer's vector with another's. Measured on palace trusty-tools: vector_id 988 shared by 5 uuids, only 1 retrievable, and palace_reembed reports missing 0 for that palace (it checks key presence, not id uniqueness).

The gap is fed by four conditions that can cancel each other: unembedded drawers (real gap, e.g. cto 80/71 gap 9 missing 9 — correct by coincidence), id aliasing (trusty-tools gap 4, missing 0 — nonzero gap, nothing unembedded), orphan vectors masking a real hole (embed_repair.rs:16-22), and the two surfaces filtering drawer_count differently (console_metrics.rs:181 vs embed_repair.rs:135-141) while reading different tables for vector_count (hnsw_store.rs:462 vs hnsw_store.rs:493).

I've edited the body's Resolution item 1 and struck the count-gap check. What #5000 should specify instead: palace_reembed dry-run's missing count PLUS a third number, the distinct-mapped-id count against VECTOR_KEYS, or it keeps missing the aliasing class entirely.

Also add to scope: console_metrics cannot serve as a health sweep at all — MAX_PALACES_IN_REPORT = 20 (console_metrics.rs:32) and an uncached palace reports 0/0 (console_metrics.rs:203-210), reading as healthy. A live sweep found palace localLLM fully unembedded, 15 drawers and zero vectors, invisible to it.

🤖🤖🤖 Generated with trusty-mpm — https://github.com/bobmatnyc/trusty-tools