fix(fmem): stop ingesting entities that can never be found again - #21
Merged
Conversation
IngestOptions defaulted embed_missing to false, with the comment "forge generates embeddings separately; set false here." forge does not generate embeddings. It has no embedding client anywhere in the workspace, and WireEntity -- the authoritative wire type for this call -- has no embedding field at all, so it is structurally incapable of attaching a vector. The option promised something the type system already ruled out. fmem stores an entity vector only when the caller supplies one or embed_missing is set. Neither held, so every entity forge has ever ingested on this path landed with entity_embedding = None. Each one is invisible to every ANN retrieval source, and an entity is lexically findable only by its NAME -- its content body is not fts-indexed -- so a content query returns nothing at all. None of that surfaces anywhere: the ingest succeeds, the entity and edge counts are correct, and retrieval is silently dead. A fresh FerrosaAi install stored 262 entities and 222 edges from the starter corpus and could not return one of them for any query (t_2618e286). - default embed_missing to true, so fmem embeds server-side - reject embed_missing=false at the call site rather than writing unreadable memory, naming the consequence and not just the flag. This is the last point that still knows why; by the time a user searches, the cause is hundreds of writes in the past - smart_paper_loader passed false explicitly; same fix Verified end to end against a real stack (ferrosa-memory v0.26.0 with the bundled llama.cpp + nomic runtime): embedding requests served during the same 262-entity ingest went from ~144 to ~424. The entities now carry vectors. Semantic search over them still returns nothing, which is a SEPARATE defect and is not fixed here -- current evidence points at session scoping, since forge ingests under its own session_id while a later search defaults to scope=both (its own session + global) and therefore never looks at forge's. Tracked on t_2618e286.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
IngestOptionsdefaultedembed_missingtofalse, with the comment:forge does not generate embeddings. There is no embedding client anywhere in the workspace, and
WireEntity— the authoritative wire type for this call — has no embedding field at all, so it is structurally incapable of attaching a vector. The option promised something the type system had already ruled out.fmem stores an entity vector only when the caller supplies one or
embed_missingis set. Neither held, so every entity forge has ever ingested on this path landed withentity_embedding = None.Each one is invisible to every ANN retrieval source. And an entity is lexically findable only by its name — its content body is not fts-indexed — so a content query returns nothing at all.
Why nobody noticed
Nothing surfaces it. The ingest succeeds, the entity and edge counts are correct, and retrieval is silently dead.
A fresh FerrosaAi install stored 262 entities and 222 edges from the starter corpus and could not return one of them for any query.
Changes
embed_missingtotrue, so fmem embeds server-sideembed_missing=falseat the call site rather than writing unreadable memory. The error names the consequence, not the flag. This is the last point that still knows why — by the time a user runs a search, the cause is hundreds of writes in the pastsmart_paper_loader.rs:148passedfalseexplicitly; same fixVerification
Tests written first; both red for the right reasons before the fix.
forge-fmem-client79/79,forge-ingest379/379.Verified end-to-end against a real stack (ferrosa-memory v0.26.0 with a bundled llama.cpp + nomic runtime), not just by unit test: embedding requests served during the same 262-entity ingest rose from ~144 to ~424. The entities now carry vectors.
Scope
Semantic search over that corpus had a second, independent cause — the client config was ingesting into a session partition no search reads — fixed separately in ferrosa-workbench. The combined end-to-end claim is not yet proven; see
t_2618e286.