You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The embeddings index is written as {df_id, embedding} and nothing else (embed.py:141). It carries no record of what produced it: not the model name, not the build date, not the text that was embedded.
So the index cannot tell you it is stale, and nothing else can either. Two ways it silently goes wrong today:
Model change._EMBED_MODEL is nomic-embed-text-v2-moe. Change it — or pull a different tag of the same name — and every subsequent --semantic search compares fresh query vectors against vectors from another model. Results degrade with no signal.
Indexed-text change. The text is composed at build time from df_id + df_description + cat_context (embed.py:112-123). Any change to that composition leaves existing indexes describing the old text. This just happened in feat(search): match the category name, not just the title #56, which widened what search matches; the embeddings path was deliberately left alone precisely because there is no way to tell users their index no longer reflects the catalogue.
This is a pre-existing hole, not a regression from #56 — it is filed separately because it was noticed while planning that change.
Proposal
Persist provenance alongside the vectors and warn on mismatch, without blocking:
record the model name, the build timestamp, and a marker of the indexed-text composition (a version integer is enough; a hash of the text template is better)
on load (embed.py:174), compare and, on mismatch, logging.warning naming opensdmx embed as the fix — then serve the results anyway
a missing marker (any index built before this lands) counts as a mismatch
Non-blocking on purpose: a stale index still returns useful results, and requiring a rebuild would demand a running Ollama just to search.
Note warnings.warn is not used in this codebase — logging.warning only, so the message never leaks internal paths to a CLI user.
Acceptance
index built by the current version → silent
index without a marker → warns once, still returns results
Problem
The embeddings index is written as
{df_id, embedding}and nothing else (embed.py:141). It carries no record of what produced it: not the model name, not the build date, not the text that was embedded.So the index cannot tell you it is stale, and nothing else can either. Two ways it silently goes wrong today:
_EMBED_MODELisnomic-embed-text-v2-moe. Change it — or pull a different tag of the same name — and every subsequent--semanticsearch compares fresh query vectors against vectors from another model. Results degrade with no signal.df_id + df_description + cat_context(embed.py:112-123). Any change to that composition leaves existing indexes describing the old text. This just happened in feat(search): match the category name, not just the title #56, which widened what search matches; the embeddings path was deliberately left alone precisely because there is no way to tell users their index no longer reflects the catalogue.This is a pre-existing hole, not a regression from #56 — it is filed separately because it was noticed while planning that change.
Proposal
Persist provenance alongside the vectors and warn on mismatch, without blocking:
embed.py:174), compare and, on mismatch,logging.warningnamingopensdmx embedas the fix — then serve the results anywayNon-blocking on purpose: a stale index still returns useful results, and requiring a rebuild would demand a running Ollama just to search.
Note
warnings.warnis not used in this codebase —logging.warningonly, so the message never leaks internal paths to a CLI user.Acceptance