Skip to content

embeddings: the index cannot tell you it is stale #57

Description

@aborruso

Problem

The embeddings index is written as {df_id, embedding} and nothing else (embed.py:141). It carries no record of what produced it: not the model name, not the build date, not the text that was embedded.

So the index cannot tell you it is stale, and nothing else can either. Two ways it silently goes wrong today:

  1. Model change. _EMBED_MODEL is nomic-embed-text-v2-moe. Change it — or pull a different tag of the same name — and every subsequent --semantic search compares fresh query vectors against vectors from another model. Results degrade with no signal.
  2. Indexed-text change. The text is composed at build time from df_id + df_description + cat_context (embed.py:112-123). Any change to that composition leaves existing indexes describing the old text. This just happened in feat(search): match the category name, not just the title #56, which widened what search matches; the embeddings path was deliberately left alone precisely because there is no way to tell users their index no longer reflects the catalogue.

This is a pre-existing hole, not a regression from #56 — it is filed separately because it was noticed while planning that change.

Proposal

Persist provenance alongside the vectors and warn on mismatch, without blocking:

  • record the model name, the build timestamp, and a marker of the indexed-text composition (a version integer is enough; a hash of the text template is better)
  • on load (embed.py:174), compare and, on mismatch, logging.warning naming opensdmx embed as the fix — then serve the results anyway
  • a missing marker (any index built before this lands) counts as a mismatch

Non-blocking on purpose: a stale index still returns useful results, and requiring a rebuild would demand a running Ollama just to search.

Note warnings.warn is not used in this codebase — logging.warning only, so the message never leaks internal paths to a CLI user.

Acceptance

  • index built by the current version → silent
  • index without a marker → warns once, still returns results
  • both branches covered by tests

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions