docs: realign the public record of how search actually works

[aborruso]

Semantic search shipped in February 2026 and the documentation has drifted ever since. This PR makes the public docs describe what the code does, and marks the plans that are no longer executable as such.

Why the docs drifted

July and August improved the corpus — category context (v0.17.0), df_keywords (v0.19.0), harvested ISTAT prose (v0.20.0), df_notes — while the engine stayed on Ollama. Each document written along the way described a different intended future in the present tense, and none was amended when that future changed. The result was five public documents describing five different states.

The state, measured

• search --semantic works. Backend: Ollama with nomic-embed-text-v2-moe, a hardcoded constant in embed.py. No flag, no env var, no fallback backend.
• The index is built on demand, per provider, by opensdmx embed into <cache>/<provider>/embeddings.parquet. Nothing is precomputed and nothing ships in the package.
• What differs per provider is the corpus, not the capability. df_prose comes from a data/descriptions/<provider>.parquet inside the wheel and only istat.parquet exists, so elsewhere the embedded document is id + title + category context. The eval numbers are an ISTAT ceiling, not a per-provider promise.
• Latency on ISTAT (4,896 dataflows): ~10 s on the first query of a session while Ollama loads the model, ~1 s after.

Corrections to public claims

Where	Was	Is
README.md	keyword default 20	50 (cli.py)
README.md	"exact substring match on dataset title"	scored token match on title, ID and category name
README.md	semantic returns "the 10 most similar"	50, --n to change
README.md	"The model is fixed"	fixed within a release; the index does not record which model built it (#57)
README.md	"3.4× more often"	named as S@10 — 57% vs 17%
docs/PRD.md, references/istat-flow.md	search matches "descriptions"	titles, IDs and category names

The "descriptions" wording is the same error LOG.md records fixing on 2026-07-16; dataflows expose an SDMX Name, not a Description.

docs/search.md gains a Current state section up front, and its "already-approved plan" claim is amended (below).

The self-contained-backend plan is a direction, not an approved plan

Its premise is falsified and nothing said so:

• The 2026-07-16 plan named google/embeddinggemma-300m with multilingual-e5-small as fallback. fastembed 0.8.0 exposes neither.
• The plan document is no longer on disk.
• fastembed had already been tried and reverted on 2026-03-31 — with a different model (nomic-embed-text-v1.5-Q), for poor quality on Italian queries.

So the repo carried both "fastembed reverted" (LOG.md) and "fastembed validated" (docs/future-ideas.md) with nothing to reconcile them. Each entry now names its model. No candidate has been scored against the targets docs/search.md sets — MRR 0.135 to beat BM25, ~0.327 to replace nomic — which is tracked as #77.

Behaviour change: --semantic no longer drops flags in silence

--category, --page and --all were accepted and ignored — the semantic branch returns before the code that reads them. A user who restricted a search to a category read results drawn from the whole catalogue, with no signal that the filter had been dropped. They now fail with an explicit error; --n and --grep are unaffected.

The "cache not found" message now says the index is per-provider and suggests the right --provider. Ollama-unreachable was already reported distinctly — _check_ollama() runs before the parquet read — verified with OLLAMA_HOST pointed at a dead port.

The skill was routing agents to the weaker arm

SKILL.md told the agent to offer --semantic only after keyword returned fewer than 3 results, and priced it at "10–30 s". Measured on ISTAT (eval/results/2026-08-10/retrieval.md):

register	keyword MRR	semantic MRR
naturally phrased	0.041	0.252
technical	0.107	0.407

On English queries against Italian metadata the keyword path scores 0.012 and semantic 0.310.

Routing is now conditional: where an index exists and the question is in natural language, semantic goes first; otherwise the tree-first default stands. This follows the measurement without promising what the install does not provide.

Removed from the repository

docs/evaluation.md, docs/evaluation-v0.14.0.md, docs/evaluation-v0.2.6.md — dated audits of superseded versions, kept public by an accident of .gitignore ordering (docs/evaluation-v*.md was added after two of them were already committed). evaluation.md still told readers that search --semantic requires a Google API key, removed on 2026-04-03.

Untracked, not deleted; .gitignore extended so they stay out. No public document links to them.

Issues

Closed #53 (ISTAT descriptions from METADATA_API — shipped in v0.20.0, refreshed monthly, last run 2026-08-01 green).

Opened #75 ([semantic] extra), #76 (BM25 on search_dataset, +85% MRR, no new dependency), #77 (static retriever measured against the targets).

Verification

ruff clean · mypy strict clean on 15 files · 384 tests green, 5 new covering the rejected flag combinations.

Not addressed, and worth stating: the doc-first counterweight block in the gold set, and measuring the extended-text gain on providers other than ISTAT. Both are prerequisites for reading the eval numbers as anything other than an ISTAT ceiling.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TekhaUTgxCgKVyiNEBCN36

[greptile-apps]

Greptile Summary

This PR realigns the search documentation and agent guidance with the current Ollama-backed, per-provider semantic-search implementation.

• Corrects public descriptions of keyword and semantic search behavior, defaults, corpus inputs, latency, and reproducibility.
• Rejects keyword-only flags when used with semantic search and adds regression coverage, including explicit --page 1.
• Updates the explorer skill to prefer semantic search for suitable natural-language requests while retaining an explicit tree-first fallback.
• Retires obsolete evaluation documents and clarifies that the self-contained backend remains an unvalidated direction.

Important Files Changed

Filename	Overview
src/opensdmx/cli.py	Rejects unsupported semantic-search options using parameter-source detection, fully addressing the explicit-default bypass.
tests/test_cli.py	Adds focused coverage for each rejected option, their combinations, explicit --page 1, and supported --n/--grep.
skills/sdmx-explorer/SKILL.md	Revises discovery routing and explicitly continues through the tree-first path when semantic prerequisites are unavailable.
README.md	Corrects search defaults, prerequisites, corpus behavior, evaluation claims, and rebuild reproducibility wording.
docs/search.md	Documents the current semantic backend, per-provider index, distinct search contracts, measured results, and unresolved backend direction.
LOG.md	Records the documentation realignment, semantic flag validation, routing update, and removal of obsolete evaluation records.

_{Reviews (2): Last reviewed commit: "fix: address review on #78 — flag detect..." | Re-trigger Greptile}

[Copilot]

Pull request overview

This PR realigns public documentation and the sdmx-explorer skill guidance with the current, shipped behavior of opensdmx search (keyword vs semantic via Ollama), and introduces a small CLI behavior change to avoid silently ignoring incompatible flags under --semantic.

Changes:

• Update docs/README/skill references to accurately describe current keyword scoring, semantic search prerequisites (Ollama + per-provider on-demand embeddings), and measured retrieval results.
• Change CLI behavior so search --semantic rejects keyword-only flags (--category, --page, --all) instead of accepting/ignoring them silently; add tests for these combinations.
• Remove/ignore outdated evaluation documents from the public repo surface (delete files, extend .gitignore).

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/test_cli.py	Adds regression tests ensuring keyword-only flags are rejected under --semantic, while --n/--grep remain allowed.
src/opensdmx/cli.py	Enforces incompatible-flag rejection in the semantic path; improves missing-embeddings messaging.
skills/sdmx-explorer/SKILL.md	Updates the agent’s routing guidance and semantic-search expectations (latency, top-N, compatible flags).
skills/sdmx-explorer/references/thematic-tree.md	Updates routing table and adds clarified semantic-search prerequisites and behavior.
skills/sdmx-explorer/references/istat-flow.md	Corrects search matching description (titles/IDs/categories, not “descriptions”).
README.md	Updates public CLI docs and semantic-search section to match actual behavior and defaults.
LOG.md	Adds a dated entry documenting the doc realignment and the --semantic flag-behavior change.
docs/search.md	Adds an upfront “Current state” section and amends the self-contained-backend plan status.
docs/PRD.md	Corrects dataset discovery and semantic-search statements to match the actual implementation.
docs/future-ideas.md	Clarifies the historical fastembed attempts and corrects the “approved plan” framing.
docs/evaluation.md	Removed (superseded/outdated evaluation doc).
docs/evaluation-v0.2.6.md	Removed (superseded evaluation doc).
docs/evaluation-v0.14.0.md	Removed (superseded evaluation doc).
.gitignore	Ensures docs/evaluation.md stays untracked going forward.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[greptile-apps]

Thanks for the thorough writeup — the ParameterSource.name comparison to avoid importing the enum is exactly the right call given the typer dependency constraint, and wrapping it in a test that enforces that boundary is solid. The resolved-cache-key fix for the provider hint is a nice cleanup too; using whatever actually determined the index path is cleaner than reconstructing it from the CLI flag.

All four fixes look well-reasoned. Nothing further from me on this PR.