docs: realign the public record of how search actually works (#78)

* docs: realign the public record of how search actually works

Semantic search shipped in February and the docs have drifted ever since. July
and August improved the corpus (category context, df_keywords, harvested prose,
df_notes) while the engine stayed on Ollama, and each document written along the
way described a different intended future in the present tense.

State, measured: the index is built on demand per provider by `opensdmx embed`;
nothing is precomputed and nothing ships. What differs per provider is the
corpus, not the capability — only istat.parquet ships, so elsewhere the embedded
document is id + title + category context. Latency on ISTAT: ~10 s on the first
query while Ollama loads the model, ~1 s after.

Correct the claims: keyword default 20 -> 50; "exact substring match on dataset
title" -> the scored token match on title, ID and category name; semantic "10
most similar" -> 50; "The model is fixed" -> fixed within a release, and the
index does not record which model built it (#57); the 3.4x figure now names its
metric. PRD and istat-flow still said search matches "descriptions" — the same
wording fixed once on 2026-07-16.

Mark the self-contained-backend plan as a direction, not an approved plan: it
named embeddinggemma-300m with multilingual-e5-small as fallback and fastembed
0.8.0 exposes neither. Say which model each fastembed result used, so "reverted"
(2026-03-31, nomic-embed-text-v1.5-Q) and "validated" (2026-08-04,
paraphrase-multilingual-MiniLM-L12-v2) stop reading as a contradiction.

`--semantic` no longer drops flags in silence: --category, --page and --all were
accepted and ignored, so a user who restricted a search to a category read
results drawn from the whole catalogue with no signal. They now fail explicitly.

The skill was routing agents to the weaker arm — it offered semantic only after
keyword returned fewer than 3 results, and priced it at "10-30 s". On naturally
phrased questions the keyword scorer measures MRR 0.041 against 0.252 for
semantic. Routing is now conditional on an index existing.

Untrack three dated audits of superseded versions, kept public by an accident of
.gitignore ordering; evaluation.md still told readers that `search --semantic`
needs a Google API key, removed on 2026-04-03.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TekhaUTgxCgKVyiNEBCN36

* docs(future-ideas): say which model each fastembed result used

LOG.md records fastembed tried and reverted (2026-03-31) for poor quality on
Italian queries; this section records a fastembed prototype validated
(2026-08-04). Different models — nomic-embed-text-v1.5-Q and
paraphrase-multilingual-MiniLM-L12-v2 — but neither text said so, so the repo
handed a reader two opposite verdicts with nothing to reconcile them.

Also mark the plan this prototype was said to confirm: it named
embeddinggemma-300m with multilingual-e5-small as fallback, and fastembed 0.8.0
exposes neither, so its premise no longer holds. The prototype was qualitative,
on descriptions only, and has never been scored against the targets in
docs/search.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TekhaUTgxCgKVyiNEBCN36

* fix: address review on #78 — flag detection, provider naming, two overstated claims

- `--page 1` was accepted under `--semantic`: the value equals the default, so a
  value comparison cannot tell "not given" from "given as 1". Read the parameter
  source instead. Uses `ParameterSource.name` rather than importing the enum,
  since `click` is not a declared dependency.
- The "no index" message named the provider only when `--provider` was passed on
  the command line, so a provider selected via OPENSDMX_PROVIDER or by default
  produced a hint the user could not act on. Use the resolved cache key, which is
  what actually determines the index path.
- README claimed a rebuild "reproduces the same index". Only the *model* is fixed
  within a release: `build_embeddings` re-reads the catalog, the category tree and
  the harvested descriptions as they are at that moment, so a rebuild after the
  caches refresh embeds different text.
- SKILL.md told the agent to probe for an index with `search --semantic`, which
  fails on the Ollama check first — so a stopped server was indistinguishable from
  a missing index. Name both error messages and, either way, fall back to the
  tree-first path instead of stopping.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TekhaUTgxCgKVyiNEBCN36

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: aborruso

.gitignore

@@ -65,3 +65,4 @@ video/
 docs/evaluation-v0.10.3.md
 .vscode/settings.json
 docs/evaluation-v*.md
+docs/evaluation.md

LOG.md

@@ -1,5 +1,17 @@
 # LOG
 
+## 2026-08-21 - docs: realign the public record of how search actually works
+
+- **The state, measured, not assumed.** `search --semantic` ships and works: Ollama with `nomic-embed-text-v2-moe`, a hardcoded constant in `embed.py`, no flag and no fallback backend. The index is built **on demand, per provider**, by `opensdmx embed` into `<cache>/<provider>/embeddings.parquet` — nothing is precomputed and nothing ships. Latency on ISTAT (4,896 dataflows): ~10 s on the first query of a session while Ollama loads the model, ~1 s after. What differs per provider is the *corpus*, not the capability: `df_prose` comes from a `data/descriptions/<provider>.parquet` inside the wheel and only `istat.parquet` exists, so elsewhere the embedded document is id + title + category context.
+- **Why the docs had drifted.** July and August improved the corpus (category context v0.17.0, `df_keywords` v0.19.0, harvested prose v0.20.0, `df_notes`) while the engine stayed on Ollama. Each document written along the way described a different intended future in the present tense and none was amended when that future changed.
+- **The self-contained-backend plan is a direction, not an approved plan, and now says so.** Its premise is falsified: the 2026-07-16 plan named `google/embeddinggemma-300m` with `multilingual-e5-small` as fallback and fastembed 0.8.0 exposes neither; the plan document is gone; and fastembed had already been tried and reverted on 2026-03-31 with a *different* model (`nomic-embed-text-v1.5-Q`) for poor Italian quality. `docs/future-ideas.md` now states which model each result used, so the "reverted" and "validated" entries stop reading as a contradiction. No candidate has been scored against the targets `docs/search.md` sets (MRR 0.135 to beat BM25, ~0.327 to replace nomic) — issue #77.
+- **Corrections to public claims.** README: keyword default 20 → 50; "exact substring match on dataset title" → the scored token match on title, ID and category name; semantic "10 most similar" → 50; "The model is fixed" → the model is fixed *within a release*, and the index does not record which model built it (#57); the 3.4× figure now names its metric (S@10, 57% vs 17%). `docs/PRD.md` and `references/istat-flow.md` still said search matches "descriptions" — the same wording `LOG.md` records fixing on 2026-07-16; dataflows expose an SDMX `Name`.
+- **`--semantic` no longer drops flags in silence.** `--category`, `--page` and `--all` were accepted and ignored — the semantic branch returns before the code that reads them — so a user who restricted a search to a category read results drawn from the whole catalogue with no signal. They now fail with an explicit error. `--n` and `--grep` are unaffected. The "cache not found" message now says the index is per-provider and suggests the right `--provider`. Ollama-unreachable was already reported distinctly, since `_check_ollama()` runs first.
+- **The skill was routing agents to the weaker arm.** `SKILL.md` told the agent to offer `--semantic` only after keyword returned fewer than 3 results, and priced it at "10-30 s". Measured on ISTAT: on naturally-phrased questions the keyword scorer reaches MRR 0.041 against 0.252 for semantic, and on English queries against Italian metadata keyword search finds essentially nothing. Routing is now conditional — where an index exists and the question is in natural language, semantic goes first; otherwise the `tree`-first default stands. Latency and the top-N figure corrected.
+- **Removed from the repository**: `docs/evaluation.md`, `docs/evaluation-v0.14.0.md`, `docs/evaluation-v0.2.6.md`. Dated audits of superseded versions, kept public by an accident of `.gitignore` ordering; `evaluation.md` still told readers that `search --semantic` needs a Google API key, removed on 2026-04-03. Untracked, not deleted.
+- Issue #53 (ISTAT descriptions from METADATA_API) closed — shipped in v0.20.0 and refreshed monthly since. Opened #75 (`[semantic]` extra), #76 (BM25 on `search_dataset`), #77 (static retriever).
+- Gate: ruff clean, mypy strict clean on 15 files, 384 tests green (5 new, covering the rejected flag combinations). `skills/sdmx-explorer/` updated (SKILL.md + `references/thematic-tree.md` + `references/istat-flow.md`).
+
 ## 2026-08-19 - v0.22.3 - fix: `tree` probes the category tree of an unlisted provider
 
 - A provider given as a URL (`--provider https://.../rest`) got `Active provider does not expose /categoryscheme` even when it does: `categories_supported` is a *declared* capability of the 15 providers in `portals.json`, and a custom dict never carries the key, so the falsy default was read as a denial. The gate now fires only when a listed provider declares `false`; an undeclared provider is probed live and an `httpx.HTTPStatusError` is translated into `CategoriesNotSupported` naming the URL and the status code.

README.md

@@ -53,6 +53,10 @@ uv lock --upgrade-package opensdmx  # if used as a library dependency
 pip install --upgrade opensdmx      # if installed with pip
 ```
 
+Everything except semantic search works with this install alone. `opensdmx
+search --semantic` additionally needs a local [Ollama](https://ollama.com)
+server and a one-off index build — see [Semantic search](#semantic-search).
+
 ## CLI quick start
 
 ```bash
@@ -265,8 +269,8 @@ All commands accept `--provider` (`-p`) to select the provider.
 
 | Command | Description |
 |---|---|
-| `opensdmx search <keyword> [--n N] [--grep pattern] [-p provider]` | Keyword search in dataset titles and IDs (default: 20 results); optionally filter by regex |
-| `opensdmx search --semantic <query> [--n N]` | Semantic search (requires `opensdmx embed`) |
+| `opensdmx search <keyword> [--n N] [--grep pattern] [-p provider]` | Keyword search in dataset titles, IDs and category names (default: 50 results per page); optionally filter by regex |
+| `opensdmx search --semantic <query> [--n N] [--grep pattern]` | Semantic search (requires a running Ollama and `opensdmx embed`). Only `--n` and `--grep` apply; `--category`, `--page` and `--all` are rejected |
 | `opensdmx embed [-p provider]` | Build semantic embeddings cache via Ollama |
 | `opensdmx info <id> [-p provider]` | Show dataset metadata and dimensions |
 | `opensdmx values <id> <dim> [--grep pattern] [-p provider]` | Show codelist values for a dimension (case-insensitive); optionally filter by regex |
@@ -475,19 +479,29 @@ opensdmx tree --scheme t_economy --depth 1
 
 | Mode | How it works | Best for |
 |---|---|---|
-| Keyword (default) | Exact substring match on dataset title | When you know the right technical term |
+| Keyword (default) | Scored token match on title, ID and category name — all tokens must match, falling back to any | When you know the right technical term |
 | `--semantic` | Embedding similarity via Ollama | When you don't know the exact wording, or want conceptually related datasets |
 
 #### Setup
 
-Requires [Ollama](https://ollama.com) with the `nomic-embed-text-v2-moe` model:
+Requires a running [Ollama](https://ollama.com) server with the
+`nomic-embed-text-v2-moe` model, plus one index build per provider:
 
 ```bash
-ollama pull nomic-embed-text-v2-moe
+ollama serve &                        # if not already running
+ollama pull nomic-embed-text-v2-moe   # ~1 GB, once
 opensdmx embed              # build embeddings for default provider (eurostat)
 opensdmx embed -p istat     # build embeddings for ISTAT
 ```
 
+The index is built **on demand, per provider**, and stored in the local cache as
+`embeddings.parquet`. Until you run `opensdmx embed` for a provider,
+`--semantic` reports that no index exists for it. Rebuilding takes one pass over
+the whole catalog; there is no incremental update.
+
+Every provider can be indexed, but the quality of the index depends on how much
+text there is to embed. That is where ISTAT stands apart — see below.
+
 For providers that expose a thematic catalog (Eurostat, ISTAT, ECB, OECD, INSEE,
 ABS, BIS), running `opensdmx tree` once before `opensdmx embed` enriches each
 embedding with the names of the categories the dataflow belongs to. This
@@ -501,8 +515,9 @@ how it is produced, not just its title. See
 [Dataflow descriptions for semantic search](docs/descriptions.md) for the design,
 and [Dataset search](docs/search.md) for how the keyword and semantic paths
 compare — including the measured result that semantic retrieval finds the right
-dataflow 3.4× more often, and that on English queries against Italian metadata
-keyword search finds essentially nothing.
+dataflow in the top 10 for 57% of queries against 17% for the keyword path, and
+that on English queries against Italian metadata keyword search finds
+essentially nothing.
 
 #### Tips for better results
 
@@ -568,7 +583,7 @@ opensdmx search --semantic "underemployment" # finds involuntary part-time datas
 
 **When keyword search is enough**
 
-When you already know the technical term, keyword search is faster and returns all matching datasets (not capped at 10). `search "unemployment"` returns 114 results; `search --semantic "unemployment"` returns the 10 most similar by score — useful to surface the most relevant ones quickly.
+When you already know the technical term, keyword search is faster and can return every matching dataset with `--all`. `search "unemployment"` returns 114 results; `search --semantic "unemployment"` returns the 50 most similar by score (change with `--n`) — useful to surface the most relevant ones quickly.
 
 **Rule of thumb:** start with a keyword search. If results are empty or off-target, switch to `--semantic`.
 
@@ -578,7 +593,7 @@ The `score` column is the **[cosine similarity](https://en.wikipedia.org/wiki/Co
 
 The model converts text into high-dimensional vectors such that semantically related phrases point in similar directions, regardless of the exact words used. Cosine similarity measures the angle between two such vectors: a score of 1 means identical direction, 0 means orthogonal (unrelated).
 
-The ranking therefore depends entirely on the model: a different model would produce different vectors and a different ordering. The model is fixed — if you rebuild embeddings with `opensdmx embed`, the same model is used.
+The ranking therefore depends entirely on the model: a different model would produce different vectors and a different ordering. Within a given release the model is fixed, so the ranking will not shift under you because the backend changed. It can still shift for a different reason: `opensdmx embed` re-reads the catalog, the category tree and the harvested descriptions as they are *at that moment*, so a rebuild after the caches refresh embeds different text and produces different vectors. Should the backend change in a future release, embeddings will have to be rebuilt — the index does not record which model produced it ([#57](https://github.com/ondata/opensdmx/issues/57)).
 
 ### Caching
 

docs/PRD.md

@@ -56,7 +56,7 @@ Statistical data from international agencies is distributed via SDMX 2.1 REST AP
 ### Dataset discovery
 
 - List all available datasets for the active provider (`all_available()`).
-- Search datasets by keyword in their description (`search_dataset()`).
+- Search datasets by keyword in their title, ID and category name (`search_dataset()`). Dataflows expose an SDMX `Name`, not a `Description`.
 - Load a dataset object by dataflow ID, structure ID, or description (`load_dataset()`).
 - Show dimension metadata including position, codelist ID, and description (`dimensions_info()`).
 - Show available dimension values from the codelist (`get_dimension_values()`).
@@ -71,7 +71,7 @@ Statistical data from international agencies is distributed via SDMX 2.1 REST AP
 
 ### Semantic search
 
-- Build and store Ollama embedding vectors for all dataset descriptions (`embed` command / `build_embeddings()`).
+- Build and store Ollama embedding vectors for the catalog (`embed` command / `build_embeddings()`), one index per provider, built on demand.
 - Search by semantic similarity against locally built embeddings (`semantic_search()`).
 - Embeddings are stored per provider in the provider cache directory, for example `~/.cache/opensdmx/eurostat/embeddings.parquet`.
 
@@ -99,7 +99,7 @@ Statistical data from international agencies is distributed via SDMX 2.1 REST AP
 
 | Command | Description |
 |---|---|
-| `opensdmx search <keyword>` | Keyword search in dataset descriptions |
+| `opensdmx search <keyword>` | Keyword search in dataset titles, IDs and category names |
 | `opensdmx search --semantic <query>` | Semantic search (requires `opensdmx embed`) |
 | `opensdmx embed` | Build semantic embeddings cache via Ollama |
 | `opensdmx info <id>` | Show dataset metadata and dimensions |