Skip to content

Split fuzzy search from the LLM path: /search/fuzzy and /geometry/{id}#7

Open
yellowcap wants to merge 4 commits into
mainfrom
feature/fuzzy-search-endpoint
Open

Split fuzzy search from the LLM path: /search/fuzzy and /geometry/{id}#7
yellowcap wants to merge 4 commits into
mainfrom
feature/fuzzy-search-endpoint

Conversation

@yellowcap

@yellowcap yellowcap commented Jul 23, 2026

Copy link
Copy Markdown
Member

Summary

  • Adds GET /search/fuzzy — pure Jaro-Winkler place-name matching against divisions_area/natural_earth, no LLM. Returns the combined top-limit matches as a GeoJSON FeatureCollection (raw or simplified geometry via simplify). Pass ids_only=true to skip geometry fetching entirely and get back {"ids": [{"source", "id", "name", "country", "subtype", "admin_level", "bbox"}, ...]}bbox ([minx, miny, maxx, maxy]) is computed via ST_XMin/YMin/XMax/YMax, far cheaper than full geometry.
  • Adds GET /geometry/{id} — direct single-feature geometry lookup by ID (source inferred from the ne_ prefix if omitted), no fuzzy matching or LLM. Pairs with ids_only above for a cheap search-then-fetch workflow.
  • Extracts geometry simplification/GeoJSON normalization out of sql.py into a standalone geometry.py with zero dependency on lm.py, so the fuzzy/ID-lookup paths never import DSPy or any LLM machinery — this is what actually enforces "no generative elements," not just a docstring claim.
  • Loads the spatial extension once at startup via FastAPI lifespan onto a shared DuckDB connection (app.state.duckdb_con); request handlers use cheap cursor() calls instead of reconnecting per request. Removes ~90ms of redundant LOAD spatial overhead per call (previously paid twice on the simplify path — once in the endpoint, once hidden inside the geometry-normalization helper).
  • gazet_demo.py: backend is now configurable via GAZET_BACKEND env var instead of hardcoded "gguf", so the demo can be pointed at either backend without editing source.

🐛 Bug fix: fuzzy matching was silently broken for most of divisions_area

Found while testing this PR manually — not introduced by this PR, but fixed here since it affects the new endpoints directly and shares code with the existing generative /search pipeline (both go through search_candidates):

  • search_divisions_area matched against names.common.en, which is NULL for 824,502 of 1,071,266 rows (77%) of divisions_area. Since the query filtered on names.common.en IS NOT NULL, the large majority of admin divisions worldwide were silently invisible to every fuzzy search, in both /search/fuzzy and the existing /search LLM pipeline. Fixed by matching names.primary instead, consistent with how search_natural_earth already worked.
  • Even for rows that did have a names.common.en value, an exact query like "Loja" scored worse against "Loja Province" (Jaro-Winkler 0.862) than against unrelated near-miss strings like "Lodja" (0.947), "Lojane" (0.933), "Loulja" (0.911) — so the real match ranked 4th+ instead of 1st. Fixed by adding a substring-match tier: candidates where the query is a literal substring of the name (contains(lower(name), lower(query))) are now ranked ahead of pure edit-distance near-misses, with similarity as the tiebreaker within each tier.
  • Verified this also fixes real query results in the existing generative pipeline: /search?q=Loja&backend=dspy now correctly resolves to the actual Ecuador "Loja" region instead of a wrong candidate ("Lodja", a DR Congo county) that used to win purely on higher Jaro-Winkler score against the truncated names.common.en field.
  • Also added country/subtype/admin_level to ids_only responses, since fixing the coverage bug surfaces multiple genuinely distinct real-world places sharing the same name (e.g. q=Loja now correctly returns Ecuador's "Loja" region, its nested "Loja" county, a "Loja" locality in Spain, and one in Latvia) — bare name alone can't disambiguate those.

Test plan

  • ruff check clean on all touched files (pre-existing unrelated issues in sql.py left as-is)
  • /search/fuzzy: verified geometry attached, sources filter, invalid-source 400, simplify=true/false size difference (14.6KB raw vs 4.5KB simplified), ids_only=true minimal response including country/subtype/admin_level/bbox
  • /geometry/{id}: verified explicit source, auto-inference from ne_ prefix, 404 on missing ID, 400 on bad source
  • /search and /search/stream (both gguf and dspy backends) still work end-to-end after the shared-connection refactor
  • Confirmed per-request latency drop after the connection-reuse fix (~1200ms → ~980-1140ms for repeated /search/fuzzy calls)
  • Confirmed the fuzzy-matching bug fix: q=Loja now returns all 5 real "Loja" divisions (Ecuador ×3 admin levels, Spain, Latvia) ranked at similarity 1.0, ahead of near-misses like "Aloja"; q=Georgia still returns exact matches first (now 5 instead of 4, since more of the dataset is reachable)

…e LLM path

Splits the generative pipeline from fuzzy matching so raw geometry can be
fetched without any LLM involvement:

- GET /search/fuzzy: Jaro-Winkler match against a place name, returns the
  combined top-N GeoJSON FeatureCollection across divisions_area/natural_earth.
  Supports ids_only=true for a cheap candidate-ID-only response.
- GET /geometry/{id}: direct single-feature geometry lookup by ID (source
  inferred from the ne_ prefix if omitted), no fuzzy matching or LLM.
- Moves geometry simplification out of sql.py into a standalone geometry.py
  with no dependency on lm.py, so the fuzzy/ID-lookup paths never import
  DSPy/LLM machinery.
- Loads the spatial extension once via FastAPI lifespan onto a shared
  connection; requests use cursor() instead of reconnecting, cutting ~90ms
  of redundant LOAD spatial overhead per call (previously paid twice per
  request on the simplify path).
- gazet_demo.py: backend now configurable via GAZET_BACKEND env var instead
  of hardcoded "gguf".
Gives minimal spatial context (bounding box) alongside bare IDs, without
paying for full geometry. bbox is computed via ST_XMin/YMin/XMax/YMax
directly in SQL rather than fetching and simplifying full geometry.
- divisions_area now matches against names.primary instead of
  names.common.en, which is null for 77% of rows and silently excluded
  most of the dataset from every fuzzy search (both /search/fuzzy and
  the existing generative /search pipeline, which share search_candidates).
- Ranks exact-substring matches (query literally contained in the name,
  e.g. "Loja" in "Loja Province") ahead of pure edit-distance near-misses
  (e.g. "Lodja"), even when the near-miss scores a slightly higher raw
  Jaro-Winkler similarity.
- ids_only responses now include country and admin_level to disambiguate
  identically-named places (e.g. multiple real "Loja"s across Ecuador,
  Spain, and Latvia at different admin levels).
admin_level alone doesn't disambiguate well: the same subtype (e.g.
"county") occurs at different admin_levels, and locality-type subtypes
have no admin_level at all. subtype gives the human-readable category
(region/county/locality/...) alongside the numeric admin_level.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant