Split fuzzy search from the LLM path: /search/fuzzy and /geometry/{id}#7
Open
yellowcap wants to merge 4 commits into
Open
Split fuzzy search from the LLM path: /search/fuzzy and /geometry/{id}#7yellowcap wants to merge 4 commits into
yellowcap wants to merge 4 commits into
Conversation
…e LLM path
Splits the generative pipeline from fuzzy matching so raw geometry can be
fetched without any LLM involvement:
- GET /search/fuzzy: Jaro-Winkler match against a place name, returns the
combined top-N GeoJSON FeatureCollection across divisions_area/natural_earth.
Supports ids_only=true for a cheap candidate-ID-only response.
- GET /geometry/{id}: direct single-feature geometry lookup by ID (source
inferred from the ne_ prefix if omitted), no fuzzy matching or LLM.
- Moves geometry simplification out of sql.py into a standalone geometry.py
with no dependency on lm.py, so the fuzzy/ID-lookup paths never import
DSPy/LLM machinery.
- Loads the spatial extension once via FastAPI lifespan onto a shared
connection; requests use cursor() instead of reconnecting, cutting ~90ms
of redundant LOAD spatial overhead per call (previously paid twice per
request on the simplify path).
- gazet_demo.py: backend now configurable via GAZET_BACKEND env var instead
of hardcoded "gguf".
Gives minimal spatial context (bounding box) alongside bare IDs, without paying for full geometry. bbox is computed via ST_XMin/YMin/XMax/YMax directly in SQL rather than fetching and simplifying full geometry.
- divisions_area now matches against names.primary instead of names.common.en, which is null for 77% of rows and silently excluded most of the dataset from every fuzzy search (both /search/fuzzy and the existing generative /search pipeline, which share search_candidates). - Ranks exact-substring matches (query literally contained in the name, e.g. "Loja" in "Loja Province") ahead of pure edit-distance near-misses (e.g. "Lodja"), even when the near-miss scores a slightly higher raw Jaro-Winkler similarity. - ids_only responses now include country and admin_level to disambiguate identically-named places (e.g. multiple real "Loja"s across Ecuador, Spain, and Latvia at different admin levels).
admin_level alone doesn't disambiguate well: the same subtype (e.g. "county") occurs at different admin_levels, and locality-type subtypes have no admin_level at all. subtype gives the human-readable category (region/county/locality/...) alongside the numeric admin_level.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GET /search/fuzzy— pure Jaro-Winkler place-name matching againstdivisions_area/natural_earth, no LLM. Returns the combined top-limitmatches as a GeoJSON FeatureCollection (raw or simplified geometry viasimplify). Passids_only=trueto skip geometry fetching entirely and get back{"ids": [{"source", "id", "name", "country", "subtype", "admin_level", "bbox"}, ...]}—bbox([minx, miny, maxx, maxy]) is computed via ST_XMin/YMin/XMax/YMax, far cheaper than full geometry.GET /geometry/{id}— direct single-feature geometry lookup by ID (source inferred from thene_prefix if omitted), no fuzzy matching or LLM. Pairs withids_onlyabove for a cheap search-then-fetch workflow.sql.pyinto a standalonegeometry.pywith zero dependency onlm.py, so the fuzzy/ID-lookup paths never import DSPy or any LLM machinery — this is what actually enforces "no generative elements," not just a docstring claim.lifespanonto a shared DuckDB connection (app.state.duckdb_con); request handlers use cheapcursor()calls instead of reconnecting per request. Removes ~90ms of redundantLOAD spatialoverhead per call (previously paid twice on thesimplifypath — once in the endpoint, once hidden inside the geometry-normalization helper).gazet_demo.py: backend is now configurable viaGAZET_BACKENDenv var instead of hardcoded"gguf", so the demo can be pointed at either backend without editing source.🐛 Bug fix: fuzzy matching was silently broken for most of
divisions_areaFound while testing this PR manually — not introduced by this PR, but fixed here since it affects the new endpoints directly and shares code with the existing generative
/searchpipeline (both go throughsearch_candidates):search_divisions_areamatched againstnames.common.en, which isNULLfor 824,502 of 1,071,266 rows (77%) ofdivisions_area. Since the query filtered onnames.common.en IS NOT NULL, the large majority of admin divisions worldwide were silently invisible to every fuzzy search, in both/search/fuzzyand the existing/searchLLM pipeline. Fixed by matchingnames.primaryinstead, consistent with howsearch_natural_earthalready worked.names.common.envalue, an exact query like"Loja"scored worse against"Loja Province"(Jaro-Winkler 0.862) than against unrelated near-miss strings like"Lodja"(0.947),"Lojane"(0.933),"Loulja"(0.911) — so the real match ranked 4th+ instead of 1st. Fixed by adding a substring-match tier: candidates where the query is a literal substring of the name (contains(lower(name), lower(query))) are now ranked ahead of pure edit-distance near-misses, with similarity as the tiebreaker within each tier./search?q=Loja&backend=dspynow correctly resolves to the actual Ecuador "Loja" region instead of a wrong candidate ("Lodja", a DR Congo county) that used to win purely on higher Jaro-Winkler score against the truncatednames.common.enfield.country/subtype/admin_leveltoids_onlyresponses, since fixing the coverage bug surfaces multiple genuinely distinct real-world places sharing the same name (e.g.q=Lojanow correctly returns Ecuador's "Loja" region, its nested "Loja" county, a "Loja" locality in Spain, and one in Latvia) — barenamealone can't disambiguate those.Test plan
ruff checkclean on all touched files (pre-existing unrelated issues insql.pyleft as-is)/search/fuzzy: verified geometry attached,sourcesfilter, invalid-source 400,simplify=true/falsesize difference (14.6KB raw vs 4.5KB simplified),ids_only=trueminimal response includingcountry/subtype/admin_level/bbox/geometry/{id}: verified explicitsource, auto-inference fromne_prefix, 404 on missing ID, 400 on bad source/searchand/search/stream(bothggufanddspybackends) still work end-to-end after the shared-connection refactor/search/fuzzycalls)q=Lojanow returns all 5 real "Loja" divisions (Ecuador ×3 admin levels, Spain, Latvia) ranked at similarity 1.0, ahead of near-misses like "Aloja";q=Georgiastill returns exact matches first (now 5 instead of 4, since more of the dataset is reachable)