Skip to content

feat(data): POI build pipeline + chunked binary (PR 2 of 5) - #4

Merged
taddyb merged 7 commits into
mainfrom
feat/richer-geocoding-pr2
May 20, 2026
Merged

feat(data): POI build pipeline + chunked binary (PR 2 of 5)#4
taddyb merged 7 commits into
mainfrom
feat/richer-geocoding-pr2

Conversation

@taddyb

@taddyb taddyb commented May 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Second PR in the richer-geocoding rollout. Extracts ~80k categorized POIs from the same OSM PBF the walking graph already uses, pre-snaps each to a walk-graph node in the largest connected component (so future routing can't silently fail), and ships the result as a chunked binary.

No runtime code touched — this PR is data infrastructure. PR 3 will wire the POI binary into the search bar.

What's in the binary

  • 79,863 POIs across the basemap (NYC + LI + south Westchester/Conn + NJ)
  • 10 categories: food, transit, park, culture, attraction, shop, school, health, service, worship
  • Each record carries a pre-snapped walk_node index (correctness win — fixes the class of bug where a POI lands inside an OSM-island park path and becomes unrouteable)
  • File size: 2.5 MB → single chunk tiles/pois.part-aa (well under the 50 MB / GitHub 100 MB cap)

New files

Path Purpose
pipelines/walk_graph_reader.py Decode walk_graph.bin (NWLK format) for offline tools; LCC + local snap
pipelines/poi_emit.py POI1 binary writer (24B header, 20B records, names blob)
pipelines/pois.py pyosmium extractor: classify tags → category, pre-snap, clip to bbox
scripts/build-pois.sh Bash wrapper; runs the pipeline + chunks to tiles/pois.part-*
scripts/assemble-pois.sh Idempotent reverse: cat chunks → tiles/pois.bin
tiles/pois.part-aa Committed POI chunk
pipelines/tests/test_*.py 8 new pytest cases (binary round-trip, classify, extract, LCC snap)

CI

  • pages.yml now assembles pois.bin from chunks alongside the walk-graph
  • pr-checks.yml gains a pipeline-tests job that runs uv run pytest against the assembled walk-graph

Test plan

  • `uv run --directory pipelines pytest -v` → 15/15 pass locally
  • `bash scripts/build-pois.sh` produces a sane binary (79,863 POIs / 2.5 MB)
  • `bash scripts/assemble-pois.sh` round-trips chunks back to identical `pois.bin`
  • CI's new `pipeline-tests` job runs pytest with the assembled walk graph
  • CI's existing deploy job assembles `pois.bin` without error

Subsequent PRs

  • PR 3: local POI lane in the search dropdown (consumes `pois.bin` at runtime)
  • PR 4: WASM `reachable_nodes` method
  • PR 5: search-driven category discovery + map pins

🤖 Generated with Claude Code

taddyb and others added 7 commits May 19, 2026 22:03
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Builds tiles/pois.bin from the cached OSM PBF: 79,863 POIs across 10
categories, each pre-snapped to a walk-graph node in the LCC so the
runtime search never returns an unrouteable result. Chunked to
tiles/pois.part-* (50 MB cap; only one chunk needed at this size).

Dual-import in pipelines/pois.py: package-style for pytest, sibling-
style for direct script invocation matching walk_graph.py's pattern.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@taddyb
taddyb merged commit fa900f2 into main May 20, 2026
2 checks passed
@taddyb
taddyb deleted the feat/richer-geocoding-pr2 branch May 20, 2026 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant