Skip to content

OSM/HydroLAKES layers, opt-in rivers/ponds, and nearest-water API - #16

Merged
dillonstreator merged 7 commits into
mainfrom
feat/osm-hydrolakes-dataset
Aug 1, 2026
Merged

OSM/HydroLAKES layers, opt-in rivers/ponds, and nearest-water API#16
dillonstreator merged 7 commits into
mainfrom
feat/osm-hydrolakes-dataset

Conversation

@dillonstreator

@dillonstreator dillonstreator commented Jul 25, 2026

Copy link
Copy Markdown
Member

Summary

Re-source water coverage from upstream OSM and HydroLAKES, make layers opt-in per host, add optional inland rivers/ponds, and ship a nearest-shoreline API with a map demo control.

1. Replace geo-maps with OSM oceans/seas + HydroLAKES

The @geo-maps/earth-waterbodies-1m package is unmaintained (~2017 OSM vintage). Data now comes from upstream:

Lake Superior, the Caspian Sea, and ~90k other lakes now return water: true. Dataset update checks poll OSM Last-Modified / ETag instead of the npm registry.

2. Opt-in coverage layers

Coverage is {feature}:{precision} via WATER_LAYERS:

WATER_LAYERS=oceans:medium,lakes:medium   # default
WATER_LAYERS=oceans:full,lakes:high       # finer shorelines, more lakes
WATER_LAYERS=oceans                       # oceans only

data/layers.json declares delivery (bundled, download, or range). WATER_LAYERS_REGISTRY can point at a custom registry.

3. Query FlatGeobuf’s packed R-tree directly

GDAL already writes a Hilbert R-tree into the FGB; the old in-memory RBush rebuild was redundant. Lookups use a point-sized bbox against the file index.

before after
resident memory ~1049 MB ~294 MB (oceans+lakes medium)
startup ~2.0 s ~554 ms

4. Optional rivers & ponds (Geofabrik extracts)

Global rivers:medium and ponds:medium are built from continent PBFs and published as GitHub release assets (data-v1). They stay opt-in (delivery: download); default coverage remains oceans + lakes.

5. GET /api/nearest + demo map control

  • Query: lat, lon, optional count (1–25), type, maxKm
  • Response: shoreline snaps ranked by distance, then area
  • Homepage: on land results, a Leaflet Find nearest water control plots hits and fits bounds

Notes for the reviewer

  • isOnWater() is async and returns layer. water: false means “not in any enabled layer”; use GET /api/layers for what an instance has loaded.
  • Nearest hits are shoreline coordinates — they may not return water: true from /api/water (ring is outside polygon fill).
  • Oceans/lakes medium stay bundled; rivers/ponds download from the data-v1 release.

Test plan

  • pnpm test
  • pnpm exec tsc --noEmit / pnpm build
  • Layer selector + opt-in runtime tests
  • Nearest API + homepage content assertions
  • Confirm nightly dataset-update workflow on CI runner memory
  • Smoke the demo: land click → Find nearest water → pins + JSON

dillonstreator and others added 2 commits July 25, 2026 09:28
The @geo-maps/earth-waterbodies-1m package is unmaintained and its
OpenStreetMap vintage is ~2017, so shorelines had drifted and inland
water was effectively unusable. Source directly from upstream instead:
osmdata.openstreetmap.de water polygons for oceans and seas, and
HydroLAKES v1.0 (>= 2 km2) for inland lakes and reservoirs.

Both layers are merged with ogrmerge and Douglas-Peucker simplified at
0.003 degrees so the gzipped FlatGeobuf stays under GitHub's 100 MB
blob limit. The dataset-update workflow now polls the OSM Last-Modified
and ETag headers rather than the npm registry.

Co-authored-by: Cursor <cursoragent@cursor.com>
Coverage is now addressed as {feature}:{precision} pairs selected with
WATER_LAYERS, so a host can trade artifact size for finer shorelines and
smaller water bodies. A feature listed twice collapses to the highest
precision requested, and the registry in data/layers.json declares how
each artifact is delivered: bundled in the repo, downloaded once at boot
and checksum-verified, or queried in place over HTTP range requests.
Hosts can point WATER_LAYERS_REGISTRY at their own registry to self-host
artifacts this project does not ship.

Startup no longer rebuilds an RBush over every polygon. GDAL already
writes a packed Hilbert R-tree into the FlatGeobuf files, so lookups
query it with a point-sized bbox and parse only the candidates whose
envelope contains the coordinate. That drops resident memory from about
1049 MB to 294 MB and startup from roughly two seconds to 554 ms, with
no change to results, and it makes each additional layer cost roughly
its file size rather than hundreds of megabytes of heap.

Because coverage now varies by deployment, a false result is no longer
self-describing. Water responses report the layer that matched, and
GET /api/layers reports which layers an instance has enabled.

The single merged waterbodies.fgb.gz is replaced by per-layer artifacts
under data/layers/, and manifest.json is folded into the registry. Build
metadata is recorded beside each cached source zip at download time so
the registry describes the bytes actually built from rather than
whatever upstream published since.

Also stop copying the ~1.7 GB of cached source zips into dist/ and the
Docker image, which nothing at runtime reads.

Co-authored-by: Cursor <cursoragent@cursor.com>
@dillonstreator dillonstreator changed the title Replace geo-maps with OSM oceans/seas and HydroLAKES inland lakes Re-source waterbodies from OSM + HydroLAKES and make coverage opt-in Jul 25, 2026
Riverbank / river / canal area polygons and small inland water bodies
(≤ 2 km²) are extracted from Geofabrik continent OSM PBFs via GDAL's
OSM driver, then published as rivers:{precision} and ponds:{precision}
artifacts selectable with WATER_LAYERS.

The medium tiers are bundled (≈20 MB rivers, ≈29 MB ponds) for the
continents built so far: Antarctica, Central America, Australia-Oceania,
and South America. Remaining continents (Africa, Europe, Asia, North
America) can be merged with GEOFABRIK_REGIONS once Docker has enough
disk; the registry scope field records which regions each artifact
covers so a false result stays interpretable.

Co-authored-by: Cursor <cursoragent@cursor.com>
@dillonstreator

Copy link
Copy Markdown
Member Author

Rivers & ponds follow-up

Added opt-in rivers and ponds layers sourced from Geofabrik continent OSM PBFs (GDAL OSM driver → FlatGeobuf):

Layer What it covers Medium artifact
rivers OSM water=river / water=canal / waterway=riverbank area polygons 58k features, 19.5 MB
ponds Other inland natural=water / reservoirs in 0.01–2 km² (complements HydroLAKES) 325k features, 28.8 MB

Enable with WATER_LAYERS=oceans,lakes,rivers,ponds (default stays oceans+lakes).

Coverage caveat: the bundled medium artifacts currently include Antarctica, Central America, Australia-Oceania, and South America. Africa / Europe / Asia / North America PBFs are cached locally but the Africa extract OOMed Docker after filling the C: volume; re-run GEOFABRIK_REGIONS=… pnpm dataset:build:inland once Docker has disk headroom to extend the same artifacts. The registry scope field lists which regions each artifact covers.

Stream centerlines remain out of scope — only mapped riverbank polygons count, which is why a point on a narrow stream may still return water: false.

Full-planet inland layers exceed git limits (~138 MB / ~820 MB), so they
are removed from the repo, registered as download delivery, and served
from the data-v1 GitHub Release. Inland build script gains streaming
gzip for >2 GiB FGBs and Docker Desktop mount support.

Co-authored-by: Cursor <cursoragent@cursor.com>
@cursor

cursor Bot commented Aug 1, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@dillonstreator

Copy link
Copy Markdown
Member Author

Global inland layers

Rivers and ponds now cover all 8 Geofabrik continents and are published as release assets (too large for git):

delivery: download in data/layers.json. Oceans/lakes remain bundled. Default WATER_LAYERS is still oceans+lakes only; rivers/ponds stay opt-in.

Land checks can discover nearby shoreline hits ranked by distance, with the homepage button mounted as a Leaflet map control.

Co-authored-by: Cursor <cursoragent@cursor.com>
@dillonstreator dillonstreator changed the title Re-source waterbodies from OSM + HydroLAKES and make coverage opt-in OSM/HydroLAKES layers, opt-in rivers/ponds, and nearest-water API Aug 1, 2026
dillonstreator and others added 2 commits July 31, 2026 21:07
Global ponds:medium is too large to download and load within the suite timeout on GitHub Actions.

Co-authored-by: Cursor <cursoragent@cursor.com>
Use t.test({ skip }) so tsc and docker build accept the CI skip path.

Co-authored-by: Cursor <cursoragent@cursor.com>
@dillonstreator
dillonstreator merged commit 2a599db into main Aug 1, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant