OSM/HydroLAKES layers, opt-in rivers/ponds, and nearest-water API - #16
Conversation
The @geo-maps/earth-waterbodies-1m package is unmaintained and its OpenStreetMap vintage is ~2017, so shorelines had drifted and inland water was effectively unusable. Source directly from upstream instead: osmdata.openstreetmap.de water polygons for oceans and seas, and HydroLAKES v1.0 (>= 2 km2) for inland lakes and reservoirs. Both layers are merged with ogrmerge and Douglas-Peucker simplified at 0.003 degrees so the gzipped FlatGeobuf stays under GitHub's 100 MB blob limit. The dataset-update workflow now polls the OSM Last-Modified and ETag headers rather than the npm registry. Co-authored-by: Cursor <cursoragent@cursor.com>
Coverage is now addressed as {feature}:{precision} pairs selected with
WATER_LAYERS, so a host can trade artifact size for finer shorelines and
smaller water bodies. A feature listed twice collapses to the highest
precision requested, and the registry in data/layers.json declares how
each artifact is delivered: bundled in the repo, downloaded once at boot
and checksum-verified, or queried in place over HTTP range requests.
Hosts can point WATER_LAYERS_REGISTRY at their own registry to self-host
artifacts this project does not ship.
Startup no longer rebuilds an RBush over every polygon. GDAL already
writes a packed Hilbert R-tree into the FlatGeobuf files, so lookups
query it with a point-sized bbox and parse only the candidates whose
envelope contains the coordinate. That drops resident memory from about
1049 MB to 294 MB and startup from roughly two seconds to 554 ms, with
no change to results, and it makes each additional layer cost roughly
its file size rather than hundreds of megabytes of heap.
Because coverage now varies by deployment, a false result is no longer
self-describing. Water responses report the layer that matched, and
GET /api/layers reports which layers an instance has enabled.
The single merged waterbodies.fgb.gz is replaced by per-layer artifacts
under data/layers/, and manifest.json is folded into the registry. Build
metadata is recorded beside each cached source zip at download time so
the registry describes the bytes actually built from rather than
whatever upstream published since.
Also stop copying the ~1.7 GB of cached source zips into dist/ and the
Docker image, which nothing at runtime reads.
Co-authored-by: Cursor <cursoragent@cursor.com>
Riverbank / river / canal area polygons and small inland water bodies
(≤ 2 km²) are extracted from Geofabrik continent OSM PBFs via GDAL's
OSM driver, then published as rivers:{precision} and ponds:{precision}
artifacts selectable with WATER_LAYERS.
The medium tiers are bundled (≈20 MB rivers, ≈29 MB ponds) for the
continents built so far: Antarctica, Central America, Australia-Oceania,
and South America. Remaining continents (Africa, Europe, Asia, North
America) can be merged with GEOFABRIK_REGIONS once Docker has enough
disk; the registry scope field records which regions each artifact
covers so a false result stays interpretable.
Co-authored-by: Cursor <cursoragent@cursor.com>
Rivers & ponds follow-upAdded opt-in
Enable with Coverage caveat: the bundled medium artifacts currently include Antarctica, Central America, Australia-Oceania, and South America. Africa / Europe / Asia / North America PBFs are cached locally but the Africa extract OOMed Docker after filling the C: volume; re-run Stream centerlines remain out of scope — only mapped riverbank polygons count, which is why a point on a narrow stream may still return |
Full-planet inland layers exceed git limits (~138 MB / ~820 MB), so they are removed from the repo, registered as download delivery, and served from the data-v1 GitHub Release. Inland build script gains streaming gzip for >2 GiB FGBs and Docker Desktop mount support. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
Global inland layersRivers and ponds now cover all 8 Geofabrik continents and are published as release assets (too large for git):
|
Land checks can discover nearby shoreline hits ranked by distance, with the homepage button mounted as a Leaflet map control. Co-authored-by: Cursor <cursoragent@cursor.com>
Global ponds:medium is too large to download and load within the suite timeout on GitHub Actions. Co-authored-by: Cursor <cursoragent@cursor.com>
Use t.test({ skip }) so tsc and docker build accept the CI skip path.
Co-authored-by: Cursor <cursoragent@cursor.com>
Summary
Re-source water coverage from upstream OSM and HydroLAKES, make layers opt-in per host, add optional inland rivers/ponds, and ship a nearest-shoreline API with a map demo control.
1. Replace
geo-mapswith OSM oceans/seas + HydroLAKESThe
@geo-maps/earth-waterbodies-1mpackage is unmaintained (~2017 OSM vintage). Data now comes from upstream:Lake Superior, the Caspian Sea, and ~90k other lakes now return
water: true. Dataset update checks poll OSMLast-Modified/ ETag instead of the npm registry.2. Opt-in coverage layers
Coverage is
{feature}:{precision}viaWATER_LAYERS:data/layers.jsondeclares delivery (bundled,download, orrange).WATER_LAYERS_REGISTRYcan point at a custom registry.3. Query FlatGeobuf’s packed R-tree directly
GDAL already writes a Hilbert R-tree into the FGB; the old in-memory RBush rebuild was redundant. Lookups use a point-sized bbox against the file index.
4. Optional rivers & ponds (Geofabrik extracts)
Global
rivers:mediumandponds:mediumare built from continent PBFs and published as GitHub release assets (data-v1). They stay opt-in (delivery: download); default coverage remains oceans + lakes.5.
GET /api/nearest+ demo map controllat,lon, optionalcount(1–25),type,maxKmNotes for the reviewer
isOnWater()is async and returnslayer.water: falsemeans “not in any enabled layer”; useGET /api/layersfor what an instance has loaded.water: truefrom/api/water(ring is outside polygon fill).data-v1release.Test plan
pnpm testpnpm exec tsc --noEmit/pnpm builddataset-updateworkflow on CI runner memory