Brain dump: OSM exports into the future

[spwoodcock]

@dulcetberg plans to do some volunteering, to assist with our OSM Export pipelines.

The following is a rough brain dump of info.

Context

• raw-data-api is a queueing system for OSM exports. It is attached to a large many terrabytes databases of OSM data imported every 2 minutes using osm2pgsql. People make requests, it queries the database, then returns a file format of geometries in e.g. geojson format.
• OpenStreetMap doesn't actually store geometries. Instead it stores nodes (points), ways (lines), and relations (that link up points and lines with metadata). The osm2pgsql import actually converts all the OSM data to PostGIS geometries.
• OSM Export Tool is due for a revamp. It currently uses raw-data-api to do the actual exports. We'll come back to this later.
• raw-data-api has two different queue systems. One is used for regular queries, and another is a priority queue for integrations with our tools such as Tasking Manager, Field Tasking Manager, etc.
• One other thing that raw-data-api handles is a cron/scheduled export of osm-country-exports, which end up in the Humanitarian Data Exchange (monthly exports, but frequency can be adjusted as needed).

Plan

• Based on and evolved from Long term maintenance plan: raw-data-api & export tool #300
• Basically we need to handle two types of OSM Exports: (1) large country scale exports (2) smaller every-day exports of smaller AOIs.
• For (1) we need a new system, described in Document new Windmill workflows approach, before deploy osm-country-exports#11. This uses DuckDB and the OSM planet.pbf to do country-side exports, via a Python task runner called Windmill. Our instance (I'll give you access).
• For (2) I think we can replace raw-data-api entirely. It was designed back in the days when there were no alternatives. It served it's purpose well, but we can probably scale it down & remove the very expensive AWS RDS database storing all that OSM data. I really like PostPass (it probably can't support country-scale exports, but its fine for most every-day use).

Step 1: General Exports via PostPass

• Finalise these PRs: Separate sql query builder & yaml parser package #304 feat: add PostPass query builder module to osm2pgsql-query-builder #305
• Once we have a usable Python package wrapper around PostPass, it can be installed into our tools that need it (Tasking Manager, FieldTM, Export Tool).
• We could also consider revamping osm-rawdata and putting the export logic there, overriding Simplify this package to be a Python API wrapper for raw-data-api osm-rawdata#50:
  1. Local postgres export
  2. PostPass export
  3. Geofabrik export that already exists - perhaps test it still works well
  4. raw-data-api export using raw-data-api-py logic (secondary, as this may be deprecated)
  5. (optional) we could even consider adding SliceOSM support as another provider, if this makes sense
• There is a good example async implementation of raw-data-api export here: https://github.com/kshitijrajsharma/geoml-toolkits/blob/master/src/geomltoolkits/downloader/osm.py
• The downside of bundling is all into osm-rawdata, is that if we need to install into an application that only needs PostPass, then that is entirely HTTP based, and requiring psycopg and it's C postgres driver is a bit overkill... modularity is probably preferred

Step 2: Export Tool Revamp

• Refactor Export Tool to be based on the PostPass package. If we queries are fast enough, we could consider making this frontend-only, removing a need for a queue system on our end, and even user accounts. Just simple (1) select AOI, data required, data format (2) click export (3) receive a file.
• If we really need to retain the existing queue system and users, then we migrate to a new simplified and modern system.

Step 3: Country Exports

• A lot of work has already been put into this by Kshitij (& some infra by Sam).
• This becomes a OSM planet.pbf + DuckDB via oex export, as described in the linked issue above.
• It runs via Windmill in our Kubernetes cluster.
• Each job publishes the data to the Humanitarian Data Exchange.
• We can ideally tweak the frequency to be daily, if needed during a disaster activation. But primarily it's monthly.
• Countries can be toggled on and off easily too.

Step 4: Tasking Manager Data Exports

• For TM, we need to export the resulting data after mapping is complete.
• We plan to use a similar pipeline to the Country Exports above.
• This is triggered on demand though, rather than scheduled.
• More details to come.

[dulcetberg]

Hey @spwoodcock,

Wanted to post an update on my understanding of the plan here, partly to make sure I actually have it right.

Today, raw-data-api answers every export request, small or large, out of its own PostGIS database, kept in sync with OSM every 2 minutes via osm2pgsql. That's the expensive part this whole plan is trying to get away from: running a full, always-on copy of planet-scale OSM data just in case someone asks for a small export.

The plan splits exports into two genuinely different problems instead of one problem at two sizes:

For small, everyday requests, hand them off to PostPass instead. It's a separate, free, public service Geofabrik already runs and maintains, and it already answers exactly this kind of query over plain HTTP. No giant database needed on our side at all.

For country-scale requests, that's too much to put on a shared public service like PostPass without hurting everyone else using it, so that stays on infrastructure HOT controls directly: read the full planet.pbf file with oex and DuckDB, run as jobs on the Kubernetes cluster through Windmill.

On Step 1 specifically: #304 and #305 build the actual PostPass query-building package. As of this week they're merged locally with the full test suite passing (42/42), and testing PostPass end to end (not just the unit tests) turned up a real bug: wrap_postpass_geojson strips the geom column PostPass needs to auto-detect the geometry, so every wrapped query comes back with a 400. Ken and I have been working through it on #305, he's reproducing it on his end now with the exact requests I sent him.

There's also a design question still open once that package is solid: should it stay a standalone package installed wherever it's needed, or fold into osm-rawdata as one of several interchangeable export backends per issue #50. Pulling in psycopg for something that only ever needs PostPass's pure HTTP path seems like real unnecessary weight, so modularity looks like the better direction to me, but I know that's not settled.

Let me know if I've still got anything backwards.

Best regards,
Brian

[spwoodcock]

Thanks for the summary - all sounds correct to me 👍

I'm thinking we can probably merge everything into osm-rawdata but just have dependency groups so people can install osm-rawdata[postgres] or something, for psycopg and local db functionality