Overhaul Overpass usage: batched queries, server status probing, background prefetch - #93
Open
shizumaat wants to merge 2 commits into
Open
Overhaul Overpass usage: batched queries, server status probing, background prefetch#93shizumaat wants to merge 2 commits into
shizumaat wants to merge 2 commits into
Conversation
Each vector layer (big_roads, small_roads, water, airports) previously issued one HTTP request per query statement -- around 16 Overpass transactions per tile, each acquiring its own rate-limit slot, paying a fresh TCP/TLS handshake, and re-downloading nodes shared between statements through repeated recurse-down. On busy public servers this multiplied the chances of 429/timeout denials. Layers also downloaded strictly inline, so all network waits added directly to build time. O4_OSM_Utils.py: - OSM_queries_to_OSM_layer now collects every statement not satisfied from cache and downloads them in ONE Overpass QL union request per layer (a tile now makes 4 requests instead of ~16); the recurse-down runs once over the union so shared nodes transfer once. - Before the first download of a session (and after any failure), all candidate servers' /api/status endpoints are probed in parallel -- nearly free for the server -- and the real query goes to the most available one: free slot for our IP and fastest probe answer, else soonest slot to free up. - Server selection is sticky-with-failover: keep the server that last answered (preserves keep-alive and the rate-limit slot already held), reprobe only after a failure; explicit server pinning still honoured. - One persistent requests.Session (connection keep-alive) instead of a new session per request. - POST instead of splicing the query into the URL, per Overpass API guidance for generated queries. - Explicit [timeout:180] setting sent to the server, and the client read timeout raised to outlast it (previously 60 s, which gave up on queries the server was still legitimately computing). - Detect mid-stream server aborts (<remark> trailers on runtime timeout / memory exhaustion) that were previously accepted as complete answers, silently truncating data. - Retry-After parsed defensively (may be an HTTP-date) and capped at 120 s, since the next attempt rotates to a different server. - Full console feedback: every attempt is announced before it is sent, a reassurance line prints every 10 s while a slow request is in flight, and the GUI stop button works both mid-request and during backoff waits (sleeps sliced per second, in-flight waits abandoned immediately). O4_Vector_Map.py: - As soon as the airports layer (the only download the next pipeline stages need immediately) has arrived, a background thread prefetches the tile's remaining missing layer caches (big_roads, small_roads, coastline, water -- honouring road_level gates and custom coastline/water data), sequentially, one request at a time: the same server load as the old inline order, just overlapped with airport processing instead of after it. The include_* encoders join the prefetch and then recycle the cache exactly as before. Each layer's statements/tags are extracted to module-level specifications shared by the prefetch and the encoders, so the cache a prefetch writes is exactly the cache the encoder would have written. Per-layer caching, cache file formats, and all public signatures are unchanged. Verified live on tile +14-024: one union request yields node-for-node identical layer contents to six individual requests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
|
These proposed changes make sense but in practice, I'm not seeing it working very well. The issues I see:
At least for me, the existing logic works better. Examples of trying to use these changes: First attempt: Second attempt: I reverted to the original logic and although I see a couple rejections, switching to another server seems to work pretty well and the build process continues. |
* Accumulating per-request failure exclusion: every server that failed for the current request is excluded until the pool is exhausted, then the round resets (a status probe can call a server available while it answers queries with garbage or timeouts). * A failed attempt drops the stickiness of the server it failed on, so the first attempt of the NEXT request no longer returns to a server whose previous request just timed out; stickiness is re-earned by a success. * A short rate-limit penalty (Retry-After <= 30 s, or, when the 429 carries no Retry-After, the server's own /api/status next-slot time) is sat out on the SAME server instead of rotating to a slower mirror. A probe-derived slot time binds only that same-server wait: when it is too long to sit out, it is discarded and the rotation happens on the plain exponential backoff, since that server's slot time says nothing about the one we rotate to. * Escalating patience: attempts 1-2 declare [timeout:60] and wait 75 s, attempts 3+ declare [timeout:180] and wait 210 s, so a hung mirror costs ~1 min rather than 3.5 and no server is asked for more time than we will actually wait. * /api/status probes get one parallel retry round for the servers that answered nothing (transient 504s / timeouts), and the probe timeout is shortened to (3, 5) s connect/read. Headless tests for all five in tests/test_osm_server_selection.py. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author
|
Thanks for the detailed logs — you're right on both points, and they pinned down real defects. The status probes are flakier than assumed (I measured ~1-in-8 transient timeouts on the mirrors), and a probe failure wrongly removed a server from consideration entirely; worse, a server that failed was still "sticky" for the next layer's request, and the 210 s read timeout made every wrong pick cost 3.5 minutes — which is why the old logic felt better. I've pushed a fix (22daf4b): failed servers are excluded for the whole request, stickiness is dropped on failure, silent probes get one retry, the first two attempts use a 60 s window so a hung server costs ~1 min, and a short rate-limit (like the DE 429 in your first log, slot free in ~2 s) is now waited out on the same fast server instead of rotating to a slow mirror. Would appreciate a retest on the same tile. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Each vector layer (big_roads, small_roads, water, airports) issues one HTTP request per query statement — around 16 Overpass transactions per tile. Every request acquires its own rate-limit slot on the server, pays a fresh TCP/TLS handshake, and re-downloads nodes shared between statements (the
(._;>>;);recurse-down runs per request, so e.g. an intersection node between a primary and a secondary road transfers twice). On busy public servers this multiplies the chances of 429s and timeouts, and all downloads run strictly inline, so every network wait adds directly to build time.Changes
O4_OSM_Utils.pyOSM_queries_to_OSM_layercollects every statement not satisfied from cache and downloads them in a single Overpass QL union request — a tile now makes 4 requests instead of ~16, and the recurse-down runs once over the union so shared nodes transfer once./api/statusendpoints are probed in parallel — nearly free for the server — and the real query goes to the most available one: a server with a free slot for our IP (fastest probe answer wins), else the one whose next slot frees soonest. This also preemptively avoids 429s the status page already predicts.overpass_server_choiceis honoured as before.requests.Session(connection keep-alive) instead of a new session per request.[timeout:180]sent to the server, with the client read timeout raised to outlast it (previously 60 s, which gave up on queries the server was still legitimately computing).<remark>trailers on runtime timeout / memory exhaustion) were previously accepted as complete, silently truncating data; they now retry.Retry-Afterparsed defensively (the header may be an HTTP-date) and capped at 120 s, since the next attempt rotates to a different server anyway.OSM server DE (big_roads) is working on our request (20s)...), and the GUI stop button now works both mid-request and during backoff waits.O4_Vector_Map.pyroad_levelgates and custom coastline/water data — sequentially, one request at a time. Same server load as the old inline order, just overlapped with airport processing instead of serialized after it. Theinclude_*encoders join the prefetch and then recycle the cache exactly as before.Compatibility
Per-layer caching, cache file formats,
overpass_servers.txt, and all public function signatures are unchanged; no other call sites needed changes.Verification
🤖 Generated with Claude Code