Phase 38: Dual-Path Index Rebuild + Stale Detection - #41
Merged
Pascal-ZeGerman merged 11 commits intoAug 14, 2026
Conversation
The lazy `from ... import async_create as pn_create` inside _async_check_stale_and_rebuild's try block makes `pn_create` a function-local name for the whole method scope. Any exception raised before that import ran left the name unbound, so the `except` handler died with UnboundLocalError instead of posting its error notification. - Re-import pn_create inside the except handler (additive; keeps the sys.modules patchability the tests rely on) - Add regression test reproducing the path via a tz-naive _last_rebuilt Verified RED before the fix (UnboundLocalError at coordinator.py:1107), GREEN after. Full offline suite 818 passed.
…URCE async_request_rebuild() overwrote self._last_button_press with "now" before _async_decide_rebuild_path() read it, making the 24h double-press check a self-comparison that was always true. Every button press (including the first) was misrouted to the slow FROM_SOURCE path, defeating the dual-path routing feature. Fix threads the previous press snapshot through _async_do_rebuild -> _async_decide_rebuild_path as an explicit argument instead of re-reading the mutated instance attribute. Adds a real async_request_rebuild() end-to-end regression test (fired twice) that the prior isolated-stub tests could not catch.
_async_do_rebuild went straight from _sync_atomic_swap to posting a success notification without ever calling the existing _sync_verify_index() / IndexIntegrityError integrity check (already used by the first-time-setup path). A partial/corrupt write (disk full mid-write, SD-card corruption, truncated extraction) would be silently promoted to the live index with a false "success" message. Adds the verify call right after the swap so a raised IndexIntegrityError is caught by the existing except block and reported as a normal rebuild failure instead of an opaque rtree crash later. Updates test spy helpers in the affected suites to stub the new call.
_async_check_stale_and_rebuild posts the "index is stale, auto-rebuilding" notification then awaits async_request_rebuild(triggered_by="stale_check"). If the rebuild failed, _async_do_rebuild's except block dismissed only "asp_parking_index_rebuild" and posted "asp_parking_index_rebuild_error" -- it never dismissed "asp_parking_index_stale". Users ended up with a stale, now-inaccurate "auto-rebuilding" banner sitting alongside the new "Rebuild Failed" notification until the next 24h stale-check cycle. Dismissing it is idempotent (safe even when the notification was never posted, e.g. button-triggered rebuilds).
_sync_fetch_asp_signs paginated the SODA parking-signs endpoint with the same while-True/offset-increment shape as _sync_fetch_cscl_features but had no equivalent MAX_CSCL_PAGES-style cap. A misbehaving or compromised SODA endpoint that kept returning exactly SIGNS_BATCH_SIZE records would loop indefinitely in the executor thread. Adds MAX_SIGNS_PAGES (mirroring MAX_CSCL_PAGES) and breaks with a warning + partial results once the cap is hit, preserving the fetcher's existing fail-soft contract (unlike CSCL's fail-hard RuntimeError).
_sync_fetch_cscl_features guards its response shape
(features = body.get(...) if isinstance(body, dict) else []) but
_sync_fetch_asp_signs did not: `records = resp.json()` followed by
`if not records: break` then `for record in records:`. A truthy
non-list JSON body (e.g. {"error": "..."} on an HTTP-200 soft error)
made `not records` False, so the loop iterated the dict's string KEYS
and `record.get(...)` raised AttributeError -- an exception type not in
the caught tuple, turning the documented "SODA outages must NOT block a
CSCL rebuild" fail-soft contract (T-38-01-02) into a hard failure of the
entire from-source rebuild. Adds an isinstance(records, list) guard that
treats a non-list body as "no more data", mirroring the CSCL fetcher.
…ex backup _sync_cleanup_stale's crash-recovery path treats _bak as the LAST viable copy of the index when index_dir is absent. If os.replace(bak, index_dir) raised OSError (e.g. EXDEV when _bak and the parent dir unexpectedly end up on different filesystems, or a transient EBUSY), the handler unconditionally rmtree'd _bak -- permanently discarding a state that was often still recoverable via a plain copy. Adds a shutil.copytree fallback before giving up; _bak is wiped either way once the fallback is resolved (redundant on copy success, unusable on copy failure). Splits the old combined regression test into a copy-succeeds case (index recovered) and a both-fallbacks-fail case (bak wiped, index absent, matching the previous behavior only as a true last resort).
…ild invariants Retroactive Nyquist audit found the existing tests only checked file existence, not the invariants the threat model actually cares about: R-tree population (guards rtree bug #159's silent-empty-index failure mode), graph.json.zst round-tripping through StreetGraph, and manifest.json never picking up a heavy GIS dependency. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| # -- First press: no prior press exists ------------------------------ | ||
| await request_rebuild(triggered_by="button") | ||
| assert len(captured_coros) == 1 | ||
| await captured_coros[0] # run _async_do_rebuild -> _async_decide_rebuild_path |
| # -- Second press: within the 24h double-press window ---------------- | ||
| await request_rebuild(triggered_by="button") | ||
| assert len(captured_coros) == 2 | ||
| await captured_coros[1] |
…est_index_io.py CI's ruff format check flagged these two pre-existing files as unformatted. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pascal-ZeGerman
deleted the
gsd/phase-38-dual-path-index-rebuild-stale-detection
branch
August 14, 2026 03:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 38: Dual-Path Index Rebuild + Stale Detection
Goal: Users can choose between a fast prebuilt-index redownload (existing button) and a true from-source CSCL rebuild (new capability), and the integration auto-redownloads when the local index is stale (>60 days) so the index never silently rots.
Status: Verified ✓ (Nyquist-compliant, 96/96 automated tests green)
Adds a
_sync_build_from_source()pipeline that rebuilds the full NYC spatial index directly from the live CSCL GeoJSON + SODA ASP-signs APIs (pure shapely/rtree/httpx/zstandard — no geopandas, no newmanifest.jsondependency), smart coordinator routing that picks between a fast prebuilt-release download and a slow from-source rebuild based on remote-asset age and press history, and a daily stale-index check that auto-triggers a redownload plus a persistent HA notification when the index passes 60 days old.Changes
Plan 01 — From-source CSCL index rebuild (IDX-06)
Pure-shapely
_sync_build_from_source(index_dir): CSCL fetch → filter/reproject → node lookup → cross streets → SODA fetch → BFS ASP propagation → R-tree +segments.json→graph.json.zst→build_info.json, writing only to<index_dir>_tmp(caller owns the atomic swap). Also patches the existing download path to stampsource: "github_release"provenance (D-04) and silently skip malformedbuild_info.json(D-05).Key files:
custom_components/asp_parking/index_io.py,custom_components/asp_parking/const.py,tests/test_index_io_build_from_source.py,tests/fixtures/cscl_geojson_sample.json,tests/fixtures/soda_asp_signs_sample.jsonPlan 02 — Coordinator dual-path routing (IDX-05)
RebuildPathenum +_async_decide_rebuild_path(triggered_by): remote release <30 days old → download; ≥30 days stale, a double-press within 24h, or a GitHub API failure → from-source.triggered_by="stale_check"skips the 24h override. Path decision logged at INFO.Key files:
custom_components/asp_parking/coordinator.py,tests/test_coordinator_path_selection.pyPlan 03 — Stale detection (IDX-07)
Daily
async_track_time_intervalcheck againstbuild_info.json's age; >60 days triggers an auto rebuild + persistent notification (asp_parking_index_stale), guarded against first-install (_last_rebuilt is None) and concurrent rebuilds (_is_rebuilding).Key files:
custom_components/asp_parking/coordinator.py,tests/test_coordinator_stale.pyPost-verification fixes (CR-01, WR-01–05)
Code-review and Nyquist-audit follow-up commits: fixed button-triggered rebuild always routing to FROM_SOURCE (CR-01), added a pagination cap to the SODA ASP-signs fetcher (WR-02), guarded it against a non-list JSON body (WR-03), dismissed the stale-index notification on rebuild failure (WR-04), added copy-based recovery before discarding the last index backup (WR-05), verified a newly-swapped-in index before trusting it (WR-01), fixed a stale-check
UnboundLocalError, and closed 3 Nyquist coverage gaps (R-tree non-empty check,graph.json.zstround-trip viaStreetGraph.load, and an evergreen guard against heavy GIS deps re-enteringmanifest.json).Requirements Addressed
build_info.json, and atomically swaps the index directory using the Phase 33_sync_atomic_swap+SpatialIndex.reset()sequencebuild_info.json) at coordinator startup and during daily checks; auto-triggers a re-download (fast path) and notifies the user via persistent notificationVerification
.planning/phases/38-.../38-VERIFICATION.md)test_coordinator_path_selection.py,test_coordinator_stale.py,test_index_io_build_from_source.py,test_index_io.pyKey Decisions
GITHUB_INDEX_RELEASE_TAG = "index-v1"instead of/releases/latest— a research probe found/latestreturns a release with zero assets whileindex.ziplives on tagindex-v1.scripts/build_index.pypipeline in pure shapely rather than importing it — geopandas pulls in GDAL (~500MB), which would violate themanifest.json"no new external deps" constraint.build_info.jsoncases (absent, unparseable, non-dict) without failing the download path.TDD Audit
9daf1e4fix(38): WR-05 attempt copy-based recovery before discarding last index backupe1097e7fix(38): WR-03 guard SODA ASP-signs fetcher against non-list JSON body2044c0ffix(38): WR-02 add pagination cap to SODA ASP-signs fetcher56536dffix(38): WR-04 dismiss stale-index notification on rebuild failurefd3db64fix(38): WR-01 verify newly-swapped-in index before trusting it7a24238fix(38): CR-01 fix button-triggered rebuild always routing to FROM_SOURCE10c13d3fix(38-03): stale-check error handler crashed with UnboundLocalError0a2e557test(phase-38): add Nyquist validation tests for from-source index build invariantsAggregate: 0 skill, 0 fallback, 0 exempt — 8 missing. (No commit in this range carries a
gate_status:trailer; the underlying phase-38 execution/review-fix commits predate this repo's trailer convention, and this audit trail is informational only — it does not block the ship.)gate_status: skill=0, fallback=0, exempt=0, missing=8