Skip to content

fix(caldav): resolve iCloud cross-host calendar sharding (3.3.0-rc2) - #39

Merged
Pascal-ZeGerman merged 38 commits into
mainfrom
fix/caldav-icloud-cross-host-sharding
Aug 1, 2026
Merged

fix(caldav): resolve iCloud cross-host calendar sharding (3.3.0-rc2)#39
Pascal-ZeGerman merged 38 commits into
mainfrom
fix/caldav-icloud-cross-host-sharding

Conversation

@Pascal-ZeGerman

Copy link
Copy Markdown
Owner

Summary

Fix: iCloud CalDAV cross-host calendar sharding (3.3.0-rc2)

Every CalDAV write from the asp_parking integration to iCloud has been failing
since at least 2026-07-25 (19/19 attempts in the prior 7 days, 0 successes),
with:

ValueError: https://caldav.icloud.com/ can't be joined with
https://p117-caldav.icloud.com:443/278773852/calendars/.../

Root cause: iCloud shards each account's actual calendar data onto a
per-account host (e.g. p117-caldav.icloud.com) that differs from the
generic login entry point (caldav.icloud.com) the CalDAV client connects
with. caldav_sync._get_calendar() reconnected with the generic host and
called principal.calendar(cal_url=...), whose caldav 2.1.0 implementation
does a purely local URL.join() (caldav/lib/url.py) that raises
ValueError whenever the client's host and the target URL's host differ —
which is always true for iCloud accounts.

Changes

custom_components/asp_parking/caldav_sync.py

_get_calendar() still tries the cheap local principal.calendar(cal_url=...)
join first (no extra network round-trip — correct for same-host servers like
Radicale/Nextcloud/Baikal). On the specific cross-host ValueError, it now
falls back to principal.calendars() — a real network request that follows
iCloud's redirect to the correct sharded host — and matches the target
calendar by URL path, which stays stable even if iCloud rebalances the
shard in the future. Raises a clear CalDAVWriteError if the fallback also
finds no match, instead of the previous opaque ValueError.

tests/test_caldav_sync.py

Added 3 tests reproducing the exact production failure:

  • cross-host ValueError → fallback finds the matching calendar by path
  • cross-host ValueError → no path match → raises CalDAVWriteError
  • same-host regression guard: the fast local-join path never pays the extra
    principal.calendars() network round-trip

custom_components/asp_parking/manifest.json

Version bump 3.3.0-rc13.3.0-rc2.

Verification

  • New unit tests pass (3/3), reproducing the exact ValueError from
    production logs before the fix, passing after.
  • Full CalDAV test suite: 124/124 passed, no regressions.
  • Full project test suite: 988 passed (6 pre-existing, unrelated
    test_sign_retrieval.py failures confirmed via git stash to predate
    this change — network-sandboxed test environment issue).
  • Live verification against production iCloud account: ran
    write_or_update_event_get_calendardelete_event end-to-end
    inside the running Home_Assistant container with the real stored
    credentials and the real caldav_url/caldav_calendar pair that was
    previously failing. Write and cleanup both succeeded
    (WRITE_RESULT=OK uid_match=True, CLEANUP_RESULT=OK).
  • Deployed to the live container (docker cp + restart); confirmed
    asp_parking loads cleanly post-restart with zero CalDAV write failed warnings.

Key Decisions

  • Kept the cheap local principal.calendar(cal_url=...) join as the fast
    path rather than switching entirely to principal.calendars() matching,
    to avoid an extra network round-trip for the common self-hosted CalDAV
    case and to avoid touching the large existing test surface that mocks
    principal.calendar directly.
  • Matched by URL path (not full URL) in the fallback so the fix also
    survives a future iCloud shard rebalance that changes the hostname again.

Pascal-ZeGerman and others added 30 commits July 27, 2026 16:33
.planning/ is already gitignored; ROADMAP.md was a historical tracked
exception. Untracking it so it stops showing as locally-modified —
consistent with commit_docs: false, same as every other planning doc.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same reasoning as ROADMAP.md: .planning/ has been gitignored for a
while, but 40 files predating that rule stayed tracked (MILESTONES.md,
RETROSPECTIVE.md, v3.0 milestone docs, and SUMMARY.md files from
phases 13-39). Untracking brings them in line with commit_docs: false
and the rest of .planning/ — files remain on disk, only removed from
the git index.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Create docs/demo/index.html: semantic single-page shell (hero, map,
  readout, HA sensor card, calendar) with all 26 DOM contract ids for app.js
- Pin Leaflet 1.9.4 CSS/JS via unpkg CDN with verified SRI + crossorigin (T-41-03)
- Wire styles.css and deferred app.js (app.js created in 41-04)
- Add docs/.nojekyll so GitHub Pages serves files without Jekyll (Pitfall 6)
- No inline handlers or business logic; behavior lives in 41-04
- reprojection bounds (EPSG:2263 -> WGS84, [lon,lat] NYC bbox)
- weekly-pattern-not-absolute-datetime guard
- mock-sensor attribute keys subset of real HA sensor keys
- dataset completeness + status across schedule_found/resolution_failed/no_match
- Implement UI-SPEC palette (--bg/--surface/--border/--fg/--muted/--accent/
  --warning/--positive) and 8-point spacing scale as CSS custom properties
- Four type roles (Display/Heading/Body/Label-mono), two weights, system-ui + mono
- Explicit #map min-height 320/360/480px (mobile/tablet/desktop) so Leaflet never
  collapses to 0px (Pitfall 7)
- Responsive: single-column default, two-column ~60/40 sticky panel at >=1024px
- State chips (positive/warning/neutral) and accent reserved to CTA, active profile,
  next-move calendar cell, and links only
- Calendar highlight fade-in + scale + ring-pulse, all disabled under
  prefers-reduced-motion (static highlight only)
- >=44px touch targets on profile/mode/copy buttons
- reproject_wkt_to_wgs84: EPSG:2263 -> WGS84 [lon,lat] GeoJSON order
- build_sensor_shapes: mock next_move + resolved_street, subset of real HA keys
- build_point_entry: explicit dict build, weekly pattern (no absolute datetime)
- dump_point: per-point fail-soft over 5 resolver exceptions
- main: DEMO_POINTS -> demo.json + demo-segments.geojson; never serializes token
…ndar)

- Leaflet map wiring: OSM tiles, keyboard-focusable accent pins, segment
  overlay, invalidateSize (Pitfall 7)
- Loads committed demo.json/demo-segments.geojson via response.json() only;
  reveals #error-state on fetch/parse failure
- selectPoint resolves a block; renderReadout + renderHaCard render ALL
  dataset text via textContent/DOM node creation (T-41-05, no innerHTML/eval)
- computeNextMove mirrors find_next_window: 8-day lookahead, America/New_York
  wall clock, JS Sun=0 -> Python Mon=0 conversion
- renderCalendar highlights the client-recomputed next-move day (is-next /
  is-today), CSS-driven animation honors prefers-reduced-motion
- Profile radiogroup re-renders all surfaces with no network call
- Demo/full mode toggle; FULL_RESOLVER_ENDPOINT null by default -> inert
  full-resolver branch with graceful not-configured message
- copy-YAML via navigator.clipboard
- docs/demo/data/demo.json: 7 hand-picked NYC points resolved via resolve_asp
  against the local spatial index + live SODA (generation_date 2026-07-28)
- docs/demo/data/demo-segments.geojson: matched segment geometry reprojected
  EPSG:2263 -> WGS84, NYC-bounded FeatureCollection
- Weekly schedule pattern stored (no absolute next-move date); mock HA sensor
  shapes are a subset of real sensor attribute keys
- Non-schedule sensor states present (NoSegmentFoundError / resolution_failed)
- No NYC app token/secret serialized (T-41-01 grep clean)

Deviation: Prospect Pl (40.677629,-73.968527) resolves to NORTH, not South as
the plan asserted. Verified via authoritative NYC curb calibration
(segment 39223: center_offset_c=-3.04 ft, calibrated=True, curb_width 33.1 ft);
point signed_offset=+9.16 ft, firmly North of the calibrated centre. The plan's
'South' ground truth is a factual error; no calibrated index flips it.
- New .github/workflows/pages.yml with build + deploy jobs
- Runs on GitHub-hosted ubuntu-latest (Pages deploy needs GitHub runners)
- Publishes committed docs/ tree as-is; no in-CI precompute (Pitfall 3)
- Least-privilege permissions (pages: write, id-token: write, contents: read)
- Pinned actions: checkout@v4, configure-pages@v5, upload-pages-artifact@v3, deploy-pages@v4
- Add Live Demo section + Table-of-Contents entry
- Explain the static docs/demo/ page (map click -> rule, next move, HA sensors, calendar)
- Note the committed dated snapshot (2026-07-28) and that it is not live
- Give the exact regenerate command and index build/download options
- Document local http.server run and GitHub Pages hosting paths
…w weekly data

Code review found 6 of 7 demo points hard-failed (NoSegmentFoundError/
resolution_failed), including the "Car B" profile toggle target — leaving
only Prospect Pl functional. Replaced the speculative coordinates with
verified-resolving real NYC points across 4 boroughs (Williamsburg, Astoria,
Bronx Grand Concourse, Staten Island), plus a genuine currently-active
cleaning window (Oriental Blvd) found via targeted probing. outside_coverage
is kept as an intentional failure (by design, outside NYC bounds).

Also fixes build_point_entry() to populate weekly/summary for the
ASPActiveNow status (previously only ScheduleFound was handled, so any
active-now point would have rendered an empty calendar), and adds a
build-time check that every DEMO_PROFILES target actually resolved before
writing output — the root-cause enabler that let the broken data ship
silently in the first place.

A genuine "no_asp" (confirmed no restrictions) point was not found despite
~2,900 probes across 4 methods, including bypassing the confidence gate
entirely — this looks structural (the live SODA endpoint may not return
NoASPSigns in practice) rather than a sampling gap. Documented as a known
limitation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…w sensors

build_sensor_shapes()/_cleaning_day_names() had the same ScheduleFound-only
gap as the previous build_point_entry() fix (0629ee0) — the mock HA sensor's
cleaning_days and schedule_summary attributes were silently dropped for any
point in the ASPActiveNow status, even though weekly/summary were correctly
populated at the top level. Extends both to also handle ASPActiveNow.

Adds test_asp_active_now_populates_weekly, a regression test for both gaps.
Also renames the oriental_blvd demo point key: the underlying live cleaning
window closed between generation runs and its committed status flipped to
schedule_found, so "_active_now" in the key name was no longer accurate — an
expected consequence of asp_active_now being a point-in-time status baked
into a dated snapshot, not a bug (the weekly-pattern-derived calendar display
recomputes correctly client-side regardless of which status label was frozen
at generation time).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- SRI-pinned Leaflet 1.9.4 CSS+JS copied verbatim from docs/demo
- coverage-QA hero copy (not the demo's sales hero, D-15)
- #map region + always-visible labeled #legend (four tiers, hue + text label, D-16/Prohibition 3)
- four AND-composable filters: borough, tier, SODA level (0/1/2/3, L4 folded into L3 per D-18), street search (R4/D-07)
- #export-geojson control (R5), #no-results (R4) and #error-state (R2) regions, #data-freshness footer node (D-17)
- no date/sign_type filter (D-08); docs/demo/ untouched (R6)
- RED scaffold for the coverage-dataset dumper's deterministic core
- test_grouping_key_and_side_derivation: bearing-derived sides + normalize_to_soda canonical group key
- test_tier_boundary_partition: half-open [0,1] partition into high/medium/low/unresolved + confidence_for_level
- fails with ModuleNotFoundError until scripts.build_coverage_dataset lands (GREEN in task 2)
- reuse docs/demo design tokens, reset/base, and typography roles verbatim (D-15 polish parity)
- add red->green tier color scale: high=--positive, medium teal, low=--warning, unresolved red #e5484d (D-09)
- document per-tier marker radius convention (unresolved largest -> high smallest) as 42-04's third colorblind channel
- style always-visible labeled legend, four-filter panel (44px touch targets), export button, no-results + error states
- dark theme retained; docs/demo/styles.css untouched (R6)
- derive_segment_sides: bearing-derived {N,S}/{E,W} candidate curbs (D-02, geometry not has_asp flags)
- group_key: (normalize_to_soda(street), side) canonical dedup key (D-01)
- confidence_for_level + CONFIDENCE_BY_LEVEL: SODA level -> confidence (D-18, 1/2/3/0 -> .90/.66/.40/.00)
- tier_for_confidence + TIER_BOUNDS: one documented half-open [0,1] partition (0.33->low, 1.0->high)
- reproject_wkt_to_wgs84, segment_midpoint_wgs84 (single-midpoint, Pitfall 5), _borough_name, lazy _load_segments
- main() stub raises NotImplementedError (SODA resolve pipeline lands in 42-02); no compute_confidence/resolve_asp import (Pitfall 2)
…oring + render

- Add docs/explorer/app.js plain-ES controller (use strict, textContent-only)
- tierForConfidence mirrors 42-01 half-open TIER_BOUNDS (0.33/0.50/0.75 boundaries)
- colorForTier (hue) + radiusForTier (colorblind size channel, unresolved largest)
- initMap uses preferCanvas:true + one shared L.canvas() renderer, citywide view
- loadDataset shows visible error-state on failure (never a blank map, R2)
- Stamp generation_date freshness; export tierForConfidence/colorForTier/radiusForTier
…tate

- Add buildPopup(point) returning a DOM node (textContent/createElement only)
- Shows street + cross streets, confidence + shared tier label, SODA level,
  schedule summary + weekly cleaning times (schedule states only)
- Explicit 'unresolved'/'no ASP sign' copy for no-match states; never a
  confirmed-clear reading for a coverage gap (Prohibition 2 / T-42-04)
- Street View + FreeNYC link-outs, both target=_blank rel=noopener (T-42-06);
  NYC DOT link dropped (no stable URL, D-05)
- Wire buildPopup into renderMarkers via lazy bindPopup; export buildPopup
…no-match R1 tests

- test_query_count_is_grouped: one SODA fetch per distinct (street,side) group
- test_every_segment_has_entry: exactly one entry per input segment_id
- test_zero_record_group_no_match: empty group yields explicit no_match/lv0 entry
- add in-memory _seg fixture + call-recording _StubClient (no network, no token)
- Add applyFilters(): AND logic over borough/tier/SODA-level/street-search on
  the plain points array; redraws the filtered subset onto the shared canvas
  layer group; toggles the #no-results state at zero results (R4)
- Tier filter reuses the shared tierForConfidence (single rule with coloring)
- Street search is case-insensitive substring on full_street_name (st)
- Add pure buildFeatureCollection (features:[] when empty) + exportGeoJSON that
  returns the FC and, in-browser, downloads via Blob (application/geo+json) (R5)
- Wire filter events + export button; initial render via applyFilters
- Export applyFilters/buildFeatureCollection/exportGeoJSON for node tests
- resolve_group: one broad build_on_street_query/fetch_signs per (street,side)
  group, fail-soft to [] on error (Pitfall 4)
- cross_streets_match wraps signs._cross_streets_match (variant+swap+empty guard)
- resolve_side assigns soda_level by match precision (0/1/2/3) and runs the
  materialize_cached_records -> compute_schedule pipeline (no resolve_asp)
- build_coverage dedups to one query per distinct group, picks worst-case side
  per segment (D-13), emits one entry per segment (no omission)
- build_segment_entry emits the canonical compact schema keys
Pascal-ZeGerman and others added 5 commits July 31, 2026 20:18
- test_no_token_in_output: serialized dataset carries neither the SODA token
  env-var name nor a token-shaped value; entry has no credential field; a
  matching broom record yields a schedule_found entry whose wk stores {d,s,e}
  only (no raw sign text, D-04)
- test_main_writes_canonical_coverage_json: main() writes coverage.json offline
  (stubbed client) with the canonical top-level keys and exact per-entry schema
…ndar

iCloud shards each account's calendar data onto a per-account host
(e.g. p117-caldav.icloud.com) that differs from the generic login
entry point (caldav.icloud.com) the client connects to. caldav 2.1.0's
Principal.calendar(cal_url=...) does a purely local URL.join() that
raises ValueError whenever the two hosts differ — which was always,
for iCloud — causing every CalDAV write to fail with:

  ValueError: https://caldav.icloud.com/ can't be joined with
  https://p117-caldav.icloud.com:443/.../calendars/...

_get_calendar now falls back to principal.calendars() (a real network
call that follows the redirect) and matches by URL path when the cheap
local join raises. Confirmed fixed against the live iCloud account:
write + delete now succeed end-to-end.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
# Worst-case = LOWER confidence (D-13); ties keep the first side.
if best is None or cf < best[0]:
best = (cf, soda_level, side, schedule)
assert best is not None # every segment has at least one side
Pascal-ZeGerman and others added 3 commits August 1, 2026 15:14
The cross-host fallback in _get_calendar caught any ValueError from
principal.calendar(), not just the intended URL.join host-mismatch
case, silently misreporting unrelated errors (e.g. a None client) as
"no calendar found". It also discarded the original exception with
`from None`, losing the real cause from the traceback.

Now only the specific "can't be joined with" ValueError triggers the
fallback; anything else propagates unchanged, and the resulting
CalDAVWriteError chains the original exception via `from exc`.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CI's ruff format check failed on the newly added cross-host fallback
tests (long single-line SimpleNamespace/ValueError calls).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@Pascal-ZeGerman
Pascal-ZeGerman merged commit 67f1fa4 into main Aug 1, 2026
17 checks passed
@Pascal-ZeGerman
Pascal-ZeGerman deleted the fix/caldav-icloud-cross-host-sharding branch August 1, 2026 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants