Plan: moczarr — sparse-DGGS xarray reader for morton-hive stores

[espg]

🤖 from Claude

Plan: moczarr — sparse-DGGS xarray reader for morton-hive stores

The O3 standalone extension from englacial/zagg#198 (resolution comment), and the Phase-2 gate deliverable for englacial/zagg#251. moczarr reads morton-hive stores (the convention frozen on espg/mortie#62, written by zagg) into lazy xarray Datasets, with optional xdggs integration via its public grid registry.

What it is

Three layers (from the O3 resolution):

1. Core reader (no xdggs dependency): moczarr.open_hive(store_root, aoi=..., window=...) → lazy xr.Dataset. Flow per sparse_coverage.md §5: manifest GET → root coverage.moc ∩ AOI → arithmetic hive paths → stamped-leaf opens (tier-0 box / bitmap-sidecar AOI reject) → concat cells (+ time across windowed leaves). LIST-walk as fallback/discovery only.
   Acceptance (the zagg#251 Phase-2 criterion): as ergonomic as xr.open_zarr() for the debug/interop case.
2. xdggs layer (moczarr[xdggs] extra): MortonInfo(DGGSInfo) + MortonIndex(DGGSIndex) registered via register_dggs("morton"), backed by mortie (mort2geo, geo2mort, mort2polygon, clip2order/generate_morton_children for zoom_to). Plus a MOC-backed lazy index with a from_moc() constructor — bootstrapped from coverage.moc without reading cell arrays (the HealpixMocIndex analog). This layer is the future upstream offering (Upstream (xdggs): support MortonIndexDtype as a native DGGS coordinate englacial/zagg#72, pitched at Design/Datamodel Decision: Very large datasets xarray-contrib/xdggs#143).
3. Cross-resolution join (ours permanently): the truncation join (fine_id.startswith(coarse_id); §5 rules) across two stores at different orders — upstream xdggs forbids mixed-grid alignment by design, so this never migrates.

Non-goals: no write path (zagg owns it); no pyramid sweep (zagg §7); no FITS/IVOA MOC interop (per mortie#50).

Dependencies

Core: mortie>=0.9.0 (kernels: moc_and, hive_path, decimal↔word, clip2order, children/parents), zarr>=3.1.5, xarray, numpy>=2.0, obstore, numcodecs (zstd sidecar decode). Extras: xdggs (heavy — lonboard/matplotlib/etc. are its hard deps, which is exactly why it's an extra), test, docs group. Python >=3.12, matching zagg (Q5 below).

Phases

• Phase 0 — scaffold + CI harness. (done — PR Scaffold + convention core (phases 0-1 of issue #1) #2) src/moczarr/ layout, hatchling + hatch-vcs (tag-driven version, pip-installable from day one), MIT. CI ported from zagg's template: test.yml (uv, 3.12/3.13 matrix, pytest + coverage), lint.yml (ruff PR-review bot), docs.yml (mkdocs → GitHub Pages), .pre-commit-config.yaml (ruff + mypy + codespell), README.
• Phase 1 — convention core (pure functions, no network). (done — PR Scaffold + convention core (phases 0-1 of issue #1) #2) Read-side implementation of morton-hive/1+/2 and morton-moc/1: decimal-id helpers (order/base/rank), hive path arithmetic + node invariant, windowed leaf-name grammar (first-_ split), manifest parse/validate, coverage envelopes (ranges | bitmap | full), zstd bitmap decode, ranges↔words. Golden vectors ported from zagg's tests so the two implementations can never drift silently.
• Phase 2 — store layer. (done — PR Scaffold + convention core (phases 0-1 of issue #1) #2) obstore-backed root/leaf access (s3 + local file), manifest/root-MOC/commit-stamp readers with zagg's postures (debris = absent; corrupt sidecar raises; unusable root MOC degrades to walk, never wrong answers — D9), delimiter-LIST discovery walk with the digit-children termination rule.
• Phase 3 — the opener. (done — PR open_hive() + zagg-written SERC fixture (phase 3 of issue #1) #3; cross-window time concat deferred, see checklist below) open_hive() assembling the lazy Dataset: shard set from root MOC ∩ AOI (walk fallback), box/bitmap leaf rejects, per-leaf xr.open_zarr, concat along the cells dim, window selection + time concat for /2 stores, convention attrs preserved on the result. Integration-tested against a zagg-written fixture store committed to the tree.
• Phase 4 — xdggs extra. (done — PR xdggs integration: grid_name morton (phase 4 of issue #1) #6, incl. cell_boundaries backend= argument) MortonInfo/MortonIndex + register_dggs("morton"); open_hive(..., decode=True); accessor round-trip tests (sel_latlon, cell_centers, cell_boundaries, zoom_to).
• Phase 5 — MOC-backed lazy index. from_moc() from the store/leaf coverage tiers; lazy coordinate fabrication; sel/isel without materializing the domain.
• Phase 6 — cross-resolution join. Truncation-join API across two open stores (exact xarray surface is zagg#198's O4 — design question to settle on this issue before the phase starts).
• Phase 7 — docs + binder notebook + publish workflow. Runnable-on-binder example against a public store or in-tree fixture (zagg §4 docs convention); publish.yml mirroring zagg's tag-driven TestPyPI→PyPI flow (tags are espg's to push).

Phases 0–3 are the critical path to the zagg#251 Phase-2 gate; 4–6 can interleave after 3.

Added scope (post-plan, from ratified decisions — state as of 2026-07-17)

• cell_ids fabrication layer (the zagg#262 gate) — PR cell_ids fabrication layer — the zagg#262 gate (issue #1) #7.
• Shared store handle + concurrent metadata (issue open_hive: shared store handle — O(leaves) store opens must collapse before thousand-leaf stores #5) — PR Shared store handle + concurrent metadata (issue #5) #9 (ready for review, unmerged).
• Sharded-leaf fixture arm (also the zagg#122 standing proof).
• Cross-window time concat (needs a /2 fixture from zagg's temporal path).
• Empty-AOI semantics (issue open_hive: AOI over empty coverage raises — decide vs schema-correct empty Dataset #4).
• Mixed-order adoption (issue Adopt mixed-order morton support when mortie#116 lands (swap posture tests + guards) #8 — blocked on mortie#116).

Design questions for espg

1. Code-sharing direction with zagg. moczarr re-implements the read-side convention from the spec (goldens ported, so drift fails tests). Longer term: should zagg's consumption layer (zagg/coverage.py, the read half of zagg/hive.py) eventually delegate to moczarr as a dependency, or do both sides keep independent implementations pinned by shared golden vectors? My lean: independent now, zagg→moczarr dep after moczarr stabilizes (deletes ~300 lines from zagg).
2. Store backend. obstore for parity with zagg (and it's already proven on the write side) vs fsspec for broader ecosystem reach. My lean: obstore core; fsspec never.
3. Naming. open_hive() vs plain moczarr.open(); and the accessor name for layer-3 ops (.moc?). My lean: open_hive (says what it reads), accessor decision deferred to Phase 6.
4. Fixture store. Phase 3 needs a small zagg-written hive store committed in-tree (also serves the binder docs). Generate from the NEON SERC config? Needs to be small (<~5 MB).
5. Python floor. >=3.12 (zagg parity, and mortie targets it) vs >=3.11 (xdggs parity). My lean: 3.12.

I'll proceed on the leans above unless redirected; each phase lands as a draft-PR commit chain per the zagg workflow conventions.

[espg]

whether zagg eventually deletes its read-side and depends on moczarr

probably? Can you actually walk me through what read side functionality we have so far (in zagg)?

Code-sharing direction with zagg. moczarr re-implements the read-side convention from the spec (goldens ported, so drift fails tests). Longer term: should zagg's consumption layer (zagg/coverage.py, the read half of zagg/hive.py) eventually delegate to moczarr as a dependency, or do both sides keep independent implementations pinned by shared golden vectors? My lean: independent now, zagg→moczarr dep after moczarr stabilizes (deletes ~300 lines from zagg).

I would love to reduce the scope of zagg and remove the reader side code. Your lean is approved, with a goal of stabilizing soon to enable that removal from zagg

Store backend. obstore for parity with zagg (and it's already proven on the write side) vs fsspec for broader ecosystem reach. My lean: obstore core; fsspec never.

I like obstore and am happy to approve that. Why fsspec never? (mostly agree, but want your reasoning)

Fixture store. Phase 3 needs a small zagg-written hive store committed in-tree (also serves the binder docs). Generate from the NEON SERC config? Needs to be small (<~5 MB).

yeah, this is fine for now. If we ever get the spec nailed down so it isn't drifting, by preference will be to pin this to source.coop public anonymous read-allowed bucket/prefix

Python floor. >=3.12 (zagg parity, and mortie targets it) vs >=3.11 (xdggs parity). My lean: 3.12.

approved, 3.12

Naming. open_hive() vs plain moczarr.open(); and the accessor name for layer-3 ops (.moc?). My lean: open_hive (says what it reads), accessor decision deferred to Phase 6.

Love open_hive() , strongly approved!

[espg]

🤖 from Claude

All five leans ratified — recorded. Answers to the two questions back:

zagg's current read-side inventory (what a future zagg→moczarr dep would delete or migrate)

Surveyed at zagg main (b3fe71e). Four pieces, in decreasing "moves to moczarr" confidence:

1. zagg/coverage.py — ~320 lines, 100% read-side, zero in-package consumers. The "§5-lite" consumption layer from PR Coverage MOCs: worker-owned leaf tier + end-of-run root (issue 200) englacial/zagg#208 phase 4: load_coverage (tolerant root-MOC load), root_coverage_and / box_and / bitmap_and (the tier-cascade AOI intersects), warn_if_stale (the O7 lazy staleness latch), refresh_root_coverage (the sanctioned LIST-walk rebuild). Nothing inside zagg calls it — it exists for tests, notebooks, and the reader that is now moczarr. This module is the cleanest deletion candidate; moczarr phase 2 reimplements its surface.
2. The read half of zagg/hive.py (~250 of 1055 lines): read_manifest, read_commit, read_coverage, read_coverage_bitmap + decode_coverage_bitmap, read_root_coverage, root_coverage_words, plus the shared id/rank helpers. Caveat: the write path uses some of these too (ensure_manifest verifies via the read path; retry logic reads stamps), so this is a "zagg imports moczarr's primitives" migration, not a pure deletion.
3. zagg/readers/tdigest_tensor.py — the client-side ragged t-digest → dense tensor reconstruction (issues Read helper(s) englacial/zagg#79/#209). Genuinely read-side and substantial, but product-schema-specific (decodes the ragged element attrs convention). Candidate for a later moczarr module (leaf ragged-field decoding is squarely reader territory) — flagging it now, not scheduling it.
4. Shared infra that stays in zagg: zagg/store.py (open_store/open_object_store — the write path needs them regardless; moczarr carries its own thin copy) and zagg/windows.py grammar helpers (used by the writer to produce leaf names; moczarr has its read-side port with goldens).

Also worth naming: the current "debug read path" is xr.open_zarr on flat stores — that's the story englacial/zagg#251 deletes, replaced by open_hive(). Net: once moczarr stabilizes, zagg sheds coverage.py outright, thins hive.py toward write-only, and (optionally, later) hands off the t-digest tensor readers.

Why "fsspec never"

Four reasons, strongest first:

1. The termination rule is a correctness invariant, not plumbing. The walk's "no digit children ⇒ nothing finer exists" and the D4 debris model both lean on strongly-consistent, uncached delimiter-LIST semantics reaching the code unmediated. fsspec's filesystem instances carry directory caches (dircache, instance caching across calls) that are exactly the mechanism to serve a stale listing — and a stale listing here isn't slow, it's a false "nothing finer exists", i.e. silently missing data. obstore is a thin typed binding over the rust object-store crate: every LIST is a real LIST.
2. One backend = one error/consistency surface. Supporting both means every posture in the reader (NotFound→absent vs corrupt→raise) needs verifying against two libraries' exception taxonomies and retry behaviors, forever. The D9 discipline ("degrade, never wrong answers") is only as strong as the weakest backend's semantics.
3. Dependency surface. fsspec's S3 path pulls s3fs→aiobotocore, whose botocore pins historically fight boto3 — a resolver tax on every consumer. obstore is one wheel, no transitive Python deps.
4. The escape hatch already exists at the right level. moczarr's job is computing which leaves to open; the leaf opens go through zarr's store API, and anyone who truly needs an fsspec-backed store can hand zarr/xarray one for a single leaf themselves. Interop doesn't require moczarr to carry fsspec in its hot path.

(1) is the one I'd defend to the death; (2)–(4) are cost/benefit.

Pages target (for the record, from the session)

docs.yml deploys to https://espg.github.io/moczarr/ — a project page under its own subpath, coexisting with the espg.github.io user page; no collision with englacial's docs domain. site_url gets pinned in mkdocs.yml in the next commit so canonical URLs/sitemap resolve. A custom domain remains a per-repo CNAME option later if wanted.

Proceeding with phase 2 (store layer) on PR #2 under the ratified answers. One posture note recorded there: refresh_root_coverage is a cache write — phase 2 ships the read-only discovery walk, and whether moczarr ever writes regenerable cache objects (my lean: eventually yes, else the zagg deletion strands the rebuild tool) is deferred to the phase that needs it.