switch to source.coop for downloads, fronted by Cloudflare - #337
Open
avsm wants to merge 13 commits into
Open
Conversation
zarr-init used mode="w", which deletes the destination prefix before creating the group -- pointed at an existing store it would have wiped it. Use mode="w-" instead: create, never clobber, and no list or delete permission needed. Object stores answer 403, not 404, for a missing key when the caller cannot list the prefix, so existence probes cannot distinguish "absent" from "not allowed". remote.exists()/listdir() take an on_denied argument; init warns and proceeds, registry and lock reads treat it as absent, and the zone check in zarr-fill opens the group (GetObject) rather than probing the prefix (ListBucket). Reading a parquet no longer probes first, halving its round trips. s3fs surfaces a 403 as a builtin PermissionError rather than a botocore exception, so the CLI handler missed it and printed a traceback; catch the OSError family too and explain the 403-instead-of-404 behaviour. Document that Source Cooperative serves reads from data.source.coop but takes writes only on the backing bucket with no --endpoint-url, which is the likeliest cause of an AccessDenied here. Also adds the zarr test suite omitted from the previous commit.
The ingestion registry is only written when a (zone, year) finishes, so a run killed partway -- an OOM kill leaves no traceback -- loses that year's bookkeeping even though the shards it wrote are safely in the store. The shard objects survive anything, so use them as the resume signal: zarr-fill --skip-existing-shards lists the shards already present for a (zone, year), skips them, and records their tiles so later runs need no flag. It assumes the tile inventory has not grown since those shards were written. Where the credentials cannot list the store it probes only the shards the run would touch, and an empty listing is distinguished from a denied one so a fresh zone costs no probes. zarr-scan reuses that scan to inventory a store against the manifest without writing anything. Shards are classified written, missing, or empty -- no manifest tiles fall in an empty one, so it is ocean or outside coverage and will never be filled. Keeping those separate makes the percentages meaningful: they are over land, not over each zone's bounding box. Prints per-zone/year and per-year summaries and can write the full per-shard index as parquet. Also warn when the worker count cannot fit in RAM. Each worker holds a full (128, 4096, 4096) int8 shard buffer plus scales, about 2.1 GiB, so --workers 16 needs 33 GiB resident and is silently OOM-killed on a smaller box.
Scan a store on its own to report how much is left to fill. The store argument comes first and the tile mirror is now optional: what decides whether a shard can ever hold data is land coverage, and that comes from the landmask registry (~19 MB, fetched and cached) rather than the ~200 MB manifest. Supplying a mirror switches the denominator to each year's actual embedding coverage, which only matters where a year covers less than the full land area. Shards are classified written, missing, or empty -- no land falls in an empty one, so it is ocean or outside coverage and will never be filled. Keeping those separate makes the percentages meaningful: they are over land, not over each zone's bounding box, which for a coastal zone is mostly sea. Factors the tile-footprint projection out of gather_tile_infos so it can be applied to landmask tiles too, which have no embeddings to look up.
avsm
force-pushed
the
switch-source-coop
branch
from
August 1, 2026 06:17
de18122 to
01cf155
Compare
Making the tile mirror optional also flipped zarr-scan's positionals to <store> <source>, which no other command takes. A script passing the usual <source> <store> therefore read the store URL as the tile source, went looking for a manifest inside the store, and signed the request with source credentials the caller had no reason to set -- surfacing as an unrelated NoCredentialsError. Restore <source> <store> with the source optional. argparse binds a lone positional to store_path, so both `zarr-scan <store>` and `zarr-scan <mirror> <store>` do the obvious thing. A source that is really a store now says so, rather than failing later inside botocore. And _resolve_source sat outside the error handling in zarr-init, zarr-fill and zarr-scan, so any credential failure while resolving the manifest escaped as a traceback instead of the clean message.
The ingestion registry is written only when a (zone, year) finishes, so a run killed partway loses that year's bookkeeping and re-uploads everything. For a zone that is 97% done that means rebuilding 1,398 shards to add 48. The shard objects survive anything, and a shard is always written from every tile covering it, so their presence is proof of completeness. Scan for them before doing any work and skip what is already there. Where the credentials cannot list the store, fall back to probing just the shards the run would touch. --rewrite-existing-shards forces a rebuild, which is needed only when the tile inventory has grown, since a newly-added tile falls inside an existing shard and would otherwise be skipped rather than merged in. --skip-existing-shards stays accepted as a no-op so scripts keep working. Also exit cleanly on Ctrl-C. The pool was shut down with wait=True, so an interrupt blocked until every in-flight shard finished; with workers holding 2 GiB each that looks like a hang and invites a second Ctrl-C and a second traceback. Cancel what has not started, release the zone lock, and exit 130.
Computing the global RGB stretch by re-reading shards costs terabytes; every
pixel it needs was already in a fill worker's buffer exactly once. Collect
per-(zone, year) statistics at fill time instead, as six ordinary arrays in
each zone group (spec: docs/specs/zarr-stretch-stats.md):
stretch_stats_{count,sum,prod} exact N / Sum(x) / Sum(x x^T), additive
stretch_sample{,_scales,_count} weighted raw-pixel sample, int8+scale
The sums are the sufficient statistics for the covariance, so the global
PCA derived from them is exact -- verified |cos| = 1.0 per component
against a full-population fit -- and additive across zones. Quantiles are
not additive and PC-space projections need the global axes, hence the raw
sample (~2.6 MiB per zone-year at K=20000), drawn per shard proportional
to valid-pixel count and merged by weighted reservoir. Zone groups gain no
attributes: the geoemb: convention keeps those on the root, so the filled
slot count is itself an array.
zarr-stretch defaults to aggregating these (a few MiB of reads, works on
remote stores, seconds); --from-shards keeps the legacy sampling path. A
drift check compares the stats covariance against one refitted from the
stored sample -- relative Frobenius distance with a sqrt(d/n) noise-aware
limit, after the specified per-component |cos| check false-alarmed on
near-degenerate eigenvalues -- and flags double-counted rewrites.
zarr-fill --backfill-stretch-stats rebuilds a zone's arrays from its
shards (the repair for pre-feature stores and interrupted fills), and
zarr-extend refuses to desynchronise a group that lacks them.
Also included from the same working set: --spill-dir to memory-map shard
buffers (RssAnon 1.73 -> 0.35 GiB per worker, for memory-tight VMs);
--source/--store-path-style for S3-compatible endpoints without wildcard
DNS; and --source-npy-root/--source-landmask-root for mirrors in the
legacy flat layout.
avsm
force-pushed
the
switch-source-coop
branch
from
August 2, 2026 11:28
c047526 to
5e50272
Compare
zarr-fill is now stateless: no build directory, no ingestion registry, no advisory locks. The store's shard objects are the only record of progress -- a shard is always written from every tile covering it, so its presence is proof of completion -- and the start-of-run scan is the resume. A spot instance killed at any moment leaves nothing to clean up or take over; relaunching the same command continues where the objects stop. Stretch statistics collect themselves by the same mechanism. A seventh per-zone array, stretch_stats_shards (T, shard_rows, shard_cols) uint8, records which shards are folded into the sums; the fill diffs it against the scan and reads back any shard whose statistics are missing as catch-up tasks in the same worker pool. That covers a crash between shard write and stats fold (sums are written before the mask, so the worst-timed kill re-folds one shard rather than silently dropping it), shards written by builds that predate collection (arrays are created on the zone's next fill), and sums whose provenance the missing mask makes unknowable (reset and recomputed). --backfill-stretch-stats survives only as the explicit repair for suspected double-counting. One fill per (zone, year) at a time remains the operating contract but is no longer enforced: identical concurrent fills write identical shards, so the failure mode is wasted work or a drift-detectable stats double-fold, not data corruption. --state-url and --force-lock are accepted as no-ops so existing scripts keep working; zarr-consolidate still reads --state-url to merge ingestion registries written by older builds, and skips the merge entirely rather than conjure a state directory for an empty result.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See https://eeg.zulipchat.com/#narrow/channel/527258-Tessera/topic/Switching.20hosting.20of.20embeddings.20to.20source.2Ecoop/with/613621733