Skip to content

switch to source.coop for downloads, fronted by Cloudflare #337

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
avsm wants to merge 13 commits into
base: main
Choose a base branch
from
Open

switch to source.coop for downloads, fronted by Cloudflare #337

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
d642b6d
switch to source.coop for downloads, fronted by Cloudflare
avsm Jul 30, 2026
44b453a
cleanups
avsm Jul 30, 2026
08198ec
updates
avsm Jul 30, 2026
fb29328
flesh out the zarr fill commands with s3 support and make s3fs optional
avsm Jul 31, 2026
a3259f6
more reliability fixes for zarr-init
avsm Jul 31, 2026
96d6dbe
resume a crashed zarr-fill from the store itself, and add zarr-scan
avsm Aug 1, 2026
01cf155
update zarr-scan for optional source dir
avsm Aug 1, 2026
567f195
let scan work with no local mirror
avsm Aug 1, 2026
063e7f7
scan before fill
avsm Aug 1, 2026
701c3c5
fix deadlocks
avsm Aug 1, 2026
5e50272
zarr stats gathering during fill
avsm Aug 2, 2026
f276bcf
simplify for serverless deployment
avsm Aug 2, 2026
0427e3d
update for remote stretch
avsm Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
171 changes: 169 additions & 2 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,177 @@
## Unreleased

### Breaking Changes

- **All downloads now come from Source Cooperative**, fronted by CloudFlare.
Embeddings, landmasks, manifests, and the zarr store are served from the public
`https://data.source.coop/tessera/tessera` repository over HTTPS,
replacing the retired `tessera-embeddings` AWS S3 bucket. The repository
is organised by media type (`npy/{version}/`, `landmasks/{version}/`,
`zarr/{version}/`) with per-version `manifest.parquet` and
`landmasks.parquet` files colocated with their data, and carries one
embedding tree per version. (@avsm, @mtelvers)
- **Dependencies removed**: `botocore` and `awscrt` are no longer required.
Downloads use the standard library with retry/backoff, `If-Modified-Since`
conditional-GET caching, and integrity verification against the response
`Content-Length` plus a streamed MD5 whenever the server's `ETag` is a
content MD5 (single-part uploads) (@avsm)

### New Features

- **Per-version default variant**: omitting `dataset_variant` /
`--dataset-variant` now selects the version's published variant (`vultr`
for v1, `cambridge` for v1.1) instead of always `vultr`, so
`GeoTessera(dataset_version="v1.1")` works without an explicit variant.
(@avsm)
- **Remote zarr builds**: `zarr-init` and `zarr-fill` now take locations
rather than paths — both the tile source and the output store may be
fsspec URLs (`s3://bucket/prefix`), so a store on one S3 node can be
filled from tiles on another with no local mirror. Remote tiles are read
with byte-range GETs sized to the rows each shard needs (an `.npy` is a
header plus a flat C-ordered buffer), so no scratch disk is involved.
`--source-*` and `--store-*` flags configure the two endpoints
independently; credentials come from the environment, a named profile
(`--store-profile`), or an instance role rather than argv, and
`--store-acl` stamps a canned ACL such as `bucket-owner-full-control` on
every object written. `s3://` locations need the new optional
`s3` extra (`pip install 'geotessera[s3]'`), which is what pulls in
`s3fs` and `botocore`; the core install stays free of both and `https://`
sources work without it. (@avsm)
- **Parallel per-zone fills**: `zarr-fill --zones N` is now safe to run as
one process per UTM zone against a shared store. Ingestion tracking is one
object per zone/year, each zone/year takes an advisory lock
(`--force-lock` to take over a dead run's), and root-metadata
consolidation is skipped by default for a zone-restricted fill. See
the architecture guide for the sweep recipe. (@avsm)
- **The store now contains only Zarr**: build bookkeeping — the ingestion
registry, fill locks and global-preview resume markers — moved out of the
store into a sibling location, `<store>.build` by default and relocatable
with `--state-url`. Previously these sat at the store root, where every
hierarchy listing and `consolidate_metadata` call warned about
unrecognised objects and readers saw non-Zarr entries. A `_registry.parquet`
left inside an older store is still read, so existing stores resume
correctly; nothing is written back into them. (@avsm)
- **`geotessera-registry zarr-scan`**: New subcommand that inventories a
store's shards without writing anything, classifying each as `written`,
`missing`, or `empty` (no land falls in it, so it is ocean or outside
coverage and will never be filled). Prints per-zone/year and per-year
summaries of how much is left to fill — percentages are over land, not
over each zone's bounding box — and optionally writes the per-shard index
as parquet. Takes the store alone: the land denominator comes from the
landmask registry (~19 MB, cached), so no tile mirror or manifest is
needed. An optional tile mirror switches the denominator to each year's
actual embedding coverage from the manifest. (@avsm)
- **`zarr-fill` scans before writing and uploads only what is missing.**
The ingestion registry is written only when a (zone, year) finishes, so a
run killed partway loses that year's bookkeeping and would re-upload
everything — for a zone that is 97% done, rebuilding 1,398 shards to add
48. The shard objects survive anything and a shard is always written from
every tile covering it, so their presence is proof of completeness. Fills
now scan for them by default and skip what is there; falls back to
probing just the shards in hand where the credentials cannot list the
store. `--rewrite-existing-shards` forces a rebuild, needed only when the
tile inventory has grown, since a newly-added tile falls inside an
existing shard. `--skip-existing-shards` is still accepted as a no-op.
(@avsm)
- **Fills are stateless and stretch statistics collect themselves.** The
ingestion registry and advisory locks are gone from `zarr-fill`: the
store's shard objects are the only record of progress, so a preemptible
(spot) instance that dies mid-run leaves nothing to clean up or take
over — relaunching the same command scans and continues. A per-zone
coverage mask (`stretch_stats_shards`) records which shards are folded
into the stretch sums; the fill diffs it against the same scan and reads
back any shard whose statistics are missing, so interrupted runs and
stores from older builds converge automatically — no separate backfill
step, which now exists only as the explicit repair for suspected
double-counting. `--state-url` and `--force-lock` are accepted as no-ops
for existing scripts; `zarr-consolidate` still reads `--state-url` to
merge registries written by older builds. (@avsm)
- **Per-zone stretch statistics, collected at fill time** (see
`docs/specs/zarr-stretch-stats.md`): each zone group gains six arrays —
exact mean/covariance sufficient statistics per (zone, year), additive
across zones, plus a weighted 20k-pixel sample for quantiles — folded in
by `zarr-fill` from the shard buffers it already holds, at no extra I/O.
`zarr-stretch` now aggregates these by default: a few MiB of reads and an
*exact* global PCA (verified |cos| = 1.0 against a full-population fit)
instead of terabytes of shard re-reads, and it works against remote
stores. A drift check compares the stats-derived covariance against one
refitted from the stored sample and warns when rewritten shards have
double-counted. `--from-shards` keeps the legacy path;
`zarr-fill --backfill-stretch-stats` rebuilds statistics for stores
filled before this existed (and is required before `zarr-extend` will
touch such stores); `zarr-init --stretch-sample-size` tunes the sample.
(@avsm)
- **`zarr-fill` reports the shard arithmetic**: `Shards: 1,373 land, 48
recorded done, 43 found in store, 1,282 to write`. The previous line
showed only the last figure, which could not be reconciled with
`zarr-scan` — that counts every land shard, whereas a fill considers only
those covering tiles the registry has not already recorded. (@avsm)
- **Object-store libraries no longer log over the progress bar**: botocore
logs "Found credentials in shared credentials file" at INFO every time a
client is built, once per worker process, and the workers share a
terminal with the progress bar. botocore, boto3, aiobotocore, s3fs,
urllib3 and aiohttp are now capped at WARNING in both the parent and the
workers. (@avsm)
- **Fills no longer deadlock against an object store.** `s3fs` runs its
client on a background event-loop thread and `fsspec` caches both the loop
and its filesystem instances globally; a forked worker inherits those
objects but not the thread running the loop, so its first call to the
store waits forever (main thread in `futex_do_wait`, loop thread idle in
`ep_poll`). Workers are now started with "spawn" rather than the Linux
default of "fork", and reset any inherited fsspec state on startup.
(@avsm)
- **Workers die with their parent.** A fill killed outright left its
workers running, each holding gigabytes and accumulating across runs —
19 orphans holding 8 GB were observed on one host. Workers now set
`PR_SET_PDEATHSIG` on Linux. (@avsm)
- **Ctrl-C during a fill exits cleanly** with status 130 instead of two
tracebacks. The process pool was shut down with `wait=True`, so an
interrupt blocked until every in-flight shard finished — with
multi-gigabyte workers that looks like a hang and invites a second
Ctrl-C. (@avsm)
- **`zarr-fill` warns when the worker count will not fit in memory**, using
`MemAvailable` rather than physical RAM since these hosts are usually
shared. The estimate covers the real peak — a worker holds a 2.1 GiB
shard buffer, then zarr's sharding codec compresses every inner chunk and
assembles the shard while s3fs holds the upload body, measured at 4.3 GiB
and observed being OOM-killed at 12 GiB — so it budgets ~6.2 GiB each
rather than the raw buffer. An OOM kill leaves no traceback, so the
warning is the only diagnosis. (@avsm)
- **`geotessera-registry zarr-extend`**: New subcommand that appends years
to an existing store's time axis, so a new year can be added without
rebuilding. Time is chunked one year per chunk, making this a
metadata-only edit — existing chunks are never rewritten — and the new
slice reads back with the same sentinels a freshly initialised year has.
Years may only be appended (inserting an earlier one would renumber every
chunk, so it is refused), and it will not run while a fill lock is held.
(@avsm)
- **`geotessera-registry zarr-consolidate`**: New subcommand that
re-consolidates a store's root metadata after in-place changes.
Mostly only for repairs and not regular use.
re-consolidates a store's root metadata after in-place changes, and
merges the per-zone ingestion registries into `_registry.parquet`. This
is the single-writer step that finishes a parallel sweep; also useful for
repairs. Accepts a local path or a remote store URL.

### Bug Fixes

- **Incremental fills no longer erase neighbouring tiles**: a shard write
replaces the whole shard, so a fill that touched a shard already holding
data would zero out the tiles it did not re-read. Touched shards are now
rebuilt from every tile overlapping them. (@avsm)
- **Failed shards are no longer recorded as written**: tiles are only added
to the ingestion registry once every shard covering them succeeded, so
re-running a fill retries exactly the unfinished work. A fill with any
failed shard now reports an error. (@avsm)
- **`geotessera-registry` propagates exit status**: command return codes
were discarded, so failures reported success to the shell. (@avsm)
- **`geotessera-registry s3scan` scans Source Cooperative**: listings are
path-style against the S3-compatible endpoint given by `--endpoint-url`
(default `https://data.source.coop`); the AWS virtual-hosted addressing
for the retired bucket is gone, and `--region` with it. Discovery
understands the flat `{version}/{year}/` layout with the variant supplied
via `--variant`, and `--landmasks-uri` points the landmask scan at a
separate tree (e.g. `s3://tessera/tessera/landmasks/`), including
landmasks-only runs. This allows manifests and landmask registries to be
regenerated directly from the Source Cooperative repository. (@avsm)

## v0.9.0 (2026-06-09)

Expand Down
61 changes: 34 additions & 27 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,9 +101,9 @@ The Tessera embeddings use a **0.1-degree grid system**:

### File Structure and Downloads

When you request embeddings, GeoTessera downloads files from the public S3
bucket (using anonymous, unsigned requests) into the output directory you
specify, where they persist for re-use:
When you request embeddings, GeoTessera downloads files over HTTPS from the
public Source Cooperative repository into the output directory you specify,
where they persist for re-use:

#### Embedding Files (via `fetch_embedding`)
1. **Quantized embeddings** (`grid_X.XX_Y.YY.npy`):
Expand Down Expand Up @@ -136,7 +136,7 @@ User Request (lat/lon bbox)
↓
Parquet Registry Lookup (find available tiles from manifest.parquet)
↓
Anonymous S3 Downloads to Output Directory (CRC64NVMe verified)
HTTPS Downloads from Source Cooperative to Output Directory (integrity verified)
├── embedding.npy (quantized) → output dir
└── embedding_scales.npy → output dir
↓
Expand Down Expand Up @@ -379,7 +379,7 @@ Options:
-f, --format TEXT Output format: 'tiff' or 'npy' (default: tiff)
--year INT Year of embeddings (default: 2024)
--dataset-version TEXT Tessera dataset version (e.g. v1, v1.1)
--dataset-variant TEXT Tessera dataset variant (default: vultr)
--dataset-variant TEXT Tessera dataset variant (default: the version's published variant)
--bands TEXT Comma-separated band indices (default: all 128)
--compress TEXT Compression for TIFF format (default: lzw)
--dry-run Calculate total download size without downloading
Expand Down Expand Up @@ -449,7 +449,7 @@ Options:
--tile TEXT Single tile by any point within it: 'lon,lat'
--by-source Render each (version, variant) source in a distinct colour
--dataset-version TEXT Tessera dataset version (e.g. v1, v1.1; or 'all' with --by-source)
--dataset-variant TEXT Tessera dataset variant (default: vultr; or 'all' with --by-source)
--dataset-variant TEXT Tessera dataset variant (default: the version's published variant; or 'all' with --by-source)
--region-file PATH GeoJSON/Shapefile to focus on specific region
--country TEXT Country name to focus on (e.g., 'United Kingdom')
--tile-color TEXT Color for tiles (default: red)
Expand Down Expand Up @@ -483,7 +483,7 @@ geotessera info [OPTIONS]
Options:
--tiles PATH Analyze tile files/directory (GeoTIFF or NPY format)
--dataset-version TEXT Tessera dataset version (e.g. v1, v1.1)
--dataset-variant TEXT Tessera dataset variant (default: vultr)
--dataset-variant TEXT Tessera dataset variant (default: the version's published variant)
-v, --verbose Verbose output
```

Expand All @@ -498,15 +498,18 @@ GeoTessera uses a Parquet-based registry system to efficiently manage and access
- **Fast queries**: Uses pandas DataFrames for efficient spatial and temporal filtering
- **Block-based organization**: Internal 5×5 degree geographic blocks for efficient queries
- **Minimal storage**: Manifest files are ~few MB each and cached locally
- **Integrity checking**: End-to-end CRC64NVMe checksums verified against S3's
`x-amz-checksum-crc64nvme` response header during each download
- **Always enforced** for data integrity — a checksum mismatch (or a missing checksum header) rejects the download
- **Integrity checking**: Every download is verified against the response
`Content-Length`, and against an MD5 computed over the streamed body whenever
the server's `ETag` is a content MD5 (single-part uploads)
- A mismatch rejects the download and triggers a retry, so corrupt or
truncated files never reach the cache

### Dataset Versions and Variants

Tessera embeddings are published as dataset *versions* (e.g. `v1`, `v1.1`) and,
within a version, as *variants* produced by different model runs (e.g. the
default `vultr`, or `cambridge`). Select them on the CLI with `--dataset-version`
within a version, as *variants* produced by different model runs (`vultr` for
v1, `cambridge` for v1.1; each is the version's default). Select them on the
CLI with `--dataset-version`
and `--dataset-variant`, or in Python:

```python
Expand All @@ -523,7 +526,7 @@ The registry can be loaded from multiple sources (in priority order):
1. **Local file** (via `registry_path` parameter)
2. **Local directory** (via `--registry-dir` or `registry_dir` parameter, looks for `manifest.parquet`, falling back to the legacy `registry.parquet`)
3. **Remote URL** (via `registry_url` parameter)
4. **Default remote** (from `https://s3.us-west-2.amazonaws.com/tessera-embeddings/{version}/manifest.parquet`)
4. **Default remote** (from `https://data.source.coop/tessera/tessera/npy/{version}/manifest.parquet`)

```python
# Use local manifest file
Expand Down Expand Up @@ -556,27 +559,31 @@ print(manifest.head())
2. **Request tiles for bbox** → Query DataFrame for tiles in region
3. **Filter by year and variant** → Select tiles matching the requested year/variant
4. **Find available tiles** → Return list of matching tiles
5. **Anonymous S3 download** → Fetch tiles on demand into the output directory, verified with CRC64NVMe
5. **HTTPS download** → Fetch tiles on demand from the Source Cooperative mirror into the output directory, with integrity checks
6. **Persist** → Downloaded tiles stay in the output directory and are skipped on rerun

## Data Organization

### Tessera Data Structure

```
Remote Server (https://s3.us-west-2.amazonaws.com/tessera-embeddings)
├── v1/ # Dataset version 1.0
│ ├── manifest.parquet # Per-version tile manifest
│ ├── landmasks.parquet # Landmask manifest
│ ├── global_0.1_degree_representation/ # vultr variant (default)
Remote Server (https://data.source.coop/tessera/tessera)
├── npy/ # NPY embeddings + scales
│ ├── v1/ # Dataset version 1.0
│ │ ├── manifest.parquet # Per-version tile manifest
│ │ └── 2024/grid_0.15_52.05/grid_0.15_52.05{,_scales}.npy
│ └── global_0.1_degree_tiff_all/
│ └── grid_0.15_52.05.tiff # Landmask with projection info
└── v1.1/ # Dataset version 1.1
├── manifest.parquet
├── landmasks.parquet
└── global_0.1_degree_representation.cambridge/
└── 2024/grid_0.15_52.05/grid_0.15_52.05{,_scales}.npy
│ └── v1.1/ # Dataset version 1.1
│ ├── manifest.parquet
│ └── 2024/grid_0.15_52.05/grid_0.15_52.05{,_scales}.npy
├── landmasks/ # Landmask TIFFs
│ ├── v1/
│ │ ├── landmasks.parquet # Landmask manifest
│ │ └── grid_0.15_52.05.tiff # Landmask with projection info
│ └── v1.1/
│ ├── landmasks.parquet
│ └── grid_0.15_52.05.tiff
└── zarr/ # Cloud-native zarr store
└── v1/
```

### Local Cache Structure
Expand Down Expand Up @@ -634,7 +641,7 @@ When `cache_dir` is not specified, the registry is cached in platform-appropriat

## Hash Verification

GeoTessera verifies end-to-end CRC64NVMe checksums for all downloaded files (embeddings, scales, and landmasks) against S3's `x-amz-checksum-crc64nvme` response header to ensure data integrity. This check is always enforced: a download whose checksum does not match — or whose S3 object is missing the checksum header — is rejected rather than used, so corrupt or truncated files never reach the cache.
GeoTessera verifies every downloaded file (embeddings, scales, and landmasks) against the response `Content-Length`, and additionally against an MD5 computed over the streamed body whenever the server's `ETag` is a content MD5 (a single-part upload; this covers landmask TIFFs and scales files). Large multipart-uploaded embedding tiles carry a composite ETag that is not a content hash, so they are length-checked only. A mismatch rejects the download and triggers a retry with backoff, so corrupt or truncated files never reach the cache.

## Contributing

Expand Down
Loading
Loading