Skip to content

feat(dataset): windowed crop-from-path/URL to avoid full reads of huge /vsicurl rasters #957

Description

@MAfarrag

Context

While building the JRC European Flood Hazard Map backend in earthlens (serapeum-org/earthlens#958), Dataset.crop(bbox=)
could not be used against the EFHM rasters (110162 x 51992) served over /vsicurl: it triggers a full read of the
source, which hung over the network. The backend had to hand-roll read_array(window=) + GeoTIFF write instead.

Problem / Current Behaviour

Dataset.crop(bbox=) reads the entire source array before cropping. For a very large or network-backed (/vsicurl
https) raster this is prohibitively slow or hangs, defeating the point of a spatial crop. Every consumer reading a
small AOI out of a large remote raster must re-implement windowed reads.

Steps to Reproduce

from pyramids.dataset import Dataset
ds = Dataset.read_file("/vsicurl/https://<host>/efhm_rp100.tif")  # 110162 x 51992
ds.crop(bbox=[west, south, east, north])   # full read of the source -> hangs over the network

Proposed Solution

# Offer a windowed crop that reads only the AOI window straight from the path/URL:
Dataset.crop_from_path(src, bbox=[...], out_path=...)   # computes the pixel window, reads only it
# or an option on the existing API:
ds.crop(bbox=[...], windowed=True)                      # read_array(window=) under the hood

So consumers need not hand-roll read_array(window=) + origin math + GeoTIFF write.

Out of Scope

  • The earthlens-side consolidation of the current hand-rolled helpers (tracked in earthlens).

Effort Estimate

Size: M
Rationale: window/geotransform math plus a new API path and tests; self-contained in pyramids.

Definition of Done

  • a windowed crop reads only the AOI window from a local path or /vsicurl URL (no full read)
  • correct geotransform/CRS on the output; nodata preserved
  • tests cover a windowed crop from a local file and a range-read URL
  • existing crop behaviour unchanged

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions