Skip to content

Commit 099673f

Browse files
gggravitygggravity
committed
stac-fetch: query STAC catalogs, download COGs, build xarray datacubes
CLI and library over pystac-client with catalog presets, friendly datetime and filter syntax, correct pagination with explicit truncation reporting, resumable and verified concurrent downloads, optional Planetary Computer asset signing, and lazy datacube construction via stackstac or odc-stac. 244 hermetic tests: no network, static STAC fixtures plus mocked HTTP.
0 parents  commit 099673f

35 files changed

Lines changed: 4937 additions & 0 deletions

‎.github/workflows/ci.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
name: CI
2+
on:
3+
push:
4+
branches: [main, build]
5+
pull_request:
6+
branches: [main]
7+
jobs:
8+
test:
9+
runs-on: ubuntu-latest
10+
strategy:
11+
fail-fast: false
12+
matrix:
13+
python-version: ["3.12", "3.13", "3.14"]
14+
steps:
15+
- uses: actions/checkout@v5
16+
- uses: actions/setup-python@v6
17+
with:
18+
python-version: ${{ matrix.python-version }}
19+
cache: pip
20+
- run: python -m pip install --upgrade pip
21+
- run: pip install -r requirements-dev.txt
22+
- run: ruff check .
23+
- run: ruff format --check .
24+
- run: python -m pytest

‎.gitignore‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
__pycache__/
2+
*.py[cod]
3+
.venv/
4+
venv/
5+
.pytest_cache/
6+
.ruff_cache/
7+
*.egg-info/
8+
.coverage
9+
htmlcov/
10+
*.tif
11+
*.zarr/
12+
*.part
13+
scenes/

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 python-remote-sensing.com
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 305 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,305 @@
1+
# stac-fetch
2+
3+
Query any STAC catalog from the command line and turn the result into a listing, a set of
4+
downloaded COGs, or a lazy xarray datacube.
5+
6+
## The problem
7+
8+
Every satellite-imagery project starts with the same twenty minutes of friction. You want to know
9+
what Sentinel-2 scenes cover a basin in July under 20% cloud, so you open a notebook, look up the
10+
Earth Search endpoint again, remember that `datetime` wants an RFC 3339 interval and not
11+
`2024-07`, remember that `eo:cloud_cover` goes in `query` and not in the top level, then discover
12+
that `search.items()` quietly stopped at the first page and you have been reasoning about 100
13+
scenes when there were 1,400.
14+
15+
Then you want the pixels. So you write the download loop again: threads, a `.part` file so an
16+
interrupted run does not start from zero, a size check so a truncated GeoTIFF does not poison the
17+
pipeline three steps later, a skip-if-present check so re-running is cheap. And if the catalog is
18+
Planetary Computer, everything 403s until you remember the assets need signing.
19+
20+
`stac-fetch` is that twenty minutes, packaged. It is a reconnaissance tool first — `--print` tells
21+
you what exists before you commit to moving bytes — and a transfer tool second.
22+
23+
## Install
24+
25+
Python 3.12 or newer.
26+
27+
```bash
28+
git clone https://github.com/python-remote-sensing/stac-fetch.git
29+
cd stac-fetch
30+
pip install -r requirements.txt
31+
```
32+
33+
That is enough for searching, listing and downloading. The datacube mode (`--to-zarr`,
34+
`to_datacube()`) needs a heavier optional stack:
35+
36+
```bash
37+
pip install -r requirements-datacube.txt # stackstac, xarray, dask, rioxarray, zarr
38+
```
39+
40+
There is nothing to install into site-packages — run it out of the clone with
41+
`python -m stac_fetch`.
42+
43+
## Usage
44+
45+
### What is out there? (reconnaissance)
46+
47+
```console
48+
$ python -m stac_fetch -u earth-search -c sentinel-2-l2a \
49+
--bbox 5.9,45.8,6.5,46.4 -d 2024-07 --cloud-cover 20 --sortby eo:cloud_cover --limit 5
50+
warning: Result set truncated: returned 5 of 14 matching items (--limit 5). Raise or drop --limit to fetch the rest.
51+
5 item(s) of 14 matching from https://earth-search.aws.element84.com/v1
52+
ID DATETIME CLOUD% ASSETS
53+
------------------------ -------------------- ------ --------------------------------------
54+
S2A_31TGM_20240725_0_L2A 2024-07-25T10:48:05Z 4.4 aot, aot-jp2, blue, blue-jp2, +31 more
55+
S2B_31TGL_20240717_0_L2A 2024-07-17T10:38:25Z 5.0 aot, aot-jp2, blue, blue-jp2, +31 more
56+
S2B_32TLS_20240720_0_L2A 2024-07-20T10:48:01Z 5.8 aot, aot-jp2, blue, blue-jp2, +31 more
57+
S2B_31TGL_20240727_0_L2A 2024-07-27T10:38:24Z 6.3 aot, aot-jp2, blue, blue-jp2, +31 more
58+
S2A_32TLS_20240725_0_L2A 2024-07-25T10:48:00Z 6.6 aot, aot-jp2, blue, blue-jp2, +31 more
59+
```
60+
61+
The table goes to stdout, the summary and any truncation warning go to stderr, so
62+
`... | head` stays clean.
63+
64+
### JSON for pipelines
65+
66+
```console
67+
$ python -m stac_fetch -u earth-search -c sentinel-2-l2a --aoi basin.geojson \
68+
-d last-30-days --cloud-cover 20 --json
69+
{
70+
"search": {
71+
"catalog": "https://earth-search.aws.element84.com/v1",
72+
"returned": 2,
73+
"matched": 14,
74+
"limit": 2,
75+
"truncated": true
76+
},
77+
"items": [
78+
{
79+
"id": "S2A_31TGM_20240725_0_L2A",
80+
"collection": "sentinel-2-l2a",
81+
"datetime": "2024-07-25T10:48:05Z",
82+
"cloud_cover": 4.447231,
83+
"bbox": [5.5795144, 45.9187689, 6.7387394, 46.9235637],
84+
"assets": ["aot", "blue", "green", "nir", "red", "scl", "..."],
85+
"asset_details": {
86+
"red": {
87+
"href": "https://.../B04.tif",
88+
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
89+
"title": "Red (band 4) - 10m"
90+
}
91+
}
92+
}
93+
]
94+
}
95+
```
96+
97+
`truncated` is the field to assert on in a pipeline: it is `true` whenever `--limit` stopped the
98+
walk before the result set was exhausted.
99+
100+
### Download assets
101+
102+
```console
103+
$ python -m stac_fetch -u earth-search -c sentinel-2-l2a --bbox 5.9,45.8,6.5,46.4 \
104+
-d 2024-07 --cloud-cover 20 --limit 3 -a red -a nir --download ./scenes --workers 8
105+
[1/6] downloaded S2B_31TGL_20240730_0_L2A/red 112.4 MiB
106+
[2/6] downloaded S2B_31TGL_20240730_0_L2A/nir 118.9 MiB
107+
[3/6] downloaded S2B_31TGL_20240727_0_L2A/red 111.7 MiB
108+
...
109+
6 asset(s): 6 downloaded -> ./scenes
110+
```
111+
112+
Run it again and nothing moves:
113+
114+
```console
115+
$ python -m stac_fetch -u earth-search -c sentinel-2-l2a --bbox 5.9,45.8,6.5,46.4 \
116+
-d 2024-07 --cloud-cover 20 --limit 3 -a red -a nir --download ./scenes
117+
[1/6] skipped S2B_31TGL_20240730_0_L2A/red -
118+
[2/6] skipped S2B_31TGL_20240730_0_L2A/nir -
119+
...
120+
6 asset(s): 6 skipped -> ./scenes
121+
```
122+
123+
Kill it half way through and start it again, and each partial file resumes from where it stopped
124+
rather than restarting.
125+
126+
### Planetary Computer
127+
128+
```bash
129+
pip install planetary-computer
130+
python -m stac_fetch -u pc -c sentinel-2-l2a --bbox 5.9,45.8,6.5,46.4 \
131+
-d last-30-days -a B04 -a B08 --download ./scenes
132+
```
133+
134+
Signing is automatic for that host. Without the optional package you get
135+
`error: Signing Microsoft Planetary Computer assets requires the 'planetary-computer' package,
136+
which is not installed. Install it with: pip install planetary-computer` — not a 403 twenty
137+
minutes into a transfer.
138+
139+
### Build a datacube
140+
141+
```bash
142+
python -m stac_fetch -u earth-search -c sentinel-2-l2a --bbox 5.9,45.8,6.5,46.4 \
143+
-d 2024-07 --cloud-cover 20 -a red -a nir \
144+
--epsg 32632 --resolution 20 --chunksize 2048 --to-zarr cube.zarr
145+
```
146+
147+
### As a library
148+
149+
```python
150+
from stac_fetch import search, download, to_datacube
151+
152+
result = search(
153+
"earth-search",
154+
collections="sentinel-2-l2a",
155+
bbox="5.9,45.8,6.5,46.4",
156+
datetime="last-30-days",
157+
cloud_cover=20,
158+
sortby="eo:cloud_cover",
159+
limit=20,
160+
)
161+
print(len(result), "of", result.matched, "truncated:", result.truncated)
162+
163+
download(result, ["red", "nir"], "./scenes", workers=8)
164+
165+
cube = to_datacube(result, assets=["red", "nir"], resolution=20, epsg=32632)
166+
ndvi = (cube.sel(band="nir") - cube.sel(band="red")) / (cube.sel(band="nir") + cube.sel(band="red"))
167+
```
168+
169+
`search()` returns a `SearchResult`, which is a `pystac.ItemCollection` subclass — it hands
170+
straight to `stackstac.stack()`, `odc.stac.load()` or anything else that eats items — with
171+
`matched`, `truncated` and `limit` attached.
172+
173+
## How it works
174+
175+
**Query construction.** The friendly forms are translated to exactly what the STAC API spec wants
176+
before a request is made. `2024-07` becomes `2024-07-01T00:00:00Z/2024-07-31T23:59:59Z` — note
177+
that the end snaps to the *last* instant of the period, which is the difference between catching
178+
and missing the last scene of the month. `last-30-days` is resolved against the current UTC time.
179+
`--cloud-cover 20` becomes `{"eo:cloud_cover": {"lte": 20}}` under the Query extension, or
180+
`{"op": "<=", "args": [{"property": "eo:cloud_cover"}, 20]}` under `--filter-lang cql2-json` for
181+
catalogs that implement the Filter extension instead. Bounding boxes are validated (range, and
182+
`minx <= maxx`) locally, because a transposed bbox otherwise comes back as a confusing empty
183+
result rather than an error.
184+
185+
**Pagination.** A STAC search response is one page with a `next` link; the item count you get from
186+
a naive `search()` is the page size, not the match count. `stac-fetch` walks every `next` link to
187+
exhaustion. When you deliberately cap the walk with `--limit`, the cap is reported — on stderr as
188+
a warning and in `--json` as `search.truncated` — so a truncated result can never be mistaken for
189+
a complete one. The `numberMatched` the server reports is carried through as `matched` for the
190+
same reason.
191+
192+
**Signing.** Planetary Computer asset hrefs are blob-store URLs that need a short-lived SAS token.
193+
Signing is applied *after* the search and only in the modes that actually read bytes, so listing
194+
stays cheap and no token is minted for a query you were only eyeballing. `--sign auto` (the
195+
default) keys off the catalog host; `--sign none` and `--sign planetary-computer` override it.
196+
197+
**Downloads.** Each asset goes to `<name>.part` first and is `os.replace`d into place only after
198+
it verifies, so a destination file is either absent or complete — never a truncated GeoTIFF that
199+
fails three pipeline stages later. If a `.part` file survives an interrupted run, the next run
200+
sends `Range: bytes=N-`; a server that ignores the range and answers `200` triggers a clean
201+
restart rather than a corrupt append, and a `416` discards the partial and starts over.
202+
Verification uses `file:size` from the File extension when the catalog publishes it, otherwise the
203+
`Content-Length` of the response; a `file:checksum` multihash is verified when it is a sha2-256
204+
one. Skip-if-present compares the existing file's size to `file:size` where known, so a file
205+
truncated by an earlier crash is refetched instead of trusted. Transfers run on a thread pool,
206+
which is the right shape for this: the work is entirely I/O-bound on remote object storage.
207+
208+
**Datacube.** `stackstac.stack()` (or `odc.stac.load()`) reads only STAC metadata to lay out the
209+
grid, so building the cube costs no pixel reads — the Dask graph is materialised lazily when you
210+
compute or write. `--chunksize` sets the Dask chunk in pixels along x and y, which is the knob
211+
that decides whether the eventual compute fits in memory.
212+
213+
## Options
214+
215+
| Flag | Meaning |
216+
| --- | --- |
217+
| `-u, --url URL\|PRESET` | STAC API root, or a preset: `planetary-computer` (`pc`), `earth-search` (`aws`), `landsatlook` (`usgs`). Default `earth-search` |
218+
| `-c, --collection ID` | Collection to search; repeatable |
219+
| `--header K=V` | Extra HTTP header, e.g. `--header 'Authorization=Bearer ...'`; repeatable |
220+
| `--sign {auto,planetary-computer,none}` | Asset signing mode. Default `auto` |
221+
| `-d, --datetime SPEC` | `2024-06-01`, `2024-06`, `2024`, `2024-06-01/2024-08-31`, `2024-06-01/..`, `last-30-days`, `last-6-months`, `today`, `yesterday` |
222+
| `--bbox MINX,MINY,MAXX,MAXY` | Bounding box in EPSG:4326 |
223+
| `--aoi PATH\|WKT` | GeoJSON file, inline GeoJSON, or WKT geometry; its bounding box is used |
224+
| `-q, --query EXPR` | Property filter, e.g. `platform=sentinel-2a`, `view:off_nadir<=10`; repeatable |
225+
| `--cloud-cover PCT` | Shortcut for `eo:cloud_cover<=PCT` |
226+
| `--filter-lang {query,cql2-json}` | Filter encoding. Default `query` |
227+
| `--limit N` | Stop after N items in total (reported as truncation) |
228+
| `--page-size N` | Items per HTTP request. Default 100 |
229+
| `--sortby KEYS` | Comma-separated sort keys, `-` prefix for descending |
230+
| `--print` / `--json` | Table (default) or machine-readable JSON |
231+
| `--download DIR` | Download `--asset` keys into DIR |
232+
| `--to-zarr PATH` | Build a datacube and write it to a Zarr store |
233+
| `-a, --asset KEY` | Asset key to download or stack; repeatable |
234+
| `--workers N` | Concurrent downloads. Default 4 |
235+
| `--overwrite` | Re-download files that already exist |
236+
| `--per-item-dirs` | One subdirectory per item instead of a flat directory |
237+
| `--no-progress` | Suppress per-file progress lines |
238+
| `--engine {auto,stackstac,odc-stac}` | Datacube backend. Default `auto` |
239+
| `--resolution M`, `--epsg CODE`, `--bounds`, `--chunksize N` | Datacube grid controls |
240+
| `--list-catalogs` | Print the presets and exit |
241+
| `--quiet` | Suppress informational stderr messages |
242+
243+
Exit codes: `0` success, `1` the operation failed (no matches, unreachable catalog, missing asset
244+
key, failed transfer), `2` the invocation was invalid (bad bbox, unparseable datetime, conflicting
245+
flags). Errors are always a single `error: ...` line, never a traceback.
246+
247+
### Library API
248+
249+
```python
250+
search(url, *, collections=None, datetime=None, bbox=None, aoi=None, intersects=None,
251+
query=None, cloud_cover=None, limit=None, page_size=None, sortby=None,
252+
filter_lang=None, headers=None, client=None) -> SearchResult
253+
254+
download(items, assets, dest, *, workers=4, overwrite=False, flat=True,
255+
progress=True, session=None, timeout=(10, 120)) -> list[DownloadResult]
256+
257+
to_datacube(items, *, assets=None, resolution=None, epsg=None, bounds=None,
258+
chunksize=1024, engine="auto", fill_value=None)
259+
260+
write_zarr(cube, path, *, mode="w") -> str
261+
```
262+
263+
## Limitations
264+
265+
- `--aoi` uses the geometry's **bounding box**, not the polygon itself. For true polygon
266+
intersection pass a GeoJSON geometry to `search(intersects=...)` from Python.
267+
- Bounding boxes that cross the antimeridian are rejected rather than silently interpreted. Split
268+
them into two searches.
269+
- `--sortby` is forwarded to the catalog. Not every STAC API implements the Sort extension, and
270+
several ignore it silently — check the returned order before relying on it.
271+
- `--query` covers the six comparison operators (`=`, `!=`, `<`, `<=`, `>`, `>=`). There is no
272+
`IN`, `LIKE` or spatial predicate; for those, build a CQL2 document and use `pystac-client`
273+
directly.
274+
- Checksums are only verified for sha2-256 `file:checksum` multihashes. Other algorithms are
275+
ignored rather than treated as failures.
276+
- Resume relies on the server honouring HTTP `Range`. Most object stores do; one that does not
277+
causes a full re-download, not a corrupt file.
278+
- The datacube mode is a thin, well-typed wrapper over `stackstac`/`odc-stac`. Anything beyond
279+
resolution, CRS, bounds and chunking is better done on the returned xarray object.
280+
281+
## Further reading
282+
283+
Background guides that go deeper on the mechanics this tool automates:
284+
285+
- [Paginating large STAC searches with pystac-client](https://www.python-remote-sensing.com/core-raster-fundamentals-stac-mapping/querying-stac-catalogs-programmatically/paginating-large-stac-searches-with-pystac-client/) — why a naive `items()` call under-reports, and how the `next`-link walk works.
286+
- [Using pystac-client to filter Sentinel-2 imagery by date](https://www.python-remote-sensing.com/core-raster-fundamentals-stac-mapping/querying-stac-catalogs-programmatically/using-pystac-client-to-filter-sentinel-2-imagery-by-date/) — the datetime interval semantics behind `--datetime`.
287+
- [stackstac vs odc-stac for STAC-to-array](https://www.python-remote-sensing.com/core-raster-fundamentals-stac-mapping/stackstac-vs-odc-stac-for-stac-to-array/) — which backend to pick for `--engine`, and how they differ.
288+
- [Reading a COG over S3 without downloading](https://www.python-remote-sensing.com/core-raster-fundamentals-stac-mapping/understanding-cloud-optimized-geotiff-structure/reading-a-cog-over-s3-without-downloading/) — when you should skip `--download` entirely.
289+
- [Reducing S3 egress costs in raster pipelines](https://www.python-remote-sensing.com/cloud-execution-and-orchestration/optimizing-pipeline-cost-and-performance/reducing-s3-egress-costs-in-raster-pipelines/) — why skip-if-present and resume matter once transfers are billed.
290+
291+
## Contributing
292+
293+
Issues and pull requests are welcome. Please keep the test suite hermetic — no test may touch the
294+
network; add STAC JSON to `tests/fixtures/` and mock HTTP with `responses`. Before opening a PR:
295+
296+
```bash
297+
pip install -r requirements-dev.txt
298+
ruff check . && ruff format --check . && python -m pytest
299+
```
300+
301+
## License
302+
303+
MIT — see [LICENSE](LICENSE).
304+
305+
Built and maintained alongside [Python Remote Sensing & Raster Processing Pipelines](https://www.python-remote-sensing.com/).

‎pyproject.toml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
[tool.ruff]
2+
line-length = 100
3+
target-version = "py312"
4+
5+
[tool.ruff.lint]
6+
select = ["E", "F", "I", "UP", "B", "SIM", "C4"]
7+
8+
[tool.pytest.ini_options]
9+
testpaths = ["tests"]
10+
addopts = "-q --strict-markers"
11+
markers = [
12+
"datacube: requires an installed datacube backend (stackstac or odc-stac)",
13+
]

‎requirements-datacube.txt‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# Optional: needed only for --to-zarr / to_datacube().
2+
-r requirements.txt
3+
stackstac>=0.5.1
4+
xarray>=2024.3
5+
dask[array]>=2024.3
6+
rioxarray>=0.15
7+
zarr>=2.17
8+
# Optional: needed only for Microsoft Planetary Computer asset signing.
9+
planetary-computer>=1.0

‎requirements-dev.txt‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
-r requirements.txt
2+
pytest>=8.0
3+
responses>=0.25
4+
ruff>=0.6

0 commit comments

Comments
 (0)