Skip to content

Exit non-zero on a failed download, and pin fetchez to a version with the auth fix - #100

Merged
mmacferrin merged 3 commits into
mainfrom
earthdata-download-failures
Sep 1, 2026
Merged

Exit non-zero on a failed download, and pin fetchez to a version with the auth fix#100
mmacferrin merged 3 commits into
mainfrom
earthdata-download-failures

Conversation

@mmacferrin

@mmacferrin mmacferrin commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Two loose ends from #23, which is otherwise resolved between fetchez's
redirect fix and IVERT's .netrc handling (#46, #74).

Pin fetchez to >= 0.6.13

The JSONDecodeError in #23 came from fetchez injecting the Earthdata
credentials into the initial Harmony request headers: the redirect to
Earthdata Login dropped them, Earthdata served its HTML login page, and
parsing that page as JSON blew up. matth-love fixed this in
continuous-dems/fetchez#277 with a redirect-safe requests.Session,
first released in fetchez v0.6.13.

IVERT listed fetchez unpinned, so a fresh install could still resolve a
version with the bug and reproduce the original report.

Exit non-zero when part of a download fails

The last open comment on #23: the error was reported, then "skipping"
logic ran and the program exited 0.

A Harmony request that came back with nothing logged a warning, skipped
that sub-region and moved on, and ivert database download still exited
0 — so a download that retrieved nothing at all looked exactly like a
successful one, and a script driving IVERT had no way to tell.

download_new_granules() now returns a DownloadSummary counting how
each sub-region turned out, and the CLI raises a ClickException when
any part failed.

The distinction that matters: only a Harmony request that returned
nothing counts as a failure. A sub-region Harmony served correctly
that added no new granules — the region holds no ICESat-2 data, or every
granule over it is already in the database — is counted separately and is
still a success. Errors are still counted rather than raised inside the
loop, so one unreachable sub-region does not abandon the rest of the
request; whatever did download is saved, and re-running retries only the
parts still missing.

Partial failure also exits non-zero, since a failed part leaves a hole in
the region that was asked for.

Testing

Exercised through the real CLI path with a stubbed IS2Database:

Summary Exit Message
total failure 1 "The download failed. No ICESat-2 data was retrieved."
partial (1 of 3 failed, 5 granules saved) 1 names the counts, says a re-run retries only what's missing
datum mismatch 1 the specific logger.error prints at every verbosity level
clean success 0
region with no data 0
nothing new to download 0

An AST pass over download_new_granules confirms no bare return
remains, so the CLI cannot receive None. prek run --all-files passes.

Follow-up

The conda-forge feedstock recipe still lists fetchez unpinned and needs
the same >= 0.6.13 bound, or a conda install keeps the hole this closes.


🔍 Docs preview: https://ivert--100.org.readthedocs.build/en/100/

Earlier fetchez releases injected the Earthdata credentials into the
initial Harmony request headers, and the redirect to Earthdata Login
dropped them. Earthdata served its HTML login page instead, and the
download died parsing that page as JSON. fetchez 0.6.13 is the first
release carrying the redirect-safe requests.Session that fixes it.

The dependency was unpinned, so a fresh install could still resolve a
version with the bug.
A Harmony request that came back with nothing logged a warning, skipped
that sub-region and moved on to the next, and 'ivert database download'
still exited 0. A download that retrieved no data at all was therefore
indistinguishable from a successful one, and a script driving IVERT had
no way to tell.

download_new_granules() now returns a DownloadSummary counting how each
sub-region of the request turned out, and the CLI raises a
ClickException when any part failed.

Only a Harmony request that returned nothing counts as a failure. A
sub-region Harmony served correctly that added no new granules -- it
holds no ICESat-2 data, or every granule over it is already in the
database -- is counted separately and is still a success. Whatever did
download is saved, and re-running the command retries only the parts
still missing.
@mmacferrin
mmacferrin merged commit 7a1b3ce into main Sep 1, 2026
5 checks passed
@mmacferrin
mmacferrin deleted the earthdata-download-failures branch September 1, 2026 22:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant