Abort on failed gdalcubes chunk reads before caching (#87)

[NewGraphEnvironment]

Summary

• A failed gdalcubes image open now aborts (drift_incomplete_cube) with nothing cached. This covers dft_stac_cube(), dft_stac_composite() and dft_stac_fetch(), tiled and untiled. The failure is an expired, corrupted or refused signed URL, which held 15 of 30 tiles in Dated reference imagery: true-colour composites, NDWI/MNDWI, and RGB layers in dft_map_interactive (+ HLS source) #79. cube_write_ncdf() is now drift's single write_ncdf() call. It checks two things:
  • the per-chunk chunk_status gdalcubes writes into every output, which reaches R from worker processes too, unlike the stderr line;
  • gdalcubes' "could not be added to output" merge warning. The abort comes after the write returns, so gdalcubes' C++ cleanup runs.
• Untiled fetch caches written before this that record failed chunks are treated as a cache miss and re-fetched.
• Supporting changes:
  • GDAL HTTP retries in the cube session and in every fetch;
  • tiled fetch cleans up its tile files on abort;
  • the offset-split path no longer loses the abort class inside terra::cover()'s S4 dispatch, a bug the new tests found;
  • ncdf4 added to Suggests (gdalcubes imports it).

What this does NOT catch

gdalcubes records only one failure: a band image that will not open. Every other I/O step drops its return code. Code-check round 3 enumerated 38 failure paths in the write pipeline: drift now detects 25, and 13 stay silent. The silent ones include:

• an image that opens and then fails mid-read (throttling);
• a mask image that opens and then fails mid-read, which leaves the scene unmasked;
• missing or unreadable worker chunk files;
• a worker killed by a signal, which hangs write_ncdf().

These are #99, with an upstream gdalcubes draft that has not been posted. For caches other than untiled fetch (cube, composite, tiled fetch .tif), entries written before this version cannot be re-checked. Long tiled runs from before it are worth re-running with force = TRUE, and the NEWS entry should say so.

Measurement

• Offline fixture (one image deleted after the collection is built): chunk_status 9 × INCOMPLETE of 279 chunks at parallel 1 and 4. The stderr line appeared at 1 only. A monthly median filled 1600 of 1600 cells from the surviving scene, so a pixel post-check could never have caught it.
• Live, 8 of 8 (data-raw/probe_chunk_status_live.R, log data-raw/logs/probe_chunk_status_live/20261002T140832Z.txt):
  • Bad tokens on every asset: composite untiled and tiled, cube, and fetch untiled and tiled all aborted with 0 cache files in 2–10 s.
  • Good-token controls returned and cached: cube over months 6:9 at parallel 4 (266 s), tiled composite at parallel 4 (142 s), untiled fetch (8 s).
• Scale (CLAUDE.md BULK rule). The check reads one integer per chunk and no pixels: 2.3 ms on a 2000 × 2000 cube of 7,936 chunks, the size of one BULK tile at tile_size = 20000. No BULK RSS run, since there is no new raster work to measure.
• devtools::test(): FAIL 0 | PASS 1549 | SKIP 16.

Related Issues

• Fixes Failed gdalcubes chunk reads are silent: a partial cube passes the empty check and is cached #87
• Follow-ups: gdalcubes read failures after open, mask-open failures and worker crashes are still silent #99 (failures gdalcubes does not record), NewGraphEnvironment/soul#316 (the code-check rule this corrects)
• Relates to Dated reference imagery: true-colour composites, NDWI/MNDWI, and RGB layers in dft_map_interactive (+ HLS source) #79, Cube read path: date-only end bound drops the last day, no #51 paging guards, Jan 2022 cover bias, miss/hit drift, temp NetCDFs #83
• Relates to NewGraphEnvironment/sred-2025-2026#16

Test plan

• Offline integration tests (real gdalcubes writer, local fixture, parallel = 4): red before the wiring, green after, red again with the wrapper reduced to a bare write_ncdf()
• Composite: a holed year aborts rather than being skipped as "no scenes"; offset split with the failure on the post side
• Clean-read controls cached as before; cache keys unchanged (frozen-key tests untouched)
• Live corrupted-token probe, 8 of 8
• /code-check: plan review plus three rounds, ended by enumeration (planning/archive/2026-10-issue-87-silent-chunk-read-failures/)

Notes

• Reviewer agents for this task: four (plan review, code-check rounds 1–3; round 3 carried the enumeration). Rounds 2 and 3 each found a defect inside the previous round's fix.
• The gdalcubes [WARNING] n out of m chunks line still prints during the broken-fixture tests. It is C++ stdio, and no R handler can silence it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PRhUJsuKABLfpBGktPoiBN