Problem
6 item JSONs from mapsheet 092p045 return 403 Forbidden from S3, preventing STAC API registration. Discovered during baseline registration benchmark of 58,019 items on geodev.
Affected items:
092-092p-2024-dem-bc_092p045_1_2_2_xli1m_utm10_20240721_20240721
092-092p-2024-dem-bc_092p045_1_2_3_xli1m_utm10_20240721_20240721
092-092p-2024-dem-bc_092p045_1_2_4_xli1m_utm10_20240721_20240721
092-092p-2024-dem-bc_092p045_1_3_1_xli1m_utm10_20240721_20240721
092-092p-2024-dem-bc_092p045_1_3_2_xli1m_utm10_20240721_20240721
092-092p-2024-dem-bc_092p045_1_3_3_xli1m_utm10_20240721_20240721
All are consecutive tiles from the same mapsheet and date (2024-07-21), suggesting they were synced with incorrect ACL or missed by the sync entirely.
Proposed Solution
- Fix the 6 items: re-sync with
--acl public-read or check if files exist locally
- Add a permissions validation step to the S3 sync workflow (
s3_sync.R) that verifies public read access after upload
- Consider a post-sync smoke test that spot-checks a sample of uploaded items for accessibility
Context
- Registration log: 58,013/58,019 items succeeded, 6 failed with S3
AccessDenied
- The
s3_sync.R script uses aws s3 sync --delete --quiet but doesn't verify ACLs
- Catching permission issues early prevents silent data gaps in the STAC catalog
Relates to NewGraphEnvironment/sred-2025-2026#3
Problem
6 item JSONs from mapsheet 092p045 return 403 Forbidden from S3, preventing STAC API registration. Discovered during baseline registration benchmark of 58,019 items on geodev.
Affected items:
092-092p-2024-dem-bc_092p045_1_2_2_xli1m_utm10_20240721_20240721092-092p-2024-dem-bc_092p045_1_2_3_xli1m_utm10_20240721_20240721092-092p-2024-dem-bc_092p045_1_2_4_xli1m_utm10_20240721_20240721092-092p-2024-dem-bc_092p045_1_3_1_xli1m_utm10_20240721_20240721092-092p-2024-dem-bc_092p045_1_3_2_xli1m_utm10_20240721_20240721092-092p-2024-dem-bc_092p045_1_3_3_xli1m_utm10_20240721_20240721All are consecutive tiles from the same mapsheet and date (2024-07-21), suggesting they were synced with incorrect ACL or missed by the sync entirely.
Proposed Solution
--acl public-reador check if files exist locallys3_sync.R) that verifies public read access after uploadContext
AccessDenieds3_sync.Rscript usesaws s3 sync --delete --quietbut doesn't verify ACLsRelates to NewGraphEnvironment/sred-2025-2026#3