Skip to content

Fix 6 item JSONs with wrong S3 permissions (092p045) #13

Description

@NewGraphEnvironment

Problem

6 item JSONs from mapsheet 092p045 return 403 Forbidden from S3, preventing STAC API registration. Discovered during baseline registration benchmark of 58,019 items on geodev.

Affected items:

  • 092-092p-2024-dem-bc_092p045_1_2_2_xli1m_utm10_20240721_20240721
  • 092-092p-2024-dem-bc_092p045_1_2_3_xli1m_utm10_20240721_20240721
  • 092-092p-2024-dem-bc_092p045_1_2_4_xli1m_utm10_20240721_20240721
  • 092-092p-2024-dem-bc_092p045_1_3_1_xli1m_utm10_20240721_20240721
  • 092-092p-2024-dem-bc_092p045_1_3_2_xli1m_utm10_20240721_20240721
  • 092-092p-2024-dem-bc_092p045_1_3_3_xli1m_utm10_20240721_20240721

All are consecutive tiles from the same mapsheet and date (2024-07-21), suggesting they were synced with incorrect ACL or missed by the sync entirely.

Proposed Solution

  1. Fix the 6 items: re-sync with --acl public-read or check if files exist locally
  2. Add a permissions validation step to the S3 sync workflow (s3_sync.R) that verifies public read access after upload
  3. Consider a post-sync smoke test that spot-checks a sample of uploaded items for accessibility

Context

  • Registration log: 58,013/58,019 items succeeded, 6 failed with S3 AccessDenied
  • The s3_sync.R script uses aws s3 sync --delete --quiet but doesn't verify ACLs
  • Catching permission issues early prevents silent data gaps in the STAC catalog

Relates to NewGraphEnvironment/sred-2025-2026#3

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions