Shard-map invoke payloads exceed the 256 KB async cap — normalize to ids-only + zstd envelope; presigned-URL staging as the durable fix

[espg]

🤖 from Claude

Problem

Shard-map invoke payloads are hitting the Lambda async event cap (256 KB). The orchestrator holds no S3 credentials (deliberate divide: lambdas write, the orchestrator doesn't), so today the only channel for a shard map is the invoke payload itself — and measured against the real benchmark fixtures, the as-shipped JSON is over the cap across the board, not just on monster shards:

fixture (tests/data/benchmark/shardmaps/)	granules	JSON as b64 event	vs 256 KB
sm_healpix_o10_88s.json (1 shard)	4,605	2.15 MB	8.4× over
sm_healpix_o9_88s.json (1 shard)	5,620	2.62 MB	10.2× over
sm_healpix_o11.json (22 shards)	737	347 KB	1.35× over

(b64: an event payload is JSON, so binary/compressed bytes pay a 4/3 base64 tax; all "event" numbers below include it.)

Why the payload is fat: derivable-URL denormalization

Each granule record is {id, s3, https} — and both URLs are derivable from the id (same filename; the date path parses out of ATL03_YYYYMMDDHHMMSS…). ~89% of every record is redundancy. That shape also decides the format question below.

Measured options (real fixtures, zstd level 3 via pyarrow)

encoding	o10_88s (4,605)	o9_88s (5,620)	o11 (737)
json as shipped (event)	2.15 MB ❌	2.62 MB ❌	347 KB ❌
json + zstd (event)	93 KB ✅ 23×	111 KB ✅ 24×	4.9 KB ✅ 70×
parquet + zstd (event)	172 KB ✅ 12.5×	205 KB ✅ 13×	8.6 KB ✅ 40×
json ids-only (event)	259 KB ❌ 8.3×	316 KB ❌ 8.3×	43 KB ✅ 8×
json ids-only + zstd (event)	45 KB ✅ 48×	54 KB ✅ 48×	2.4 KB ✅ 144×

Notable:

• Parquet-over-the-wire loses to plain zstd-JSON — by 2×. On string-heavy, cross-record-redundant records, whole-blob zstd exploits the redundancy globally; parquet pays per-column page overhead and per-value string lengths at these sizes. (Parquet's wins — columnar scans, selective reads — don't apply to "decode the whole map once in a worker".) An "orchestrator sends parquet, worker re-emits JSON" pipeline was considered and is dominated on this data; it also would have squeaked under the cap with only ~3% headroom at the 6k-granule scale on synthetic records.
• Normalization alone is not enough (o9_88s ids-only is still over) — it needs the zstd too.
• The records carry no geometry — the one shape where parquet/WKB would win. If footprints ever move into these payloads, re-measure.

Recommendation (sequenced)

1. Now: ids-only + zstd envelope. Ship granule ids plus URL templates in metadata; workers reconstruct s3/https (two-line template expansion — the collection is already known). Wrap every invoke in a versioned, unconditional envelope, e.g. {v: 1, codec: "zstd+json", data: <b64>} — no size threshold: a threshold is two codepaths plus a cliff rediscovered in production. Add an orchestrator-side size guard that fails loudly before the Invoke API rejects. Result: worst real fixture becomes a 54 KB event (48×); headroom extrapolates to ~25–30k granules/shard.
2. Durable: presigned-URL staging (claim-check that honors the creds divide). A lambda mints a scoped, expiring presigned PUT URL (tiny sync invoke); the orchestrator HTTP-PUTs the map directly — never holding S3 credentials — and async-invokes workers with the object key + size/hash. Removes the ceiling category entirely; events stay small, loggable, replayable; the at-rest format becomes a pure reader-side choice. (A simpler "stager" sync-invoke variant — worker PUTs the 6 MB-capped sync payload and returns a key — works but caps at 6 MB and moves the bytes through Lambda twice.)

Rejected: parquet/arrow-IPC on the wire (dominated, above — arrow IPC additionally doesn't fit at 6k granules); threshold-switched dual formats (permanent two-path reader complexity for every consumer, cliff merely postponed).

Why the durable fix matters even at 48×

The _88s fixtures are the polar-convergence shard: ICESat-2's 92° inclination drives essentially every track through the 88°S ring, so that shard's granule count grows ~linearly with mission duration. ~25–30k granules of headroom is comfortable today and is still a date, not a fix — which is what item 2 is for.

Side note: this closes the "why is the root MOC JSON?" question too

The same measurement run covered mortie spec §7.3 (coverage.moc root ranges): on real Antarctic-basin covers, the spec's runs transform does the heavy lifting (12–98× cardinality reduction, growing with order), leaving the worst matrix case (191k cells) at 70 KB of spec JSON for a read-once-per-open object — with parquet saving only ~64 KB more at the cost of a browser-side arrow-wasm dependency (gridlook phase 6 reads it with JSON.parse) and a spec change. The spec's tiering is already right: declarations in JSON, the size-sensitive tier (leaf bitmaps, §7.2) already binary+zstd, bulk cell data in arrow/parquet via the interchange docs. No change recommended there.

Repro

Fixtures: tests/data/benchmark/shardmaps/. Encoders: zstd via pyarrow.compress (level default), parquet compression="zstd" with a shard-index column preserving bucketing. Numbers above are from the current fixtures on main; the MOC-side numbers are from mortie main (morton_coverage flat covers at orders 8/10/12, §7.3 run computation, §1 word layout).

[espg]

🤖 from Claude

Implementation plan

Grounded against current main (d104dbb): the payload that blows up is assembled in _event(shard_key, granules, ...) (src/zagg/runner.py:1310), which inlines the full ShardMap granules entries built by _granule_entry() (src/zagg/catalog/shardmap.py:~533). A byte guard already exists on one dispatch path (_ASYNC_PAYLOAD_CAP_BYTES, runner.py:~3971, raises with guidance), and issue #313 established the inline-vs-prefix fallback precedent for stats rows — this plan generalizes both.

Phase 1 — ids-only + zstd envelope (now)

1. Map schema v2 (ids + templates). _granule_entry() emits {"id": ...} only; ShardMap metadata gains url_templates (s3/https patterns with {id}/date-derived fields) and a shardmap_version: 2 marker. One expansion helper in catalog/shardmap.py — expand_granule(id, templates) -> {id, s3, https} — so every consumer sees the v1 record shape after load. Readers (worker input parse, viz/shardmap.py, catalog loaders) accept v1 and v2 during transition; writers emit v2 only. Benchmark fixtures get v2 twins; a golden test asserts v2-expanded == v1 records byte-for-byte.
2. Versioned wire envelope, unconditional. Granule-bearing invokes wrap as {"v": 1, "codec": "zstd+json", "data": <b64>} — no size threshold (a threshold is two codepaths plus a cliff rediscovered in production). Applied in _event() and the other granule-carrying invoke assembly points; the worker handler detects codec, decompresses, and proceeds through the existing parse. Compression via numcodecs.Zstd (already in the worker image with zarr — no new dependency).
3. Guard everywhere. Extend the existing _ASYNC_PAYLOAD_CAP_BYTES check to every InvocationType="Event" assembly point, measuring the post-envelope payload, failing loudly with the fixture-calibrated numbers in the message (and, once Phase 2 lands, pointing at staging instead of failing).
4. Tests / acceptance. Envelope round-trip; size-regression test pinning sm_healpix_o9_88s.json (worst real fixture) to a ≤ ~60 KB event (measured: 54 KB, 48×); guard raises on a synthetic overflow; existing fixture-driven tests pass against v2 twins.
5. Rollout order. Workers (readers) deploy first — accepting envelope + v2 — then the orchestrator (writer) starts emitting them. One release window, no flag day.

Phase 2 — presigned-URL staging (follow-up)

Design principle: one format, two transports. The staged object is the same Phase-1 envelope bytes; the worker has a single decode path — fetch bytes if by-reference, then the identical envelope decode. This is the claim-check pattern shaped to the creds divide (the orchestrator never holds S3 credentials), and it mirrors the existing result_url / result_fetch machinery in reverse (outbound results already land at pinned keys; this does the same for inbound payloads).

1. Presign mode on the worker lambda (mode flag, not a new function): a small sync invoke returns a scoped, expiring (~15 min) presigned PUT for a deterministic key — runs/{run_id}/shardmap-{sha256[:16]}.json.zst — under the existing bucket allowlist.
2. Orchestrator dispatch: request URL → HTTP PUT the envelope bytes (no AWS creds involved) → async-invoke with {"payload_ref": {"uri", "size", "sha256"}} instead of inline data. Once stable, make by-reference the default for shard dispatch (by-value remains for small control events) — the reader can't tell the difference except for one GET.
3. Worker: payload_ref present → GET (workers already read S3), verify size + sha256 (mismatch fails loudly, no partial processing), then the Phase-1 decode.
4. Hygiene: presign-expiry retries mint a fresh URL; staged objects are covered by a lifecycle rule on the runs/ prefix (or folded into existing run cleanup); the deterministic hash-key makes re-fires idempotent (re-PUT of identical bytes is a no-op).
5. Acceptance: the 88S fixtures dispatch by reference end-to-end in the integration test; the Phase-1 guard becomes an assertion that never trips; a chaos test covers expired-URL retry and hash-mismatch rejection.

Sequencing rationale

Phase 1 is small, ships inside the current architecture, and buys ~25–30k granules/shard of headroom (48× on real fixtures) — it un-blocks the 88S shards immediately. Phase 2 removes the ceiling category (the polar-convergence shard grows with mission duration, so Phase 1 alone is a date, not a fix) and can land at leisure since Phase 1's envelope is exactly the payload it stages. Refs: #313 (budget/fallback precedent), measurement tables above.

[espg]

I think that phase 1 basically covers all our issues with switching from async to sync based on payload size; seems robust and solves our problem.

Phase 2 is more about for follow on operations... things like what we implemented in #300 / #311 ; phase 2 opens up options to have those be orchestrator operations that run client side and just call lambda for temp write privileges. The question is... is that a good idea? Is the sweep work computationally small enough that we want to run it client side, or does it still have worker / fan out shaped model that makes sense? And are we happy with the security model? What are your thoughts here? Can you walk me through how #311 is setup on pyrimid building / sweeps now?