Skip to content

feat: bound CAS downloads with per-file size caps (#31)#37

Merged
LiranCohen merged 1 commit into
masterfrom
feat/bounded-cas-file-caps
Jun 4, 2026
Merged

feat: bound CAS downloads with per-file size caps (#31)#37
LiranCohen merged 1 commit into
masterfrom
feat/bounded-cas-file-caps

Conversation

@LiranCohen

Copy link
Copy Markdown
Contributor

Summary

Sidetree caps each anchored file's size and bounds gzip expansion (zip-bomb guard). Our reader enforced none of it — the five fetch* methods called cas.Get unbounded, and two // TODO Check Max … File Size markers (core_index.go, chunk.go) were never resolved. A malicious/corrupt CID could expand into an unbounded in-memory parse.

This threads the per-file cap into the download, mirroring the reference DownloadManager.download(uri, maxSizeInBytes). Resolves both TODOs. Plan: docs/plans/2026-06-04-001-feat-ion-value-locking-protocol-rules-plan.md.

Per-file caps (from params.go, #30)

File Cap
core index MaxCoreIndexFileSizeInBytes (1 MB)
core proof MaxProofFileSizeInBytes (2.5 MB, shared)
provisional index MaxProvisionalIndexFileSizeInBytes (1 MB)
provisional proof MaxProofFileSizeInBytes (2.5 MB, shared)
chunk MaxChunkFileSizeInBytes (10 MB)

Change

  • CAS.Get gains a maxSizeInBytes parameter. Its documented contract: bound the (compressed) download to maxSizeInBytes, and bound decompression to maxSizeInBytes × MaxMemoryDecompressionFactor (3×); oversized content is permanently invalid (immutable CID) → return an ErrMalformed-class error.
  • Each fetch* passes the protocol cap for its file type and defensively re-checks the returned (decompressed) length against cap × MaxMemoryDecompressionFactor — the maximum legal decompressed size — in case a CAS doesn't honor the contract. This defensive bound never rejects a valid anchor: it is exactly ION's decompression ceiling.
  • Oversized files route through classifyMalformedErrMalformed (permanent skip). New sentinel ErrFileTooLarge.

⚠️ Downstream (cross-repo)

ion-node's CAS adapter must add the maxSizeInBytes parameter to Get at its next sidetree-go repin (#53), and should enforce the bounded download + 3× decompression cap there (the CAS owns the compressed stream). ion-node pins a fixed sidetree-go commit, so its build is unaffected until that repin — no breakage now.

Testing

  • TestProcessorRejectsOversizedFile — a > cap×3 file → ErrFileTooLarge, asserted ErrMalformed.
  • TestFetchPassesPerFileSizeCaps — a full five-file batch processes cleanly and each fetch passes the correct per-file cap (captured by the test CAS).
  • The test/contract CAS implementations adopt the new signature. gofmt clean; go build, go vet, go test -race -count=1 ./... all green.

Post-Deploy Monitoring & Validation

  • What to monitor: observer logs for ErrMalformed skips carrying ErrFileTooLarge ("sidetree file exceeds its maximum size").
  • Expected healthy signal: effectively zero on ION mainnet — canonical files are well under cap. The sidetree-go defensive check only fires above cap×3 (ION's own ceiling), so it cannot reject a valid anchor.
  • Failure signal / rollback trigger: ErrFileTooLarge on a known-good mainnet anchor would indicate a wrong cap or a decompression-ratio surprise → revert and investigate.
  • Validation window & owner: first full mainnet resync post-deploy; owner = indexer maintainer.

🤖 Generated with Claude Code

Sidetree caps the size of each anchored file (core index 1MB, core/provisional
proof 2.5MB shared, provisional index 1MB, chunk 10MB) and bounds gzip
expansion to MaxMemoryDecompressionFactor (3x) as a zip-bomb guard. Our reader
enforced none of it: fetchCoreIndexFile/CoreProof/ProvisionalIndex/Provisional
Proof/Chunk called cas.Get unbounded, and two `// TODO Check Max ... File Size`
markers (core_index.go, chunk.go) were never resolved. A malicious or corrupt
CID could expand into an unbounded in-memory parse.

Thread the per-file cap into the download the way the reference
DownloadManager.download(uri, maxSizeInBytes) does:

  - CAS.Get gains a maxSizeInBytes parameter. Its contract: bound the
    (compressed) download to maxSizeInBytes and bound decompression to
    maxSizeInBytes * MaxMemoryDecompressionFactor; oversized content is
    permanently invalid (immutable CID), so return an ErrMalformed-class error.
  - Each fetch* method passes the protocol cap for the file type it fetches and
    defensively re-checks the returned (decompressed) length against
    cap * MaxMemoryDecompressionFactor (the maximum legal decompressed size), in
    case a CAS does not honor the contract. This bound never rejects a valid
    anchor — it is exactly ION's decompression ceiling.

Oversized files route through classifyMalformed -> ErrMalformed (permanent skip).
New sentinel ErrFileTooLarge. Resolves the two file-size TODOs.

Downstream: ion-node's CAS adapter must add the maxSizeInBytes parameter to Get
at its next sidetree-go repin (#53) and should enforce the bounded download +
3x decompression cap there (the CAS owns the compressed stream). ion-node pins a
fixed sidetree-go commit, so its build is unaffected until that repin.

Tested: TestProcessorRejectsOversizedFile (a >cap*3 file -> ErrFileTooLarge,
asserted ErrMalformed) and TestFetchPassesPerFileSizeCaps (a full five-file
batch processes cleanly and each fetch passes the correct per-file cap, captured
by the test CAS). The test/contract CAS implementations adopt the new signature.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@LiranCohen
LiranCohen merged commit 7de9e58 into master Jun 4, 2026
1 check passed
@LiranCohen
LiranCohen deleted the feat/bounded-cas-file-caps branch June 4, 2026 16:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant