diff --git a/rfcs/2026-10-blake3-object-digest.md b/rfcs/2026-10-blake3-object-digest.md new file mode 100644 index 0000000..f250064 --- /dev/null +++ b/rfcs/2026-10-blake3-object-digest.md @@ -0,0 +1,313 @@ +# RFC: BLAKE3 object digest + +Status: Experimental + +## Authors + +- [Alan Shaw](https://github.com/alanshaw) + +## Motivation + +The granularity at which a client can verify an Ingot object today depends on how the object was uploaded. A single `PUT` records nothing in the manifest (the per-object metadata record Ingot keeps in the bucket's catalog) about any piece of the object smaller than the whole. A client can verify such an object only in full, against a digest it obtained elsewhere, which means downloading all of it first and is impractical for a large object. A multipart object records per-part values (described next) but no whole-object digest at all, because SHA-256 cannot be assembled from the parts and re-reading every part at `CompleteMultipartUpload` is too slow. + +The one verifiable value is the per-part checksum list, and it exists only if the uploader asked for it. A multipart object uploaded with a SHA checksum algorithm, `--checksum-algorithm SHA256` or its SDK equivalent, has a checksum per part and a composite over them, and `GetObjectAttributes` returns the list after the upload completes. That is a two-level tree the client can check part-aligned ranges against, and check against the composite. No client does this by default: the AWS SDKs and CLI now send a checksum unprompted, but it is a CRC, which gives a full-object value with no per-part list; rclone, s5cmd and the MinIO client send none. The list covers nothing else: not a single `PUT`, not an upload made without a SHA algorithm, and no range finer than the uploader's parts, whose size the reader did not choose. The part ETags the object ETag is built from are not returned once the upload completes, and a per-part MD5 is available only when the uploader chose MD5 as the checksum algorithm, an opt-in like SHA-256; the ETag itself, an MD5 or an MD5 of part MD5s, identifies the object loosely and gives a reader no cryptographic assurance. The CRC family (CRC32, CRC32C, CRC64NVME) can be a full-object value, which Ingot derives at `CompleteMultipartUpload` by combining the parts' CRCs, but a CRC detects accidental corruption and nothing more. + +Nothing lets a client verify a byte range the uploader did not happen to make a part. + +This RFC proposes a whole-object BLAKE3 digest for every object, single `PUT` or multipart, returned to clients as a CID, plus a small tree of chaining values stored in the manifest that lets a client verify ranged reads. BLAKE3 is a Merkle tree, so the digest of a multipart object is assembled from per-part subtrees at `Complete` with no re-read of the data, and the same tree structure gives range verification for free. + +Blob addressing on the Forge network stays SHA-256. This RFC adds an object-level digest alongside it and changes nothing about how blobs are stored, located or proven. + +## Goals + +- Every object has a whole-object BLAKE3 digest, computed in the same streaming pass that writes its blobs. +- A multipart object's digest is assembled at `Complete` from values recorded per part. Re-reading part data is a bounded fallback for the few parts whose position was assumed wrongly, never the common path. An upload that would need more than the bound is refused at `Complete`, so every object that commits has a digest. +- A client can verify a whole-object read with the digest alone and a standard BLAKE3 implementation. +- A client can verify a ranged read with an off-the-shelf Bao library by fetching the object's outboard once: under a KiB for a small object, growing with the square root of the size. +- The S3 data path is unchanged. Bodies are raw bytes, ETags and S3 checksums behave as they do today, and a client that ignores the extension sees nothing new. +- Manifest growth stays small for ordinary objects and under 1 MiB for the largest. The block list grows with the square root of the object size and is capped. + +## Background: the BLAKE3 tree + +BLAKE3 hashes input as a binary Merkle tree over 1 KiB chunks. Each chunk is compressed to a 32-byte chaining value (CV). The compression of a chunk mixes in the chunk's index within the input, so the CV of a chunk depends on where in the input it sits. Pairs of CVs are compressed into parent CVs, with the index set to zero. The topmost compression is flagged as the root and its output is the hash. + +The tree shape is fixed by the input length. For an input of more than one chunk, the left subtree holds the largest power-of-two number of chunks that is strictly less than the total, and the right subtree holds the rest, recursively. A consequence this RFC relies on: any aligned power-of-two block of chunks that lies within the input is a node of the tree. A block of 2^k chunks starting at a multiple of 2^k chunks has a CV that appears in the final tree, whatever the total length turns out to be. + +Two further consequences follow. A contiguous byte range that starts on a chunk boundary decomposes into a short sequence of such aligned blocks, at most two per tree level, and its CVs can be computed knowing only the bytes and the starting offset. And the hash of the whole input can be computed from the CVs of any set of aligned blocks that partition it, with one parent compression per merge and the root flag applied at the last. + +This is the structure that [Bao] and iroh's [bao-tree] use for verified streaming. Their "outboard" is the list of parent nodes, which lets a reader holding only the root verify any range. This RFC stores the bottom of that outboard at a coarse block size, serves the outboard itself to clients, and leaves the Bao streaming encoding for a possible later mode. + +## Design + +### The object digest + +The object digest is the BLAKE3 hash of the object's plaintext body. It is stored in the manifest as a [multihash] with code `0x1e` (`blake3`) and a 32-byte digest. + +Clients receive it as a [CID]: version 1, codec `raw` (`0x55`), with that multihash, encoded with [multibase] base32 lowercase. A CID is self-describing, names content the same way the rest of the Forge stack does, and parses with any CID library. The `raw` codec states that the identified bytes are the object body itself, with no framing. + +The CID is returned in a response header on `GetObject` and `HeadObject`, including ranged and `?partNumber` reads. The header name is provisional: + +``` +x-cid: bafkr4i... +``` + +A client verifying a whole-object read hashes the body with any BLAKE3 implementation and compares. It needs nothing else from this RFC. + +### Single `PUT` + +The body already streams through a SHA-256 hasher and an MD5 hasher as it is split into blobs. A BLAKE3 hasher joins the same pass. It produces the object digest and the block CVs described under [Range verification](#range-verification). Nothing is read twice. + +### Multipart upload + +Parts arrive in any order and the server learns a part's true byte offset only once every lower-numbered part exists. Each `UploadPart` therefore hashes its body at an assumed offset and records what `CompleteMultipartUpload` needs to finish the tree. `CompleteMultipartUpload` checks each assumed offset against the true one and re-hashes only the parts whose assumption was wrong, within a budget; a completion that would need more is refused. + +#### What `UploadPart` records + +Each part record gains: + +- `TreeOffset`: the byte offset the part was hashed at. +- `TreeNodes`: the CVs of the part's aligned subtrees, in order, covering the whole chunks between the part's first and last chunk boundaries. A span of whole chunks decomposes into at most two aligned power-of-two blocks per tree level, so this list holds a few dozen 32-byte entries at most. +- `TreeBlocks`: the part's block CVs at the block size of an object that ends where the part ends (see [Range verification](#range-verification)). Part 1 of 5 GiB records 10,240 entries (320 KiB); each later part records fewer, down to 3 for part 10,000 of a 10,000 × 5 GiB upload. +- `TreeHead` and `TreeTail`: the part's bytes before its first chunk boundary and after its last one, under 1 KiB each. They are empty for a part whose offset and length are 1 KiB multiples. Every mainstream client sizes its parts in MiB, so both spans are empty for nearly every part ever uploaded. The one known exception is the AWS SDKs for Go and Java above the 10,000-part limit, objects over about 48 GiB, where they size parts as the object size divided by 10,000, an arbitrary byte count; those uploads are a small minority, and these two spans are what let them complete without re-reading anything. The spans live in the part's record in the catalog database with the rest of its tree material, since the part's local copy is removed once the part is parked on its storage node; nothing is held in memory between requests. +- For part 1 only, `TreeRoot`: the BLAKE3 hash of the part on its own. Part 1 is always at offset 0, so its standalone hash falls out of the same pass with a root-flagged final compression. + +A part cannot know whether it is the last, so it never treats a short final chunk as the object's end; those bytes go to `TreeTail`, and `CompleteMultipartUpload` compresses them once it knows what follows them. + +A part that supersedes an earlier upload of the same part number replaces its record, as today. + +#### Assuming the offset + +- Part 1 is at offset 0. +- Part _n_ when every part below _n_ is already recorded: the sum of their sizes. This is exact. +- Part _n_ when some lower part is recorded: assume a uniform part size and use (_n_ − 1) times the size of the lowest-numbered recorded part. +- Part _n_ when no lower part is recorded: (_n_ − 1) times this part's own size. + +Clients upload parts of one size in parallel, with a shorter final part. The rules above give the true offset for every part of a uniform-size upload, whatever that size is, except a final part that lands before any of its predecessors, which is re-hashed at `Complete` and is the smallest part of the upload. An assumed offset need not be a 1 KiB multiple: the part's first chunk boundary is wherever the offset puts it. + +A part created by copy is hashed as its bytes are read, like an uploaded part. Where the copy does not read the bytes, the part records no tree and takes the re-hash path. + +#### What `CompleteMultipartUpload` does + +`CompleteMultipartUpload` already walks the requested parts in order and computes each part's true offset. A part's record is usable when its true offset equals `TreeOffset`. + +The parts whose assumed offset was wrong are re-hashed at their true offsets by reading their blobs back from the storage node that holds them, and the results stand in for their records. The final part is always re-hashed when wrong: in a uniform-size upload it is the only part whose assumption can fail, when it lands before any predecessor and its own size stands in, and the 5 GiB part maximum bounds the cost. The other parts are re-hashed within a **re-hash budget**, one maximum part, 5 GiB, by default, and configurable. `Complete` adds up the wrong non-final parts before it accepts any part or reads anything. A sum over the budget is refused with a 400, `InvalidPartLayout`, whose message gives the bytes that would have to be read back, the server's limit, and the two ways out: parts of one size, or parts uploaded in part-number order. The upload stays open, so the client can resend parts and complete again. Every object that commits therefore has a digest. The budget bounds how long `Complete` spends reading from storage nodes and nothing else, so it can be set generously. Reaching it is rare. No mainstream client varies its part size within an upload, the final part aside, and a uniform-size upload assumes every offset exactly in any arrival order. A sequential upload assumes every offset exactly whatever the sizes, which covers the usual way objects are concatenated: each source object copied in as one part, in order. What is left is a parallel compose-by-copy of differently sized source objects, or a hand-rolled uploader sending variable-size parts in parallel, and those work up to the budget, about 5 GiB of such parts plus the final one with the default. + +A chunk that straddles a part boundary is rebuilt from the earlier part's `TreeTail` and the later part's `TreeHead` and compressed at its chunk index, one compression per boundary. The last part's `TreeTail` is the object's final chunk and is compressed as such. Part lengths that are not 1 KiB multiples therefore add little: under 2 KiB of raw bytes per part record, under 20 MiB across the 10,000-part maximum, and one compression per boundary at `Complete`. + +With every part's nodes at their true offsets, the object digest is computed by merging adjacent aligned subtrees into parents and applying the root flag at the final merge. This is one compression per recorded subtree and reads no data. The work scales with the number of parts: a part contributes at most two subtrees per tree level, so an upload of 10,000 parts merges a few tens of thousands of nodes, each one BLAKE3 compression, which is milliseconds. + +An upload completed with a single part has no merge, and its digest is that part's root as a body of its own. For part 1 that is the recorded `TreeRoot`. For any other part the recorded offset is not 0, so the offset check above re-hashes it at offset 0, and that re-hash yields the root directly. S3 requires the completed parts to be in ascending order and nothing more: the list need not start at part 1 or be contiguous, so a client that re-uploads a bad part under a new number and completes without the old one produces exactly this case. + +The block CVs for the object are built from the parts' `TreeBlocks` and `TreeNodes` as described in the next section. The root computed by merging the block CVs up must equal the root computed from the parts' subtrees. This is a consistency check on the assembly at the cost of one compression per block, 32,767 at the cap. Should it ever fail, `Complete` returns a server error and the upload stays open; the object is never committed with a digest that might be wrong, and never without one. Nothing re-reads the body. + +### Range verification + +Two terms, since BLAKE3 and Bao both have "leaves" and they are not the same thing. A **chunk** is BLAKE3's 1 KiB unit; a chunk's CV is a leaf of BLAKE3's own tree. A **block** is $2^{\text{ChunkLog}}$ chunks, the unit a client verifies; a block's CV is an interior node of BLAKE3's tree and a leaf of the Bao tree at that block size. This RFC stores and serves block CVs and never stores chunk CVs. The word "leaf" is not used below. + +A client that reads ranges needs more than the root. It needs the CVs of the tree nodes covering the bytes it read, and enough of the rest of the tree to connect them to the root. The full Bao outboard provides this at 1 KiB granularity and costs about 6% of the object size. Storing it is out of the question in a manifest, and even at bao-tree's 16 KiB default it costs 4 MiB per GiB. + +This RFC scales the block size with the object. Objects up to 8 MiB are recorded at 16 KiB blocks, bao-tree's own granularity, and above that the block grows with the square root of the size. The block list of a small object is a few KiB, and the smallest verifiable read on a large object stays a small fraction of it. + +#### The block + +The **block** is the unit of the stored tree. Its target is the square root of the object size times 32 bytes, the size of one CV: + +$$\text{target} = \sqrt{\text{size} \times 32\,\text{B}}$$ + +$$\text{block} = \max\bigl(16\,\text{KiB},\ 2^{\lceil \log_2 \text{target} \rceil}\bigr)$$ + +The target reaches the 16 KiB floor at 8 MiB, so every object up to 8 MiB has 16 KiB blocks, 512 of them at 8 MiB. Above that the block doubles each time the object quadruples: for a 1 GiB object the target is $\sqrt{2^{30} \times 2^{5}} = \sqrt{2^{35}}$, about 181 KiB, so the block is 256 KiB and there are 4,096 blocks. A hard cap of 32,768 blocks, 1 MiB of CVs, binds from 32 GiB, where the block grows with the size instead. The cap is a metadata-cost choice: the manifest is a catalog block rewritten on every version, so its size is storage the tenant pays for on each write, and 1 MiB leaves headroom under the 2 MiB byte-field limit of the catalog's CBOR encoding. A larger cap, up to that limit, buys finer verification on objects over 32 GiB at that cost. The manifest records the block size as a **chunk log**: a base-2 exponent of 1 KiB BLAKE3 chunks, the unit Bao libraries take as the block size, so 16 KiB is 4 and 1 MiB is 10. The block affects only what is stored and what granularity a client can verify at. The object digest is the same whatever block is chosen. + +| Object size | Block size | Chunk log | Block count | Manifest bytes | +|---|---|---|---|---| +| 1 KiB | 16 KiB | 4 | 1 | 32 B | +| 64 KiB | 16 KiB | 4 | 4 | 128 B | +| 1 MiB | 16 KiB | 4 | 64 | 2 KiB | +| 8 MiB | 16 KiB | 4 | 512 | 16 KiB | +| 16 MiB | 32 KiB | 5 | 512 | 16 KiB | +| 100 MiB | 64 KiB | 6 | 1,600 | 50 KiB | +| 1 GiB | 256 KiB | 8 | 4,096 | 128 KiB | +| 10 GiB | 1 MiB | 10 | 10,240 | 320 KiB | +| 32 GiB | 1 MiB | 10 | 32,768 | 1 MiB | +| 100 GiB | 4 MiB | 12 | 25,600 | 800 KiB | +| 1 TiB | 32 MiB | 15 | 32,768 | 1 MiB | +| 5 TiB | 256 MiB | 18 | 20,480 | 640 KiB | +| 50 TB | 2 GiB | 21 | 23,284 | 728 KiB | + +Sizes with binary prefixes are powers of two; the 50 TB row is decimal, 50 × 10¹² bytes, just under S3's current maximum object size of 48.8 TiB (10,000 parts of 5 GiB). Block count is the object size divided by the block size, rounded up, since a final partial block still has a CV. An object of 16 KiB or less is one block. Up to 8 MiB the manifest cost is 32 bytes per 16 KiB, 0.2% of the object. From there it grows with the square root of the size: 50 KiB at 100 MiB, 128 KiB at 1 GiB, 320 KiB at 10 GiB. The cap binds from 32 GiB and holds every larger object to 1 MiB or less, 728 KiB for a 50 TB object. + +The smallest range a client can verify independently is one block: 16 KiB on anything up to 8 MiB, 256 KiB on a 1 GiB object, 32 MiB on a 1 TiB object, 2 GiB on a 50 TB object. That is finer than the per-part checksum list of any multipart upload of the same object, whose granularity is its parts. S3 caps an upload at 10,000 parts of at least 5 MiB, so a part averages at least the larger of 5 MiB and the object size divided by 10,000. The block is at most 1 MiB up to 32 GiB, at most 4 MiB up to 128 GiB, and under the object size divided by 16,384 beyond: on a 1 TiB object a part averages at least 105 MiB where the block is 32 MiB. The tree also exists for every object, even if the uploader does not request per-part SHA-256 checksums. Three constants tune the rule. The 16 KiB floor and the 32 B factor set where the scaling starts, and the cap bounds the manifest. An object needing finer verification than its block would want the outboard-as-blob design under [Alternatives](#alternatives-considered). + +#### Block CVs + +The manifest stores the CV of every block of the object, in object order from offset 0: the **block CVs**, 32 bytes each. They are the nodes of BLAKE3's tree at the block level. Every node above them is derivable with one compression per node, so nothing above the blocks is stored, and nothing below them: chunk CVs are never stored. + +A single `PUT` computes the block CVs in its hashing pass, since the object length is known from `Content-Length` and so is the block. + +A multipart upload does not know the object length until `CompleteMultipartUpload`, so each part computes `TreeBlocks` at the block size the same rule gives for its assumed offset plus its length. A record is used only when its assumed offset is the true one, and the object then holds at least that many bytes, so the object's block is never smaller than the part's, because the rule is monotone in size. Sizing the blocks by the part's own length would also be correct, but finer than the object needs: at the 10,000 × 5 GiB maximum that is 3 GiB of CVs against about 6 MiB. At `CompleteMultipartUpload`, each object block is one of: + +- A block lying within one part. Its CV is the merge of that part's `TreeBlocks` that it covers. +- A block straddling a part boundary. The piece on each side is a union of that part's edge subtrees, which `TreeNodes` holds, so its CV is the merge of those entries. + +A 1 KiB chunk split across a part boundary belongs to neither part's subtrees. Each part's `TreeNodes` covers only its whole chunks, and the split chunk's bytes sit in the earlier part's `TreeTail` and the later part's `TreeHead`. `CompleteMultipartUpload` joins the two, compresses the chunk at its index, and merges the result into the block containing that index. Blocks are whole chunks, so the chunk lies in exactly one block. + +Either way the block CVs are built from recorded values with no data read. + +#### The tree in `GetObjectAttributes` + +A client fetches the tree with `GetObjectAttributes` (`?attributes`), which already returns a per-part checksum list for a multipart object and already takes `versionId`. This RFC adds an attribute name, `Blake3`, to the `x-amz-object-attributes` request header. Like the existing attributes it is returned only when requested, so responses to clients that do not ask for it are unchanged. The name is provisional. + +``` +GET /{bucket}/{key}?attributes[&versionId=...] +x-amz-object-attributes: ETag,Blake3 +``` + +```xml + + ... + + bafkr4i... + 12 + base64 + + +``` + +- `CID` is the object digest as returned in `x-cid` +- `ChunkLog` is the block size as a base-2 exponent of 1 KiB BLAKE3 chunks, the value a Bao library takes as its block size unchanged. iroh's fixed block is 4; the original Bao format is 0. +- `Outboard` is the standard pre-order Bao outboard at that block size, base64-encoded: the object size as 8 little-endian bytes, then the chaining-value pair of every parent node above the blocks, root first and left subtree before right. It is built from the stored block CVs on request, one compression per parent, and is byte-identical to what the Bao libraries produce for the same object and block size. It is an 8-byte prefix plus 64 bytes per parent, one fewer than the blocks: a few hundred bytes for a small object, 32 KiB at 8 MiB, 256 KiB at 1 GiB, and 2 MiB at the 32,768-block cap from 32 GiB, about 2.7 MiB after base64. A tree belongs to one version. An overwrite can never pair an old tree with new bytes, because both live in the same manifest, and the response carries the ETag of the version the tree describes for the client's `If-Match`. + +AWS rejects an attribute name it does not know with `InvalidArgument`, so accepting `Blake3` is a visible divergence. SDKs whose typed response omits unknown elements cannot read the tree, but the same SDKs could not call a custom endpoint either, and a raw HTTP client parses this XML with what it already has for every other S3 response. + +#### Client procedure + +1. Fetch the attribute once with `GetObjectAttributes`, and load the CID's digest, the block size and the outboard into a Bao library (bao-tree in Rust, the Go BLAKE3 library's `bao` package). A client that uploaded the object itself can skip this step: the digest and the outboard are functions of the bytes alone, so it can compute both as it uploads (`ingot blake3 hash --bao` does exactly this) and never ask the server. The library checks the outboard against the root as it goes; no merge code of our own is needed, and no BLAKE3 API beyond what a Bao library already wraps. +2. Read ranges with ordinary S3 `Range` requests aligned to block boundaries, sending `If-Match` with the ETag so the bytes cannot change between the attribute fetch and the reads. A Bao library can name the byte ranges needed for the blocks a client wants. +3. Verify each block received against the outboard. An object of one block has an empty outboard, and the client verifies by hashing the whole object against the CID. + +Every byte of data travels as a plain S3 body. SDKs, caches and CDNs behave as they do today. + +### Manifest changes + +The body gains three fields. Sizes are the maximum for any object. + +```go +type Body struct { + // ...existing fields + + // BLAKE3 is the whole-object blake3 multihash (34 bytes). + BLAKE3 []byte `cborgen:"b3"` + // TreeChunkLog is the block size of TreeBlocks as a base-2 exponent of chunks (one byte). + TreeChunkLog uint8 `cborgen:"tc"` + // TreeBlocks holds the chaining values of the blocks of the + // body, 32 bytes each, in order (8 KiB at 1 GiB, at most 1 MiB). + TreeBlocks []byte `cborgen:"tb"` +} +``` + +Objects written before this change have no digest. For them, GET and HEAD return no `x-cid` header and `GetObjectAttributes` returns its response without the `Blake3` element. + +That makes two situations look alike from a GET or HEAD: an object without a digest, and a server that does not implement this RFC. `GetObjectAttributes` tells them apart. The request names the attributes it wants in `x-amz-object-attributes`, and a server validates the names: one that implements this RFC accepts `Blake3` and answers, with or without the element depending on the object, while one that does not, AWS included, rejects the request with `InvalidArgument`. So a client that wants to know whether verification is available at all asks for `Blake3` once; an error means the server cannot provide it for any object, and a success without the element means it cannot provide it for this one. + +### Cost + +Ingest adds one BLAKE3 pass per body or part, alongside the MD5 pass already there. BLAKE3 with vector instructions is faster than SHA-256, so it costs less than the whole-body SHA-256 pass it makes redundant. + +`CompleteMultipartUpload` adds one compression per recorded subtree for the digest and one per block for the consistency check, a few tens of thousands at most for the largest uploads, which is milliseconds. The re-hash path reads the affected parts back through the plaintext opener, which decrypts in an encrypting deployment. A part's local copy is removed as soon as the part is parked on its provider, so the re-hash fetches the part's blobs from the storage node with ranged retrievals, the same path a `GET` takes. The work is bounded by the final part plus the budget, about 10 GiB of retrieval at most with the default, so `Complete` on such an upload takes as long as fetching that much from the provider, under a minute and a half at a gigabit per second. A variable-size parallel upload beyond the bound is refused, as described above. + +The manifest grows by the block list, 128 KiB at 1 GiB and at most 1 MiB, plus the 34-byte multihash and a few bytes of framing. The part record grows by its tree material for the life of the upload: a few KiB for parts of a few MiB, at most about 320 KiB for a 5 GiB part, nearly all of it block CVs, and less the further into the object the part sits. `Complete` loads every part's tree material, about 6 MiB for the largest upload S3 allows, 10,000 parts of 5 GiB, and about 4 MiB for a 1 TiB upload in 10,000 parts. + +## Alternatives considered + +### Whole-object SHA-256 for multipart objects + +Keeping the existing SHA-256 and computing it for multipart objects means re-reading every part at `Complete`. For a 5 GiB object that is a 5 GiB read, decrypting where encryption is on, inside a request a client expects to return promptly. It also offers no range verification. BLAKE3 removes the re-read and adds range verification with the same stored bytes. + +### A root SHA-256 over per-part SHA-256 hashes + +S3 already defines this for uploads made with `--checksum-algorithm SHA256`: a SHA-256 per part and a composite over the list, which `GetObjectAttributes` returns. The alternative is to compute that for every multipart upload, server side, whatever the client asked for, and treat the composite as the object's digest. It needs no new primitive, and for a client that already understands S3 composite checksums it is familiar. It falls short of this RFC in several ways. + +- **The root is not a hash of the bytes.** A composite depends on where the part boundaries fell, so the same bytes uploaded with 8 MiB parts, with 16 MiB parts, or as a single `PUT` get three different roots. It identifies an upload, not an object: two copies of the same data cannot be recognised as equal, nothing can be addressed by it, and a single-`PUT` object has a plain SHA-256 while a multipart object has a hash of hashes, so a client needs two code paths and two meanings for "the digest". +- **Verifying a download takes a list, not a value.** The verifier must fetch the per-part hash list and the part sizes, up to 10,000 of each, keep them for the whole download, and apply the right one to the right byte range. The BLAKE3 digest is one 32-byte value that a standard library verifies against a stream. Walking a 10,000-entry list is custom code in every client, and the list itself is hundreds of KiB of XML on every fetch, with no compact form. +- **The granularity is the uploader's, and coarser than the tree's.** Verification happens per part, so a client cannot check anything until it has a whole part. A part is at least 5 MiB, at least the object size divided by 10,000, and up to 5 GiB, and a single-`PUT` object cannot be range-verified at all. The Bao tree verifies 16 KiB on objects up to 8 MiB, and a server-chosen block on large ones, with proofs that are logarithmic in the object rather than linear in the part count. +- **No standard verifies it.** There is no library that takes a composite SHA-256 and a part list and verifies a stream or a range, so every consumer writes its own; the Bao outboard is loadable by existing libraries in Rust and Go. +- **It costs about the same.** A SHA-256 pass per part is no cheaper than the BLAKE3 pass, which is faster per byte, and the composite still has to be assembled at `Complete` from recorded values, as the tree is. + +What it does offer over the current state is universality within multipart uploads: every multipart object would have a verifiable per-part list rather than only those uploaded with SHA checksums. BLAKE3 gives that and the rest. + +### Re-addressing blobs to BLAKE3 + +Making BLAKE3 the network's content address would let storage nodes serve verified ranges directly. It touches every component that handles a multihash, the proof-tree assumptions of PDP, public gateway resolvability, and every stored byte. This RFC is independent of that decision. The object digest proposed here is the same value such a change would use, so nothing here is thrown away if it happens. The [BLAKE3 shard digests](./2026-10-blake3-shard-digest.md) RFC explores it for new shards. + +### A fixed 16 KiB block with the outboard as a blob + +bao-tree's default gives 16 KiB verification granularity at any size, with the outboard stored as a blob of its own on the Forge network and referenced from the manifest by digest. The manifest then stays a few dozen bytes however large the object, the tree is content-addressed so any node holding it can serve it and a client can check it against its SHA-256 digest, and the 2 MiB field limit of the catalog no longer bounds the tree. It is the right design if fine-grained verified reads of very large objects are a requirement. + +The client makes a second request in both designs. What differs is what the request returns and what the server manages behind it. + +- **The response is 0.4% of the object.** The outboard at 16 KiB is 64 bytes per block: 4 MiB for a 1 GiB object, 4 GiB for a 1 TiB object, about 190 GiB at the 48 TiB S3 maximum. The manifest tree is 256 KiB at 1 GiB and 2 MiB from 32 GiB up, and a client that wants one range needs only the proof slice for it, which `GetObjectAttributes` could extract from either. +- **A blob per object.** Every `PUT` writes a second blob: allocate, upload, accept, a claim, replication to every replica, a PDP piece to prove, and a release on delete. Under one object, one piece, that doubles the on-chain piece count for small objects, which are most objects, to carry a tree of 32 bytes to a few KiB. The upload also sits on the commit path, since the tree is known only once the body is hashed and the manifest is written only after its blobs are accepted. +- **Multipart grows with it.** A part's block list at 16 KiB is 10 MiB for a 5 GiB part, against about 15 KiB today, and `Complete` on the largest object reads them all back and writes the outboard out. The outboard passes the blob ceiling at about 60 GiB of object and becomes a multi-blob body of its own. +- **Encryption removes the independence.** In an encrypting bucket the tree blob is encrypted like the body, so only Ingot can serve it and the client gains nothing over the manifest. A plaintext tree would publish block-level hashes of encrypted data, which lets anyone confirm guesses about low-entropy blocks. +- **Two storage paths for one value.** A blob per 1 KiB object is a poor trade, so the blob would join the manifest path above some size rather than replace it, and both paths would be maintained for good. + +The bounded manifest tree serves the common case with no new blob and can coexist with this design later: the stored block CVs are the top of the finer tree, so an object that needs 16 KiB verification could gain a tree blob without changing what the manifest holds. + +### Returning the Bao encoded stream + +iroh serves data interleaved with the parent nodes so a client verifies as bytes arrive. S3 SDKs cannot consume such a body, so it would be a separate mode rather than the S3 `GET`. At the block sizes this RFC uses a client must buffer a whole block before verifying anything, so interleaving saves only the one tree request. The outboard served by `GetObjectAttributes` is already the Bao form, so the encoded stream can be added later as its own subresource without changing what is stored. + +### Carrying the digest in an S3 checksum header + +The `x-amz-checksum-*` headers have a fixed set of algorithms and SDKs validate the names, so BLAKE3 cannot ride in one. + +### A dedicated subresource for the tree + +A `?blake3` subresource returning a CBOR document would keep `GetObjectAttributes` identical to AWS. It costs a new endpoint, a second place that must honour `versionId`, and a body format no S3 client parses today. No SDK can call it, so it has no advantage over an attribute for SDK users, and a raw HTTP client is better served by the XML it already handles. + +### Returning the block list instead of the outboard + +The stored block CVs are half the size of the outboard, and a client could merge them to the root itself. But checking a block against its CV means hashing the block as a non-root subtree at its chunk offset, which standard BLAKE3 APIs do not expose, so every client would carry custom tree code. Serving the Bao outboard costs twice the bytes and lets any Bao library do the verification. + +### Sending the block CVs in a header + +Base64 of an 8 MiB object's 32 KiB outboard is about 43 KiB, over the 8 KiB per-header default of common proxies, larger objects have far more, and splitting it across numbered headers is a hack. The root fits in a header; the tree does not. + +## Dependencies + +The Go implementation uses `lukechampine.com/blake3` for the hash, its `guts` package for the tree primitives the hasher needs (chunk and parent compression at a given chunk index, which the standard BLAKE3 API does not expose), and its `bao` package in tests to check the outboard against an independent encoder. The library has two maintainers and releases infrequently. The exposure is small: the `bao` package is about 300 lines and the primitives a few hundred more, with no further dependencies, so a fork or a vendored copy is the fallback if a fix is ever needed faster than upstream provides it. The other maintained Go implementation, `zeebo/blake3`, exposes no tree internals and could replace only the plain hashing. + +## Open questions + +- The re-hash budget's default for non-final parts. The final part is exempt, so the budget only governs variable-size parallel uploads; a larger default trades `Complete` latency for accepting more of them. + +- Names: `x-cid` and the `Blake3` attribute are placeholders. +- The scaling constants: the 16 KiB floor, the 32 B factor that holds it to 8 MiB, and the 32,768-block cap. The exponent is a knob too; cube-root scaling would hold manifests smaller at the cost of coarser blocks on large objects. +- Whether a `GET` with `?partNumber` should also return the block CVs covering that part, so a client that already reads by part need not align to blocks. +- Whether the object CID should reach the Forge network, for example in the upload index, or remain an Ingot-level value until blob addressing is revisited. +- Whether to expose the digest on `ListObjects` responses. + +## Evaluation criteria + +- The S3 conformance suite passes unchanged: no ETag, checksum or body behaviour differs. +- Ingest throughput with the extra hash is within a few percent of today's. +- The share of `Complete` calls that take the re-hash path, measured against the AWS CLI, boto3, the Go SDK's uploader above the 10,000-part threshold, rclone, s5cmd and the MinIO client at their default part sizes, is near zero, and none exceeds the budget. +- Manifest size distribution matches the table above. +- A reference client verifies whole-object and ranged reads against a running Ingot, including a multipart object with non-power-of-two parts and a mis-ordered final part. + +## References + +- [BLAKE3 specification](https://github.com/BLAKE3-team/BLAKE3-specs/blob/master/blake3.pdf) +- [Bao] +- [bao-tree] +- [multihash] and the [multicodec table](https://github.com/multiformats/multicodec/blob/master/table.csv) +- [CID] +- [multibase] +- [S3 GetObjectAttributes](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectAttributes.html) +- [lukechampine.com/blake3](https://github.com/lukechampine/blake3), a Go implementation exposing the tree primitives (`guts`) and Bao with chunk groups + +[Bao]: https://github.com/oconnor663/bao +[bao-tree]: https://github.com/n0-computer/bao-tree +[multihash]: https://github.com/multiformats/multihash +[CID]: https://github.com/multiformats/cid +[multibase]: https://github.com/multiformats/multibase