From d036bbda3edde97fb141ee7bc526e42c9bf610bb Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Mon, 5 Oct 2026 22:21:11 +0100 Subject: [PATCH 01/19] rfc: BLAKE3 object digest --- rfcs/2026-10-blake3-object-digest.md | 254 +++++++++++++++++++++++++++ 1 file changed, 254 insertions(+) create mode 100644 rfcs/2026-10-blake3-object-digest.md diff --git a/rfcs/2026-10-blake3-object-digest.md b/rfcs/2026-10-blake3-object-digest.md new file mode 100644 index 0000000..ef239a8 --- /dev/null +++ b/rfcs/2026-10-blake3-object-digest.md @@ -0,0 +1,254 @@ +# RFC: BLAKE3 object digest + +Status: Experimental + +## Authors + +- [Alan Shaw](https://github.com/alanshaw) + +## Motivation + +An Ingot object has no digest a client can verify against. A single `PUT` records a whole-object SHA-256 in the manifest, but it is internal and never returned. A multipart object records no whole-object digest at all, because SHA-256 cannot be assembled from the parts and re-reading every part at `CompleteMultipartUpload` is too slow. A client that wants to check what it downloaded has the S3 ETag, which for a multipart object is a hash of part MD5s and verifies nothing about the bytes, and the optional S3 checksums, which for multipart are composite values with the same limitation. Nothing lets a client verify a byte range. + +This RFC proposes a whole-object BLAKE3 digest for every object, single `PUT` or multipart, returned to clients as a CID, plus a small tree of chaining values stored in the manifest that lets a client verify ranged reads. BLAKE3 is a Merkle tree, so the digest of a multipart object is assembled from per-part subtrees at `Complete` with no re-read of the data, and the same tree structure gives range verification for free. + +Blob addressing on the Forge network stays SHA-256. This RFC adds an object-level digest alongside it and changes nothing about how blobs are stored, located or proven. + +## Goals + +- Every object has a whole-object BLAKE3 digest, computed in the same streaming pass that writes its blobs. +- A multipart object's digest is assembled at `Complete` from values recorded per part. Re-reading part data is a fallback for unusual clients, never the common path. +- A client can verify a whole-object read with the digest alone and a standard BLAKE3 implementation. +- A client can verify a ranged read by fetching at most a few KiB of tree data once per object. +- The S3 data path is unchanged. Bodies are raw bytes, ETags and S3 checksums behave as they do today, and a client that ignores the extension sees nothing new. +- Manifest growth is bounded at a few KiB regardless of object size. + +## Background: the BLAKE3 tree + +BLAKE3 hashes input as a binary Merkle tree over 1 KiB chunks. Each chunk is compressed to a 32-byte chaining value (CV). The compression of a chunk mixes in the chunk's index within the input, so the CV of a chunk depends on where in the input it sits. Pairs of CVs are compressed into parent CVs, with the index set to zero. The topmost compression is flagged as the root and its output is the hash. + +The tree shape is fixed by the input length. For an input of more than one chunk, the left subtree holds the largest power-of-two number of chunks that is strictly less than the total, and the right subtree holds the rest, recursively. A consequence this RFC relies on: any aligned power-of-two block of chunks that lies within the input is a node of the tree. A block of 2^k chunks starting at a multiple of 2^k chunks has a CV that appears in the final tree, whatever the total length turns out to be. + +Two further consequences follow. A contiguous byte range that starts on a chunk boundary decomposes into a short sequence of such aligned blocks, at most two per tree level, and its CVs can be computed knowing only the bytes and the starting offset. And the hash of the whole input can be computed from the CVs of any set of aligned blocks that partition it, with one parent compression per merge and the root flag applied at the last. + +This is the structure that [Bao] and iroh's [bao-tree] use for verified streaming. Their "outboard" is the list of parent nodes, which lets a reader holding only the root verify any range. This RFC stores a reduced form of that outboard and leaves the Bao wire format for a possible later mode. + +## Design + +### The object digest + +The object digest is the BLAKE3 hash of the object's plaintext body. It is stored in the manifest as a [multihash] with code `0x1e` (`blake3`) and a 32-byte digest. + +Clients receive it as a [CID]: version 1, codec `raw` (`0x55`), with that multihash, encoded with [multibase] base32 lowercase. A CID is self-describing, names content the same way the rest of the Forge stack does, and parses with any CID library. The `raw` codec states that the identified bytes are the object body itself, with no framing. + +The CID is returned in a response header on `GetObject` and `HeadObject`, including ranged and `?partNumber` reads. The header name is provisional: + +``` +x-cid: bafkr4i... +``` + +A client verifying a whole-object read hashes the body with any BLAKE3 implementation and compares. It needs nothing else from this RFC. + +### Single `PUT` + +The body already streams through a SHA-256 hasher and an MD5 hasher as it is split into blobs. A BLAKE3 hasher joins the same pass. It produces the object digest and the leaf CVs described under [Range verification](#range-verification). Nothing is read twice. + +### Multipart upload + +Parts arrive in any order and the server learns a part's true byte offset only once every lower-numbered part exists. Each `UploadPart` therefore hashes its body at a guessed offset and records what `CompleteMultipartUpload` needs to finish the tree. `CompleteMultipartUpload` checks the guesses against the true offsets and re-hashes only the parts whose guess was wrong. + +#### What `UploadPart` records + +Each part record gains: + +- `TreeOffset`: the byte offset the part was hashed at. +- `TreeNodes`: the CVs of the part's aligned subtrees, in order. A part of any length starting on a chunk boundary decomposes into at most two aligned power-of-two blocks per tree level, so this list holds a few dozen 32-byte entries at most. +- `TreeLeaves`: the part's leaf CVs at the part's own group (see [Range verification](#range-verification)), at most 256 entries. +- For part 1 only, `TreeRoot`: the BLAKE3 hash of the part on its own. Part 1 is always at offset 0, so its standalone hash falls out of the same pass with a root-flagged final compression. + +A part that supersedes an earlier upload of the same part number replaces its record, as today. + +#### Guessing the offset + +- Part 1 is at offset 0. +- Part _n_ when every part below _n_ is already recorded: the sum of their sizes. This is exact. +- Part _n_ when some lower part is recorded: assume a uniform part size and use (_n_ − 1) times the size of the lowest-numbered recorded part. +- Part _n_ when no lower part is recorded: (_n_ − 1) times this part's own size. + +Clients upload parts of one size in parallel, with a shorter final part. The rules above guess right for every part of a uniform-size upload except a final part that lands before any of its predecessors, which is re-hashed at `Complete` and is the smallest part of the upload. + +A part created by copy is hashed as its bytes are read, like an uploaded part. Where the copy does not read the bytes, the part records no tree and takes the re-hash path. + +#### What `CompleteMultipartUpload` does + +`CompleteMultipartUpload` already walks the requested parts in order and computes each part's true offset. For each part it checks that the true offset is a multiple of 1 KiB and equals `TreeOffset`. A part that fails either check is re-hashed at its true offset by reading its blobs back, and its record is replaced before the merge proceeds. A part offset that is not a multiple of 1 KiB means some earlier part has a length that is not a multiple of 1 KiB, which places a chunk boundary inside a part. Such parts and every later part are re-hashed. This is rare: clients size parts in MiB. + +With every part's `TreeNodes` at correct offsets, the object digest is computed by merging adjacent aligned subtrees into parents and applying the root flag at the final merge. This is one compression per node, a few hundred at most, and reads no data. + +An upload completed with a single part has no merge. Its digest is part 1's `TreeRoot`. + +The leaf CVs for the object are built from the parts' `TreeLeaves` and `TreeNodes` as described in the next section. The root computed by merging the leaf CVs up must equal the root computed from the parts' subtrees. This is a free consistency check on the assembly. + +### Range verification + +A client that reads ranges needs more than the root. It needs the CVs of the tree nodes covering the bytes it read, and enough of the rest of the tree to connect them to the root. The full Bao outboard provides this at 1 KiB granularity and costs about 6% of the object size. Storing it is out of the question in a manifest, and even at bao-tree's 16 KiB default it costs 4 MiB per GiB. + +This RFC bounds the cost by fixing the number of leaves rather than the leaf size. + +#### The group + +The **group** is the leaf size of the stored tree: a power of two, at least 16 KiB, and the smallest such value that gives the object at most 256 leaves. The manifest records the group as a base-2 exponent. The group affects only what is stored and what granularity a client can verify at. The object digest is the same whatever group is chosen. + +| Object size | Group | Leaves | Manifest bytes | +|---|---|---|---| +| 1 KiB | 16 KiB | 1 | 32 B | +| 1 MiB | 16 KiB | 64 | 2 KiB | +| 4 MiB | 16 KiB | 256 | 8 KiB | +| 5 MiB | 32 KiB | 160 | 5 KiB | +| 100 MiB | 512 KiB | 200 | 6.25 KiB | +| 1 GiB | 4 MiB | 256 | 8 KiB | +| 50 GiB | 256 MiB | 200 | 6.25 KiB | +| 5 TiB | 32 GiB | 160 | 5 KiB | + +Below 4 MiB the cost is 1/512 of the object size. Above 4 MiB it stays between 4 KiB and 8 KiB: each time the size crosses a power of two the group doubles and the leaf count halves. + +The smallest range a client can verify independently is one group. A 1 GiB object verifies in 4 MiB pieces. A multi-TiB object verifies in pieces of tens of GiB, which is coarse, and an object of that size with a need for fine-grained verification would want the outboard-as-blob design under [Alternatives](#alternatives-considered). The cap of 256 is a tuning constant: halving it halves the manifest cost and doubles the smallest verifiable read. + +#### Leaf CVs + +The manifest stores the CV of every group-aligned block of the object, in order: the **leaf CVs**. These are the nodes of the BLAKE3 tree at the group level, each 32 bytes. Every node above them is derivable with one compression per node, so nothing above the leaves is stored. + +A single `PUT` computes the leaf CVs in its hashing pass, since the object length is known from `Content-Length` and so is the group. + +A multipart upload does not know the object length until `CompleteMultipartUpload`, so each part computes `TreeLeaves` at its own group, chosen by the same rule from the part's own size. The object's group is never smaller than any part's group, because the rule is monotone in size and the object is at least as large as each part. At `CompleteMultipartUpload`, each object leaf is one of: + +- A block lying within one part. Its CV is the merge of that part's `TreeLeaves` that it covers. +- A block straddling a part boundary. The piece on each side is a union of that part's edge subtrees, which `TreeNodes` holds, so its CV is the merge of those entries. + +Either way the leaf CVs are built from recorded values with no data read. + +#### The tree in `GetObjectAttributes` + +A client fetches the tree with `GetObjectAttributes` (`?attributes`), which already returns a per-part checksum list for a multipart object and already takes `versionId`. This RFC adds an attribute name, `Blake3`, to the `x-amz-object-attributes` request header. Like the existing attributes it is returned only when requested, so responses to clients that do not ask for it are unchanged. The name is provisional. + +``` +GET /{bucket}/{key}?attributes[&versionId=...] +x-amz-object-attributes: ETag,Blake3 +``` + +```xml + + ... + + bafkr4i... + 22 + + base64 of 32 bytes + ... + + + +``` + +`CID` is the object digest as returned in `x-cid`, `Group` is the group exponent and each `Leaf` is a leaf CV in order, encoded in base64 like the checksums in `ObjectParts`. At the 256-leaf cap the element is 15 to 20 KiB. A tree belongs to one version. An overwrite can never pair an old tree with new bytes, because both live in the same manifest, and the response carries the ETag of the version the tree describes for the client's `If-Match`. + +AWS rejects an attribute name it does not know with `InvalidArgument`, so accepting `Blake3` is a visible divergence. SDKs whose typed response omits unknown elements cannot read the tree, but the same SDKs could not call a custom endpoint either, and a raw HTTP client parses this XML with what it already has for every other S3 response. + +#### Client procedure + +1. Fetch the tree once with `GetObjectAttributes`. Compute the parents from the leaf CVs up to the root and compare with the CID. This is at most 255 compressions. When the tree holds a single leaf the object is at most one group long, and the client verifies by hashing the whole object instead. +2. Read ranges with ordinary S3 `Range` requests aligned to group boundaries, sending `If-Match` with the ETag so the bytes cannot change between the tree fetch and the reads. +3. Hash each group received as a non-root BLAKE3 subtree at its chunk offset and compare with its leaf CV. + +Every byte of data travels as a plain S3 body. SDKs, caches and CDNs behave as they do today. + +### Manifest changes + +The body gains three fields. Sizes are the maximum for any object. + +```go +type Body struct { + // ...existing fields + + // BLAKE3 is the whole-object blake3 multihash (34 bytes). + BLAKE3 []byte `cborgen:"b3"` + // TreeGroup is the base-2 exponent of the leaf size of TreeLeaves (one byte). + TreeGroup uint8 `cborgen:"tg"` + // TreeLeaves holds the chaining values of the group-aligned blocks of the + // body, 32 bytes each, in order (at most 8 KiB). + TreeLeaves []byte `cborgen:"tl"` +} +``` + +Objects written before this change have none of these fields. A read of such an object returns no `x-cid` header and `GetObjectAttributes` omits the `Blake3` element, so a client can tell the difference between an object without a digest and a server without the feature. + +### Cost + +Ingest adds one BLAKE3 pass per body or part, alongside the SHA-256 and MD5 passes already there. BLAKE3 with vector instructions is faster than SHA-256, so the added CPU cost is below the two hashes already paid. + +`CompleteMultipartUpload` adds a few hundred compressions in the common path. The re-hash path reads the affected parts back through the plaintext opener, which decrypts in an encrypting deployment, and is bounded by the size of the mis-guessed parts. + +The manifest grows by at most 8 KiB plus the 34-byte multihash and a few bytes of framing. The part record grows by at most about 10 KiB for the life of the upload. + +## Alternatives considered + +### Whole-object SHA-256 for multipart objects + +Keeping the existing SHA-256 and computing it for multipart objects means re-reading every part at `Complete`. For a 5 GiB object that is a 5 GiB read, decrypting where encryption is on, inside a request a client expects to return promptly. It also offers no range verification. BLAKE3 removes the re-read and adds range verification with the same stored bytes. + +### Re-addressing blobs to BLAKE3 + +Making BLAKE3 the network's content address would let storage nodes serve verified ranges directly. It touches every component that handles a multihash, the proof-tree assumptions of PDP, public gateway resolvability, and every stored byte. This RFC is independent of that decision. The object digest proposed here is the same value such a migration would use, so nothing here is thrown away if it happens. + +### A fixed 16 KiB group with the outboard as a blob + +bao-tree's default gives 16 KiB verification granularity at any size, with the outboard stored as its own blob referenced from the manifest, about 0.4% of the object. It is the right design if fine-grained verified reads of very large objects are a requirement. It adds a blob per object to write, locate and clean up, and a second request to fetch something that is usually megabytes. The bounded manifest tree serves the common case with no new blob and can coexist with this design later: the stored leaves are the top of the finer tree. + +### Returning the Bao encoded stream + +iroh serves data interleaved with the parent nodes so a client verifies as bytes arrive. S3 SDKs cannot consume such a body, so it would be a separate mode rather than the S3 `GET`. At the groups this RFC uses a client must buffer a whole group before verifying anything, so interleaving saves only the one tree request. The server can expand the stored leaf CVs into a pair-form outboard on the fly, so the encoded stream can be added later as its own subresource without changing what is stored. + +### Carrying the digest in an S3 checksum header + +The `x-amz-checksum-*` headers have a fixed set of algorithms and SDKs validate the names, so BLAKE3 cannot ride in one. + +### A dedicated subresource for the tree + +A `?blake3` subresource returning a CBOR document would keep `GetObjectAttributes` identical to AWS. It costs a new endpoint, a second place that must honour `versionId`, and a body format no S3 client parses today. No SDK can call it, so it has no advantage over an attribute for SDK users, and a raw HTTP client is better served by the XML it already handles. + +### Sending the leaf CVs in a header + +Base64 of 8 KiB is about 11 KiB, over the 8 KiB per-header default of common proxies, and splitting it across numbered headers is a hack. The root fits in a header; the tree does not. + +## Open questions + +- Names: `x-cid` and the `Blake3` attribute are placeholders. +- The leaf cap of 256. A lower cap is cheaper and coarser. +- Whether a `GET` with `?partNumber` should also return the leaf CVs covering that part, so a client that already reads by part need not align to groups. +- Whether the object CID should reach the Forge network, for example in the upload index, or remain an Ingot-level value until blob addressing is revisited. +- Whether to expose the digest on `ListObjects` responses. + +## Evaluation criteria + +- The S3 conformance suite passes unchanged: no ETag, checksum or body behaviour differs. +- Ingest throughput with the extra hash is within a few percent of today's. +- The share of `Complete` calls that take the re-hash path, measured against the AWS CLI, boto3, rclone, s5cmd and the MinIO client at their default part sizes, is near zero. +- Manifest size distribution matches the table above. +- A reference client verifies whole-object and ranged reads against a running Ingot, including a multipart object with non-power-of-two parts and a mis-ordered final part. + +## References + +- [BLAKE3 specification](https://github.com/BLAKE3-team/BLAKE3-specs/blob/master/blake3.pdf) +- [Bao] +- [bao-tree] +- [multihash] and the [multicodec table](https://github.com/multiformats/multicodec/blob/master/table.csv) +- [CID] +- [multibase] +- [S3 GetObjectAttributes](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectAttributes.html) +- [lukechampine.com/blake3](https://github.com/lukechampine/blake3), a Go implementation exposing the tree primitives (`guts`) and Bao with chunk groups + +[Bao]: https://github.com/oconnor663/bao +[bao-tree]: https://github.com/n0-computer/bao-tree +[multihash]: https://github.com/multiformats/multihash +[CID]: https://github.com/multiformats/cid +[multibase]: https://github.com/multiformats/multibase From b7908f16c6f36aa263fa6a31851f528634b28e41 Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Tue, 6 Oct 2026 10:22:22 +0100 Subject: [PATCH 02/19] refactor: adjust group sizing --- rfcs/2026-10-blake3-object-digest.md | 39 +++++++++++++++------------- 1 file changed, 21 insertions(+), 18 deletions(-) diff --git a/rfcs/2026-10-blake3-object-digest.md b/rfcs/2026-10-blake3-object-digest.md index ef239a8..7cc9428 100644 --- a/rfcs/2026-10-blake3-object-digest.md +++ b/rfcs/2026-10-blake3-object-digest.md @@ -19,9 +19,9 @@ Blob addressing on the Forge network stays SHA-256. This RFC adds an object-leve - Every object has a whole-object BLAKE3 digest, computed in the same streaming pass that writes its blobs. - A multipart object's digest is assembled at `Complete` from values recorded per part. Re-reading part data is a fallback for unusual clients, never the common path. - A client can verify a whole-object read with the digest alone and a standard BLAKE3 implementation. -- A client can verify a ranged read by fetching at most a few KiB of tree data once per object. +- A client can verify a ranged read by fetching the object's leaf list once: a few hundred bytes for a small object, growing with the square root of the size. - The S3 data path is unchanged. Bodies are raw bytes, ETags and S3 checksums behave as they do today, and a client that ignores the extension sees nothing new. -- Manifest growth is bounded at a few KiB regardless of object size. +- Manifest growth stays small for ordinary objects and under 1 MiB for the largest. The leaf list grows with the square root of the object size and is capped. ## Background: the BLAKE3 tree @@ -63,7 +63,7 @@ Each part record gains: - `TreeOffset`: the byte offset the part was hashed at. - `TreeNodes`: the CVs of the part's aligned subtrees, in order. A part of any length starting on a chunk boundary decomposes into at most two aligned power-of-two blocks per tree level, so this list holds a few dozen 32-byte entries at most. -- `TreeLeaves`: the part's leaf CVs at the part's own group (see [Range verification](#range-verification)), at most 256 entries. +- `TreeLeaves`: the part's leaf CVs at the part's own group (see [Range verification](#range-verification)), a few hundred entries at most for a 5 GiB part. - For part 1 only, `TreeRoot`: the BLAKE3 hash of the part on its own. Part 1 is always at offset 0, so its standalone hash falls out of the same pass with a root-flagged final compression. A part that supersedes an earlier upload of the same part number replaces its record, as today. @@ -93,26 +93,29 @@ The leaf CVs for the object are built from the parts' `TreeLeaves` and `TreeNode A client that reads ranges needs more than the root. It needs the CVs of the tree nodes covering the bytes it read, and enough of the rest of the tree to connect them to the root. The full Bao outboard provides this at 1 KiB granularity and costs about 6% of the object size. Storing it is out of the question in a manifest, and even at bao-tree's 16 KiB default it costs 4 MiB per GiB. -This RFC bounds the cost by fixing the number of leaves rather than the leaf size. +This RFC scales the leaf size with the object. The group is the geometric mean of the object size and 16 KiB, so the number of leaves grows with the square root of the size. A small object keeps a leaf list of a few hundred bytes, and the smallest verifiable read on a large object stays a small fraction of it. #### The group -The **group** is the leaf size of the stored tree: a power of two, at least 16 KiB, and the smallest such value that gives the object at most 256 leaves. The manifest records the group as a base-2 exponent. The group affects only what is stored and what granularity a client can verify at. The object digest is the same whatever group is chosen. +The **group** is the leaf size of the stored tree: a power of two, at least 16 KiB, and the smallest such value that is at least the square root of the object size times 16 KiB. The group doubles each time the object quadruples. A hard cap of 32,768 leaves, 1 MiB of CVs, applies beyond about 32 TB, where the group grows with the size instead. The manifest records the group as a base-2 exponent. The group affects only what is stored and what granularity a client can verify at. The object digest is the same whatever group is chosen. | Object size | Group | Leaves | Manifest bytes | |---|---|---|---| | 1 KiB | 16 KiB | 1 | 32 B | -| 1 MiB | 16 KiB | 64 | 2 KiB | -| 4 MiB | 16 KiB | 256 | 8 KiB | -| 5 MiB | 32 KiB | 160 | 5 KiB | -| 100 MiB | 512 KiB | 200 | 6.25 KiB | +| 64 KiB | 32 KiB | 2 | 64 B | +| 1 MiB | 128 KiB | 8 | 256 B | +| 4 MiB | 256 KiB | 16 | 512 B | +| 100 MiB | 2 MiB | 50 | 1.6 KiB | | 1 GiB | 4 MiB | 256 | 8 KiB | -| 50 GiB | 256 MiB | 200 | 6.25 KiB | -| 5 TiB | 32 GiB | 160 | 5 KiB | +| 10 GiB | 16 MiB | 640 | 20 KiB | +| 100 GiB | 64 MiB | 1,600 | 50 KiB | +| 1 TiB | 128 MiB | 8,192 | 256 KiB | +| 5 TiB | 512 MiB | 10,240 | 320 KiB | +| 50 TB | 2 GiB | 23,283 | 728 KiB | -Below 4 MiB the cost is 1/512 of the object size. Above 4 MiB it stays between 4 KiB and 8 KiB: each time the size crosses a power of two the group doubles and the leaf count halves. +An object of 16 KiB or less is one leaf. From there the manifest cost grows with the square root of the size: a quarter of a KiB at 1 MiB, 8 KiB at 1 GiB, 256 KiB at 1 TiB. The cap applies past about 32 TB and holds a 50 TB object to 728 KiB. -The smallest range a client can verify independently is one group. A 1 GiB object verifies in 4 MiB pieces. A multi-TiB object verifies in pieces of tens of GiB, which is coarse, and an object of that size with a need for fine-grained verification would want the outboard-as-blob design under [Alternatives](#alternatives-considered). The cap of 256 is a tuning constant: halving it halves the manifest cost and doubles the smallest verifiable read. +The smallest range a client can verify independently is one group: 256 KiB on a 4 MiB object, 4 MiB on a 1 GiB object, 128 MiB on a 1 TiB object, 2 GiB on a 50 TB object. Two constants tune the rule. The 16 KiB floor sets where the scaling starts, and the cap bounds the manifest. An object needing finer verification than its group would want the outboard-as-blob design under [Alternatives](#alternatives-considered). #### Leaf CVs @@ -150,7 +153,7 @@ x-amz-object-attributes: ETag,Blake3 ``` -`CID` is the object digest as returned in `x-cid`, `Group` is the group exponent and each `Leaf` is a leaf CV in order, encoded in base64 like the checksums in `ObjectParts`. At the 256-leaf cap the element is 15 to 20 KiB. A tree belongs to one version. An overwrite can never pair an old tree with new bytes, because both live in the same manifest, and the response carries the ETag of the version the tree describes for the client's `If-Match`. +`CID` is the object digest as returned in `x-cid`, `Group` is the group exponent and each `Leaf` is a leaf CV in order, encoded in base64 like the checksums in `ObjectParts`. The element is a few hundred bytes for a small object, about 12 KiB at 1 GiB and up to 1.6 MiB at the leaf cap. A tree belongs to one version. An overwrite can never pair an old tree with new bytes, because both live in the same manifest, and the response carries the ETag of the version the tree describes for the client's `If-Match`. AWS rejects an attribute name it does not know with `InvalidArgument`, so accepting `Blake3` is a visible divergence. SDKs whose typed response omits unknown elements cannot read the tree, but the same SDKs could not call a custom endpoint either, and a raw HTTP client parses this XML with what it already has for every other S3 response. @@ -175,7 +178,7 @@ type Body struct { // TreeGroup is the base-2 exponent of the leaf size of TreeLeaves (one byte). TreeGroup uint8 `cborgen:"tg"` // TreeLeaves holds the chaining values of the group-aligned blocks of the - // body, 32 bytes each, in order (at most 8 KiB). + // body, 32 bytes each, in order (8 KiB at 1 GiB, at most 1 MiB). TreeLeaves []byte `cborgen:"tl"` } ``` @@ -188,7 +191,7 @@ Ingest adds one BLAKE3 pass per body or part, alongside the SHA-256 and MD5 pass `CompleteMultipartUpload` adds a few hundred compressions in the common path. The re-hash path reads the affected parts back through the plaintext opener, which decrypts in an encrypting deployment, and is bounded by the size of the mis-guessed parts. -The manifest grows by at most 8 KiB plus the 34-byte multihash and a few bytes of framing. The part record grows by at most about 10 KiB for the life of the upload. +The manifest grows by the leaf list, 8 KiB at 1 GiB and at most 1 MiB, plus the 34-byte multihash and a few bytes of framing. The part record grows by at most about 15 KiB for the life of the upload. ## Alternatives considered @@ -218,12 +221,12 @@ A `?blake3` subresource returning a CBOR document would keep `GetObjectAttribute ### Sending the leaf CVs in a header -Base64 of 8 KiB is about 11 KiB, over the 8 KiB per-header default of common proxies, and splitting it across numbered headers is a hack. The root fits in a header; the tree does not. +Base64 of a 1 GiB object's 8 KiB leaf list is about 11 KiB, over the 8 KiB per-header default of common proxies, larger objects have far more, and splitting it across numbered headers is a hack. The root fits in a header; the tree does not. ## Open questions - Names: `x-cid` and the `Blake3` attribute are placeholders. -- The leaf cap of 256. A lower cap is cheaper and coarser. +- The scaling constants: the 16 KiB floor and the 32,768-leaf cap. The exponent is a knob too; cube-root scaling would give a 50 TB object 1,456 leaves at a 32 GiB group. - Whether a `GET` with `?partNumber` should also return the leaf CVs covering that part, so a client that already reads by part need not align to groups. - Whether the object CID should reach the Forge network, for example in the upload index, or remain an Ingot-level value until blob addressing is revisited. - Whether to expose the digest on `ListObjects` responses. From f9cce396ec554e0ac14c3aa9e5672ef423731d4e Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Tue, 6 Oct 2026 11:01:33 +0100 Subject: [PATCH 03/19] refactor: encode outboard in standard form --- rfcs/2026-10-blake3-object-digest.md | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/rfcs/2026-10-blake3-object-digest.md b/rfcs/2026-10-blake3-object-digest.md index 7cc9428..156cc65 100644 --- a/rfcs/2026-10-blake3-object-digest.md +++ b/rfcs/2026-10-blake3-object-digest.md @@ -19,7 +19,7 @@ Blob addressing on the Forge network stays SHA-256. This RFC adds an object-leve - Every object has a whole-object BLAKE3 digest, computed in the same streaming pass that writes its blobs. - A multipart object's digest is assembled at `Complete` from values recorded per part. Re-reading part data is a fallback for unusual clients, never the common path. - A client can verify a whole-object read with the digest alone and a standard BLAKE3 implementation. -- A client can verify a ranged read by fetching the object's leaf list once: a few hundred bytes for a small object, growing with the square root of the size. +- A client can verify a ranged read with an off-the-shelf Bao library by fetching the object's outboard once: under a KiB for a small object, growing with the square root of the size. - The S3 data path is unchanged. Bodies are raw bytes, ETags and S3 checksums behave as they do today, and a client that ignores the extension sees nothing new. - Manifest growth stays small for ordinary objects and under 1 MiB for the largest. The leaf list grows with the square root of the object size and is capped. @@ -31,7 +31,7 @@ The tree shape is fixed by the input length. For an input of more than one chunk Two further consequences follow. A contiguous byte range that starts on a chunk boundary decomposes into a short sequence of such aligned blocks, at most two per tree level, and its CVs can be computed knowing only the bytes and the starting offset. And the hash of the whole input can be computed from the CVs of any set of aligned blocks that partition it, with one parent compression per merge and the root flag applied at the last. -This is the structure that [Bao] and iroh's [bao-tree] use for verified streaming. Their "outboard" is the list of parent nodes, which lets a reader holding only the root verify any range. This RFC stores a reduced form of that outboard and leaves the Bao wire format for a possible later mode. +This is the structure that [Bao] and iroh's [bao-tree] use for verified streaming. Their "outboard" is the list of parent nodes, which lets a reader holding only the root verify any range. This RFC stores the bottom of that outboard at a coarse block size, serves the outboard itself to clients, and leaves the Bao streaming encoding for a possible later mode. ## Design @@ -145,23 +145,20 @@ x-amz-object-attributes: ETag,Blake3 bafkr4i... 22 - - base64 of 32 bytes - ... - + base64 ``` -`CID` is the object digest as returned in `x-cid`, `Group` is the group exponent and each `Leaf` is a leaf CV in order, encoded in base64 like the checksums in `ObjectParts`. The element is a few hundred bytes for a small object, about 12 KiB at 1 GiB and up to 1.6 MiB at the leaf cap. A tree belongs to one version. An overwrite can never pair an old tree with new bytes, because both live in the same manifest, and the response carries the ETag of the version the tree describes for the client's `If-Match`. +`CID` is the object digest as returned in `x-cid` and `Group` is the group exponent, which is the Bao block size: a Bao library's chunk log is `Group` minus 10. `Outboard` is the standard pre-order Bao outboard at that block size, base64-encoded: the object size as 8 little-endian bytes, then the chaining-value pair of every parent node above the leaves, root first and left subtree before right. It is built from the stored leaves on request, one compression per parent, and is byte-identical to what the Bao libraries produce for the same object and block size. It is about 64 bytes per leaf: under 100 bytes for a small object, 16 KiB at 1 GiB, 1.4 MiB at the leaf cap, plus a third for base64. A tree belongs to one version. An overwrite can never pair an old tree with new bytes, because both live in the same manifest, and the response carries the ETag of the version the tree describes for the client's `If-Match`. AWS rejects an attribute name it does not know with `InvalidArgument`, so accepting `Blake3` is a visible divergence. SDKs whose typed response omits unknown elements cannot read the tree, but the same SDKs could not call a custom endpoint either, and a raw HTTP client parses this XML with what it already has for every other S3 response. #### Client procedure -1. Fetch the tree once with `GetObjectAttributes`. Compute the parents from the leaf CVs up to the root and compare with the CID. This is at most 255 compressions. When the tree holds a single leaf the object is at most one group long, and the client verifies by hashing the whole object instead. -2. Read ranges with ordinary S3 `Range` requests aligned to group boundaries, sending `If-Match` with the ETag so the bytes cannot change between the tree fetch and the reads. -3. Hash each group received as a non-root BLAKE3 subtree at its chunk offset and compare with its leaf CV. +1. Fetch the attribute once with `GetObjectAttributes`, and load the CID's digest, the block size and the outboard into a Bao library (bao-tree in Rust, the Go BLAKE3 library's `bao` package). The library checks the outboard against the root as it goes; no merge code of our own is needed, and no BLAKE3 API beyond what a Bao library already wraps. +2. Read ranges with ordinary S3 `Range` requests aligned to block boundaries, sending `If-Match` with the ETag so the bytes cannot change between the attribute fetch and the reads. A Bao library can name the byte ranges needed for the blocks a client wants. +3. Verify each block received against the outboard. An object of one block has an empty outboard, and the client verifies by hashing the whole object against the CID. Every byte of data travels as a plain S3 body. SDKs, caches and CDNs behave as they do today. @@ -209,7 +206,7 @@ bao-tree's default gives 16 KiB verification granularity at any size, with the o ### Returning the Bao encoded stream -iroh serves data interleaved with the parent nodes so a client verifies as bytes arrive. S3 SDKs cannot consume such a body, so it would be a separate mode rather than the S3 `GET`. At the groups this RFC uses a client must buffer a whole group before verifying anything, so interleaving saves only the one tree request. The server can expand the stored leaf CVs into a pair-form outboard on the fly, so the encoded stream can be added later as its own subresource without changing what is stored. +iroh serves data interleaved with the parent nodes so a client verifies as bytes arrive. S3 SDKs cannot consume such a body, so it would be a separate mode rather than the S3 `GET`. At the groups this RFC uses a client must buffer a whole group before verifying anything, so interleaving saves only the one tree request. The outboard served by `GetObjectAttributes` is already the Bao form, so the encoded stream can be added later as its own subresource without changing what is stored. ### Carrying the digest in an S3 checksum header @@ -219,9 +216,13 @@ The `x-amz-checksum-*` headers have a fixed set of algorithms and SDKs validate A `?blake3` subresource returning a CBOR document would keep `GetObjectAttributes` identical to AWS. It costs a new endpoint, a second place that must honour `versionId`, and a body format no S3 client parses today. No SDK can call it, so it has no advantage over an attribute for SDK users, and a raw HTTP client is better served by the XML it already handles. +### Returning the leaf list instead of the outboard + +The stored leaves are half the size of the outboard, and a client could merge them to the root itself. But checking a block against its leaf means hashing the block as a non-root subtree at its chunk offset, which standard BLAKE3 APIs do not expose, so every client would carry custom tree code. Serving the Bao outboard costs twice the bytes and lets any Bao library do the verification. + ### Sending the leaf CVs in a header -Base64 of a 1 GiB object's 8 KiB leaf list is about 11 KiB, over the 8 KiB per-header default of common proxies, larger objects have far more, and splitting it across numbered headers is a hack. The root fits in a header; the tree does not. +Base64 of a 1 GiB object's 16 KiB outboard is about 22 KiB, over the 8 KiB per-header default of common proxies, larger objects have far more, and splitting it across numbered headers is a hack. The root fits in a header; the tree does not. ## Open questions From 96e2bed1cb356d0874a9e18c858b61f8f2c4ea66 Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Tue, 6 Oct 2026 11:03:21 +0100 Subject: [PATCH 04/19] refactor: a list for easier reading --- rfcs/2026-10-blake3-object-digest.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/rfcs/2026-10-blake3-object-digest.md b/rfcs/2026-10-blake3-object-digest.md index 156cc65..35466cc 100644 --- a/rfcs/2026-10-blake3-object-digest.md +++ b/rfcs/2026-10-blake3-object-digest.md @@ -150,7 +150,9 @@ x-amz-object-attributes: ETag,Blake3 ``` -`CID` is the object digest as returned in `x-cid` and `Group` is the group exponent, which is the Bao block size: a Bao library's chunk log is `Group` minus 10. `Outboard` is the standard pre-order Bao outboard at that block size, base64-encoded: the object size as 8 little-endian bytes, then the chaining-value pair of every parent node above the leaves, root first and left subtree before right. It is built from the stored leaves on request, one compression per parent, and is byte-identical to what the Bao libraries produce for the same object and block size. It is about 64 bytes per leaf: under 100 bytes for a small object, 16 KiB at 1 GiB, 1.4 MiB at the leaf cap, plus a third for base64. A tree belongs to one version. An overwrite can never pair an old tree with new bytes, because both live in the same manifest, and the response carries the ETag of the version the tree describes for the client's `If-Match`. +- `CID` is the object digest as returned in `x-cid` +- `Group` is the group exponent, which is the Bao block size: a Bao library's chunk log is `Group` minus 10. +- `Outboard` is the standard pre-order Bao outboard at that block size, base64-encoded: the object size as 8 little-endian bytes, then the chaining-value pair of every parent node above the leaves, root first and left subtree before right. It is built from the stored leaves on request, one compression per parent, and is byte-identical to what the Bao libraries produce for the same object and block size. It is about 64 bytes per leaf: under 100 bytes for a small object, 16 KiB at 1 GiB, 1.4 MiB at the leaf cap, plus a third for base64. A tree belongs to one version. An overwrite can never pair an old tree with new bytes, because both live in the same manifest, and the response carries the ETag of the version the tree describes for the client's `If-Match`. AWS rejects an attribute name it does not know with `InvalidArgument`, so accepting `Blake3` is a visible divergence. SDKs whose typed response omits unknown elements cannot read the tree, but the same SDKs could not call a custom endpoint either, and a raw HTTP client parses this XML with what it already has for every other S3 response. From d72ec18a2ac80fd192b1d320e28a57a8422b3149 Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Tue, 6 Oct 2026 15:03:10 +0100 Subject: [PATCH 05/19] refactor: rename group to chunk log --- rfcs/2026-10-blake3-object-digest.md | 60 ++++++++++++++-------------- 1 file changed, 30 insertions(+), 30 deletions(-) diff --git a/rfcs/2026-10-blake3-object-digest.md b/rfcs/2026-10-blake3-object-digest.md index 35466cc..485a7fe 100644 --- a/rfcs/2026-10-blake3-object-digest.md +++ b/rfcs/2026-10-blake3-object-digest.md @@ -63,7 +63,7 @@ Each part record gains: - `TreeOffset`: the byte offset the part was hashed at. - `TreeNodes`: the CVs of the part's aligned subtrees, in order. A part of any length starting on a chunk boundary decomposes into at most two aligned power-of-two blocks per tree level, so this list holds a few dozen 32-byte entries at most. -- `TreeLeaves`: the part's leaf CVs at the part's own group (see [Range verification](#range-verification)), a few hundred entries at most for a 5 GiB part. +- `TreeLeaves`: the part's leaf CVs at the part's own block size (see [Range verification](#range-verification)), a few hundred entries at most for a 5 GiB part. - For part 1 only, `TreeRoot`: the BLAKE3 hash of the part on its own. Part 1 is always at offset 0, so its standalone hash falls out of the same pass with a root-flagged final compression. A part that supersedes an earlier upload of the same part number replaces its record, as today. @@ -93,37 +93,37 @@ The leaf CVs for the object are built from the parts' `TreeLeaves` and `TreeNode A client that reads ranges needs more than the root. It needs the CVs of the tree nodes covering the bytes it read, and enough of the rest of the tree to connect them to the root. The full Bao outboard provides this at 1 KiB granularity and costs about 6% of the object size. Storing it is out of the question in a manifest, and even at bao-tree's 16 KiB default it costs 4 MiB per GiB. -This RFC scales the leaf size with the object. The group is the geometric mean of the object size and 16 KiB, so the number of leaves grows with the square root of the size. A small object keeps a leaf list of a few hundred bytes, and the smallest verifiable read on a large object stays a small fraction of it. +This RFC scales the leaf size with the object. The block is the geometric mean of the object size and 16 KiB, so the number of leaves grows with the square root of the size. A small object keeps a leaf list of a few hundred bytes, and the smallest verifiable read on a large object stays a small fraction of it. -#### The group +#### The block -The **group** is the leaf size of the stored tree: a power of two, at least 16 KiB, and the smallest such value that is at least the square root of the object size times 16 KiB. The group doubles each time the object quadruples. A hard cap of 32,768 leaves, 1 MiB of CVs, applies beyond about 32 TB, where the group grows with the size instead. The manifest records the group as a base-2 exponent. The group affects only what is stored and what granularity a client can verify at. The object digest is the same whatever group is chosen. +The **block** is the leaf size of the stored tree: a power of two, at least 16 KiB, and the smallest such value that is at least the square root of the object size times 16 KiB. The block doubles each time the object quadruples. A hard cap of 32,768 leaves, 1 MiB of CVs, applies beyond about 32 TB, where the block grows with the size instead. The manifest records the block size as a **chunk log**: a base-2 exponent of 1 KiB BLAKE3 chunks, the unit Bao libraries take as the block size, so 16 KiB is 4 and 4 MiB is 12. The block affects only what is stored and what granularity a client can verify at. The object digest is the same whatever block is chosen. -| Object size | Group | Leaves | Manifest bytes | -|---|---|---|---| -| 1 KiB | 16 KiB | 1 | 32 B | -| 64 KiB | 32 KiB | 2 | 64 B | -| 1 MiB | 128 KiB | 8 | 256 B | -| 4 MiB | 256 KiB | 16 | 512 B | -| 100 MiB | 2 MiB | 50 | 1.6 KiB | -| 1 GiB | 4 MiB | 256 | 8 KiB | -| 10 GiB | 16 MiB | 640 | 20 KiB | -| 100 GiB | 64 MiB | 1,600 | 50 KiB | -| 1 TiB | 128 MiB | 8,192 | 256 KiB | -| 5 TiB | 512 MiB | 10,240 | 320 KiB | -| 50 TB | 2 GiB | 23,283 | 728 KiB | +| Object size | Block | Chunk log | Leaves | Manifest bytes | +|---|---|---|---|---| +| 1 KiB | 16 KiB | 4 | 1 | 32 B | +| 64 KiB | 32 KiB | 5 | 2 | 64 B | +| 1 MiB | 128 KiB | 7 | 8 | 256 B | +| 4 MiB | 256 KiB | 8 | 16 | 512 B | +| 100 MiB | 2 MiB | 11 | 50 | 1.6 KiB | +| 1 GiB | 4 MiB | 12 | 256 | 8 KiB | +| 10 GiB | 16 MiB | 14 | 640 | 20 KiB | +| 100 GiB | 64 MiB | 16 | 1,600 | 50 KiB | +| 1 TiB | 128 MiB | 17 | 8,192 | 256 KiB | +| 5 TiB | 512 MiB | 19 | 10,240 | 320 KiB | +| 50 TB | 2 GiB | 21 | 23,283 | 728 KiB | An object of 16 KiB or less is one leaf. From there the manifest cost grows with the square root of the size: a quarter of a KiB at 1 MiB, 8 KiB at 1 GiB, 256 KiB at 1 TiB. The cap applies past about 32 TB and holds a 50 TB object to 728 KiB. -The smallest range a client can verify independently is one group: 256 KiB on a 4 MiB object, 4 MiB on a 1 GiB object, 128 MiB on a 1 TiB object, 2 GiB on a 50 TB object. Two constants tune the rule. The 16 KiB floor sets where the scaling starts, and the cap bounds the manifest. An object needing finer verification than its group would want the outboard-as-blob design under [Alternatives](#alternatives-considered). +The smallest range a client can verify independently is one block: 256 KiB on a 4 MiB object, 4 MiB on a 1 GiB object, 128 MiB on a 1 TiB object, 2 GiB on a 50 TB object. Two constants tune the rule. The 16 KiB floor sets where the scaling starts, and the cap bounds the manifest. An object needing finer verification than its block would want the outboard-as-blob design under [Alternatives](#alternatives-considered). #### Leaf CVs -The manifest stores the CV of every group-aligned block of the object, in order: the **leaf CVs**. These are the nodes of the BLAKE3 tree at the group level, each 32 bytes. Every node above them is derivable with one compression per node, so nothing above the leaves is stored. +The manifest stores the CV of every block of the object, in order: the **leaf CVs**. These are the nodes of the BLAKE3 tree at the block level, each 32 bytes. Every node above them is derivable with one compression per node, so nothing above the leaves is stored. -A single `PUT` computes the leaf CVs in its hashing pass, since the object length is known from `Content-Length` and so is the group. +A single `PUT` computes the leaf CVs in its hashing pass, since the object length is known from `Content-Length` and so is the block. -A multipart upload does not know the object length until `CompleteMultipartUpload`, so each part computes `TreeLeaves` at its own group, chosen by the same rule from the part's own size. The object's group is never smaller than any part's group, because the rule is monotone in size and the object is at least as large as each part. At `CompleteMultipartUpload`, each object leaf is one of: +A multipart upload does not know the object length until `CompleteMultipartUpload`, so each part computes `TreeLeaves` at its own block size, chosen by the same rule from the part's own size. The object's block is never smaller than any part's, because the rule is monotone in size and the object is at least as large as each part. At `CompleteMultipartUpload`, each object leaf is one of: - A block lying within one part. Its CV is the merge of that part's `TreeLeaves` that it covers. - A block straddling a part boundary. The piece on each side is a union of that part's edge subtrees, which `TreeNodes` holds, so its CV is the merge of those entries. @@ -144,14 +144,14 @@ x-amz-object-attributes: ETag,Blake3 ... bafkr4i... - 22 + 12 base64 ``` - `CID` is the object digest as returned in `x-cid` -- `Group` is the group exponent, which is the Bao block size: a Bao library's chunk log is `Group` minus 10. +- `ChunkLog` is the block size as a base-2 exponent of 1 KiB BLAKE3 chunks, the value a Bao library takes as its block size unchanged. iroh's fixed block is 4; the original Bao format is 0. - `Outboard` is the standard pre-order Bao outboard at that block size, base64-encoded: the object size as 8 little-endian bytes, then the chaining-value pair of every parent node above the leaves, root first and left subtree before right. It is built from the stored leaves on request, one compression per parent, and is byte-identical to what the Bao libraries produce for the same object and block size. It is about 64 bytes per leaf: under 100 bytes for a small object, 16 KiB at 1 GiB, 1.4 MiB at the leaf cap, plus a third for base64. A tree belongs to one version. An overwrite can never pair an old tree with new bytes, because both live in the same manifest, and the response carries the ETag of the version the tree describes for the client's `If-Match`. AWS rejects an attribute name it does not know with `InvalidArgument`, so accepting `Blake3` is a visible divergence. SDKs whose typed response omits unknown elements cannot read the tree, but the same SDKs could not call a custom endpoint either, and a raw HTTP client parses this XML with what it already has for every other S3 response. @@ -174,9 +174,9 @@ type Body struct { // BLAKE3 is the whole-object blake3 multihash (34 bytes). BLAKE3 []byte `cborgen:"b3"` - // TreeGroup is the base-2 exponent of the leaf size of TreeLeaves (one byte). - TreeGroup uint8 `cborgen:"tg"` - // TreeLeaves holds the chaining values of the group-aligned blocks of the + // TreeChunkLog is the block size of TreeLeaves as a base-2 exponent of chunks (one byte). + TreeChunkLog uint8 `cborgen:"tc"` + // TreeLeaves holds the chaining values of the blocks of the // body, 32 bytes each, in order (8 KiB at 1 GiB, at most 1 MiB). TreeLeaves []byte `cborgen:"tl"` } @@ -202,13 +202,13 @@ Keeping the existing SHA-256 and computing it for multipart objects means re-rea Making BLAKE3 the network's content address would let storage nodes serve verified ranges directly. It touches every component that handles a multihash, the proof-tree assumptions of PDP, public gateway resolvability, and every stored byte. This RFC is independent of that decision. The object digest proposed here is the same value such a migration would use, so nothing here is thrown away if it happens. -### A fixed 16 KiB group with the outboard as a blob +### A fixed 16 KiB block with the outboard as a blob bao-tree's default gives 16 KiB verification granularity at any size, with the outboard stored as its own blob referenced from the manifest, about 0.4% of the object. It is the right design if fine-grained verified reads of very large objects are a requirement. It adds a blob per object to write, locate and clean up, and a second request to fetch something that is usually megabytes. The bounded manifest tree serves the common case with no new blob and can coexist with this design later: the stored leaves are the top of the finer tree. ### Returning the Bao encoded stream -iroh serves data interleaved with the parent nodes so a client verifies as bytes arrive. S3 SDKs cannot consume such a body, so it would be a separate mode rather than the S3 `GET`. At the groups this RFC uses a client must buffer a whole group before verifying anything, so interleaving saves only the one tree request. The outboard served by `GetObjectAttributes` is already the Bao form, so the encoded stream can be added later as its own subresource without changing what is stored. +iroh serves data interleaved with the parent nodes so a client verifies as bytes arrive. S3 SDKs cannot consume such a body, so it would be a separate mode rather than the S3 `GET`. At the block sizes this RFC uses a client must buffer a whole block before verifying anything, so interleaving saves only the one tree request. The outboard served by `GetObjectAttributes` is already the Bao form, so the encoded stream can be added later as its own subresource without changing what is stored. ### Carrying the digest in an S3 checksum header @@ -229,8 +229,8 @@ Base64 of a 1 GiB object's 16 KiB outboard is about 22 KiB, over the 8 KiB per-h ## Open questions - Names: `x-cid` and the `Blake3` attribute are placeholders. -- The scaling constants: the 16 KiB floor and the 32,768-leaf cap. The exponent is a knob too; cube-root scaling would give a 50 TB object 1,456 leaves at a 32 GiB group. -- Whether a `GET` with `?partNumber` should also return the leaf CVs covering that part, so a client that already reads by part need not align to groups. +- The scaling constants: the 16 KiB floor and the 32,768-leaf cap. The exponent is a knob too; cube-root scaling would give a 50 TB object 1,456 leaves at a 32 GiB block. +- Whether a `GET` with `?partNumber` should also return the leaf CVs covering that part, so a client that already reads by part need not align to blocks. - Whether the object CID should reach the Forge network, for example in the upload index, or remain an Ingot-level value until blob addressing is revisited. - Whether to expose the digest on `ListObjects` responses. From 645a675eabe9a3ac09ce62c7d8a059edfad054df Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Wed, 7 Oct 2026 11:06:10 +0100 Subject: [PATCH 06/19] refactor: address review comments --- rfcs/2026-10-blake3-object-digest.md | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/rfcs/2026-10-blake3-object-digest.md b/rfcs/2026-10-blake3-object-digest.md index 485a7fe..3f2bf0d 100644 --- a/rfcs/2026-10-blake3-object-digest.md +++ b/rfcs/2026-10-blake3-object-digest.md @@ -8,7 +8,7 @@ Status: Experimental ## Motivation -An Ingot object has no digest a client can verify against. A single `PUT` records a whole-object SHA-256 in the manifest, but it is internal and never returned. A multipart object records no whole-object digest at all, because SHA-256 cannot be assembled from the parts and re-reading every part at `CompleteMultipartUpload` is too slow. A client that wants to check what it downloaded has the S3 ETag, which for a multipart object is a hash of part MD5s and verifies nothing about the bytes, and the optional S3 checksums, which for multipart are composite values with the same limitation. Nothing lets a client verify a byte range. +An Ingot object has no digest a client can verify against. A single `PUT` records a whole-object SHA-256 in the manifest, but it is internal and never returned. A multipart object records no whole-object digest at all, because SHA-256 cannot be assembled from the parts and re-reading every part at `CompleteMultipartUpload` is too slow. A client that wants to check what it downloaded has the S3 ETag, which for a multipart object is a hash of part MD5s and verifies nothing about the bytes, and the optional S3 checksums. Of those, the SHA family are composite values for a multipart object, a checksum of the parts' checksums with the same limitation. The CRC family (CRC32, CRC32C, CRC64NVME) can be a full-object value, which Ingot derives at `CompleteMultipartUpload` by combining the parts' CRCs, but a CRC detects accidental corruption and nothing more: it is not a cryptographic digest, it does not identify the object, and it offers no range verification. Nothing lets a client verify a byte range. This RFC proposes a whole-object BLAKE3 digest for every object, single `PUT` or multipart, returned to clients as a CID, plus a small tree of chaining values stored in the manifest that lets a client verify ranged reads. BLAKE3 is a Merkle tree, so the digest of a multipart object is assembled from per-part subtrees at `Complete` with no re-read of the data, and the same tree structure gives range verification for free. @@ -55,7 +55,7 @@ The body already streams through a SHA-256 hasher and an MD5 hasher as it is spl ### Multipart upload -Parts arrive in any order and the server learns a part's true byte offset only once every lower-numbered part exists. Each `UploadPart` therefore hashes its body at a guessed offset and records what `CompleteMultipartUpload` needs to finish the tree. `CompleteMultipartUpload` checks the guesses against the true offsets and re-hashes only the parts whose guess was wrong. +Parts arrive in any order and the server learns a part's true byte offset only once every lower-numbered part exists. Each `UploadPart` therefore hashes its body at an assumed offset and records what `CompleteMultipartUpload` needs to finish the tree. `CompleteMultipartUpload` checks each assumed offset against the true one and re-hashes only the parts whose assumption was wrong. #### What `UploadPart` records @@ -68,26 +68,28 @@ Each part record gains: A part that supersedes an earlier upload of the same part number replaces its record, as today. -#### Guessing the offset +#### Assuming the offset - Part 1 is at offset 0. - Part _n_ when every part below _n_ is already recorded: the sum of their sizes. This is exact. - Part _n_ when some lower part is recorded: assume a uniform part size and use (_n_ − 1) times the size of the lowest-numbered recorded part. - Part _n_ when no lower part is recorded: (_n_ − 1) times this part's own size. -Clients upload parts of one size in parallel, with a shorter final part. The rules above guess right for every part of a uniform-size upload except a final part that lands before any of its predecessors, which is re-hashed at `Complete` and is the smallest part of the upload. +Clients upload parts of one size in parallel, with a shorter final part. The rules above give the true offset for every part of a uniform-size upload except a final part that lands before any of its predecessors, which is re-hashed at `Complete` and is the smallest part of the upload. A part created by copy is hashed as its bytes are read, like an uploaded part. Where the copy does not read the bytes, the part records no tree and takes the re-hash path. #### What `CompleteMultipartUpload` does -`CompleteMultipartUpload` already walks the requested parts in order and computes each part's true offset. For each part it checks that the true offset is a multiple of 1 KiB and equals `TreeOffset`. A part that fails either check is re-hashed at its true offset by reading its blobs back, and its record is replaced before the merge proceeds. A part offset that is not a multiple of 1 KiB means some earlier part has a length that is not a multiple of 1 KiB, which places a chunk boundary inside a part. Such parts and every later part are re-hashed. This is rare: clients size parts in MiB. +`CompleteMultipartUpload` already walks the requested parts in order and computes each part's true offset. A part's record is usable when its true offset equals `TreeOffset`. A part whose assumed offset was wrong is re-hashed at its true offset by reading its blobs back, and the result stands in for its record before the merge proceeds. -With every part's `TreeNodes` at correct offsets, the object digest is computed by merging adjacent aligned subtrees into parents and applying the root flag at the final merge. This is one compression per node, a few hundred at most, and reads no data. +A part whose length is not a multiple of 1 KiB, other than the last, puts a chunk boundary inside the next part. No record from that part onward can serve: its own tree treated its short final chunk as the object's end, and every later part was hashed at a chunk index its bytes do not start on. Re-hashing those parts one at a time would not help either, since the chunk straddling the boundary has bytes in both. The body from the first such part to its end is therefore re-hashed as one range, from an aligned offset, and that range replaces all of their records. This is rare: clients size parts in MiB. -An upload completed with a single part has no merge. Its digest is part 1's `TreeRoot`. +With every range at its correct offset, the object digest is computed by merging adjacent aligned subtrees into parents and applying the root flag at the final merge. This is one compression per recorded subtree and reads no data. The work scales with the number of parts: a part contributes at most two subtrees per tree level, so an upload of 10,000 parts merges a few tens of thousands of nodes, each one BLAKE3 compression, which is milliseconds. -The leaf CVs for the object are built from the parts' `TreeLeaves` and `TreeNodes` as described in the next section. The root computed by merging the leaf CVs up must equal the root computed from the parts' subtrees. This is a free consistency check on the assembly. +An upload completed with a single part has no merge, and its digest is that part's root as a body of its own. For part 1 that is the recorded `TreeRoot`. For any other part the recorded offset is not 0, so the offset check above re-hashes it at offset 0, and that re-hash yields the root directly. S3 requires the completed parts to be in ascending order, not to begin at part 1, so this case is legal and does occur. + +The leaf CVs for the object are built from the parts' `TreeLeaves` and `TreeNodes` as described in the next section. The root computed by merging the leaf CVs up must equal the root computed from the parts' subtrees. This is a consistency check on the assembly at the cost of one compression per leaf, 32,767 at the cap; should it ever fail, the whole body is re-hashed, so the result is correct even if a record is bad. ### Range verification @@ -111,9 +113,9 @@ The **block** is the leaf size of the stored tree: a power of two, at least 16 K | 100 GiB | 64 MiB | 16 | 1,600 | 50 KiB | | 1 TiB | 128 MiB | 17 | 8,192 | 256 KiB | | 5 TiB | 512 MiB | 19 | 10,240 | 320 KiB | -| 50 TB | 2 GiB | 21 | 23,283 | 728 KiB | +| 50 TB | 2 GiB | 21 | 23,284 | 728 KiB | -An object of 16 KiB or less is one leaf. From there the manifest cost grows with the square root of the size: a quarter of a KiB at 1 MiB, 8 KiB at 1 GiB, 256 KiB at 1 TiB. The cap applies past about 32 TB and holds a 50 TB object to 728 KiB. +Sizes with binary prefixes are powers of two; the 50 TB row is decimal, 50 × 10¹² bytes, the largest object this RFC accounts for. Leaves is the size divided by the block, rounded up, since a final partial block still has a CV. An object of 16 KiB or less is one leaf. From there the manifest cost grows with the square root of the size: a quarter of a KiB at 1 MiB, 8 KiB at 1 GiB, 256 KiB at 1 TiB. The cap applies past about 32 TB and holds a 50 TB object to 728 KiB. The smallest range a client can verify independently is one block: 256 KiB on a 4 MiB object, 4 MiB on a 1 GiB object, 128 MiB on a 1 TiB object, 2 GiB on a 50 TB object. Two constants tune the rule. The 16 KiB floor sets where the scaling starts, and the cap bounds the manifest. An object needing finer verification than its block would want the outboard-as-blob design under [Alternatives](#alternatives-considered). @@ -152,7 +154,7 @@ x-amz-object-attributes: ETag,Blake3 - `CID` is the object digest as returned in `x-cid` - `ChunkLog` is the block size as a base-2 exponent of 1 KiB BLAKE3 chunks, the value a Bao library takes as its block size unchanged. iroh's fixed block is 4; the original Bao format is 0. -- `Outboard` is the standard pre-order Bao outboard at that block size, base64-encoded: the object size as 8 little-endian bytes, then the chaining-value pair of every parent node above the leaves, root first and left subtree before right. It is built from the stored leaves on request, one compression per parent, and is byte-identical to what the Bao libraries produce for the same object and block size. It is about 64 bytes per leaf: under 100 bytes for a small object, 16 KiB at 1 GiB, 1.4 MiB at the leaf cap, plus a third for base64. A tree belongs to one version. An overwrite can never pair an old tree with new bytes, because both live in the same manifest, and the response carries the ETag of the version the tree describes for the client's `If-Match`. +- `Outboard` is the standard pre-order Bao outboard at that block size, base64-encoded: the object size as 8 little-endian bytes, then the chaining-value pair of every parent node above the leaves, root first and left subtree before right. It is built from the stored leaves on request, one compression per parent, and is byte-identical to what the Bao libraries produce for the same object and block size. It is an 8-byte prefix plus 64 bytes per parent, one fewer than the leaves: under 100 bytes for a small object, 16 KiB at 1 GiB, 1.4 MiB for a 50 TB object and 2 MiB at the 32,768-leaf cap, about 2.7 MiB after base64. A tree belongs to one version. An overwrite can never pair an old tree with new bytes, because both live in the same manifest, and the response carries the ETag of the version the tree describes for the client's `If-Match`. AWS rejects an attribute name it does not know with `InvalidArgument`, so accepting `Blake3` is a visible divergence. SDKs whose typed response omits unknown elements cannot read the tree, but the same SDKs could not call a custom endpoint either, and a raw HTTP client parses this XML with what it already has for every other S3 response. @@ -186,9 +188,9 @@ Objects written before this change have none of these fields. A read of such an ### Cost -Ingest adds one BLAKE3 pass per body or part, alongside the SHA-256 and MD5 passes already there. BLAKE3 with vector instructions is faster than SHA-256, so the added CPU cost is below the two hashes already paid. +Ingest adds one BLAKE3 pass per body or part, alongside the MD5 pass already there. BLAKE3 with vector instructions is faster than SHA-256, so it costs less than the whole-body SHA-256 pass it makes redundant. -`CompleteMultipartUpload` adds a few hundred compressions in the common path. The re-hash path reads the affected parts back through the plaintext opener, which decrypts in an encrypting deployment, and is bounded by the size of the mis-guessed parts. +`CompleteMultipartUpload` adds one compression per recorded subtree for the digest and one per leaf for the consistency check, a few tens of thousands at most for the largest uploads, which is milliseconds. The re-hash path reads the affected parts back through the plaintext opener, which decrypts in an encrypting deployment, and is bounded by the size of the parts whose assumed offset was wrong, or of the body from an odd-length part to its end. The manifest grows by the leaf list, 8 KiB at 1 GiB and at most 1 MiB, plus the 34-byte multihash and a few bytes of framing. The part record grows by at most about 15 KiB for the life of the upload. From 224c48646bd2f73e60ae7e69f9a2786d492e6eda Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Wed, 7 Oct 2026 14:54:15 +0100 Subject: [PATCH 07/19] refactor: address review comments --- rfcs/2026-10-blake3-object-digest.md | 94 +++++++++++++++++----------- 1 file changed, 57 insertions(+), 37 deletions(-) diff --git a/rfcs/2026-10-blake3-object-digest.md b/rfcs/2026-10-blake3-object-digest.md index 3f2bf0d..974125f 100644 --- a/rfcs/2026-10-blake3-object-digest.md +++ b/rfcs/2026-10-blake3-object-digest.md @@ -8,7 +8,11 @@ Status: Experimental ## Motivation -An Ingot object has no digest a client can verify against. A single `PUT` records a whole-object SHA-256 in the manifest, but it is internal and never returned. A multipart object records no whole-object digest at all, because SHA-256 cannot be assembled from the parts and re-reading every part at `CompleteMultipartUpload` is too slow. A client that wants to check what it downloaded has the S3 ETag, which for a multipart object is a hash of part MD5s and verifies nothing about the bytes, and the optional S3 checksums. Of those, the SHA family are composite values for a multipart object, a checksum of the parts' checksums with the same limitation. The CRC family (CRC32, CRC32C, CRC64NVME) can be a full-object value, which Ingot derives at `CompleteMultipartUpload` by combining the parts' CRCs, but a CRC detects accidental corruption and nothing more: it is not a cryptographic digest, it does not identify the object, and it offers no range verification. Nothing lets a client verify a byte range. +What a client can verify about an Ingot object today depends on how the object was uploaded, and for most objects it is nothing. A single `PUT` records a whole-object SHA-256 in the manifest, but it is internal and never returned. A multipart object records no whole-object digest at all, because SHA-256 cannot be assembled from the parts and re-reading every part at `CompleteMultipartUpload` is too slow. + +The one verifiable value is the per-part checksum list, and it exists only if the uploader asked for it. A multipart object uploaded with a SHA checksum algorithm, `--checksum-algorithm SHA256` or its SDK equivalent, has a checksum per part and a composite over them, and `GetObjectAttributes` returns the list after the upload completes. That is a two-level tree the client can check part-aligned ranges against, and check against the composite. No client does this by default: the AWS SDKs and CLI now send a checksum unprompted, but it is a CRC, which gives a full-object value with no per-part list; rclone, s5cmd and the MinIO client send none. The list covers nothing else: not a single `PUT`, not an upload made without a SHA algorithm, and no range finer than the uploader's parts, whose size the reader did not choose. The part ETags the object ETag is built from are not returned once the upload completes, and a per-part MD5 is available only when the uploader chose MD5 as the checksum algorithm, an opt-in like SHA-256; the ETag itself, an MD5 or an MD5 of part MD5s, identifies the object loosely and gives a reader no cryptographic assurance. The CRC family (CRC32, CRC32C, CRC64NVME) can be a full-object value, which Ingot derives at `CompleteMultipartUpload` by combining the parts' CRCs, but a CRC detects accidental corruption and nothing more. + +Nothing lets a client verify a byte range the uploader did not happen to make a part. This RFC proposes a whole-object BLAKE3 digest for every object, single `PUT` or multipart, returned to clients as a CID, plus a small tree of chaining values stored in the manifest that lets a client verify ranged reads. BLAKE3 is a Merkle tree, so the digest of a multipart object is assembled from per-part subtrees at `Complete` with no re-read of the data, and the same tree structure gives range verification for free. @@ -17,11 +21,11 @@ Blob addressing on the Forge network stays SHA-256. This RFC adds an object-leve ## Goals - Every object has a whole-object BLAKE3 digest, computed in the same streaming pass that writes its blobs. -- A multipart object's digest is assembled at `Complete` from values recorded per part. Re-reading part data is a fallback for unusual clients, never the common path. +- A multipart object's digest is assembled at `Complete` from values recorded per part. Re-reading part data is a bounded fallback for the few parts whose position was assumed wrongly, never the common path, and an upload that would need more than the bound gets no digest rather than a slow `Complete`. - A client can verify a whole-object read with the digest alone and a standard BLAKE3 implementation. - A client can verify a ranged read with an off-the-shelf Bao library by fetching the object's outboard once: under a KiB for a small object, growing with the square root of the size. - The S3 data path is unchanged. Bodies are raw bytes, ETags and S3 checksums behave as they do today, and a client that ignores the extension sees nothing new. -- Manifest growth stays small for ordinary objects and under 1 MiB for the largest. The leaf list grows with the square root of the object size and is capped. +- Manifest growth stays small for ordinary objects and under 1 MiB for the largest. The block list grows with the square root of the object size and is capped. ## Background: the BLAKE3 tree @@ -51,21 +55,24 @@ A client verifying a whole-object read hashes the body with any BLAKE3 implement ### Single `PUT` -The body already streams through a SHA-256 hasher and an MD5 hasher as it is split into blobs. A BLAKE3 hasher joins the same pass. It produces the object digest and the leaf CVs described under [Range verification](#range-verification). Nothing is read twice. +The body already streams through a SHA-256 hasher and an MD5 hasher as it is split into blobs. A BLAKE3 hasher joins the same pass. It produces the object digest and the block CVs described under [Range verification](#range-verification). Nothing is read twice. ### Multipart upload -Parts arrive in any order and the server learns a part's true byte offset only once every lower-numbered part exists. Each `UploadPart` therefore hashes its body at an assumed offset and records what `CompleteMultipartUpload` needs to finish the tree. `CompleteMultipartUpload` checks each assumed offset against the true one and re-hashes only the parts whose assumption was wrong. +Parts arrive in any order and the server learns a part's true byte offset only once every lower-numbered part exists. Each `UploadPart` therefore hashes its body at an assumed offset and records what `CompleteMultipartUpload` needs to finish the tree. `CompleteMultipartUpload` checks each assumed offset against the true one and re-hashes only the parts whose assumption was wrong, within a budget. #### What `UploadPart` records Each part record gains: - `TreeOffset`: the byte offset the part was hashed at. -- `TreeNodes`: the CVs of the part's aligned subtrees, in order. A part of any length starting on a chunk boundary decomposes into at most two aligned power-of-two blocks per tree level, so this list holds a few dozen 32-byte entries at most. -- `TreeLeaves`: the part's leaf CVs at the part's own block size (see [Range verification](#range-verification)), a few hundred entries at most for a 5 GiB part. +- `TreeNodes`: the CVs of the part's aligned subtrees, in order, covering the whole chunks between the part's first and last chunk boundaries. A span of whole chunks decomposes into at most two aligned power-of-two blocks per tree level, so this list holds a few dozen 32-byte entries at most. +- `TreeBlocks`: the part's block CVs at the part's own block size (see [Range verification](#range-verification)), a few hundred entries at most for a 5 GiB part. +- `TreeHead` and `TreeTail`: the part's bytes before its first chunk boundary and after its last one, under 1 KiB each. They are empty for a part whose offset and length are 1 KiB multiples. Every mainstream client sizes its parts in MiB, so both spans are empty for nearly every part ever uploaded. The one known exception is the AWS SDKs for Go and Java above the 10,000-part limit, objects over about 48 GiB, where they size parts as the object size divided by 10,000, an arbitrary byte count; those uploads are a small minority, and these two spans are what let them complete without re-reading anything. - For part 1 only, `TreeRoot`: the BLAKE3 hash of the part on its own. Part 1 is always at offset 0, so its standalone hash falls out of the same pass with a root-flagged final compression. +A part cannot know whether it is the last, so it never treats a short final chunk as the object's end; those bytes go to `TreeTail`, and `CompleteMultipartUpload` compresses them once it knows what follows them. + A part that supersedes an earlier upload of the same part number replaces its record, as today. #### Assuming the offset @@ -75,33 +82,44 @@ A part that supersedes an earlier upload of the same part number replaces its re - Part _n_ when some lower part is recorded: assume a uniform part size and use (_n_ − 1) times the size of the lowest-numbered recorded part. - Part _n_ when no lower part is recorded: (_n_ − 1) times this part's own size. -Clients upload parts of one size in parallel, with a shorter final part. The rules above give the true offset for every part of a uniform-size upload except a final part that lands before any of its predecessors, which is re-hashed at `Complete` and is the smallest part of the upload. +Clients upload parts of one size in parallel, with a shorter final part. The rules above give the true offset for every part of a uniform-size upload, whatever that size is, except a final part that lands before any of its predecessors, which is re-hashed at `Complete` and is the smallest part of the upload. An assumed offset need not be a 1 KiB multiple: the part's first chunk boundary is wherever the offset puts it. A part created by copy is hashed as its bytes are read, like an uploaded part. Where the copy does not read the bytes, the part records no tree and takes the re-hash path. #### What `CompleteMultipartUpload` does -`CompleteMultipartUpload` already walks the requested parts in order and computes each part's true offset. A part's record is usable when its true offset equals `TreeOffset`. A part whose assumed offset was wrong is re-hashed at its true offset by reading its blobs back, and the result stands in for its record before the merge proceeds. +`CompleteMultipartUpload` already walks the requested parts in order and computes each part's true offset. A part's record is usable when its true offset equals `TreeOffset`. -A part whose length is not a multiple of 1 KiB, other than the last, puts a chunk boundary inside the next part. No record from that part onward can serve: its own tree treated its short final chunk as the object's end, and every later part was hashed at a chunk index its bytes do not start on. Re-hashing those parts one at a time would not help either, since the chunk straddling the boundary has bytes in both. The body from the first such part to its end is therefore re-hashed as one range, from an aligned offset, and that range replaces all of their records. This is rare: clients size parts in MiB. +The parts whose assumed offset was wrong are re-hashed at their true offsets by reading their blobs back from the local spool, and the results stand in for their records. The final part is always re-hashed when wrong: in a uniform-size upload it is the only part whose assumption can fail, when it lands before any predecessor and its own size stands in, and the 5 GiB part maximum bounds the cost. The other parts are re-hashed within a **re-hash budget**, one maximum part, 5 GiB, by default, and configurable. An upload whose wrong non-final parts add up to more than the budget commits without a digest: the object's ETag is returned as always, GET and HEAD carry no `x-cid`, and `GetObjectAttributes` omits the `Blake3` element, the state defined below for an object without a digest. The budget keeps `Complete` to seconds of local reading at most. Exceeding it is an extreme edge case. No mainstream client varies its part size within an upload, the final part aside, and a uniform-size upload assumes every offset exactly in any arrival order. A sequential upload assumes every offset exactly whatever the sizes. What is left is a hand-rolled uploader sending variable-size parts in parallel, or a parallel compose-by-copy of differently sized source objects, and even there the cost is the absence of a digest, never a failed upload. -With every range at its correct offset, the object digest is computed by merging adjacent aligned subtrees into parents and applying the root flag at the final merge. This is one compression per recorded subtree and reads no data. The work scales with the number of parts: a part contributes at most two subtrees per tree level, so an upload of 10,000 parts merges a few tens of thousands of nodes, each one BLAKE3 compression, which is milliseconds. +A chunk that straddles a part boundary is rebuilt from the earlier part's `TreeTail` and the later part's `TreeHead` and compressed at its chunk index, one compression per boundary. The last part's `TreeTail` is the object's final chunk and is compressed as such. Part lengths that are not 1 KiB multiples therefore cost nothing. + +With every part's nodes at their true offsets, the object digest is computed by merging adjacent aligned subtrees into parents and applying the root flag at the final merge. This is one compression per recorded subtree and reads no data. The work scales with the number of parts: a part contributes at most two subtrees per tree level, so an upload of 10,000 parts merges a few tens of thousands of nodes, each one BLAKE3 compression, which is milliseconds. An upload completed with a single part has no merge, and its digest is that part's root as a body of its own. For part 1 that is the recorded `TreeRoot`. For any other part the recorded offset is not 0, so the offset check above re-hashes it at offset 0, and that re-hash yields the root directly. S3 requires the completed parts to be in ascending order, not to begin at part 1, so this case is legal and does occur. -The leaf CVs for the object are built from the parts' `TreeLeaves` and `TreeNodes` as described in the next section. The root computed by merging the leaf CVs up must equal the root computed from the parts' subtrees. This is a consistency check on the assembly at the cost of one compression per leaf, 32,767 at the cap; should it ever fail, the whole body is re-hashed, so the result is correct even if a record is bad. +The block CVs for the object are built from the parts' `TreeBlocks` and `TreeNodes` as described in the next section. The root computed by merging the block CVs up must equal the root computed from the parts' subtrees. This is a consistency check on the assembly at the cost of one compression per block, 32,767 at the cap. Should it ever fail, the object commits without a digest rather than with a wrong one; nothing re-reads the body. ### Range verification +Two terms, since BLAKE3 and Bao both have "leaves" and they are not the same thing. A **chunk** is BLAKE3's 1 KiB unit; a chunk's CV is a leaf of BLAKE3's own tree. A **block** is 2^ChunkLog chunks, the unit a client verifies; a block's CV is an interior node of BLAKE3's tree and a leaf of the Bao tree at that block size. This RFC stores and serves block CVs and never stores chunk CVs. The word "leaf" is not used below. + A client that reads ranges needs more than the root. It needs the CVs of the tree nodes covering the bytes it read, and enough of the rest of the tree to connect them to the root. The full Bao outboard provides this at 1 KiB granularity and costs about 6% of the object size. Storing it is out of the question in a manifest, and even at bao-tree's 16 KiB default it costs 4 MiB per GiB. -This RFC scales the leaf size with the object. The block is the geometric mean of the object size and 16 KiB, so the number of leaves grows with the square root of the size. A small object keeps a leaf list of a few hundred bytes, and the smallest verifiable read on a large object stays a small fraction of it. +This RFC scales the block size with the object. The block is the geometric mean of the object size and 16 KiB, so the number of blocks grows with the square root of the size. A small object keeps a block list of a few hundred bytes, and the smallest verifiable read on a large object stays a small fraction of it. #### The block -The **block** is the leaf size of the stored tree: a power of two, at least 16 KiB, and the smallest such value that is at least the square root of the object size times 16 KiB. The block doubles each time the object quadruples. A hard cap of 32,768 leaves, 1 MiB of CVs, applies beyond about 32 TB, where the block grows with the size instead. The manifest records the block size as a **chunk log**: a base-2 exponent of 1 KiB BLAKE3 chunks, the unit Bao libraries take as the block size, so 16 KiB is 4 and 4 MiB is 12. The block affects only what is stored and what granularity a client can verify at. The object digest is the same whatever block is chosen. +The **block** is the unit of the stored tree. Its target is the geometric mean of the object size and 16 KiB, the square root of their product: -| Object size | Block | Chunk log | Leaves | Manifest bytes | +``` +target = sqrt(size × 16 KiB) +block = the smallest power of two ≥ target, and at least 16 KiB +``` + +For a 1 MiB object the target is sqrt(1 MiB × 16 KiB) = sqrt(2^34) = 128 KiB, so the block is 128 KiB and there are 8 blocks; for a 1 GiB object it is sqrt(2^44) = 4 MiB, 256 blocks. The block doubles each time the object quadruples. A hard cap of 32,768 blocks, 1 MiB of CVs, applies beyond about 32 TB, where the block grows with the size instead. The manifest records the block size as a **chunk log**: a base-2 exponent of 1 KiB BLAKE3 chunks, the unit Bao libraries take as the block size, so 16 KiB is 4 and 4 MiB is 12. The block affects only what is stored and what granularity a client can verify at. The object digest is the same whatever block is chosen. + +| Object size | Block | Chunk log | Blocks | Manifest bytes | |---|---|---|---|---| | 1 KiB | 16 KiB | 4 | 1 | 32 B | | 64 KiB | 32 KiB | 5 | 2 | 64 B | @@ -115,22 +133,22 @@ The **block** is the leaf size of the stored tree: a power of two, at least 16 K | 5 TiB | 512 MiB | 19 | 10,240 | 320 KiB | | 50 TB | 2 GiB | 21 | 23,284 | 728 KiB | -Sizes with binary prefixes are powers of two; the 50 TB row is decimal, 50 × 10¹² bytes, the largest object this RFC accounts for. Leaves is the size divided by the block, rounded up, since a final partial block still has a CV. An object of 16 KiB or less is one leaf. From there the manifest cost grows with the square root of the size: a quarter of a KiB at 1 MiB, 8 KiB at 1 GiB, 256 KiB at 1 TiB. The cap applies past about 32 TB and holds a 50 TB object to 728 KiB. +Sizes with binary prefixes are powers of two; the 50 TB row is decimal, 50 × 10¹² bytes, just under S3's current maximum object size of 48.8 TiB (10,000 parts of 5 GiB). Blocks is the size divided by the block size, rounded up, since a final partial block still has a CV. An object of 16 KiB or less is one block. From there the manifest cost grows with the square root of the size: a quarter of a KiB at 1 MiB, 8 KiB at 1 GiB, 256 KiB at 1 TiB. The cap applies past about 32 TB and holds a 50 TB object to 728 KiB. -The smallest range a client can verify independently is one block: 256 KiB on a 4 MiB object, 4 MiB on a 1 GiB object, 128 MiB on a 1 TiB object, 2 GiB on a 50 TB object. Two constants tune the rule. The 16 KiB floor sets where the scaling starts, and the cap bounds the manifest. An object needing finer verification than its block would want the outboard-as-blob design under [Alternatives](#alternatives-considered). +The smallest range a client can verify independently is one block: 256 KiB on a 4 MiB object, 4 MiB on a 1 GiB object, 128 MiB on a 1 TiB object, 2 GiB on a 50 TB object. For a large object that is coarser than the per-part checksum list of a multipart upload made with SHA checksums: the block passes an 8 MiB part above 4 GiB, so on a 1 TiB object such a list verifies at 8 MiB where this tree verifies at 128 MiB. The tree's advantage is that it exists for every object, whatever the uploader did. Two constants tune the rule. The 16 KiB floor sets where the scaling starts, and the cap bounds the manifest. An object needing finer verification than its block would want the outboard-as-blob design under [Alternatives](#alternatives-considered). -#### Leaf CVs +#### Block CVs -The manifest stores the CV of every block of the object, in order: the **leaf CVs**. These are the nodes of the BLAKE3 tree at the block level, each 32 bytes. Every node above them is derivable with one compression per node, so nothing above the leaves is stored. +The manifest stores the CV of every block of the object, in object order from offset 0: the **block CVs**, 32 bytes each. They are the nodes of BLAKE3's tree at the block level. Every node above them is derivable with one compression per node, so nothing above the blocks is stored, and nothing below them: chunk CVs are never stored. -A single `PUT` computes the leaf CVs in its hashing pass, since the object length is known from `Content-Length` and so is the block. +A single `PUT` computes the block CVs in its hashing pass, since the object length is known from `Content-Length` and so is the block. -A multipart upload does not know the object length until `CompleteMultipartUpload`, so each part computes `TreeLeaves` at its own block size, chosen by the same rule from the part's own size. The object's block is never smaller than any part's, because the rule is monotone in size and the object is at least as large as each part. At `CompleteMultipartUpload`, each object leaf is one of: +A multipart upload does not know the object length until `CompleteMultipartUpload`, so each part computes `TreeBlocks` at its own block size, chosen by the same rule from the part's own size. The object's block is never smaller than any part's, because the rule is monotone in size and the object is at least as large as each part. At `CompleteMultipartUpload`, each object block is one of: -- A block lying within one part. Its CV is the merge of that part's `TreeLeaves` that it covers. +- A block lying within one part. Its CV is the merge of that part's `TreeBlocks` that it covers. - A block straddling a part boundary. The piece on each side is a union of that part's edge subtrees, which `TreeNodes` holds, so its CV is the merge of those entries. -Either way the leaf CVs are built from recorded values with no data read. +Either way the block CVs are built from recorded values with no data read. #### The tree in `GetObjectAttributes` @@ -154,7 +172,7 @@ x-amz-object-attributes: ETag,Blake3 - `CID` is the object digest as returned in `x-cid` - `ChunkLog` is the block size as a base-2 exponent of 1 KiB BLAKE3 chunks, the value a Bao library takes as its block size unchanged. iroh's fixed block is 4; the original Bao format is 0. -- `Outboard` is the standard pre-order Bao outboard at that block size, base64-encoded: the object size as 8 little-endian bytes, then the chaining-value pair of every parent node above the leaves, root first and left subtree before right. It is built from the stored leaves on request, one compression per parent, and is byte-identical to what the Bao libraries produce for the same object and block size. It is an 8-byte prefix plus 64 bytes per parent, one fewer than the leaves: under 100 bytes for a small object, 16 KiB at 1 GiB, 1.4 MiB for a 50 TB object and 2 MiB at the 32,768-leaf cap, about 2.7 MiB after base64. A tree belongs to one version. An overwrite can never pair an old tree with new bytes, because both live in the same manifest, and the response carries the ETag of the version the tree describes for the client's `If-Match`. +- `Outboard` is the standard pre-order Bao outboard at that block size, base64-encoded: the object size as 8 little-endian bytes, then the chaining-value pair of every parent node above the blocks, root first and left subtree before right. It is built from the stored block CVs on request, one compression per parent, and is byte-identical to what the Bao libraries produce for the same object and block size. It is an 8-byte prefix plus 64 bytes per parent, one fewer than the blocks: under 100 bytes for a small object, 16 KiB at 1 GiB, 1.4 MiB for a 50 TB object and 2 MiB at the 32,768-block cap, about 2.7 MiB after base64. A tree belongs to one version. An overwrite can never pair an old tree with new bytes, because both live in the same manifest, and the response carries the ETag of the version the tree describes for the client's `If-Match`. AWS rejects an attribute name it does not know with `InvalidArgument`, so accepting `Blake3` is a visible divergence. SDKs whose typed response omits unknown elements cannot read the tree, but the same SDKs could not call a custom endpoint either, and a raw HTTP client parses this XML with what it already has for every other S3 response. @@ -176,23 +194,23 @@ type Body struct { // BLAKE3 is the whole-object blake3 multihash (34 bytes). BLAKE3 []byte `cborgen:"b3"` - // TreeChunkLog is the block size of TreeLeaves as a base-2 exponent of chunks (one byte). + // TreeChunkLog is the block size of TreeBlocks as a base-2 exponent of chunks (one byte). TreeChunkLog uint8 `cborgen:"tc"` - // TreeLeaves holds the chaining values of the blocks of the + // TreeBlocks holds the chaining values of the blocks of the // body, 32 bytes each, in order (8 KiB at 1 GiB, at most 1 MiB). - TreeLeaves []byte `cborgen:"tl"` + TreeBlocks []byte `cborgen:"tb"` } ``` -Objects written before this change have none of these fields. A read of such an object returns no `x-cid` header and `GetObjectAttributes` omits the `Blake3` element, so a client can tell the difference between an object without a digest and a server without the feature. +Objects written before this change have none of these fields, and neither does a multipart object whose completion exceeded the re-hash budget. A read of such an object returns no `x-cid` header and `GetObjectAttributes` omits the `Blake3` element. A client can tell an object without a digest from a server without the feature by asking for the `Blake3` attribute by name: a server without the feature rejects the name with `InvalidArgument`, as AWS does, while a server with it returns the response without the element. ### Cost Ingest adds one BLAKE3 pass per body or part, alongside the MD5 pass already there. BLAKE3 with vector instructions is faster than SHA-256, so it costs less than the whole-body SHA-256 pass it makes redundant. -`CompleteMultipartUpload` adds one compression per recorded subtree for the digest and one per leaf for the consistency check, a few tens of thousands at most for the largest uploads, which is milliseconds. The re-hash path reads the affected parts back through the plaintext opener, which decrypts in an encrypting deployment, and is bounded by the size of the parts whose assumed offset was wrong, or of the body from an odd-length part to its end. +`CompleteMultipartUpload` adds one compression per recorded subtree for the digest and one per block for the consistency check, a few tens of thousands at most for the largest uploads, which is milliseconds. The re-hash path reads the affected parts back through the plaintext opener, which decrypts in an encrypting deployment, and is bounded by the budget, one maximum part by default. The price of the bound is that a variable-size parallel upload beyond it gets no digest. -The manifest grows by the leaf list, 8 KiB at 1 GiB and at most 1 MiB, plus the 34-byte multihash and a few bytes of framing. The part record grows by at most about 15 KiB for the life of the upload. +The manifest grows by the block list, 8 KiB at 1 GiB and at most 1 MiB, plus the 34-byte multihash and a few bytes of framing. The part record grows by at most about 15 KiB for the life of the upload. ## Alternatives considered @@ -206,7 +224,7 @@ Making BLAKE3 the network's content address would let storage nodes serve verifi ### A fixed 16 KiB block with the outboard as a blob -bao-tree's default gives 16 KiB verification granularity at any size, with the outboard stored as its own blob referenced from the manifest, about 0.4% of the object. It is the right design if fine-grained verified reads of very large objects are a requirement. It adds a blob per object to write, locate and clean up, and a second request to fetch something that is usually megabytes. The bounded manifest tree serves the common case with no new blob and can coexist with this design later: the stored leaves are the top of the finer tree. +bao-tree's default gives 16 KiB verification granularity at any size, with the outboard stored as its own blob referenced from the manifest, about 0.4% of the object. It is the right design if fine-grained verified reads of very large objects are a requirement. It adds a blob per object to write, locate and clean up, and a second request to fetch something that is usually megabytes. The bounded manifest tree serves the common case with no new blob and can coexist with this design later: the stored block CVs are the top of the finer tree. ### Returning the Bao encoded stream @@ -220,19 +238,21 @@ The `x-amz-checksum-*` headers have a fixed set of algorithms and SDKs validate A `?blake3` subresource returning a CBOR document would keep `GetObjectAttributes` identical to AWS. It costs a new endpoint, a second place that must honour `versionId`, and a body format no S3 client parses today. No SDK can call it, so it has no advantage over an attribute for SDK users, and a raw HTTP client is better served by the XML it already handles. -### Returning the leaf list instead of the outboard +### Returning the block list instead of the outboard -The stored leaves are half the size of the outboard, and a client could merge them to the root itself. But checking a block against its leaf means hashing the block as a non-root subtree at its chunk offset, which standard BLAKE3 APIs do not expose, so every client would carry custom tree code. Serving the Bao outboard costs twice the bytes and lets any Bao library do the verification. +The stored block CVs are half the size of the outboard, and a client could merge them to the root itself. But checking a block against its CV means hashing the block as a non-root subtree at its chunk offset, which standard BLAKE3 APIs do not expose, so every client would carry custom tree code. Serving the Bao outboard costs twice the bytes and lets any Bao library do the verification. -### Sending the leaf CVs in a header +### Sending the block CVs in a header Base64 of a 1 GiB object's 16 KiB outboard is about 22 KiB, over the 8 KiB per-header default of common proxies, larger objects have far more, and splitting it across numbered headers is a hack. The root fits in a header; the tree does not. ## Open questions +- The re-hash budget's default for non-final parts. The final part is exempt, so the budget only governs variable-size parallel uploads; a larger default trades `Complete` latency for digests on more of them. + - Names: `x-cid` and the `Blake3` attribute are placeholders. -- The scaling constants: the 16 KiB floor and the 32,768-leaf cap. The exponent is a knob too; cube-root scaling would give a 50 TB object 1,456 leaves at a 32 GiB block. -- Whether a `GET` with `?partNumber` should also return the leaf CVs covering that part, so a client that already reads by part need not align to blocks. +- The scaling constants: the 16 KiB floor and the 32,768-block cap. The exponent is a knob too; cube-root scaling would give a 50 TB object 1,456 blocks at a 32 GiB block. +- Whether a `GET` with `?partNumber` should also return the block CVs covering that part, so a client that already reads by part need not align to blocks. - Whether the object CID should reach the Forge network, for example in the upload index, or remain an Ingot-level value until blob addressing is revisited. - Whether to expose the digest on `ListObjects` responses. @@ -240,7 +260,7 @@ Base64 of a 1 GiB object's 16 KiB outboard is about 22 KiB, over the 8 KiB per-h - The S3 conformance suite passes unchanged: no ETag, checksum or body behaviour differs. - Ingest throughput with the extra hash is within a few percent of today's. -- The share of `Complete` calls that take the re-hash path, measured against the AWS CLI, boto3, rclone, s5cmd and the MinIO client at their default part sizes, is near zero. +- The share of `Complete` calls that take the re-hash path, measured against the AWS CLI, boto3, the Go SDK's uploader above the 10,000-part threshold, rclone, s5cmd and the MinIO client at their default part sizes, is near zero, and none exceeds the budget. - Manifest size distribution matches the table above. - A reference client verifies whole-object and ranged reads against a running Ingot, including a multipart object with non-power-of-two parts and a mis-ordered final part. From 222970a1ef15e61f2c5ebb3da342f8c7dd37e48a Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Wed, 7 Oct 2026 15:02:49 +0100 Subject: [PATCH 08/19] refactor: address review comments --- rfcs/2026-10-blake3-object-digest.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/rfcs/2026-10-blake3-object-digest.md b/rfcs/2026-10-blake3-object-digest.md index 974125f..237784e 100644 --- a/rfcs/2026-10-blake3-object-digest.md +++ b/rfcs/2026-10-blake3-object-digest.md @@ -202,13 +202,15 @@ type Body struct { } ``` -Objects written before this change have none of these fields, and neither does a multipart object whose completion exceeded the re-hash budget. A read of such an object returns no `x-cid` header and `GetObjectAttributes` omits the `Blake3` element. A client can tell an object without a digest from a server without the feature by asking for the `Blake3` attribute by name: a server without the feature rejects the name with `InvalidArgument`, as AWS does, while a server with it returns the response without the element. +Some objects have no digest: those written before this change, and a multipart object whose completion exceeded the re-hash budget. For them, GET and HEAD return no `x-cid` header and `GetObjectAttributes` returns its response without the `Blake3` element. + +That makes two situations look alike from a GET or HEAD: an object without a digest, and a server that does not implement this RFC. `GetObjectAttributes` tells them apart. The request names the attributes it wants in `x-amz-object-attributes`, and a server validates the names: one that implements this RFC accepts `Blake3` and answers, with or without the element depending on the object, while one that does not, AWS included, rejects the request with `InvalidArgument`. So a client that wants to know whether verification is available at all asks for `Blake3` once; an error means the server cannot provide it for any object, and a success without the element means it cannot provide it for this one. ### Cost Ingest adds one BLAKE3 pass per body or part, alongside the MD5 pass already there. BLAKE3 with vector instructions is faster than SHA-256, so it costs less than the whole-body SHA-256 pass it makes redundant. -`CompleteMultipartUpload` adds one compression per recorded subtree for the digest and one per block for the consistency check, a few tens of thousands at most for the largest uploads, which is milliseconds. The re-hash path reads the affected parts back through the plaintext opener, which decrypts in an encrypting deployment, and is bounded by the budget, one maximum part by default. The price of the bound is that a variable-size parallel upload beyond it gets no digest. +`CompleteMultipartUpload` adds one compression per recorded subtree for the digest and one per block for the consistency check, a few tens of thousands at most for the largest uploads, which is milliseconds. The re-hash path reads the affected parts back through the plaintext opener, which decrypts in an encrypting deployment. The parts are read from the local spool: a part's blobs stay there until the object is deleted, since nothing evicts spool copies today, so no re-hash fetches from a storage node. The work is bounded by the final part plus the budget, about 10 GiB of local reading at most with the default, which is seconds to tens of seconds. The price of the bound is that a variable-size parallel upload beyond it gets no digest. The manifest grows by the block list, 8 KiB at 1 GiB and at most 1 MiB, plus the 34-byte multihash and a few bytes of framing. The part record grows by at most about 15 KiB for the life of the upload. From 4d3c54942ae14dd651906379de267bcb010de622 Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Wed, 7 Oct 2026 15:07:54 +0100 Subject: [PATCH 09/19] refactor: add alternative --- rfcs/2026-10-blake3-object-digest.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/rfcs/2026-10-blake3-object-digest.md b/rfcs/2026-10-blake3-object-digest.md index 237784e..b23c14d 100644 --- a/rfcs/2026-10-blake3-object-digest.md +++ b/rfcs/2026-10-blake3-object-digest.md @@ -220,6 +220,18 @@ The manifest grows by the block list, 8 KiB at 1 GiB and at most 1 MiB, plus the Keeping the existing SHA-256 and computing it for multipart objects means re-reading every part at `Complete`. For a 5 GiB object that is a 5 GiB read, decrypting where encryption is on, inside a request a client expects to return promptly. It also offers no range verification. BLAKE3 removes the re-read and adds range verification with the same stored bytes. +### A root SHA-256 over per-part SHA-256 hashes + +S3 already defines this for uploads made with `--checksum-algorithm SHA256`: a SHA-256 per part and a composite over the list, which `GetObjectAttributes` returns. The alternative is to compute that for every multipart upload, server side, whatever the client asked for, and treat the composite as the object's digest. It needs no new primitive, and for a client that already understands S3 composite checksums it is familiar. It falls short of this RFC in several ways. + +- **The root is not a hash of the bytes.** A composite depends on where the part boundaries fell, so the same bytes uploaded with 8 MiB parts, with 16 MiB parts, or as a single `PUT` get three different roots. It identifies an upload, not an object: two copies of the same data cannot be recognised as equal, nothing can be addressed by it, and a single-`PUT` object has a plain SHA-256 while a multipart object has a hash of hashes, so a client needs two code paths and two meanings for "the digest". +- **Verifying a download takes a list, not a value.** The verifier must fetch the per-part hash list and the part sizes, up to 10,000 of each, keep them for the whole download, and apply the right one to the right byte range. The BLAKE3 digest is one 32-byte value that a standard library verifies against a stream. Walking a 10,000-entry list is custom code in every client, and the list itself is hundreds of KiB of XML on every fetch, with no compact form. +- **The granularity is the uploader's, and often too coarse.** Verification happens per part, so a client cannot check anything until it has a whole part, 5 GiB on large uploads, and a single-`PUT` object cannot be range-verified at all. The Bao tree verifies 16 KiB on small objects, and a server-chosen block on large ones, with proofs that are logarithmic in the object rather than linear in the part count. +- **No standard verifies it.** There is no library that takes a composite SHA-256 and a part list and verifies a stream or a range, so every consumer writes its own; the Bao outboard is loadable by existing libraries in Rust and Go. +- **It costs about the same.** A SHA-256 pass per part is no cheaper than the BLAKE3 pass, which is faster per byte, and the composite still has to be assembled at `Complete` from recorded values, as the tree is. + +What it does offer over the current state is universality within multipart uploads: every multipart object would have a verifiable per-part list rather than only those uploaded with SHA checksums. BLAKE3 gives that and the rest. + ### Re-addressing blobs to BLAKE3 Making BLAKE3 the network's content address would let storage nodes serve verified ranges directly. It touches every component that handles a multihash, the proof-tree assumptions of PDP, public gateway resolvability, and every stored byte. This RFC is independent of that decision. The object digest proposed here is the same value such a migration would use, so nothing here is thrown away if it happens. From 178f324da395bbaf3666ee97326fe052fbe9a172 Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Wed, 7 Oct 2026 15:31:39 +0100 Subject: [PATCH 10/19] refactor: address review comments --- rfcs/2026-10-blake3-object-digest.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/rfcs/2026-10-blake3-object-digest.md b/rfcs/2026-10-blake3-object-digest.md index b23c14d..5dddb37 100644 --- a/rfcs/2026-10-blake3-object-digest.md +++ b/rfcs/2026-10-blake3-object-digest.md @@ -8,7 +8,7 @@ Status: Experimental ## Motivation -What a client can verify about an Ingot object today depends on how the object was uploaded, and for most objects it is nothing. A single `PUT` records a whole-object SHA-256 in the manifest, but it is internal and never returned. A multipart object records no whole-object digest at all, because SHA-256 cannot be assembled from the parts and re-reading every part at `CompleteMultipartUpload` is too slow. +What a client can verify about an Ingot object today depends on how the object was uploaded, and for most objects it is nothing. A single `PUT` records a whole-object SHA-256 in the manifest, the per-object metadata record Ingot keeps in the bucket's catalog, but it is internal and never returned. A multipart object records no whole-object digest at all, because SHA-256 cannot be assembled from the parts and re-reading every part at `CompleteMultipartUpload` is too slow. The one verifiable value is the per-part checksum list, and it exists only if the uploader asked for it. A multipart object uploaded with a SHA checksum algorithm, `--checksum-algorithm SHA256` or its SDK equivalent, has a checksum per part and a composite over them, and `GetObjectAttributes` returns the list after the upload completes. That is a two-level tree the client can check part-aligned ranges against, and check against the composite. No client does this by default: the AWS SDKs and CLI now send a checksum unprompted, but it is a CRC, which gives a full-object value with no per-part list; rclone, s5cmd and the MinIO client send none. The list covers nothing else: not a single `PUT`, not an upload made without a SHA algorithm, and no range finer than the uploader's parts, whose size the reader did not choose. The part ETags the object ETag is built from are not returned once the upload completes, and a per-part MD5 is available only when the uploader chose MD5 as the checksum algorithm, an opt-in like SHA-256; the ETag itself, an MD5 or an MD5 of part MD5s, identifies the object loosely and gives a reader no cryptographic assurance. The CRC family (CRC32, CRC32C, CRC64NVME) can be a full-object value, which Ingot derives at `CompleteMultipartUpload` by combining the parts' CRCs, but a CRC detects accidental corruption and nothing more. @@ -117,7 +117,7 @@ target = sqrt(size × 16 KiB) block = the smallest power of two ≥ target, and at least 16 KiB ``` -For a 1 MiB object the target is sqrt(1 MiB × 16 KiB) = sqrt(2^34) = 128 KiB, so the block is 128 KiB and there are 8 blocks; for a 1 GiB object it is sqrt(2^44) = 4 MiB, 256 blocks. The block doubles each time the object quadruples. A hard cap of 32,768 blocks, 1 MiB of CVs, applies beyond about 32 TB, where the block grows with the size instead. The manifest records the block size as a **chunk log**: a base-2 exponent of 1 KiB BLAKE3 chunks, the unit Bao libraries take as the block size, so 16 KiB is 4 and 4 MiB is 12. The block affects only what is stored and what granularity a client can verify at. The object digest is the same whatever block is chosen. +For a 1 MiB object the target is sqrt(1 MiB × 16 KiB) = sqrt(2^34) = 128 KiB, so the block is 128 KiB and there are 8 blocks; for a 1 GiB object it is sqrt(2^44) = 4 MiB, 256 blocks. The block doubles each time the object quadruples. A hard cap of 32,768 blocks, 1 MiB of CVs, applies beyond about 32 TB, where the block grows with the size instead. The cap is a metadata-cost choice: the manifest is a catalog block rewritten on every version, so its size is storage the tenant pays for on each write, and 1 MiB leaves headroom under the 2 MiB byte-field limit of the catalog's CBOR encoding. A larger cap, up to that limit, buys finer verification on multi-TB objects at that cost. The manifest records the block size as a **chunk log**: a base-2 exponent of 1 KiB BLAKE3 chunks, the unit Bao libraries take as the block size, so 16 KiB is 4 and 4 MiB is 12. The block affects only what is stored and what granularity a client can verify at. The object digest is the same whatever block is chosen. | Object size | Block | Chunk log | Blocks | Manifest bytes | |---|---|---|---|---| @@ -178,7 +178,7 @@ AWS rejects an attribute name it does not know with `InvalidArgument`, so accept #### Client procedure -1. Fetch the attribute once with `GetObjectAttributes`, and load the CID's digest, the block size and the outboard into a Bao library (bao-tree in Rust, the Go BLAKE3 library's `bao` package). The library checks the outboard against the root as it goes; no merge code of our own is needed, and no BLAKE3 API beyond what a Bao library already wraps. +1. Fetch the attribute once with `GetObjectAttributes`, and load the CID's digest, the block size and the outboard into a Bao library (bao-tree in Rust, the Go BLAKE3 library's `bao` package). A client that uploaded the object itself can skip this step: the digest and the outboard are functions of the bytes alone, so it can compute both as it uploads (`ingot blake3 hash --bao` does exactly this) and never ask the server. The library checks the outboard against the root as it goes; no merge code of our own is needed, and no BLAKE3 API beyond what a Bao library already wraps. 2. Read ranges with ordinary S3 `Range` requests aligned to block boundaries, sending `If-Match` with the ETag so the bytes cannot change between the attribute fetch and the reads. A Bao library can name the byte ranges needed for the blocks a client wants. 3. Verify each block received against the outboard. An object of one block has an empty outboard, and the client verifies by hashing the whole object against the CID. @@ -238,7 +238,7 @@ Making BLAKE3 the network's content address would let storage nodes serve verifi ### A fixed 16 KiB block with the outboard as a blob -bao-tree's default gives 16 KiB verification granularity at any size, with the outboard stored as its own blob referenced from the manifest, about 0.4% of the object. It is the right design if fine-grained verified reads of very large objects are a requirement. It adds a blob per object to write, locate and clean up, and a second request to fetch something that is usually megabytes. The bounded manifest tree serves the common case with no new blob and can coexist with this design later: the stored block CVs are the top of the finer tree. +bao-tree's default gives 16 KiB verification granularity at any size, with the outboard stored as its own blob referenced from the manifest, about 0.4% of the object. It is the right design if fine-grained verified reads of very large objects are a requirement. It adds a blob per object to write, locate and clean up. The client's extra request is the same either way, but it fetches megabytes instead of KiB. The bounded manifest tree serves the common case with no new blob and can coexist with this design later: the stored block CVs are the top of the finer tree. ### Returning the Bao encoded stream @@ -260,6 +260,10 @@ The stored block CVs are half the size of the outboard, and a client could merge Base64 of a 1 GiB object's 16 KiB outboard is about 22 KiB, over the 8 KiB per-header default of common proxies, larger objects have far more, and splitting it across numbered headers is a hack. The root fits in a header; the tree does not. +## Dependencies + +The Go implementation uses `lukechampine.com/blake3` for the hash, its `guts` package for the tree primitives the hasher needs (chunk and parent compression at a given chunk index, which the standard BLAKE3 API does not expose), and its `bao` package in tests to check the outboard against an independent encoder. The library has two maintainers and releases infrequently. The exposure is small: the `bao` package is about 300 lines and the primitives a few hundred more, with no further dependencies, so a fork or a vendored copy is the fallback if a fix is ever needed faster than upstream provides it. The other maintained Go implementation, `zeebo/blake3`, exposes no tree internals and could replace only the plain hashing. + ## Open questions - The re-hash budget's default for non-final parts. The final part is exempt, so the budget only governs variable-size parallel uploads; a larger default trades `Complete` latency for digests on more of them. From 8c1e8ac3a8491251301238427afa22515e6378df Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Wed, 7 Oct 2026 16:21:32 +0100 Subject: [PATCH 11/19] refactor: tweak min verifiable block size --- rfcs/2026-10-blake3-object-digest.md | 42 +++++++++++++++------------- 1 file changed, 22 insertions(+), 20 deletions(-) diff --git a/rfcs/2026-10-blake3-object-digest.md b/rfcs/2026-10-blake3-object-digest.md index 5dddb37..88e4a43 100644 --- a/rfcs/2026-10-blake3-object-digest.md +++ b/rfcs/2026-10-blake3-object-digest.md @@ -67,7 +67,7 @@ Each part record gains: - `TreeOffset`: the byte offset the part was hashed at. - `TreeNodes`: the CVs of the part's aligned subtrees, in order, covering the whole chunks between the part's first and last chunk boundaries. A span of whole chunks decomposes into at most two aligned power-of-two blocks per tree level, so this list holds a few dozen 32-byte entries at most. -- `TreeBlocks`: the part's block CVs at the part's own block size (see [Range verification](#range-verification)), a few hundred entries at most for a 5 GiB part. +- `TreeBlocks`: the part's block CVs at the part's own block size (see [Range verification](#range-verification)), at most 10,240 entries (320 KiB) for a 5 GiB part. - `TreeHead` and `TreeTail`: the part's bytes before its first chunk boundary and after its last one, under 1 KiB each. They are empty for a part whose offset and length are 1 KiB multiples. Every mainstream client sizes its parts in MiB, so both spans are empty for nearly every part ever uploaded. The one known exception is the AWS SDKs for Go and Java above the 10,000-part limit, objects over about 48 GiB, where they size parts as the object size divided by 10,000, an arbitrary byte count; those uploads are a small minority, and these two spans are what let them complete without re-reading anything. - For part 1 only, `TreeRoot`: the BLAKE3 hash of the part on its own. Part 1 is always at offset 0, so its standalone hash falls out of the same pass with a root-flagged final compression. @@ -106,36 +106,38 @@ Two terms, since BLAKE3 and Bao both have "leaves" and they are not the same thi A client that reads ranges needs more than the root. It needs the CVs of the tree nodes covering the bytes it read, and enough of the rest of the tree to connect them to the root. The full Bao outboard provides this at 1 KiB granularity and costs about 6% of the object size. Storing it is out of the question in a manifest, and even at bao-tree's 16 KiB default it costs 4 MiB per GiB. -This RFC scales the block size with the object. The block is the geometric mean of the object size and 16 KiB, so the number of blocks grows with the square root of the size. A small object keeps a block list of a few hundred bytes, and the smallest verifiable read on a large object stays a small fraction of it. +This RFC scales the block size with the object. Objects up to 8 MiB are recorded at 16 KiB blocks, bao-tree's own granularity, and above that the block grows with the square root of the size. The block list of a small object is a few KiB, and the smallest verifiable read on a large object stays a small fraction of it. #### The block -The **block** is the unit of the stored tree. Its target is the geometric mean of the object size and 16 KiB, the square root of their product: +The **block** is the unit of the stored tree. Its target is the square root of the object size times 32 bytes, the size of one CV: ``` -target = sqrt(size × 16 KiB) +target = sqrt(size × 32 B) block = the smallest power of two ≥ target, and at least 16 KiB ``` -For a 1 MiB object the target is sqrt(1 MiB × 16 KiB) = sqrt(2^34) = 128 KiB, so the block is 128 KiB and there are 8 blocks; for a 1 GiB object it is sqrt(2^44) = 4 MiB, 256 blocks. The block doubles each time the object quadruples. A hard cap of 32,768 blocks, 1 MiB of CVs, applies beyond about 32 TB, where the block grows with the size instead. The cap is a metadata-cost choice: the manifest is a catalog block rewritten on every version, so its size is storage the tenant pays for on each write, and 1 MiB leaves headroom under the 2 MiB byte-field limit of the catalog's CBOR encoding. A larger cap, up to that limit, buys finer verification on multi-TB objects at that cost. The manifest records the block size as a **chunk log**: a base-2 exponent of 1 KiB BLAKE3 chunks, the unit Bao libraries take as the block size, so 16 KiB is 4 and 4 MiB is 12. The block affects only what is stored and what granularity a client can verify at. The object digest is the same whatever block is chosen. +The target reaches the 16 KiB floor at 8 MiB, so every object up to 8 MiB has 16 KiB blocks, 512 of them at 8 MiB. Above that the block doubles each time the object quadruples: for a 1 GiB object the target is sqrt(2^30 × 2^5) = sqrt(2^35), so the block is 256 KiB and there are 4,096 blocks. A hard cap of 32,768 blocks, 1 MiB of CVs, binds from 32 GiB, where the block grows with the size instead. The cap is a metadata-cost choice: the manifest is a catalog block rewritten on every version, so its size is storage the tenant pays for on each write, and 1 MiB leaves headroom under the 2 MiB byte-field limit of the catalog's CBOR encoding. A larger cap, up to that limit, buys finer verification on objects over 32 GiB at that cost. The manifest records the block size as a **chunk log**: a base-2 exponent of 1 KiB BLAKE3 chunks, the unit Bao libraries take as the block size, so 16 KiB is 4 and 1 MiB is 10. The block affects only what is stored and what granularity a client can verify at. The object digest is the same whatever block is chosen. | Object size | Block | Chunk log | Blocks | Manifest bytes | |---|---|---|---|---| | 1 KiB | 16 KiB | 4 | 1 | 32 B | -| 64 KiB | 32 KiB | 5 | 2 | 64 B | -| 1 MiB | 128 KiB | 7 | 8 | 256 B | -| 4 MiB | 256 KiB | 8 | 16 | 512 B | -| 100 MiB | 2 MiB | 11 | 50 | 1.6 KiB | -| 1 GiB | 4 MiB | 12 | 256 | 8 KiB | -| 10 GiB | 16 MiB | 14 | 640 | 20 KiB | -| 100 GiB | 64 MiB | 16 | 1,600 | 50 KiB | -| 1 TiB | 128 MiB | 17 | 8,192 | 256 KiB | -| 5 TiB | 512 MiB | 19 | 10,240 | 320 KiB | +| 64 KiB | 16 KiB | 4 | 4 | 128 B | +| 1 MiB | 16 KiB | 4 | 64 | 2 KiB | +| 8 MiB | 16 KiB | 4 | 512 | 16 KiB | +| 16 MiB | 32 KiB | 5 | 512 | 16 KiB | +| 100 MiB | 64 KiB | 6 | 1,600 | 50 KiB | +| 1 GiB | 256 KiB | 8 | 4,096 | 128 KiB | +| 10 GiB | 1 MiB | 10 | 10,240 | 320 KiB | +| 32 GiB | 1 MiB | 10 | 32,768 | 1 MiB | +| 100 GiB | 4 MiB | 12 | 25,600 | 800 KiB | +| 1 TiB | 32 MiB | 15 | 32,768 | 1 MiB | +| 5 TiB | 256 MiB | 18 | 20,480 | 640 KiB | | 50 TB | 2 GiB | 21 | 23,284 | 728 KiB | -Sizes with binary prefixes are powers of two; the 50 TB row is decimal, 50 × 10¹² bytes, just under S3's current maximum object size of 48.8 TiB (10,000 parts of 5 GiB). Blocks is the size divided by the block size, rounded up, since a final partial block still has a CV. An object of 16 KiB or less is one block. From there the manifest cost grows with the square root of the size: a quarter of a KiB at 1 MiB, 8 KiB at 1 GiB, 256 KiB at 1 TiB. The cap applies past about 32 TB and holds a 50 TB object to 728 KiB. +Sizes with binary prefixes are powers of two; the 50 TB row is decimal, 50 × 10¹² bytes, just under S3's current maximum object size of 48.8 TiB (10,000 parts of 5 GiB). Blocks is the size divided by the block size, rounded up, since a final partial block still has a CV. An object of 16 KiB or less is one block. Up to 8 MiB the manifest cost is 32 bytes per 16 KiB, 0.2% of the object. From there it grows with the square root of the size: 50 KiB at 100 MiB, 128 KiB at 1 GiB, 320 KiB at 10 GiB. The cap binds from 32 GiB and holds every larger object to 1 MiB or less, 728 KiB for a 50 TB object. -The smallest range a client can verify independently is one block: 256 KiB on a 4 MiB object, 4 MiB on a 1 GiB object, 128 MiB on a 1 TiB object, 2 GiB on a 50 TB object. For a large object that is coarser than the per-part checksum list of a multipart upload made with SHA checksums: the block passes an 8 MiB part above 4 GiB, so on a 1 TiB object such a list verifies at 8 MiB where this tree verifies at 128 MiB. The tree's advantage is that it exists for every object, whatever the uploader did. Two constants tune the rule. The 16 KiB floor sets where the scaling starts, and the cap bounds the manifest. An object needing finer verification than its block would want the outboard-as-blob design under [Alternatives](#alternatives-considered). +The smallest range a client can verify independently is one block: 16 KiB on anything up to 8 MiB, 256 KiB on a 1 GiB object, 32 MiB on a 1 TiB object, 2 GiB on a 50 TB object. For a large object that is coarser than the per-part checksum list of a multipart upload made with SHA checksums: the block passes an 8 MiB part above 256 GiB, so on a 1 TiB object such a list verifies at 8 MiB where this tree verifies at 32 MiB. The tree's advantage is that it exists for every object, whatever the uploader did. Three constants tune the rule. The 16 KiB floor and the 32 B factor set where the scaling starts, and the cap bounds the manifest. An object needing finer verification than its block would want the outboard-as-blob design under [Alternatives](#alternatives-considered). #### Block CVs @@ -172,7 +174,7 @@ x-amz-object-attributes: ETag,Blake3 - `CID` is the object digest as returned in `x-cid` - `ChunkLog` is the block size as a base-2 exponent of 1 KiB BLAKE3 chunks, the value a Bao library takes as its block size unchanged. iroh's fixed block is 4; the original Bao format is 0. -- `Outboard` is the standard pre-order Bao outboard at that block size, base64-encoded: the object size as 8 little-endian bytes, then the chaining-value pair of every parent node above the blocks, root first and left subtree before right. It is built from the stored block CVs on request, one compression per parent, and is byte-identical to what the Bao libraries produce for the same object and block size. It is an 8-byte prefix plus 64 bytes per parent, one fewer than the blocks: under 100 bytes for a small object, 16 KiB at 1 GiB, 1.4 MiB for a 50 TB object and 2 MiB at the 32,768-block cap, about 2.7 MiB after base64. A tree belongs to one version. An overwrite can never pair an old tree with new bytes, because both live in the same manifest, and the response carries the ETag of the version the tree describes for the client's `If-Match`. +- `Outboard` is the standard pre-order Bao outboard at that block size, base64-encoded: the object size as 8 little-endian bytes, then the chaining-value pair of every parent node above the blocks, root first and left subtree before right. It is built from the stored block CVs on request, one compression per parent, and is byte-identical to what the Bao libraries produce for the same object and block size. It is an 8-byte prefix plus 64 bytes per parent, one fewer than the blocks: a few hundred bytes for a small object, 32 KiB at 8 MiB, 256 KiB at 1 GiB, and 2 MiB at the 32,768-block cap from 32 GiB, about 2.7 MiB after base64. A tree belongs to one version. An overwrite can never pair an old tree with new bytes, because both live in the same manifest, and the response carries the ETag of the version the tree describes for the client's `If-Match`. AWS rejects an attribute name it does not know with `InvalidArgument`, so accepting `Blake3` is a visible divergence. SDKs whose typed response omits unknown elements cannot read the tree, but the same SDKs could not call a custom endpoint either, and a raw HTTP client parses this XML with what it already has for every other S3 response. @@ -226,7 +228,7 @@ S3 already defines this for uploads made with `--checksum-algorithm SHA256`: a S - **The root is not a hash of the bytes.** A composite depends on where the part boundaries fell, so the same bytes uploaded with 8 MiB parts, with 16 MiB parts, or as a single `PUT` get three different roots. It identifies an upload, not an object: two copies of the same data cannot be recognised as equal, nothing can be addressed by it, and a single-`PUT` object has a plain SHA-256 while a multipart object has a hash of hashes, so a client needs two code paths and two meanings for "the digest". - **Verifying a download takes a list, not a value.** The verifier must fetch the per-part hash list and the part sizes, up to 10,000 of each, keep them for the whole download, and apply the right one to the right byte range. The BLAKE3 digest is one 32-byte value that a standard library verifies against a stream. Walking a 10,000-entry list is custom code in every client, and the list itself is hundreds of KiB of XML on every fetch, with no compact form. -- **The granularity is the uploader's, and often too coarse.** Verification happens per part, so a client cannot check anything until it has a whole part, 5 GiB on large uploads, and a single-`PUT` object cannot be range-verified at all. The Bao tree verifies 16 KiB on small objects, and a server-chosen block on large ones, with proofs that are logarithmic in the object rather than linear in the part count. +- **The granularity is the uploader's, and often too coarse.** Verification happens per part, so a client cannot check anything until it has a whole part, 5 GiB on large uploads, and a single-`PUT` object cannot be range-verified at all. The Bao tree verifies 16 KiB on objects up to 8 MiB, and a server-chosen block on large ones, with proofs that are logarithmic in the object rather than linear in the part count. - **No standard verifies it.** There is no library that takes a composite SHA-256 and a part list and verifies a stream or a range, so every consumer writes its own; the Bao outboard is loadable by existing libraries in Rust and Go. - **It costs about the same.** A SHA-256 pass per part is no cheaper than the BLAKE3 pass, which is faster per byte, and the composite still has to be assembled at `Complete` from recorded values, as the tree is. @@ -258,7 +260,7 @@ The stored block CVs are half the size of the outboard, and a client could merge ### Sending the block CVs in a header -Base64 of a 1 GiB object's 16 KiB outboard is about 22 KiB, over the 8 KiB per-header default of common proxies, larger objects have far more, and splitting it across numbered headers is a hack. The root fits in a header; the tree does not. +Base64 of an 8 MiB object's 32 KiB outboard is about 43 KiB, over the 8 KiB per-header default of common proxies, larger objects have far more, and splitting it across numbered headers is a hack. The root fits in a header; the tree does not. ## Dependencies @@ -269,7 +271,7 @@ The Go implementation uses `lukechampine.com/blake3` for the hash, its `guts` pa - The re-hash budget's default for non-final parts. The final part is exempt, so the budget only governs variable-size parallel uploads; a larger default trades `Complete` latency for digests on more of them. - Names: `x-cid` and the `Blake3` attribute are placeholders. -- The scaling constants: the 16 KiB floor and the 32,768-block cap. The exponent is a knob too; cube-root scaling would give a 50 TB object 1,456 blocks at a 32 GiB block. +- The scaling constants: the 16 KiB floor, the 32 B factor that holds it to 8 MiB, and the 32,768-block cap. The exponent is a knob too; cube-root scaling would hold manifests smaller at the cost of coarser blocks on large objects. - Whether a `GET` with `?partNumber` should also return the block CVs covering that part, so a client that already reads by part need not align to blocks. - Whether the object CID should reach the Forge network, for example in the upload index, or remain an Ingot-level value until blob addressing is revisited. - Whether to expose the digest on `ListObjects` responses. From 2bdc03a88272194a38fbadb646e85c313c4b8a2c Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Mon, 12 Oct 2026 12:32:26 +0100 Subject: [PATCH 12/19] refactor: fail after budget --- rfcs/2026-10-blake3-object-digest.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/rfcs/2026-10-blake3-object-digest.md b/rfcs/2026-10-blake3-object-digest.md index 88e4a43..ceecc59 100644 --- a/rfcs/2026-10-blake3-object-digest.md +++ b/rfcs/2026-10-blake3-object-digest.md @@ -21,7 +21,7 @@ Blob addressing on the Forge network stays SHA-256. This RFC adds an object-leve ## Goals - Every object has a whole-object BLAKE3 digest, computed in the same streaming pass that writes its blobs. -- A multipart object's digest is assembled at `Complete` from values recorded per part. Re-reading part data is a bounded fallback for the few parts whose position was assumed wrongly, never the common path, and an upload that would need more than the bound gets no digest rather than a slow `Complete`. +- A multipart object's digest is assembled at `Complete` from values recorded per part. Re-reading part data is a bounded fallback for the few parts whose position was assumed wrongly, never the common path. An upload that would need more than the bound is refused at `Complete`, so every object that commits has a digest. - A client can verify a whole-object read with the digest alone and a standard BLAKE3 implementation. - A client can verify a ranged read with an off-the-shelf Bao library by fetching the object's outboard once: under a KiB for a small object, growing with the square root of the size. - The S3 data path is unchanged. Bodies are raw bytes, ETags and S3 checksums behave as they do today, and a client that ignores the extension sees nothing new. @@ -59,7 +59,7 @@ The body already streams through a SHA-256 hasher and an MD5 hasher as it is spl ### Multipart upload -Parts arrive in any order and the server learns a part's true byte offset only once every lower-numbered part exists. Each `UploadPart` therefore hashes its body at an assumed offset and records what `CompleteMultipartUpload` needs to finish the tree. `CompleteMultipartUpload` checks each assumed offset against the true one and re-hashes only the parts whose assumption was wrong, within a budget. +Parts arrive in any order and the server learns a part's true byte offset only once every lower-numbered part exists. Each `UploadPart` therefore hashes its body at an assumed offset and records what `CompleteMultipartUpload` needs to finish the tree. `CompleteMultipartUpload` checks each assumed offset against the true one and re-hashes only the parts whose assumption was wrong, within a budget; a completion that would need more is refused. #### What `UploadPart` records @@ -90,7 +90,7 @@ A part created by copy is hashed as its bytes are read, like an uploaded part. W `CompleteMultipartUpload` already walks the requested parts in order and computes each part's true offset. A part's record is usable when its true offset equals `TreeOffset`. -The parts whose assumed offset was wrong are re-hashed at their true offsets by reading their blobs back from the local spool, and the results stand in for their records. The final part is always re-hashed when wrong: in a uniform-size upload it is the only part whose assumption can fail, when it lands before any predecessor and its own size stands in, and the 5 GiB part maximum bounds the cost. The other parts are re-hashed within a **re-hash budget**, one maximum part, 5 GiB, by default, and configurable. An upload whose wrong non-final parts add up to more than the budget commits without a digest: the object's ETag is returned as always, GET and HEAD carry no `x-cid`, and `GetObjectAttributes` omits the `Blake3` element, the state defined below for an object without a digest. The budget keeps `Complete` to seconds of local reading at most. Exceeding it is an extreme edge case. No mainstream client varies its part size within an upload, the final part aside, and a uniform-size upload assumes every offset exactly in any arrival order. A sequential upload assumes every offset exactly whatever the sizes. What is left is a hand-rolled uploader sending variable-size parts in parallel, or a parallel compose-by-copy of differently sized source objects, and even there the cost is the absence of a digest, never a failed upload. +The parts whose assumed offset was wrong are re-hashed at their true offsets by reading their blobs back from the storage node that holds them, and the results stand in for their records. The final part is always re-hashed when wrong: in a uniform-size upload it is the only part whose assumption can fail, when it lands before any predecessor and its own size stands in, and the 5 GiB part maximum bounds the cost. The other parts are re-hashed within a **re-hash budget**, one maximum part, 5 GiB, by default, and configurable. `Complete` adds up the wrong non-final parts before it accepts any part or reads anything. A sum over the budget is refused with a 400, `InvalidPartLayout`, whose message gives the bytes that would have to be read back, the server's limit, and the two ways out: parts of one size, or parts uploaded in part-number order. The upload stays open, so the client can resend parts and complete again. Every object that commits therefore has a digest. The budget bounds how long `Complete` spends reading from storage nodes and nothing else, so it can be set generously. Reaching it is rare. No mainstream client varies its part size within an upload, the final part aside, and a uniform-size upload assumes every offset exactly in any arrival order. A sequential upload assumes every offset exactly whatever the sizes, which covers the usual way objects are concatenated: each source object copied in as one part, in order. What is left is a parallel compose-by-copy of differently sized source objects, or a hand-rolled uploader sending variable-size parts in parallel, and those work up to the budget, about 5 GiB of such parts plus the final one with the default. A chunk that straddles a part boundary is rebuilt from the earlier part's `TreeTail` and the later part's `TreeHead` and compressed at its chunk index, one compression per boundary. The last part's `TreeTail` is the object's final chunk and is compressed as such. Part lengths that are not 1 KiB multiples therefore cost nothing. @@ -98,7 +98,7 @@ With every part's nodes at their true offsets, the object digest is computed by An upload completed with a single part has no merge, and its digest is that part's root as a body of its own. For part 1 that is the recorded `TreeRoot`. For any other part the recorded offset is not 0, so the offset check above re-hashes it at offset 0, and that re-hash yields the root directly. S3 requires the completed parts to be in ascending order, not to begin at part 1, so this case is legal and does occur. -The block CVs for the object are built from the parts' `TreeBlocks` and `TreeNodes` as described in the next section. The root computed by merging the block CVs up must equal the root computed from the parts' subtrees. This is a consistency check on the assembly at the cost of one compression per block, 32,767 at the cap. Should it ever fail, the object commits without a digest rather than with a wrong one; nothing re-reads the body. +The block CVs for the object are built from the parts' `TreeBlocks` and `TreeNodes` as described in the next section. The root computed by merging the block CVs up must equal the root computed from the parts' subtrees. This is a consistency check on the assembly at the cost of one compression per block, 32,767 at the cap. Should it ever fail, `Complete` returns a server error and the upload stays open; the object is never committed with a digest that might be wrong, and never without one. Nothing re-reads the body. ### Range verification @@ -204,7 +204,7 @@ type Body struct { } ``` -Some objects have no digest: those written before this change, and a multipart object whose completion exceeded the re-hash budget. For them, GET and HEAD return no `x-cid` header and `GetObjectAttributes` returns its response without the `Blake3` element. +Objects written before this change have no digest. For them, GET and HEAD return no `x-cid` header and `GetObjectAttributes` returns its response without the `Blake3` element. That makes two situations look alike from a GET or HEAD: an object without a digest, and a server that does not implement this RFC. `GetObjectAttributes` tells them apart. The request names the attributes it wants in `x-amz-object-attributes`, and a server validates the names: one that implements this RFC accepts `Blake3` and answers, with or without the element depending on the object, while one that does not, AWS included, rejects the request with `InvalidArgument`. So a client that wants to know whether verification is available at all asks for `Blake3` once; an error means the server cannot provide it for any object, and a success without the element means it cannot provide it for this one. @@ -212,7 +212,7 @@ That makes two situations look alike from a GET or HEAD: an object without a dig Ingest adds one BLAKE3 pass per body or part, alongside the MD5 pass already there. BLAKE3 with vector instructions is faster than SHA-256, so it costs less than the whole-body SHA-256 pass it makes redundant. -`CompleteMultipartUpload` adds one compression per recorded subtree for the digest and one per block for the consistency check, a few tens of thousands at most for the largest uploads, which is milliseconds. The re-hash path reads the affected parts back through the plaintext opener, which decrypts in an encrypting deployment. The parts are read from the local spool: a part's blobs stay there until the object is deleted, since nothing evicts spool copies today, so no re-hash fetches from a storage node. The work is bounded by the final part plus the budget, about 10 GiB of local reading at most with the default, which is seconds to tens of seconds. The price of the bound is that a variable-size parallel upload beyond it gets no digest. +`CompleteMultipartUpload` adds one compression per recorded subtree for the digest and one per block for the consistency check, a few tens of thousands at most for the largest uploads, which is milliseconds. The re-hash path reads the affected parts back through the plaintext opener, which decrypts in an encrypting deployment. A part's local copy is removed as soon as the part is parked on its provider, so the re-hash fetches the part's blobs from the storage node with ranged retrievals, the same path a `GET` takes. The work is bounded by the final part plus the budget, about 10 GiB of retrieval at most with the default, so `Complete` on such an upload takes as long as fetching that much from the provider, under a minute and a half at a gigabit per second. A variable-size parallel upload beyond the bound is refused, as described above. The manifest grows by the block list, 8 KiB at 1 GiB and at most 1 MiB, plus the 34-byte multihash and a few bytes of framing. The part record grows by at most about 15 KiB for the life of the upload. @@ -236,7 +236,7 @@ What it does offer over the current state is universality within multipart uploa ### Re-addressing blobs to BLAKE3 -Making BLAKE3 the network's content address would let storage nodes serve verified ranges directly. It touches every component that handles a multihash, the proof-tree assumptions of PDP, public gateway resolvability, and every stored byte. This RFC is independent of that decision. The object digest proposed here is the same value such a migration would use, so nothing here is thrown away if it happens. +Making BLAKE3 the network's content address would let storage nodes serve verified ranges directly. It touches every component that handles a multihash, the proof-tree assumptions of PDP, public gateway resolvability, and every stored byte. This RFC is independent of that decision. The object digest proposed here is the same value such a change would use, so nothing here is thrown away if it happens. The [BLAKE3 shard digests](./2026-10-blake3-shard-digest.md) RFC explores it for new shards. ### A fixed 16 KiB block with the outboard as a blob @@ -268,7 +268,7 @@ The Go implementation uses `lukechampine.com/blake3` for the hash, its `guts` pa ## Open questions -- The re-hash budget's default for non-final parts. The final part is exempt, so the budget only governs variable-size parallel uploads; a larger default trades `Complete` latency for digests on more of them. +- The re-hash budget's default for non-final parts. The final part is exempt, so the budget only governs variable-size parallel uploads; a larger default trades `Complete` latency for accepting more of them. - Names: `x-cid` and the `Blake3` attribute are placeholders. - The scaling constants: the 16 KiB floor, the 32 B factor that holds it to 8 MiB, and the 32,768-block cap. The exponent is a knob too; cube-root scaling would hold manifests smaller at the cost of coarser blocks on large objects. From 466a37ac239901406e24d7aa7b0b4f3f3e9ba6d0 Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Mon, 12 Oct 2026 14:52:43 +0100 Subject: [PATCH 13/19] refactor: take suggested opening --- rfcs/2026-10-blake3-object-digest.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/2026-10-blake3-object-digest.md b/rfcs/2026-10-blake3-object-digest.md index ceecc59..a8f0492 100644 --- a/rfcs/2026-10-blake3-object-digest.md +++ b/rfcs/2026-10-blake3-object-digest.md @@ -8,7 +8,7 @@ Status: Experimental ## Motivation -What a client can verify about an Ingot object today depends on how the object was uploaded, and for most objects it is nothing. A single `PUT` records a whole-object SHA-256 in the manifest, the per-object metadata record Ingot keeps in the bucket's catalog, but it is internal and never returned. A multipart object records no whole-object digest at all, because SHA-256 cannot be assembled from the parts and re-reading every part at `CompleteMultipartUpload` is too slow. +The granularity at which a client can verify an Ingot object today depends on how the object was uploaded. A single `PUT` records nothing in the manifest (the per-object metadata record Ingot keeps in the bucket's catalog) about any piece of the object smaller than the whole. A client can verify such an object only in full, against a digest it obtained elsewhere, which means downloading all of it first and is impractical for a large object. A multipart object records per-part values (described next) but no whole-object digest at all, because SHA-256 cannot be assembled from the parts and re-reading every part at `CompleteMultipartUpload` is too slow. The one verifiable value is the per-part checksum list, and it exists only if the uploader asked for it. A multipart object uploaded with a SHA checksum algorithm, `--checksum-algorithm SHA256` or its SDK equivalent, has a checksum per part and a composite over them, and `GetObjectAttributes` returns the list after the upload completes. That is a two-level tree the client can check part-aligned ranges against, and check against the composite. No client does this by default: the AWS SDKs and CLI now send a checksum unprompted, but it is a CRC, which gives a full-object value with no per-part list; rclone, s5cmd and the MinIO client send none. The list covers nothing else: not a single `PUT`, not an upload made without a SHA algorithm, and no range finer than the uploader's parts, whose size the reader did not choose. The part ETags the object ETag is built from are not returned once the upload completes, and a per-part MD5 is available only when the uploader chose MD5 as the checksum algorithm, an opt-in like SHA-256; the ETag itself, an MD5 or an MD5 of part MD5s, identifies the object loosely and gives a reader no cryptographic assurance. The CRC family (CRC32, CRC32C, CRC64NVME) can be a full-object value, which Ingot derives at `CompleteMultipartUpload` by combining the parts' CRCs, but a CRC detects accidental corruption and nothing more. From d4fcb8dc5bdf4b5c2323ec6ff85cff82c8601f3a Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Mon, 12 Oct 2026 15:13:07 +0100 Subject: [PATCH 14/19] refactor: add more justification for outboard as separate blob --- rfcs/2026-10-blake3-object-digest.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/rfcs/2026-10-blake3-object-digest.md b/rfcs/2026-10-blake3-object-digest.md index a8f0492..e009b8f 100644 --- a/rfcs/2026-10-blake3-object-digest.md +++ b/rfcs/2026-10-blake3-object-digest.md @@ -240,7 +240,17 @@ Making BLAKE3 the network's content address would let storage nodes serve verifi ### A fixed 16 KiB block with the outboard as a blob -bao-tree's default gives 16 KiB verification granularity at any size, with the outboard stored as its own blob referenced from the manifest, about 0.4% of the object. It is the right design if fine-grained verified reads of very large objects are a requirement. It adds a blob per object to write, locate and clean up. The client's extra request is the same either way, but it fetches megabytes instead of KiB. The bounded manifest tree serves the common case with no new blob and can coexist with this design later: the stored block CVs are the top of the finer tree. +bao-tree's default gives 16 KiB verification granularity at any size, with the outboard stored as a blob of its own on the Forge network and referenced from the manifest by digest. The manifest then stays a few dozen bytes however large the object, the tree is content-addressed so any node holding it can serve it and a client can check it against its SHA-256 digest, and the 2 MiB field limit of the catalog no longer bounds the tree. It is the right design if fine-grained verified reads of very large objects are a requirement. + +The client makes a second request in both designs. What differs is what the request returns and what the server manages behind it. + +- **The response is 0.4% of the object.** The outboard at 16 KiB is 64 bytes per block: 4 MiB for a 1 GiB object, 4 GiB for a 1 TiB object, about 190 GiB at the 48 TiB S3 maximum. The manifest tree is 256 KiB at 1 GiB and 2 MiB from 32 GiB up, and a client that wants one range needs only the proof slice for it, which `GetObjectAttributes` could extract from either. +- **A blob per object.** Every `PUT` writes a second blob: allocate, upload, accept, a claim, replication to every replica, a PDP piece to prove, and a release on delete. Under one object, one piece, that doubles the on-chain piece count for small objects, which are most objects, to carry a tree of 32 bytes to a few KiB. The upload also sits on the commit path, since the tree is known only once the body is hashed and the manifest is written only after its blobs are accepted. +- **Multipart grows with it.** A part's block list at 16 KiB is 10 MiB for a 5 GiB part, against about 15 KiB today, and `Complete` on the largest object reads them all back and writes the outboard out. The outboard passes the blob ceiling at about 60 GiB of object and becomes a multi-blob body of its own. +- **Encryption removes the independence.** In an encrypting bucket the tree blob is encrypted like the body, so only Ingot can serve it and the client gains nothing over the manifest. A plaintext tree would publish block-level hashes of encrypted data, which lets anyone confirm guesses about low-entropy blocks. +- **Two storage paths for one value.** A blob per 1 KiB object is a poor trade, so the blob would join the manifest path above some size rather than replace it, and both paths would be maintained for good. + +The bounded manifest tree serves the common case with no new blob and can coexist with this design later: the stored block CVs are the top of the finer tree, so an object that needs 16 KiB verification could gain a tree blob without changing what the manifest holds. ### Returning the Bao encoded stream From e0f739396dd27314801f7fd5d924dda16a2576aa Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Mon, 12 Oct 2026 17:10:50 +0100 Subject: [PATCH 15/19] refactor: clarify in memory sizes --- rfcs/2026-10-blake3-object-digest.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/rfcs/2026-10-blake3-object-digest.md b/rfcs/2026-10-blake3-object-digest.md index e009b8f..cefb91c 100644 --- a/rfcs/2026-10-blake3-object-digest.md +++ b/rfcs/2026-10-blake3-object-digest.md @@ -68,7 +68,7 @@ Each part record gains: - `TreeOffset`: the byte offset the part was hashed at. - `TreeNodes`: the CVs of the part's aligned subtrees, in order, covering the whole chunks between the part's first and last chunk boundaries. A span of whole chunks decomposes into at most two aligned power-of-two blocks per tree level, so this list holds a few dozen 32-byte entries at most. - `TreeBlocks`: the part's block CVs at the part's own block size (see [Range verification](#range-verification)), at most 10,240 entries (320 KiB) for a 5 GiB part. -- `TreeHead` and `TreeTail`: the part's bytes before its first chunk boundary and after its last one, under 1 KiB each. They are empty for a part whose offset and length are 1 KiB multiples. Every mainstream client sizes its parts in MiB, so both spans are empty for nearly every part ever uploaded. The one known exception is the AWS SDKs for Go and Java above the 10,000-part limit, objects over about 48 GiB, where they size parts as the object size divided by 10,000, an arbitrary byte count; those uploads are a small minority, and these two spans are what let them complete without re-reading anything. +- `TreeHead` and `TreeTail`: the part's bytes before its first chunk boundary and after its last one, under 1 KiB each. They are empty for a part whose offset and length are 1 KiB multiples. Every mainstream client sizes its parts in MiB, so both spans are empty for nearly every part ever uploaded. The one known exception is the AWS SDKs for Go and Java above the 10,000-part limit, objects over about 48 GiB, where they size parts as the object size divided by 10,000, an arbitrary byte count; those uploads are a small minority, and these two spans are what let them complete without re-reading anything. The spans live in the part's record in the catalog database with the rest of its tree material, since the part's local copy is removed once the part is parked on its storage node; nothing is held in memory between requests. - For part 1 only, `TreeRoot`: the BLAKE3 hash of the part on its own. Part 1 is always at offset 0, so its standalone hash falls out of the same pass with a root-flagged final compression. A part cannot know whether it is the last, so it never treats a short final chunk as the object's end; those bytes go to `TreeTail`, and `CompleteMultipartUpload` compresses them once it knows what follows them. @@ -92,7 +92,7 @@ A part created by copy is hashed as its bytes are read, like an uploaded part. W The parts whose assumed offset was wrong are re-hashed at their true offsets by reading their blobs back from the storage node that holds them, and the results stand in for their records. The final part is always re-hashed when wrong: in a uniform-size upload it is the only part whose assumption can fail, when it lands before any predecessor and its own size stands in, and the 5 GiB part maximum bounds the cost. The other parts are re-hashed within a **re-hash budget**, one maximum part, 5 GiB, by default, and configurable. `Complete` adds up the wrong non-final parts before it accepts any part or reads anything. A sum over the budget is refused with a 400, `InvalidPartLayout`, whose message gives the bytes that would have to be read back, the server's limit, and the two ways out: parts of one size, or parts uploaded in part-number order. The upload stays open, so the client can resend parts and complete again. Every object that commits therefore has a digest. The budget bounds how long `Complete` spends reading from storage nodes and nothing else, so it can be set generously. Reaching it is rare. No mainstream client varies its part size within an upload, the final part aside, and a uniform-size upload assumes every offset exactly in any arrival order. A sequential upload assumes every offset exactly whatever the sizes, which covers the usual way objects are concatenated: each source object copied in as one part, in order. What is left is a parallel compose-by-copy of differently sized source objects, or a hand-rolled uploader sending variable-size parts in parallel, and those work up to the budget, about 5 GiB of such parts plus the final one with the default. -A chunk that straddles a part boundary is rebuilt from the earlier part's `TreeTail` and the later part's `TreeHead` and compressed at its chunk index, one compression per boundary. The last part's `TreeTail` is the object's final chunk and is compressed as such. Part lengths that are not 1 KiB multiples therefore cost nothing. +A chunk that straddles a part boundary is rebuilt from the earlier part's `TreeTail` and the later part's `TreeHead` and compressed at its chunk index, one compression per boundary. The last part's `TreeTail` is the object's final chunk and is compressed as such. Part lengths that are not 1 KiB multiples therefore add little: under 2 KiB of raw bytes per part record, under 20 MiB across the 10,000-part maximum, and one compression per boundary at `Complete`. With every part's nodes at their true offsets, the object digest is computed by merging adjacent aligned subtrees into parents and applying the root flag at the final merge. This is one compression per recorded subtree and reads no data. The work scales with the number of parts: a part contributes at most two subtrees per tree level, so an upload of 10,000 parts merges a few tens of thousands of nodes, each one BLAKE3 compression, which is milliseconds. @@ -214,7 +214,7 @@ Ingest adds one BLAKE3 pass per body or part, alongside the MD5 pass already the `CompleteMultipartUpload` adds one compression per recorded subtree for the digest and one per block for the consistency check, a few tens of thousands at most for the largest uploads, which is milliseconds. The re-hash path reads the affected parts back through the plaintext opener, which decrypts in an encrypting deployment. A part's local copy is removed as soon as the part is parked on its provider, so the re-hash fetches the part's blobs from the storage node with ranged retrievals, the same path a `GET` takes. The work is bounded by the final part plus the budget, about 10 GiB of retrieval at most with the default, so `Complete` on such an upload takes as long as fetching that much from the provider, under a minute and a half at a gigabit per second. A variable-size parallel upload beyond the bound is refused, as described above. -The manifest grows by the block list, 8 KiB at 1 GiB and at most 1 MiB, plus the 34-byte multihash and a few bytes of framing. The part record grows by at most about 15 KiB for the life of the upload. +The manifest grows by the block list, 128 KiB at 1 GiB and at most 1 MiB, plus the 34-byte multihash and a few bytes of framing. The part record grows by its tree material for the life of the upload: a few KiB for parts of a few MiB, up to about 320 KiB for a 5 GiB part, nearly all of it block CVs. `Complete` loads every part's record, so the largest upload S3 allows, 10,000 parts of 5 GiB, holds about 3 GiB of records in memory while it completes. ## Alternatives considered From 899eb47cb19ae692e9ba8c14daf29cce40dd89ca Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Mon, 12 Oct 2026 17:17:24 +0100 Subject: [PATCH 16/19] refactor: clarify part number and offset --- rfcs/2026-10-blake3-object-digest.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/2026-10-blake3-object-digest.md b/rfcs/2026-10-blake3-object-digest.md index cefb91c..9c3cfcd 100644 --- a/rfcs/2026-10-blake3-object-digest.md +++ b/rfcs/2026-10-blake3-object-digest.md @@ -96,7 +96,7 @@ A chunk that straddles a part boundary is rebuilt from the earlier part's `TreeT With every part's nodes at their true offsets, the object digest is computed by merging adjacent aligned subtrees into parents and applying the root flag at the final merge. This is one compression per recorded subtree and reads no data. The work scales with the number of parts: a part contributes at most two subtrees per tree level, so an upload of 10,000 parts merges a few tens of thousands of nodes, each one BLAKE3 compression, which is milliseconds. -An upload completed with a single part has no merge, and its digest is that part's root as a body of its own. For part 1 that is the recorded `TreeRoot`. For any other part the recorded offset is not 0, so the offset check above re-hashes it at offset 0, and that re-hash yields the root directly. S3 requires the completed parts to be in ascending order, not to begin at part 1, so this case is legal and does occur. +An upload completed with a single part has no merge, and its digest is that part's root as a body of its own. For part 1 that is the recorded `TreeRoot`. For any other part the recorded offset is not 0, so the offset check above re-hashes it at offset 0, and that re-hash yields the root directly. S3 requires the completed parts to be in ascending order and nothing more: the list need not start at part 1 or be contiguous, so a client that re-uploads a bad part under a new number and completes without the old one produces exactly this case. The block CVs for the object are built from the parts' `TreeBlocks` and `TreeNodes` as described in the next section. The root computed by merging the block CVs up must equal the root computed from the parts' subtrees. This is a consistency check on the assembly at the cost of one compression per block, 32,767 at the cap. Should it ever fail, `Complete` returns a server error and the upload stays open; the object is never committed with a digest that might be wrong, and never without one. Nothing re-reads the body. From b5922f894ee5f1459253ad06390bfb2387af26ec Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Mon, 12 Oct 2026 17:48:09 +0100 Subject: [PATCH 17/19] refactor: gfm maths --- rfcs/2026-10-blake3-object-digest.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/rfcs/2026-10-blake3-object-digest.md b/rfcs/2026-10-blake3-object-digest.md index 9c3cfcd..637d2e1 100644 --- a/rfcs/2026-10-blake3-object-digest.md +++ b/rfcs/2026-10-blake3-object-digest.md @@ -102,7 +102,7 @@ The block CVs for the object are built from the parts' `TreeBlocks` and `TreeNod ### Range verification -Two terms, since BLAKE3 and Bao both have "leaves" and they are not the same thing. A **chunk** is BLAKE3's 1 KiB unit; a chunk's CV is a leaf of BLAKE3's own tree. A **block** is 2^ChunkLog chunks, the unit a client verifies; a block's CV is an interior node of BLAKE3's tree and a leaf of the Bao tree at that block size. This RFC stores and serves block CVs and never stores chunk CVs. The word "leaf" is not used below. +Two terms, since BLAKE3 and Bao both have "leaves" and they are not the same thing. A **chunk** is BLAKE3's 1 KiB unit; a chunk's CV is a leaf of BLAKE3's own tree. A **block** is $2^{\text{ChunkLog}}$ chunks, the unit a client verifies; a block's CV is an interior node of BLAKE3's tree and a leaf of the Bao tree at that block size. This RFC stores and serves block CVs and never stores chunk CVs. The word "leaf" is not used below. A client that reads ranges needs more than the root. It needs the CVs of the tree nodes covering the bytes it read, and enough of the rest of the tree to connect them to the root. The full Bao outboard provides this at 1 KiB granularity and costs about 6% of the object size. Storing it is out of the question in a manifest, and even at bao-tree's 16 KiB default it costs 4 MiB per GiB. @@ -112,12 +112,11 @@ This RFC scales the block size with the object. Objects up to 8 MiB are recorded The **block** is the unit of the stored tree. Its target is the square root of the object size times 32 bytes, the size of one CV: -``` -target = sqrt(size × 32 B) -block = the smallest power of two ≥ target, and at least 16 KiB -``` +$$\text{target} = \sqrt{\text{size} \times 32\,\text{B}}$$ + +$$\text{block} = \max\bigl(16\,\text{KiB},\ 2^{\lceil \log_2 \text{target} \rceil}\bigr)$$ -The target reaches the 16 KiB floor at 8 MiB, so every object up to 8 MiB has 16 KiB blocks, 512 of them at 8 MiB. Above that the block doubles each time the object quadruples: for a 1 GiB object the target is sqrt(2^30 × 2^5) = sqrt(2^35), so the block is 256 KiB and there are 4,096 blocks. A hard cap of 32,768 blocks, 1 MiB of CVs, binds from 32 GiB, where the block grows with the size instead. The cap is a metadata-cost choice: the manifest is a catalog block rewritten on every version, so its size is storage the tenant pays for on each write, and 1 MiB leaves headroom under the 2 MiB byte-field limit of the catalog's CBOR encoding. A larger cap, up to that limit, buys finer verification on objects over 32 GiB at that cost. The manifest records the block size as a **chunk log**: a base-2 exponent of 1 KiB BLAKE3 chunks, the unit Bao libraries take as the block size, so 16 KiB is 4 and 1 MiB is 10. The block affects only what is stored and what granularity a client can verify at. The object digest is the same whatever block is chosen. +The target reaches the 16 KiB floor at 8 MiB, so every object up to 8 MiB has 16 KiB blocks, 512 of them at 8 MiB. Above that the block doubles each time the object quadruples: for a 1 GiB object the target is $\sqrt{2^{30} \times 2^{5}} = \sqrt{2^{35}}$, about 181 KiB, so the block is 256 KiB and there are 4,096 blocks. A hard cap of 32,768 blocks, 1 MiB of CVs, binds from 32 GiB, where the block grows with the size instead. The cap is a metadata-cost choice: the manifest is a catalog block rewritten on every version, so its size is storage the tenant pays for on each write, and 1 MiB leaves headroom under the 2 MiB byte-field limit of the catalog's CBOR encoding. A larger cap, up to that limit, buys finer verification on objects over 32 GiB at that cost. The manifest records the block size as a **chunk log**: a base-2 exponent of 1 KiB BLAKE3 chunks, the unit Bao libraries take as the block size, so 16 KiB is 4 and 1 MiB is 10. The block affects only what is stored and what granularity a client can verify at. The object digest is the same whatever block is chosen. | Object size | Block | Chunk log | Blocks | Manifest bytes | |---|---|---|---|---| From a72b41b569c5777e8520ae96195fee1d0f6120a3 Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Mon, 12 Oct 2026 18:33:33 +0100 Subject: [PATCH 18/19] refactor: address pr comments --- rfcs/2026-10-blake3-object-digest.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/rfcs/2026-10-blake3-object-digest.md b/rfcs/2026-10-blake3-object-digest.md index 637d2e1..ad3d982 100644 --- a/rfcs/2026-10-blake3-object-digest.md +++ b/rfcs/2026-10-blake3-object-digest.md @@ -118,7 +118,7 @@ $$\text{block} = \max\bigl(16\,\text{KiB},\ 2^{\lceil \log_2 \text{target} \rcei The target reaches the 16 KiB floor at 8 MiB, so every object up to 8 MiB has 16 KiB blocks, 512 of them at 8 MiB. Above that the block doubles each time the object quadruples: for a 1 GiB object the target is $\sqrt{2^{30} \times 2^{5}} = \sqrt{2^{35}}$, about 181 KiB, so the block is 256 KiB and there are 4,096 blocks. A hard cap of 32,768 blocks, 1 MiB of CVs, binds from 32 GiB, where the block grows with the size instead. The cap is a metadata-cost choice: the manifest is a catalog block rewritten on every version, so its size is storage the tenant pays for on each write, and 1 MiB leaves headroom under the 2 MiB byte-field limit of the catalog's CBOR encoding. A larger cap, up to that limit, buys finer verification on objects over 32 GiB at that cost. The manifest records the block size as a **chunk log**: a base-2 exponent of 1 KiB BLAKE3 chunks, the unit Bao libraries take as the block size, so 16 KiB is 4 and 1 MiB is 10. The block affects only what is stored and what granularity a client can verify at. The object digest is the same whatever block is chosen. -| Object size | Block | Chunk log | Blocks | Manifest bytes | +| Object size | Block size | Chunk log | Block count | Manifest bytes | |---|---|---|---|---| | 1 KiB | 16 KiB | 4 | 1 | 32 B | | 64 KiB | 16 KiB | 4 | 4 | 128 B | @@ -134,9 +134,9 @@ The target reaches the 16 KiB floor at 8 MiB, so every object up to 8 MiB has 16 | 5 TiB | 256 MiB | 18 | 20,480 | 640 KiB | | 50 TB | 2 GiB | 21 | 23,284 | 728 KiB | -Sizes with binary prefixes are powers of two; the 50 TB row is decimal, 50 × 10¹² bytes, just under S3's current maximum object size of 48.8 TiB (10,000 parts of 5 GiB). Blocks is the size divided by the block size, rounded up, since a final partial block still has a CV. An object of 16 KiB or less is one block. Up to 8 MiB the manifest cost is 32 bytes per 16 KiB, 0.2% of the object. From there it grows with the square root of the size: 50 KiB at 100 MiB, 128 KiB at 1 GiB, 320 KiB at 10 GiB. The cap binds from 32 GiB and holds every larger object to 1 MiB or less, 728 KiB for a 50 TB object. +Sizes with binary prefixes are powers of two; the 50 TB row is decimal, 50 × 10¹² bytes, just under S3's current maximum object size of 48.8 TiB (10,000 parts of 5 GiB). Block count is the object size divided by the block size, rounded up, since a final partial block still has a CV. An object of 16 KiB or less is one block. Up to 8 MiB the manifest cost is 32 bytes per 16 KiB, 0.2% of the object. From there it grows with the square root of the size: 50 KiB at 100 MiB, 128 KiB at 1 GiB, 320 KiB at 10 GiB. The cap binds from 32 GiB and holds every larger object to 1 MiB or less, 728 KiB for a 50 TB object. -The smallest range a client can verify independently is one block: 16 KiB on anything up to 8 MiB, 256 KiB on a 1 GiB object, 32 MiB on a 1 TiB object, 2 GiB on a 50 TB object. For a large object that is coarser than the per-part checksum list of a multipart upload made with SHA checksums: the block passes an 8 MiB part above 256 GiB, so on a 1 TiB object such a list verifies at 8 MiB where this tree verifies at 32 MiB. The tree's advantage is that it exists for every object, whatever the uploader did. Three constants tune the rule. The 16 KiB floor and the 32 B factor set where the scaling starts, and the cap bounds the manifest. An object needing finer verification than its block would want the outboard-as-blob design under [Alternatives](#alternatives-considered). +The smallest range a client can verify independently is one block: 16 KiB on anything up to 8 MiB, 256 KiB on a 1 GiB object, 32 MiB on a 1 TiB object, 2 GiB on a 50 TB object. That is finer than the per-part checksum list of any multipart upload of the same object, whose granularity is its parts. S3 caps an upload at 10,000 parts of at least 5 MiB, so a part averages at least the larger of 5 MiB and the object size divided by 10,000. The block is at most 1 MiB up to 32 GiB, at most 4 MiB up to 128 GiB, and under the object size divided by 16,384 beyond: on a 1 TiB object a part averages at least 105 MiB where the block is 32 MiB. The tree also exists for every object, even if the uploader does not request per-part SHA-256 checksums. Three constants tune the rule. The 16 KiB floor and the 32 B factor set where the scaling starts, and the cap bounds the manifest. An object needing finer verification than its block would want the outboard-as-blob design under [Alternatives](#alternatives-considered). #### Block CVs @@ -149,6 +149,8 @@ A multipart upload does not know the object length until `CompleteMultipartUploa - A block lying within one part. Its CV is the merge of that part's `TreeBlocks` that it covers. - A block straddling a part boundary. The piece on each side is a union of that part's edge subtrees, which `TreeNodes` holds, so its CV is the merge of those entries. +A 1 KiB chunk split across a part boundary belongs to neither part's subtrees. Each part's `TreeNodes` covers only its whole chunks, and the split chunk's bytes sit in the earlier part's `TreeTail` and the later part's `TreeHead`. `CompleteMultipartUpload` joins the two, compresses the chunk at its index, and merges the result into the block containing that index. Blocks are whole chunks, so the chunk lies in exactly one block. + Either way the block CVs are built from recorded values with no data read. #### The tree in `GetObjectAttributes` @@ -227,7 +229,7 @@ S3 already defines this for uploads made with `--checksum-algorithm SHA256`: a S - **The root is not a hash of the bytes.** A composite depends on where the part boundaries fell, so the same bytes uploaded with 8 MiB parts, with 16 MiB parts, or as a single `PUT` get three different roots. It identifies an upload, not an object: two copies of the same data cannot be recognised as equal, nothing can be addressed by it, and a single-`PUT` object has a plain SHA-256 while a multipart object has a hash of hashes, so a client needs two code paths and two meanings for "the digest". - **Verifying a download takes a list, not a value.** The verifier must fetch the per-part hash list and the part sizes, up to 10,000 of each, keep them for the whole download, and apply the right one to the right byte range. The BLAKE3 digest is one 32-byte value that a standard library verifies against a stream. Walking a 10,000-entry list is custom code in every client, and the list itself is hundreds of KiB of XML on every fetch, with no compact form. -- **The granularity is the uploader's, and often too coarse.** Verification happens per part, so a client cannot check anything until it has a whole part, 5 GiB on large uploads, and a single-`PUT` object cannot be range-verified at all. The Bao tree verifies 16 KiB on objects up to 8 MiB, and a server-chosen block on large ones, with proofs that are logarithmic in the object rather than linear in the part count. +- **The granularity is the uploader's, and coarser than the tree's.** Verification happens per part, so a client cannot check anything until it has a whole part. A part is at least 5 MiB, at least the object size divided by 10,000, and up to 5 GiB, and a single-`PUT` object cannot be range-verified at all. The Bao tree verifies 16 KiB on objects up to 8 MiB, and a server-chosen block on large ones, with proofs that are logarithmic in the object rather than linear in the part count. - **No standard verifies it.** There is no library that takes a composite SHA-256 and a part list and verifies a stream or a range, so every consumer writes its own; the Bao outboard is loadable by existing libraries in Rust and Go. - **It costs about the same.** A SHA-256 pass per part is no cheaper than the BLAKE3 pass, which is faster per byte, and the composite still has to be assembled at `Complete` from recorded values, as the tree is. From a331a30918b653e149da09ca7e902cdca7cecdc8 Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Tue, 13 Oct 2026 12:23:14 +0100 Subject: [PATCH 19/19] refactor: store block CVs at granularity needed for size of upload --- rfcs/2026-10-blake3-object-digest.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/rfcs/2026-10-blake3-object-digest.md b/rfcs/2026-10-blake3-object-digest.md index ad3d982..f250064 100644 --- a/rfcs/2026-10-blake3-object-digest.md +++ b/rfcs/2026-10-blake3-object-digest.md @@ -67,7 +67,7 @@ Each part record gains: - `TreeOffset`: the byte offset the part was hashed at. - `TreeNodes`: the CVs of the part's aligned subtrees, in order, covering the whole chunks between the part's first and last chunk boundaries. A span of whole chunks decomposes into at most two aligned power-of-two blocks per tree level, so this list holds a few dozen 32-byte entries at most. -- `TreeBlocks`: the part's block CVs at the part's own block size (see [Range verification](#range-verification)), at most 10,240 entries (320 KiB) for a 5 GiB part. +- `TreeBlocks`: the part's block CVs at the block size of an object that ends where the part ends (see [Range verification](#range-verification)). Part 1 of 5 GiB records 10,240 entries (320 KiB); each later part records fewer, down to 3 for part 10,000 of a 10,000 × 5 GiB upload. - `TreeHead` and `TreeTail`: the part's bytes before its first chunk boundary and after its last one, under 1 KiB each. They are empty for a part whose offset and length are 1 KiB multiples. Every mainstream client sizes its parts in MiB, so both spans are empty for nearly every part ever uploaded. The one known exception is the AWS SDKs for Go and Java above the 10,000-part limit, objects over about 48 GiB, where they size parts as the object size divided by 10,000, an arbitrary byte count; those uploads are a small minority, and these two spans are what let them complete without re-reading anything. The spans live in the part's record in the catalog database with the rest of its tree material, since the part's local copy is removed once the part is parked on its storage node; nothing is held in memory between requests. - For part 1 only, `TreeRoot`: the BLAKE3 hash of the part on its own. Part 1 is always at offset 0, so its standalone hash falls out of the same pass with a root-flagged final compression. @@ -144,7 +144,7 @@ The manifest stores the CV of every block of the object, in object order from of A single `PUT` computes the block CVs in its hashing pass, since the object length is known from `Content-Length` and so is the block. -A multipart upload does not know the object length until `CompleteMultipartUpload`, so each part computes `TreeBlocks` at its own block size, chosen by the same rule from the part's own size. The object's block is never smaller than any part's, because the rule is monotone in size and the object is at least as large as each part. At `CompleteMultipartUpload`, each object block is one of: +A multipart upload does not know the object length until `CompleteMultipartUpload`, so each part computes `TreeBlocks` at the block size the same rule gives for its assumed offset plus its length. A record is used only when its assumed offset is the true one, and the object then holds at least that many bytes, so the object's block is never smaller than the part's, because the rule is monotone in size. Sizing the blocks by the part's own length would also be correct, but finer than the object needs: at the 10,000 × 5 GiB maximum that is 3 GiB of CVs against about 6 MiB. At `CompleteMultipartUpload`, each object block is one of: - A block lying within one part. Its CV is the merge of that part's `TreeBlocks` that it covers. - A block straddling a part boundary. The piece on each side is a union of that part's edge subtrees, which `TreeNodes` holds, so its CV is the merge of those entries. @@ -215,7 +215,7 @@ Ingest adds one BLAKE3 pass per body or part, alongside the MD5 pass already the `CompleteMultipartUpload` adds one compression per recorded subtree for the digest and one per block for the consistency check, a few tens of thousands at most for the largest uploads, which is milliseconds. The re-hash path reads the affected parts back through the plaintext opener, which decrypts in an encrypting deployment. A part's local copy is removed as soon as the part is parked on its provider, so the re-hash fetches the part's blobs from the storage node with ranged retrievals, the same path a `GET` takes. The work is bounded by the final part plus the budget, about 10 GiB of retrieval at most with the default, so `Complete` on such an upload takes as long as fetching that much from the provider, under a minute and a half at a gigabit per second. A variable-size parallel upload beyond the bound is refused, as described above. -The manifest grows by the block list, 128 KiB at 1 GiB and at most 1 MiB, plus the 34-byte multihash and a few bytes of framing. The part record grows by its tree material for the life of the upload: a few KiB for parts of a few MiB, up to about 320 KiB for a 5 GiB part, nearly all of it block CVs. `Complete` loads every part's record, so the largest upload S3 allows, 10,000 parts of 5 GiB, holds about 3 GiB of records in memory while it completes. +The manifest grows by the block list, 128 KiB at 1 GiB and at most 1 MiB, plus the 34-byte multihash and a few bytes of framing. The part record grows by its tree material for the life of the upload: a few KiB for parts of a few MiB, at most about 320 KiB for a 5 GiB part, nearly all of it block CVs, and less the further into the object the part sits. `Complete` loads every part's tree material, about 6 MiB for the largest upload S3 allows, 10,000 parts of 5 GiB, and about 4 MiB for a 1 TiB upload in 10,000 parts. ## Alternatives considered