rfc: BLAKE3 object digest by alanshaw · Pull Request #33 · fil-one/RFC

alanshaw · 2026-10-05T21:30:17Z

Summary

This RFC proposes exposing a full-object hash that allows plaintext data to be verified by clients. The full-object hash is a blake3 hash. The reason for Blake3 is that it can be generated at the completion of a multipart upload (CompleteMultipartUpload) without having to re-read and hash each part.

The RFC also proposes storing a reduced Bao outboard for verified streaming (in the format used by Iroh) where the minimum verifiable chunk is proportional to the total size of the object.

Copilot

Copilot review overview

🟡 Changes recommended

Non-aligned multipart boundaries cannot be independently rehashed as specified, preventing correct tree construction.

Review effort: Balanced
Findings: 1 · 3 · 3

Open (7)

Stream across misaligned boundaries when regenerating BLAKE3 nodes · New
Bound completion work by recorded subtrees, not a few hundred merges · New
Handle single non-first-part uploads without a TreeRoot · New
Account for leaf consistency reduction in completion cost · New
Acknowledge S3 full-object checksums and BLAKE3's added role · New
Use ceiling leaf counts and clarify decimal versus binary units · New
Correct the maximum Bao outboard size estimate · New

What changed in this PR

Proposes BLAKE3 object digests and Bao-based range verification for Ingot objects.

Changes:

Defines digest generation for single and multipart uploads.
Adds bounded tree metadata and S3 retrieval mechanisms.
Documents compatibility, costs, alternatives, and evaluation criteria.

File	Description
`rfcs/2026-10-blake3-object-digest.md`	Specifies the BLAKE3 digest and range-verification design.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

bajtos

This is a great experiment 👍🏻

I think the proposal is sound. It would be great to improve clarity; see my comments below.

There are three items that make me hesitant to approve:

What real customer problem are we trying to solve? Is this the right solution? (Can we solve 80% of use cases with something much simpler, like using the existing per-chunk MD5 or adding per-chunk SHA-256?)
The worst-case cost will be very noticeable; we should not ignore it on the grounds that it's unlikely to happen.
The referenced Blake3 implementation in Go has only two maintainers and no commits since May 2025.

bajtos · 2026-10-07T08:11:47Z

+
+## Motivation
+
+An Ingot object has no digest a client can verify against. A single `PUT` records a whole-object SHA-256 in the manifest, but it is internal and never returned. A multipart object records no whole-object digest at all, because SHA-256 cannot be assembled from the parts and re-reading every part at `CompleteMultipartUpload` is too slow. A client that wants to check what it downloaded has the S3 ETag, which for a multipart object is a hash of part MD5s and verifies nothing about the bytes, and the optional S3 checksums, which for multipart are composite values with the same limitation. Nothing lets a client verify a byte range.


I feel these parts are not correct:

An Ingot object has no digest a client can verify against
Nothing lets a client verify a byte range

As long as the client knows MD5 hashes of individual parts it uploaded, it can check that the ETag was constructed correctly from the per-part MD5s, and can use per-part MDF5s to verify that downloaded data chunked along the same boundaries hashes to the same MD5s.

According to the text later, GetObjectAttributes returns a per-part checksum list for a multipart object, which means the client can obtain those hashes from the server and does not need to store them.

A byte range can be verified as long as it matches one of the uploaded parts and the client has access to the MD5 of that part.

In practice, with AWS S3 client's default of 8 MB for the part size, clients can implement streaming verification with 8 MB chunk granularity.

It's not perfect, but it's more than "nothing".

You're right - "nothing" is incorrect. I'll reword.

The part ETags aren't returned after the upload completes, though. S3 returns per-part values only for uploads made with a checksum algorithm, and only for the algorithm chosen: GetObjectAttributes has no per-part ETag, and its per-part MD5 appears only if the uploader picked MD5 as the checksum algorithm, which is the same opt-in as SHA-256. Ingot does the same. A client wanting MD5-based verification would have to have recorded the part MD5s itself at upload time, and in practice S3 clients don't.

So the verifiable value a client has today exists only if the uploader asked for it. The uploader must have chosen a SHA checksum algorithm for the multipart upload, with --checksum-algorithm SHA256 or its SDK equivalent; then GetObjectAttributes returns the per-part SHA-256 list and the composite over it, which is a two-level tree, is cryptographic, and lets a reader verify ranges aligned to the parts. No client does this by default. The AWS SDKs and CLI now send a checksum unprompted, but it is a CRC, which gives a full-object value with no per-part list, but no cryptographic strength; rclone, s5cmd and the MinIO client send none. So the reader depends on an upload-time choice that, as far as I know, nobody makes without being told to. I'll say so in the RFC.

What it does not cover: a single PUT, an upload made without a checksum algorithm, and any range finer than the uploader's part size. The granularity is whatever the uploader chose, which the reader can't rely on. To be fair to it, for large objects that granularity is finer than this RFC's: the blocks here pass an 8 MiB part above 4 GiB and reach 128 MiB at 1 TiB, so a per-part SHA-256 list on a 1 TiB object verifies at 8 MiB where the Bao tree verifies at 128 MiB. What the per-part list can't do at any size is cover an object nobody uploaded with SHA checksums, which is nearly all of them. And MD5 adds no cryptographic assurance on top of any of this.

That makes BLAKE3 the generalisation rather than the first verifiable value: one digest for every object however it was uploaded, a verification granularity the server chooses independent of part size, and verification an off-the-shelf Bao library can do.

bajtos · 2026-10-07T08:47:48Z

+- A client can verify a whole-object read with the digest alone and a standard BLAKE3 implementation.
+- A client can verify a ranged read with an off-the-shelf Bao library by fetching the object's outboard once: under a KiB for a small object, growing with the square root of the size.
+- The S3 data path is unchanged. Bodies are raw bytes, ETags and S3 checksums behave as they do today, and a client that ignores the extension sees nothing new.
+- Manifest growth stays small for ordinary objects and under 1 MiB for the largest. The leaf list grows with the square root of the object size and is capped.


Is "manifest" the same thing as "object's outboard" mentioned earlier (two lines above)? Let's use the same name everywhere, please.

It's not the same - the manifest is the data structure stored at the leaves of the MST. You can think of the outboard as the Merkle proofs (the tree, not leaves) that allow verifications of the whole or part.

bajtos · 2026-10-07T09:37:30Z

+
+Ingest adds one BLAKE3 pass per body or part, alongside the SHA-256 and MD5 passes already there. BLAKE3 with vector instructions is faster than SHA-256, so the added CPU cost is below the two hashes already paid.
+
+`CompleteMultipartUpload` adds a few hundred compressions in the common path. The re-hash path reads the affected parts back through the plaintext opener, which decrypts in an encrypting deployment, and is bounded by the size of the mis-guessed parts.


What's the cost of re-hashing in the worst-case scenario?

If every uploaded part has a random length, we need to rehash all parts except the first one.

Depending on whether we keep the uploaded parts in the spool or stream them directly to Piri node(s), we may also need to retrieve the parts from Piri before we can decrypt them.

You were right about the version you reviewed: random part lengths would have sent everything from part 2 onward to a single re-hash, because an odd-length part put chunk boundaries inside its successors. That is gone. Each part record now carries the raw bytes either side of its whole chunks, under 1 KiB at each end, and Complete hashes the chunk straddling each boundary from them. Part length no longer causes any re-hash, however random, so a sequential upload of variable-size parts assumes every offset exactly and re-reads nothing.

What is left is a wrong assumed offset, which only an out-of-order arrival can cause. In a uniform-size upload that is exactly one part, the final one landing before any predecessor, and it is always re-hashed. Anything else, variable sizes uploaded in parallel, is re-hashed within a budget of 5 GiB by default; past that the object completes with its ETag and bytes but no digest, so Complete never becomes slow, only the digest goes missing. The worst case is therefore the final part plus the budget, about 10 GiB of reading, which is seconds to tens of seconds.

On where the bytes come from: the local spool. A part's blobs stay there until the object is deleted, since nothing evicts spool copies today (that is the open unbounded-spool issue), so a re-hash is a local read plus decryption, never a fetch from piri. If spool eviction is built later, the read tier already falls back to the storage node by digest, so correctness holds and only the cost of that path rises; the budget bounds that too. The cost section now says all of this.

On where the bytes come from: the local spool. A part's blobs stay there until the object is deleted, since nothing evicts spool copies today

FWIW, spool eviction is already implemented in Ingot. See fil-forge/ingot#218 and the related pull requests.

I think it's worth checking if the assumptions made here still hold with the spool eviction in place.

OK this shouldn't be a problem now we keep the head/tail in the DB.

bajtos · 2026-10-07T09:46:59Z

+
+### A fixed 16 KiB block with the outboard as a blob
+
+bao-tree's default gives 16 KiB verification granularity at any size, with the outboard stored as its own blob referenced from the manifest, about 0.4% of the object. It is the right design if fine-grained verified reads of very large objects are a requirement. It adds a blob per object to write, locate and clean up, and a second request to fetch something that is usually megabytes. The bounded manifest tree serves the common case with no new blob and can coexist with this design later: the stored leaves are the top of the finer tree.


and a second request to fetch something that is usually megabytes.

The current proposal still requires the client to send a second request to fetch the outboard via GetAttributes, doesn't it? The difference is that with your proposal, that extra request fetches at most ~1 MiB.

Yes correct. It's the management overhead of a separate blob that I wanted to avoid.

You can, of course generate both the blake3 hash and outboard as the upload happens and keep them, such that you never have to ask the server.

Can we improve this ADR paragraph to more clearly explain what's different about this alternative and why we rejected it? (I.e. the second request is needed in both alternatives; the difference is the response size + the need to manage a separate blob.)

Done in d4fcb8d

alanshaw · 2026-10-07T16:18:35Z

@bajtos

There are three items that make me hesitant to approve:

What real customer problem are we trying to solve? Is this the right solution? (Can we solve 80% of use cases with something much simpler, like using the existing per-chunk MD5 or adding per-chunk SHA-256?)

When I think about product differentiators and real customer problems I'm always brought back to Filecoin. There's plenty of reasons to use Filecoin that I don't think are necessary to restate here, but in this context keep in mind: content addressing (verifiability) and durability. There's also plenty of barriers to entry, which frankly is what we're all working to improve at Fil One.

Of all the things, I keep coming back to that phrase - "your data, verifiably stored on Filecoin". That's the crux of it for me. We're building an easy on-ramp to Filecoin, but none of it really matters if we can't prove the data is being stored on Filecoin. Step 0 of "your data, verifiably stored on Filecoin" is "your data, verifiable". If your data is not verifiable, we can't prove it's on Filecoin, period. Step 0 is what this RFC proposes to fix.

So, yes we could automatically calculate SHA2-256 of each upload part, and yes that might solve 80% of use cases, but that doesn't help us thread the needle of the "verifiable on Filecoin" story in the long run and it really isn't the easiest way to thread it i.e. there's no single hash, the checksums are not CIDs, no sub-part verifiability etc. etc.

The worst-case cost will be very noticeable; we should not ignore it on the grounds that it's unlikely to happen.

Detailed response is here: #33 (comment)

TLDR; no mainstream client sends variable sized parts and all of them align with the 1KiB boundary except a handful of them which do not for uploads >48GB. There's mitigations for both of them (written in the RFC and implemented in Ingot) and both are edge cases and very unlikely.

The referenced Blake3 implementation in Go has only two maintainers and no commits since May 2025.

I addressed this here #33 (comment)

TLDR; I'm not super worried about this - there's multiple other implementations of Blake3 and Bao is a very small thing.

bajtos

I reviewed the new version. Thank you for all the clarifications you made!

Here are the most important topics that need further discussion:

How to handle uploads that exceed the bound?
https://github.com/fil-one/RFC/pull/33/changes/d72ec18a2ac80fd192b1d320e28a57a8422b3149..8c1e8ac3a8491251301238427afa22515e6378df#r4245642249
Memory footprint of many concurrent large-file multipart uploads.
https://github.com/fil-one/RFC/pull/33/changes/d72ec18a2ac80fd192b1d320e28a57a8422b3149..8c1e8ac3a8491251301238427afa22515e6378df#r4245917802
Spool eviction is already implemented; did it break any assumptions made by this RFC?

bajtos · 2026-10-12T07:22:22Z

+
+### A fixed 16 KiB block with the outboard as a blob
+
+bao-tree's default gives 16 KiB verification granularity at any size, with the outboard stored as its own blob referenced from the manifest, about 0.4% of the object. It is the right design if fine-grained verified reads of very large objects are a requirement. It adds a blob per object to write, locate and clean up, and a second request to fetch something that is usually megabytes. The bounded manifest tree serves the common case with no new blob and can coexist with this design later: the stored leaves are the top of the finer tree.


Can we improve this ADR paragraph to more clearly explain what's different about this alternative and why we rejected it? (I.e. the second request is needed in both alternatives; the difference is the response size + the need to manage a separate blob.)

bajtos · 2026-10-12T08:15:33Z

+
+The parts whose assumed offset was wrong are re-hashed at their true offsets by reading their blobs back from the local spool, and the results stand in for their records. The final part is always re-hashed when wrong: in a uniform-size upload it is the only part whose assumption can fail, when it lands before any predecessor and its own size stands in, and the 5 GiB part maximum bounds the cost. The other parts are re-hashed within a **re-hash budget**, one maximum part, 5 GiB, by default, and configurable. An upload whose wrong non-final parts add up to more than the budget commits without a digest: the object's ETag is returned as always, GET and HEAD carry no `x-cid`, and `GetObjectAttributes` omits the `Blake3` element, the state defined below for an object without a digest. The budget keeps `Complete` to seconds of local reading at most. Exceeding it is an extreme edge case. No mainstream client varies its part size within an upload, the final part aside, and a uniform-size upload assumes every offset exactly in any arrival order. A sequential upload assumes every offset exactly whatever the sizes. What is left is a hand-rolled uploader sending variable-size parts in parallel, or a parallel compose-by-copy of differently sized source objects, and even there the cost is the absence of a digest, never a failed upload.
+
+A chunk that straddles a part boundary is rebuilt from the earlier part's `TreeTail` and the later part's `TreeHead` and compressed at its chunk index, one compression per boundary. The last part's `TreeTail` is the object's final chunk and is compressed as such. Part lengths that are not 1 KiB multiples therefore cost nothing.


Part lengths that are not 1 KiB multiples therefore cost nothing.

I feel "cost nothing" is an exaggeration. Can we say "add negligible overhead" instead?

One aspect to consider:

We need to keep tails & heads for each part that's not aligned to 1 KiB multiples. In the worst case, that's N-1 KiB, where N is the number of parts.

the AWS SDKs for Go and Java above the 10,000-part limit, objects over about 48 GiB, where they size parts as the object size divided by 10,000, an arbitrary byte count; those uploads are a small minority, and these two spans are what let them complete without re-reading anything.

In this case, we will need to keep 9,999 KiB of object bytes in tree heads & tails in memory; i.e. 9.76 MiB per concurrent large-object upload.

I am wondering how quickly this will become a problem. 100 concurrent large-object uploads require Ingot to have more than 1GiB of RAM - that seems okay on any reasonably sized bare-metal machine?

A potential fix is to read tree head & tail bytes from the spool, and change the in-memory structure to hold only the information needed to locate the head & tail bytes in the spooled files. (Byte-length may be all we need.)

If we decide to keep head & tail bytes in memory, then I propose to capture this information somewhere in the RFC or Ingot documentation, since it affects the machine-sizing calculations.

I am wondering how quickly this will become a problem. 100 concurrent large-object uploads require Ingot to have more than 1GiB of RAM - that seems okay on any reasonably sized bare-metal machine?

I would love to have 100 concurrent, >48GiB uploads...but yeah 1GiB is fine I think.

A potential fix is to read tree head & tail bytes from the spool

So yes, this is an option right now. However, I don't know how much longer all parts will be available in the spool for reading. It requires our spool to be at least as big as the biggest object that can be uploaded, so I suspect we'll need to offload blobs to the network sooner than later...and then it becomes (9,999-n)x network requests, were n is the number of blobs offloaded.

Correction, 10,000 5GB parts will actually load ~3GB of data into memory. Again, I think this is ok. The box will be operating with much more memory than that and is only for an instant. We can stream the data if necessary.

Yeah, ok I said I thought it was ok but it has been playing on my mind and have implemented a fix.

Basically we compute and store the block CVs needed for the size of the upload. It means that at the start we compute very granularly, but as we learn how big the upload is, we compute less granularly. We basically significantly reduce the amount of data we store from ~3GiB to ~8MiB for the biggest object.

Documented in a331a30

Addressed in fil-forge/ingot@2bc35ec

bajtos · 2026-10-12T09:54:41Z

 ```

-Objects written before this change have none of these fields. A read of such an object returns no `x-cid` header and `GetObjectAttributes` omits the `Blake3` element, so a client can tell the difference between an object without a digest and a server without the feature.
+Some objects have no digest: those written before this change, and a multipart object whose completion exceeded the re-hash budget. For them, GET and HEAD return no `x-cid` header and `GetObjectAttributes` returns its response without the `Blake3` element.


If we decide to reject multipart objects whose completion exceeds the re-hash budget, then this paragraph will need updating.

bajtos · 2026-10-12T09:56:18Z

+Ingest adds one BLAKE3 pass per body or part, alongside the MD5 pass already there. BLAKE3 with vector instructions is faster than SHA-256, so it costs less than the whole-body SHA-256 pass it makes redundant.

-`CompleteMultipartUpload` adds a few hundred compressions in the common path. The re-hash path reads the affected parts back through the plaintext opener, which decrypts in an encrypting deployment, and is bounded by the size of the mis-guessed parts.
+`CompleteMultipartUpload` adds one compression per recorded subtree for the digest and one per block for the consistency check, a few tens of thousands at most for the largest uploads, which is milliseconds. The re-hash path reads the affected parts back through the plaintext opener, which decrypts in an encrypting deployment. The parts are read from the local spool: a part's blobs stay there until the object is deleted, since nothing evicts spool copies today, so no re-hash fetches from a storage node. The work is bounded by the final part plus the budget, about 10 GiB of local reading at most with the default, which is seconds to tens of seconds. The price of the bound is that a variable-size parallel upload beyond it gets no digest.


a part's blobs stay there until the object is deleted, since nothing evicts spool copies today

As mentioned elsewhere, the spool eviction is implemented now.

Thanks, this was already corrected after the spool eviction landed. The cost paragraph now says a part's local copy is removed once it is parked, so the re-hash fetches the part's blobs from the storage node with ranged retrievals, about 10 GiB at most with the default budget.

bajtos · 2026-10-12T09:59:32Z

+S3 already defines this for uploads made with `--checksum-algorithm SHA256`: a SHA-256 per part and a composite over the list, which `GetObjectAttributes` returns. The alternative is to compute that for every multipart upload, server side, whatever the client asked for, and treat the composite as the object's digest. It needs no new primitive, and for a client that already understands S3 composite checksums it is familiar. It falls short of this RFC in several ways.
+
+- **The root is not a hash of the bytes.** A composite depends on where the part boundaries fell, so the same bytes uploaded with 8 MiB parts, with 16 MiB parts, or as a single `PUT` get three different roots. It identifies an upload, not an object: two copies of the same data cannot be recognised as equal, nothing can be addressed by it, and a single-`PUT` object has a plain SHA-256 while a multipart object has a hash of hashes, so a client needs two code paths and two meanings for "the digest".
+- **Verifying a download takes a list, not a value.** The verifier must fetch the per-part hash list and the part sizes, up to 10,000 of each, keep them for the whole download, and apply the right one to the right byte range. The BLAKE3 digest is one 32-byte value that a standard library verifies against a stream. Walking a 10,000-entry list is custom code in every client, and the list itself is hundreds of KiB of XML on every fetch, with no compact form.


Well, if we want to be precise, Blake3 byte-range verification requires a list too (the outboard). The other arguments in this paragraph are sound 👍🏻

bajtos · 2026-10-12T10:03:42Z

+
+- **The root is not a hash of the bytes.** A composite depends on where the part boundaries fell, so the same bytes uploaded with 8 MiB parts, with 16 MiB parts, or as a single `PUT` get three different roots. It identifies an upload, not an object: two copies of the same data cannot be recognised as equal, nothing can be addressed by it, and a single-`PUT` object has a plain SHA-256 while a multipart object has a hash of hashes, so a client needs two code paths and two meanings for "the digest".
+- **Verifying a download takes a list, not a value.** The verifier must fetch the per-part hash list and the part sizes, up to 10,000 of each, keep them for the whole download, and apply the right one to the right byte range. The BLAKE3 digest is one 32-byte value that a standard library verifies against a stream. Walking a 10,000-entry list is custom code in every client, and the list itself is hundreds of KiB of XML on every fetch, with no compact form.
+- **The granularity is the uploader's, and often too coarse.** Verification happens per part, so a client cannot check anything until it has a whole part, 5 GiB on large uploads, and a single-`PUT` object cannot be range-verified at all. The Bao tree verifies 16 KiB on objects up to 8 MiB, and a server-chosen block on large ones, with proofs that are logarithmic in the object rather than linear in the part count.


The granularity is the uploader's, and often too coarse.

Often too coarse - is that true? For large files, your proposal is the one with coarser granularity because you cap the outboard size.

Suggestions:

"often too coarse for objects under 1GB" (replace 1GB with the size when your proposal becomes more coarse than AWS client default behaviour).

remove ", and often too coarse"

Good challenge, and the numbers come out the other way: the tree is finer than the parts of any multipart upload of the same object, at every size. The RFC's earlier comparison (8 MiB parts against 32 MiB blocks at 1 TiB) was wrong, because a 1 TiB object can't be uploaded in 8 MiB parts. S3 allows at most 10,000 parts, so a part is at least the larger of 5 MiB and the object size divided by 10,000.

Object size Size ÷ 10,000 Smallest legal part Smallest verifiable block

1 KiB under 1 B 1 KiB 1 KiB, the whole object

64 KiB 6 B 64 KiB 16 KiB

1 MiB 104 B 1 MiB 16 KiB

5 MiB 524 B 5 MiB 16 KiB

8 MiB 838 B 5 MiB 16 KiB

16 MiB 1.6 KiB 5 MiB 32 KiB

100 MiB 10 KiB 5 MiB 64 KiB

1 GiB 105 KiB 5 MiB 256 KiB

5 GiB 524 KiB 5 MiB 512 KiB

10 GiB 1 MiB 5 MiB 1 MiB

32 GiB 3.3 MiB 5 MiB 1 MiB

50 GiB 5.1 MiB 5.1 MiB 2 MiB

64 GiB 6.6 MiB 6.6 MiB 2 MiB

128 GiB 13 MiB 13 MiB 4 MiB

256 GiB 26 MiB 26 MiB 8 MiB

1 TiB 105 MiB 105 MiB 32 MiB

5 TiB 524 MiB 524 MiB 256 MiB

48.8 TiB 5 GiB 5 GiB 2 GiB

Below 50 GiB the 5 MiB part minimum is what limits the parts; above it, the 10,000-part cap does. The block is at least 2.5 times smaller than the smallest legal part throughout, closest at 5 TiB. That holds by construction: the block is at most 1 MiB up to 32 GiB, at most 4 MiB up to 128 GiB, and under the object size divided by 16,384 beyond, because the 32,768-block cap is wider than the 10,000-part cap.

So the bullet now reads "The granularity is the uploader's, and coarser than the tree's", with the part-size limits stated, and the smallest-range paragraph in the block section carries the same comparison with the 1 TiB example (parts average at least 105 MiB, blocks are 32 MiB).

bajtos · 2026-10-12T10:39:23Z

 ### A fixed 16 KiB block with the outboard as a blob

-bao-tree's default gives 16 KiB verification granularity at any size, with the outboard stored as its own blob referenced from the manifest, about 0.4% of the object. It is the right design if fine-grained verified reads of very large objects are a requirement. It adds a blob per object to write, locate and clean up, and a second request to fetch something that is usually megabytes. The bounded manifest tree serves the common case with no new blob and can coexist with this design later: the stored leaves are the top of the finer tree.
+bao-tree's default gives 16 KiB verification granularity at any size, with the outboard stored as its own blob referenced from the manifest, about 0.4% of the object. It is the right design if fine-grained verified reads of very large objects are a requirement. It adds a blob per object to write, locate and clean up. The client's extra request is the same either way, but it fetches megabytes instead of KiB. The bounded manifest tree serves the common case with no new blob and can coexist with this design later: the stored block CVs are the top of the finer tree.


The client's extra request is the same either way, but it fetches megabytes instead of KiB.

I like this phrasing 👏🏻

* Adds `x-cid` to `GetObject` requests. * Adds `<Blake3/>` to `GetObjectAttributes`. Please read the RFC for details: fil-one/RFC#33 Depends on: * fil-forge/versitygw#16 resolves #226 --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

rfc: BLAKE3 object digest

d036bbd

alanshaw requested a review from a team October 5, 2026 21:31

refactor: adjust group sizing

b7908f1

alanshaw mentioned this pull request Oct 6, 2026

feat: add Blake3 object attribute extension fil-forge/versitygw#16

Merged

alanshaw added 2 commits October 6, 2026 11:01

refactor: encode outboard in standard form

f9cce39

refactor: a list for easier reading

96e2bed

alanshaw mentioned this pull request Oct 6, 2026

feat: BLAKE3 object digest fil-forge/ingot#225

Merged

refactor: rename group to chunk log

d72ec18

alanshaw mentioned this pull request Oct 6, 2026

Give every object a verifiable whole-object digest, with incremental and random-access verification fil-forge/ingot#226

Closed

alanshaw requested a balanced review from Copilot October 7, 2026 09:31

Copilot started reviewing on behalf of alanshaw October 7, 2026 09:32

View session

Copilot AI reviewed Oct 7, 2026

View reviewed changes

bajtos reviewed Oct 7, 2026

View reviewed changes

alanshaw added 6 commits October 7, 2026 11:06

refactor: address review comments

645a675

refactor: address review comments

224c486

refactor: address review comments

222970a

refactor: add alternative

4d3c549

refactor: address review comments

178f324

refactor: tweak min verifiable block size

8c1e8ac

bajtos reviewed Oct 12, 2026

View reviewed changes

alanshaw added 8 commits October 12, 2026 12:32

refactor: fail after budget

2bdc03a

refactor: take suggested opening

466a37a

refactor: add more justification for outboard as separate blob

d4fcb8d

refactor: clarify in memory sizes

e0f7393

refactor: clarify part number and offset

899eb47

refactor: gfm maths

b5922f8

refactor: address pr comments

a72b41b

refactor: store block CVs at granularity needed for size of upload

a331a30

alanshaw merged commit f8374b5 into main Oct 13, 2026
2 checks passed

alanshaw deleted the ash/rfc/blake3-object-digest branch October 13, 2026 11:45


		## Motivation

		An Ingot object has no digest a client can verify against. A single `PUT` records a whole-object SHA-256 in the manifest, but it is internal and never returned. A multipart object records no whole-object digest at all, because SHA-256 cannot be assembled from the parts and re-reading every part at `CompleteMultipartUpload` is too slow. A client that wants to check what it downloaded has the S3 ETag, which for a multipart object is a hash of part MD5s and verifies nothing about the bytes, and the optional S3 checksums, which for multipart are composite values with the same limitation. Nothing lets a client verify a byte range.


		Ingest adds one BLAKE3 pass per body or part, alongside the SHA-256 and MD5 passes already there. BLAKE3 with vector instructions is faster than SHA-256, so the added CPU cost is below the two hashes already paid.

		`CompleteMultipartUpload` adds a few hundred compressions in the common path. The re-hash path reads the affected parts back through the plaintext opener, which decrypts in an encrypting deployment, and is bounded by the size of the mis-guessed parts.


		### A fixed 16 KiB block with the outboard as a blob

		bao-tree's default gives 16 KiB verification granularity at any size, with the outboard stored as its own blob referenced from the manifest, about 0.4% of the object. It is the right design if fine-grained verified reads of very large objects are a requirement. It adds a blob per object to write, locate and clean up, and a second request to fetch something that is usually megabytes. The bounded manifest tree serves the common case with no new blob and can coexist with this design later: the stored leaves are the top of the finer tree.


		The parts whose assumed offset was wrong are re-hashed at their true offsets by reading their blobs back from the local spool, and the results stand in for their records. The final part is always re-hashed when wrong: in a uniform-size upload it is the only part whose assumption can fail, when it lands before any predecessor and its own size stands in, and the 5 GiB part maximum bounds the cost. The other parts are re-hashed within a re-hash budget, one maximum part, 5 GiB, by default, and configurable. An upload whose wrong non-final parts add up to more than the budget commits without a digest: the object's ETag is returned as always, GET and HEAD carry no `x-cid`, and `GetObjectAttributes` omits the `Blake3` element, the state defined below for an object without a digest. The budget keeps `Complete` to seconds of local reading at most. Exceeding it is an extreme edge case. No mainstream client varies its part size within an upload, the final part aside, and a uniform-size upload assumes every offset exactly in any arrival order. A sequential upload assumes every offset exactly whatever the sizes. What is left is a hand-rolled uploader sending variable-size parts in parallel, or a parallel compose-by-copy of differently sized source objects, and even there the cost is the absence of a digest, never a failed upload.

		A chunk that straddles a part boundary is rebuilt from the earlier part's `TreeTail` and the later part's `TreeHead` and compressed at its chunk index, one compression per boundary. The last part's `TreeTail` is the object's final chunk and is compressed as such. Part lengths that are not 1 KiB multiples therefore cost nothing.

Object size	Size ÷ 10,000	Smallest legal part	Smallest verifiable block
1 KiB	under 1 B	1 KiB	1 KiB, the whole object
64 KiB	6 B	64 KiB	16 KiB
1 MiB	104 B	1 MiB	16 KiB
5 MiB	524 B	5 MiB	16 KiB
8 MiB	838 B	5 MiB	16 KiB
16 MiB	1.6 KiB	5 MiB	32 KiB
100 MiB	10 KiB	5 MiB	64 KiB
1 GiB	105 KiB	5 MiB	256 KiB
5 GiB	524 KiB	5 MiB	512 KiB
10 GiB	1 MiB	5 MiB	1 MiB
32 GiB	3.3 MiB	5 MiB	1 MiB
50 GiB	5.1 MiB	5.1 MiB	2 MiB
64 GiB	6.6 MiB	6.6 MiB	2 MiB
128 GiB	13 MiB	13 MiB	4 MiB
256 GiB	26 MiB	26 MiB	8 MiB
1 TiB	105 MiB	105 MiB	32 MiB
5 TiB	524 MiB	524 MiB	256 MiB
48.8 TiB	5 GiB	5 GiB	2 GiB

Conversation

alanshaw commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bajtos left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alanshaw Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alanshaw commented Oct 7, 2026

Uh oh!

bajtos left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alanshaw commented Oct 5, 2026 •

edited

Loading

alanshaw Oct 7, 2026 •

edited

Loading