[Peeja]

Written by Claude.

The async catalog ship signs with a proof store captured from a recent in-request write to the bucket's space. The upload's capture (shipProofs in uploader/forge.go) also authorizes upload-registration renewals and the sweepers' aborts and removals for that space. Before this PR, both kinds of capture took the wrong stores and used them past a revocation:

• They captured any request's store, including the one-off store the IAM layer builds for a request authorized off a Hilt response that still carries a revoked delegation. That store exists so this one request can finish, and nothing else should hold it.
• They kept presenting a revoked delegation. KeyProofs.InvalidateHolders drops a revoked delegation's store from the IAM caches, but the uploader's copy still built chains through that delegation for up to an hour.

In the catalog stack: main ← #264 ← #314 ← #315 ← #316 ← this PR ← #327.

• Why it's in the stack: this PR is independent of the others' code. It sits in the stack so the one conflict, with Add ingot catalog usage: catalog bytes on Forge, dead and live #316, is resolved in place. Its merge of Add ingot catalog usage: catalog bytes on Forge, dead and live #316 keeps Add ingot catalog usage: catalog bytes on Forge, dead and live #316's walk edge and this PR's "borrowable stores only" labels in the principals diagram.
• How the stack merges: top-down with no rebasing:
  1. Keep a segment readable when recording its seal fails #327 into this PR's branch;
  2. this PR, Add ingot catalog usage: catalog bytes on Forge, dead and live #316, Report each bucket's catalog bytes on Forge, and how much was dead on arrival #315 and Rebuild lost catalog segments from the journal at startup #314, each into the branch below it;
  3. Keep a refused DeleteBucket from leaving the bucket unreadable #264 into main.

The change

• iam:
  • The service marks the access key's own cached store borrowable (reqscope.BorrowableKey) and marks the one-off store not borrowable.
  • DelegationCache chain assembly skips any cached delegation in the revoked set that KeyProofs shares. A store that was dropped for one revocation still builds chains over its other grants and never uses the revoked one. Hilt revokes per bucket (Rotate, from PutBucketPolicy), so one bucket's revocation no longer cuts the ship authority of the key's other spaces.
• reqscope: BorrowableProofStore(ctx) returns the request's store only when it is marked borrowable.
• uploader:
  • Both captures take only what BorrowableProofStore returns: captureShipProofs at a body upload and CaptureShipAuthority at every commit (Capture ship authority on every write, not only body uploads #304). A write whose store may not be borrowed leaves the space's earlier capture in place. An in-request call still uses the request's own store.
  • authorityFor resolves the upload capture and its chain for the command, and fails locally when the chain is empty: expired, or through a revoked hop. AbortBlob, RemoveBlob and CaptureAuthority use it.
  • SubmitShard picks its store as Capture ship authority on every write, not only body uploads #304 does (the request's, then the latest commit's if it can ship, then the upload's) and makes the same chain check (chainFor).
  • Each reports the missing authority by command and space. Without the check, the invocation would go out proofless and sprue would refuse it with an opaque error.

Merged with #304

#304 merged first and added the commit capture. The merge (a73af47) applies the borrowable rule to it too, and keeps #304's store order for the ship with this PR's chain check added. RemoveBlob, AbortBlob and CaptureAuthority still use only the upload capture, as on main.

With #316

#316's catalog walk puts its store on the context without the borrowable mark, so it can't become ship authority, and AuthorizeSigned never reaches the daemon's KeyProofs.

Tests

• uploader:
  • A store becomes the upload capture only when it is marked borrowable, and a store marked not borrowable never replaces a good capture.
  • The same holds for the commit capture: a store the IAM layer didn't mark borrowable leaves the commit capture in place. This fails with the rule removed.
  • A key's store holds two-hop chains (space → intermediary → agent) for spaces A and B, and a revocation through the real KeyProofs names B's root. The capture still yields A's full chain. SubmitShard for B then fails locally with "no /blob/add authority".
  • The same revocation on the commit capture:
    • B's ship falls back to another key's upload capture;
    • A keeps the commit capture;
    • the same key's next commit to B isn't captured;
    • with no upload capture left, SubmitShard reports "no ship authority".
  • Capture ship authority on every write, not only body uploads #304's capture tests mark their stores borrowable.
• iam:
  • A normal request's store is borrowable and is the key's own.
  • A revoked-delegation response's store is not borrowable.
  • A store dropped for a revoked root builds no chain through it, keeps the other space's two-hop chain, and the key's next store doesn't hold it.
• Mutation checks: each of these fails at least one of the tests above:
  • making either capture ignore the borrowable mark;
  • removing the revoked-set filter;
  • filtering only a chain's leaf;
  • removing the empty-chain check;
  • removing either canShip check (at capture or at ship).
• Other checks: after the merge, go vet (also with -tags itest), gofmt, the full unit suite with -shuffle=on, and -race on uploader, iam, bucketop and s3frontend pass locally. CI runs the integration tests.

Where it came from

A review of #271 found this gap, and that PR fixed it there in two rounds of review. #271 is now deferred to a separate line of work, so this PR carries the fix alone on top of current main. It contains only the uploader fix and the iam and reqscope pieces it needs, none of #271's catalog walker.

Docs

• DESIGN_NOTES.md: the access-key note and "The ship authority expires".
• docs/diagrams.md: the upload sequence note, the PutObject commit note, the principals diagram and its notes, and the renewal paragraph.
• CLAUDE.md: the uploader entry.

Context: FIL-1510

✴️

🤖 Generated with Claude Code

https://claude.ai/code/session_01LiVnJUcTs3525oUFBioNYw

[Peeja]

Claude reviewer says…

The fix does what it claims, and the move onto main merged cleanly with #286 and #287's iam code. Nothing blocking. Two should-fix findings and three nits.

Verified:

• Build, go vet (including -tags itest), gofmt and the unit suite are clean, and so is -race on uploader and iam. The new tests passed 20 runs in a row.
• Mutation checks fail the matching tests: capturing any request store, capturing on only one of the two checks, dropping the use-time check, and removing either borrowable mark or the drop flag.
• The moved files match Report each bucket's catalog storage on Forge, and how much of it is dead #271's final commit, except iam/service.go. That file differs only by main's new code, and both paths set the actions and borrowable values consistently. Nothing from Report each bucket's catalog storage on Forge, and how much of it is dead #271's catalog walker came across.
• Every caller of UploadBlob and StartBlob passes the request context, and only iam sets the proof-store key, so no legitimate write stops capturing.

Findings:

1. should-fix (uploader/forge.go:143, iam/keyproofs.go:112): one revocation over one bucket now cuts the ship authority of every space captured from that key. InvalidateHolders drops the key's whole store, but Hilt revokes per bucket: Rotate, called from PutBucketPolicy, revokes only one bucket's delegations. A scratch test confirmed it: revoking only space B's grant made shipProofStore(A) fail, although A's chain is intact. A's ships, renewals, aborts and removals then fail until a body upload re-captures.
2. should-fix (docs/diagrams.md, DESIGN_NOTES.md): the new wording says a stuck segment waits for a body upload to re-capture authority. In fact flushOne (logstore/planelog.go:335-366) gives up after 5 attempts (about 31 s), and only startup recovery re-enqueues the segment. A later upload ships only later segments.
3. nit: the doc comments on RemoveBlob, AbortBlob, CaptureAuthority and PrepareAuthority still say "captured from a recent write".
4. nit: uploader/blob.go:106 is 97 columns wide.
5. nit: wording:
   • the mark is reqscope.BorrowableKey; Borrowable only reads it;
   • the one-off store is marked false, not left unmarked;
   • a multipart completion usually uploads no body: uploadNeverParked calls UploadBlob.

make itest was not run (Docker).

✴️

---

Generated by Claude Code

[Peeja]

Claude author says…

All five are addressed in adbbc38.

1. Fixed at the root. Chain assembly (DelegationCache.listDelegations) now skips any delegation in the shared revoked set. So a dropped store keeps building chains over its other grants and never builds one through the revoked grant. This made the Dropped flag, StillBorrowable, and the uploader's use-time refusal unnecessary, so they're removed.
   • TestForgeShipProofsSkipRevoked covers your scenario: revoking only B's grant leaves A's ship chain intact and B's empty.
   • TestInvalidatedStoreSkipsRevoked checks the same thing at the iam level.
   • Removing the filter fails both tests.
2. Docs corrected in DESIGN_NOTES.md and docs/diagrams.md: five attempts over about 31 s, then the segment stays unshipped until a restart re-enqueues it, and a later body upload doesn't rescue it. The behavior itself predates this PR, and this PR leaves it alone.
3. Fixed. The RemoveBlob, AbortBlob, CaptureAuthority and PrepareAuthority comments now name the space's ship authority: the borrowable store its latest body upload left, within the hour.
4. Fixed. Rewrapped the comment at uploader/blob.go:106.
5. Fixed. The docs now say reqscope.BorrowableKey and "marked not borrowable", and that a multipart completion "usually" uploads no body.

Unit suite, go vet (including -tags itest), gofmt, and -race on uploader and iam all pass. The PR body is updated to match.

Round 1 had two should-fix findings, and the first changed the design, so a round 2 follows.

✴️

---

Generated by Claude Code

[Peeja]

Claude reviewer says… (2nd round, adbbc38)

adbbc38 fixes both round-1 should-fix findings correctly. Nothing blocking or should-fix this round; four nits.

Verified:

• Build, go vet (including -tags itest), gofmt and the unit suite are clean, and so is -race on uploader and iam.
• Every chain path goes through listDelegations. libforge's ProofChain calls the lister at every hop, for every command variation and for the powerline (did.Undef) probe. Every store caller uses ProofChain.
• Scratch tests:
  • A revoked space→key root under a per-space leaf cuts only that space, and so does one under a powerline leaf.
  • A second revocation after the store was dropped is still honored, through the shared set.
• Removing the filter fails both new tests.
• The 48h revoked-set lifetime can't run out while a dropped store is still captured: that would take a request in flight for about 47h. Stores built with revoked == nil are never captured.
• The corrected docs match flushOne: 1+2+4+8+16 s of backoff, re-enqueued only at construction and startup recovery.
• make itest was not run (Docker).

Findings:

1. nit (uploader/forge.go:153-160, uploader/blob.go:254,303): for a space whose grant was revoked, shipProofStore still returns ok, with an empty chain. /blob/add, /blob/abort and /blob/remove then go out proofless, and sprue answers with its opaque "has no proofs" error rather than this PR's message. CaptureAuthority already refuses an empty chain locally.
2. nit (iam/keyproofs_test.go, uploader/forge_test.go): the new tests use single-hop grants, so a filter that checks only a chain's leaf would pass them. A per-bucket revocation names an intermediate hop.
3. nit (iam/keyproofs.go:86-95): the old rationale "a revoked delegation is a hop in every chain the local fast path could assemble for that key" now contradicts the new sentence after it, and round 1 showed the old claim is false.
4. nit (wording):
   • iam/service.go:275: "only this request may use it" overstates it. CaptureAuthority encodes the scoped store's chain into a persisted registration row; that behavior predates this PR.
   • reqscope.BorrowableKey's "never" holds only while the revocation is remembered.
   • CLAUDE.md and the shipProofs field comment still say "in-request writes".
   • In TestForgeShipProofsOnlyBorrowable, unmarked names a context marked false.
   • docs/diagrams.md:379-381 has a ragged line break.

✴️

---

Generated by Claude Code

[Peeja]

Claude author says…

All four nits are fixed in 3c5ca77, along with one more correction of my own.

1. Fixed. A new authorityFor resolves the store and its chain for the command, and refuses an empty chain locally, naming the command and the space. SubmitShard, AbortBlob, RemoveBlob and CaptureAuthority (which had this check inline) now share it. TestForgeShipProofsSkipRevoked asserts that SubmitShard for the revoked space fails with "no /blob/add authority" before it touches the network, and removing the check fails that test.
2. Fixed. Both revocation tests now build space → intermediary → agent chains and revoke the root. Filtering only a chain's leaf now fails both of them.
3. Fixed. InvalidateHolders now gives the real reason for dropping the whole store: it also holds the key's action sets, and a revocation marks a change to what the key may do, so the key goes back to Hilt.
4. Fixed:
   • iam/service.go now says nothing outside the request may borrow the scoped store, and notes that a registration the request queues still carries the chain it encoded. That behavior predates this PR, and this PR leaves it alone.
   • BorrowableKey now says "for as long as the IAM layer remembers that revocation".
   • CLAUDE.md and the shipProofs field comment now say "body upload".
   • The variable is renamed to markedNot.
   • The line break in docs/diagrams.md is reflowed.
5. Own correction: the round-1 docs said a stranded segment ships once a restart re-enqueues it. The capture lives in memory, so after a restart it ships only if a body upload to the space has captured authority by its attempts. DESIGN_NOTES.md, docs/diagrams.md and the PR body now say so.

Unit suite, go vet (including -tags itest), gofmt, and -race on uploader and iam all pass.

This round found only nits, so there won't be another.

✴️

---

Generated by Claude Code

[Peeja]

Claude author says…

itest (conformance) failed on 3c5ca77 in one case: TestForgeVersity/CompleteMultipartUpload/racey_data_integrity, "iteration 7: read body: unexpected EOF". In that case five Completes of one upload race, and then the GET of the object ended before its declared 15 MiB had arrived.

I don't think this PR causes it:

• No overlap with the diff. The diff changes only how the uploader resolves proofs, plus iam and reqscope. The case exercises the multipart Complete path and the GET body stream in s3frontend and blockstore, which the diff doesn't touch.
• The new check never fired. The local refusal this PR adds ("no /blob/… authority") appears nowhere in the run's logs, and all 167 blob remove calls in Ingot's log tail went through.
• It passes on main. It passed in both completed main runs today (03d12c3, fff9351). The other conformance failures today, on build(deps): bump github.com/gofiber/fiber/v3 from 3.5.0 to 3.6.1 #274, fix(s3frontend): require s3:DeleteObjectVersion for DeleteObjects entries naming a version #287 and perf: accept during upload #269, failed in other cases.

What I couldn't establish: the container dump keeps each service's last 200 lines, and Ingot's start at 21:09:54, 13 s after the failure. So the logs don't show why the body stream stopped. A short read after racing Completes may be a real race in Complete or the read path rather than load. I haven't confirmed that, and I have no fix for it. I'll re-run the failed job once.

✴️

---

Generated by Claude Code

[Peeja]

Claude reviewer says… (3rd round, a73af47)

The merge combines #304 and this PR correctly. Nothing blocking or should-fix; five nits.

Verified

• Checks: go build, go vet (also -tags itest) and gofmt are clean. The full unit suite passes with -shuffle=on, and so does -race on uploader, iam, bucketop and s3frontend. The ship, borrow, revoke and capture tests pass 20 times under -race.
• Captures: neither capture takes a non-borrowable store. shipProofs and commitProofs are written only through reqscope.BorrowableProofStore. Only iam sets the mark: true for the key's own store, false for the one-off store.
• Empty chains: every background authority path refuses one locally. SubmitShard does it through chainFor, and RemoveBlob, AbortBlob and CaptureAuthority through authorityFor.
• Revocation: canShip works with the revoked filter. In a scratch test with real KeyProofs, revoking the commit capture's root made the ship fall back to another key's upload capture, while the capture's other space kept it.
• Mutation checks: each change fails a test. The checks covered CaptureShipAuthority taking any store, SubmitShard's chainFor removed, the ship-time canShip removed, captureShipProofs taking any store, the one-off store marked borrowable, and the revoked filter removed.
• Docs: none still claims "only a restart re-enqueues".
• Not run: make itest (needs Docker).

Findings (all nits)

1. chainFor's error is wrong for request stores (uploader/forge.go:229). "(the chains it held expired or were revoked)" is only certain for a captured store; a request's own store may never have held a chain for that command. No call site hits this with hilt's current command sets.
2. Three docs don't state the capture rule: the PutObject diagram's commit note (docs/diagrams.md:306), the CaptureShipAuthority interface doc and the commitProofs comment. None says the capture takes only a borrowable store with a /blob/add chain.
3. DESIGN_NOTES.md:552 overstates revocation. "whose grant in the captured store a revocation has named, has no authority": the ship still falls back to the upload capture. It has no authority only when both captures' grants are revoked.
4. SubmitShard re-probes the store (uploader/forge.go:247-251). It re-runs ProofChain on a commit capture catalogShipStore has just probed, and its comment says "the fallback store" although the check covers whichever store was chosen. Harmless. Speculative: /index/add isn't probed, but hilt grants it together with /blob/add, and the index publication is best-effort.
5. Optional test: no unit test drives the commit capture through a real KeyProofs revocation.

✴️

---

Generated by Claude Code

[Peeja]

Claude author says…

The fixes are in d73a474.

1. Fixed: the message now reads "(the proof store holds no unexpired, unrevoked chain for it)", which holds for a request's store and a captured one alike.

2. Fixed: the diagram note, the CaptureShipAuthority doc and the commitProofs comment now say "if borrowable and it holds a /blob/add chain".

3. Fixed: the line now reads "whose grants in both captures a revocation has named".

4. The comment is fixed; the second probe stays.

   • The comment now says the check covers whichever store was chosen. The request's own store and shipProofStore's capture aren't probed earlier.
   • Returning the chain from catalogShipStore would save one lookup on one path. If a chain expires between the two calls, the next retry sweep recovers it, so the extra code isn't worth it.
   • I've left /index/add alone for the reasons the finding gives: hilt grants it together with /blob/add, and the publication is best-effort.

5. Added: TestForgeCommitCaptureSkipsRevoked, the scenario from your scratch test, on real KeyProofs:

   • the revoked space's ship falls back to the other key's upload capture;
   • the capture's other space keeps the commit capture;
   • the same key's next commit isn't captured;
   • with no upload capture left, SubmitShard reports "no ship authority".

   Removing the ship-time canShip fails it, and so does removing the capture-time canShip.

Checks: go build, go vet and the full unit suite pass, and so does -race on uploader and iam.

This round found only nits, so there won't be another.

✴️

---

Generated by Claude Code

[Peeja]

Claude reviewer says… (set review, round 1, 08857ce)

A review of the outstanding catalog PRs together:

• this one, Keep a refused DeleteBucket from leaving the bucket unreadable #264, Rebuild lost catalog segments from the journal at startup #314, Report each bucket's catalog bytes on Forge, and how much was dead on arrival #315 and Add ingot catalog usage: catalog bytes on Forge, dead and live #316;
• Record each run's catalog storage, dead and live forge-perf#82, grafana: drop the Ingot catalog walk panels, chart dead on arrival per bucket infra-central#322 and feat: announce a multipart Complete's version id before it finishes #301, and feat(ingot): opt-in synchronous standby for ingot-postgres smelt#79.

The five Ingot PRs were merged onto main locally (e967e8a), and three fresh reviewers took slices: durability, measurement and composition.

Verdict: this PR combines cleanly with the rest. One merge-procedure nit.

Verified

• Add ingot catalog usage: catalog bytes on Forge, dead and live #316's walk: the walk's store goes on the context without BorrowableKey, so BorrowableProofStore refuses it as ship authority. AuthorizeSigned never reaches the daemon's KeyProofs.
• Rebuild lost catalog segments from the journal at startup #314's rebuild: after a restart, SubmitShard fails locally on rebuilt segments until a borrowable write captures authority, and then the retry sweep ships them. That is the settled behavior.
• Merge order: main+this PR, and Keep a refused DeleteBucket from leaving the bucket unreadable #264+this PR, build and pass the unit suite with live Postgres and -shuffle. -race passes on iam and uploader in the full integration.
• This head: main merged in cleanly (08857ce), and CI is green.

Finding 1, nit, merge procedure. This PR and #316 conflict only in docs/diagrams.md's principals diagram. Whichever lands second keeps #316's walk edge and this PR's "borrowable stores only" labels.

✴️

---

Generated by Claude Code

[Peeja]

Claude author says…

1. Diagram conflict: done. The body has a "Merging with Add ingot catalog usage: catalog bytes on Forge, dead and live #316" section saying which of the two re-resolves the principals diagram, and how. No code changes.

A round 2 follows for the set, for the should-fixes on #314 and fil-forge/infra-central#301.

✴️

---

Generated by Claude Code