rfc: dynamic routing candidates

[alanshaw]

📖 Preview

Summary

Proposal for an alternative to the presupposed "static" affinity routing where ALL writes to a given space are routed to a specific storage node.

[bajtos]

LGTM!

[parkan]

I am generally in favor of minimizing central state tracking as much as possible but I'm a bit anxious that this could lead to e.g. dropping multiparts on the floor without an easy way for anyone to find them after clients think everything is ACK'd = pushes dealing with that to Ingot/Guppy actual S3 client impls? if that's acceptable 👍 but I feel like it doesn't quite match S3 semantics

(ref #3 Async-ACK under https://github.com/fil-one/RFC/blob/main/rfcs/2026-05-filone-forge-deployment-proposal.md#the-upload-path)

[alanshaw]

I'm a bit anxious that this could lead to e.g. dropping multiparts on the floor without an easy way for anyone to find them after clients think everything is ACK'd = pushes dealing with that to Ingot/Guppy actual S3 client impls?

I'm not sure I follow...

[parkan]

I'm a bit anxious that this could lead to e.g. dropping multiparts on the floor without an easy way for anyone to find them after clients think everything is ACK'd = pushes dealing with that to Ingot/Guppy actual S3 client impls?

I'm not sure I follow...

consider this situation:

• client does a multi-part upload with dynamic affinity, ACK'd at S3 boundary
• node crashes/restarts partway through processing the multipart pieces (i.e. before accept on all chunks)
• if the client (or something) doesn't keep add task state until everything is actually accepted not just ACK'd -> dangling blob(s) that needs reconciliation and doesn't have a receipt that links it to a provider
• also what happens if the affinity set changes upon retry?

I might be misunderstanding the receipt retention design? (but the way I'm reading it another problem is that all receipts need to be retained for entire blob lifecycle as the only authoritative source of truth, otherwise removes would break -- the receipt chain + indexer design also triggering some IPNI PTSD but that's neither here nor there)

[parkan]

this appears to now be superseded by #20 (I will re-raise the above concern there)

[alanshaw]

consider this situation:

• client does a multi-part upload with dynamic affinity, ACK'd at S3 boundary
• node crashes/restarts partway through processing the multipart pieces (i.e. before accept on all chunks)
• if the client (or something) doesn't keep add task state until everything is actually accepted not just ACK'd -> dangling blob(s) that needs reconciliation and doesn't have a receipt that links it to a provider
• also what happens if the affinity set changes upon retry?

When we accept, we look up the allocation task receipt (referenced by the /http/put task receipt), so we know which node the data was uploaded to. So it doesn't matter if half the multipart blobs go to one node and half to another, when the affinity set is changed - it's only a problem if the first node were to disappear from the network.

[parkan]

right, the receipt chain pins placement per-blob, so mixed placement across affinity changes is fine -- so retries are ok

what I'm more worried about is the crash window: lookup at accept works when things are still churning. if ingot ACKs the S3 request and then dies/cycles before concluding all parts, the receipts exist at sprue but nothing progresses them and (per the removal RFC) a parked blob's only handle is its /blob/add task link/cause

so: ingot must durably persist per-part task links before the S3 ACK, so recovery can resume conclude/accept or abort -- otherwise the parts are stranded until allocation expiry, which I don't think is specified yet?

(encryption makes this more complicated)

[alanshaw]

so: ingot must durably persist per-part task links before the S3 ACK, so recovery can resume conclude/accept or abort

It does this already: UploadPart persists the task links and the sealed /http/put invocation before returning the part's 200, so everything needed to conclude or abort is durable locally — recovery doesn't depend on finding receipts at sprue.

The crash window you describe can't open, because accepts are performed on the Complete request path: every part is concluded and its /blob/accept receipt awaited before the 200 goes out. If ingot dies mid-Complete the client never got an ACK, so the upload is still unresolved from its side and it retries or aborts. Abandoned sessions are swept and aborted, releasing the parked allocations — no waiting on allocation expiry.

There is one narrow gap: a session a crash strands mid-Complete is currently reaped without aborting its parked blobs, which do then fall back to allocation expiry. I'll open a PR to fix that.

[alanshaw]

We're going to go with #20 but this RFC is retained (with deferred notice) for reference.

[parkan]

so: ingot must durably persist per-part task links before the S3 ACK, so recovery can resume conclude/accept or abort

It does this already: UploadPart persists the task links and the sealed /http/put invocation before returning the part's 200, so everything needed to conclude or abort is durable locally — recovery doesn't depend on finding receipts at sprue.

The crash window you describe can't open, because accepts are performed on the Complete request path: every part is concluded and its /blob/accept receipt awaited before the 200 goes out. If ingot dies mid-Complete the client never got an ACK, so the upload is still unresolved from its side and it retries or aborts. Abandoned sessions are swept and aborted, releasing the parked allocations — no waiting on allocation expiry.

There is one narrow gap: a session a crash strands mid-Complete is currently reaped without aborting its parked blobs, which do then fall back to allocation expiry. I'll open a PR to fix that.

sorry I failed to respond here previously: that design is correct, the framing led me to think it worked differently, this is reasonable to keep for future reference 👍