From 1ab72adae28d1be068917172d48956f416f1333c Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Thu, 23 Jul 2026 13:44:06 +0200 Subject: [PATCH 1/3] rfc: dynamic affinity routing --- rfcs/2026-07-dynamic-affinity-routing.md | 131 +++++++++++++++++++++++ 1 file changed, 131 insertions(+) create mode 100644 rfcs/2026-07-dynamic-affinity-routing.md diff --git a/rfcs/2026-07-dynamic-affinity-routing.md b/rfcs/2026-07-dynamic-affinity-routing.md new file mode 100644 index 0000000..dc6e6aa --- /dev/null +++ b/rfcs/2026-07-dynamic-affinity-routing.md @@ -0,0 +1,131 @@ +# RFC: Dynamic affinity routing + +Status: Experimental + +## Authors + +- [Alan Shaw](https://github.com/alanshaw) + +## Motivation + +[Ingot](2026-06-forge-s3-tenant-management.md#ingot---s3-api) is an S3 facade +that is normally co-located with a Forge Piri storage node. The default Sprue +routing policy selects an eligible storage provider using weighted random +selection among other routing factors. That policy can send an upload accepted +by Ingot to a _remote_ Piri, adding an avoidable data-plane hop and potentially +cross-region transfer. + +Ingot needs a way to express its affinity for its local Piri, that allows for +dynamic updates if needed, and that does not impose significant API changes or +state additions to Sprue. + +## Hypothesis / Goals + +Ingot should be able to dynamically choose a Piri node to write to, providing +opportunity to horizontally scale, manage downtime/maintenance and load balance +writes. + +Non-dynamic space-level affinity (referred herein as _static affinity_), where +ALL writes to a given space are routed to a specific storage node, imposes a +state management burden on Sprue, adds to routing complexity and necessitates an +additional UCAN command for affinity management. It will also disallow +multi-region buckets to exist in the future. + +Adding an optional affinity list to `/blob/add` will allow Ingot to constrain +Sprue's storage-node selection to its co-located Piri(s) without imposing +additional state management on Sprue, all via a minimal backward compatible +protocol change. + +The design has the following goals: + +1. Preserve existing `/blob/add` routing behavior when no affinity is given. +2. Allow an invoker to limit a request to one or more registered Piri nodes. +3. Keep the final choice among the allowed nodes with Sprue. +4. Ensure that an affinity-constrained request is never routed to a node outside + its affinity list. + +## Design + +### Terminology + +**Affinity** is a set of storage-node DIDs supplied by an invoker of +`/blob/add`. It is a routing constraint, not a request that any particular +node be selected, unless the set contains just a single DID. + +**Storage node** is a Piri node registered with Sprue and eligible to receive +blob allocations. + +### `/blob/add` arguments + +`/blob/add` gains an optional `affinity` field: + +```ipldsch +type BlobAddArguments struct { + # Existing /blob/add arguments are omitted. + affinity optional [DID] +} +``` + +An invocation without `affinity` has the same routing semantics as it did +before this change. + +When present, `affinity` MUST contain one or more DIDs. Every DID in the list +MUST identify a storage node registered with Sprue. An invocation with an +empty list or a DID that does not identify a registered storage node is +invalid and Sprue MUST reject it. + +The list is a candidate set, not an ordered preference list. Clients MAY +include one DID to require a particular storage node or multiple DIDs to allow +Sprue to select between them. + +### Routing + +For an invocation with `affinity`, Sprue MUST select a storage node from the +provided list. Sprue MUST NOT route that invocation to a storage node whose DID +is not in `affinity`. + +Sprue MAY use any of its normal routing considerations to choose among the +provided nodes, including availability, capacity, and weights. It MAY choose +any node in the affinity list at its discretion; the order of the list has no +meaning. + +If none of the affinity nodes can serve the request, Sprue MUST fail the +invocation. It MUST NOT fall back to a storage node outside the affinity list. + +An affinity list constrains only the storage node selected for the current +`/blob/add` invocation. It does not guarantee that the selected node remains +available, establish a durable placement policy, or request replication. + +### Ingot use + +An Ingot instance SHOULD include the DID or DIDs of its co-located Piri nodes +as `affinity` on each `/blob/add` invocation it originates. Ingot MAY omit +`affinity` when it intentionally accepts Sprue's ordinary routing policy. + +This preserves Sprue as the authority that selects the node while ensuring that +an Ingot deployment does not cause a blob to be allocated to an unrelated +storage node. + +## Alternatives Considered + +### Static affinity routing + +AKA assigning a "home" node to each space. + +Sprue could store affinity as an attribute of a space as part of Ingot's space +creation procedure. This is additional state that _Sprue_ would have to store +as well as provide additional UCAN command for management (to initially set and +subsequently change). Affinity would also have to be retrieved and considered +when routing, adding an additional complexity to an otherwise relatively simple +decision. + +Conversely, dynamic affinity routing may be useful for use cases outside of an +S3 facade. A client may have a trusted set of providers and wish to manually +route blobs to particular nodes. A space-level policy would make this routing +choice unavailable to them or require additional per-client state in Sprue. + +An additional benefit of dynamic affinity is that Ingot can add, remove, or +replace its local Piri nodes _immediately_ by changing the affinity included +with each write. A static space-level assignment instead requires a separate +update before the change takes effect, and creates a window in which Sprue may +route to an outdated node. From 5a1856ad3624bb9086bd767af9c2390ef4e42660 Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Thu, 6 Aug 2026 10:28:57 +0200 Subject: [PATCH 2/3] refactor: rename --- ... => 2026-07-dynamic-routing-candidates.md} | 50 ++++++++++--------- 1 file changed, 26 insertions(+), 24 deletions(-) rename rfcs/{2026-07-dynamic-affinity-routing.md => 2026-07-dynamic-routing-candidates.md} (70%) diff --git a/rfcs/2026-07-dynamic-affinity-routing.md b/rfcs/2026-07-dynamic-routing-candidates.md similarity index 70% rename from rfcs/2026-07-dynamic-affinity-routing.md rename to rfcs/2026-07-dynamic-routing-candidates.md index dc6e6aa..84ad715 100644 --- a/rfcs/2026-07-dynamic-affinity-routing.md +++ b/rfcs/2026-07-dynamic-routing-candidates.md @@ -1,4 +1,4 @@ -# RFC: Dynamic affinity routing +# RFC: Dynamic routing candidates Status: Experimental @@ -31,45 +31,47 @@ state management burden on Sprue, adds to routing complexity and necessitates an additional UCAN command for affinity management. It will also disallow multi-region buckets to exist in the future. -Adding an optional affinity list to `/blob/add` will allow Ingot to constrain +Adding an optional candidate list to `/blob/add` will allow Ingot to constrain Sprue's storage-node selection to its co-located Piri(s) without imposing additional state management on Sprue, all via a minimal backward compatible protocol change. The design has the following goals: -1. Preserve existing `/blob/add` routing behavior when no affinity is given. +1. Preserve existing `/blob/add` routing behavior when no candidate list is + given. 2. Allow an invoker to limit a request to one or more registered Piri nodes. 3. Keep the final choice among the allowed nodes with Sprue. -4. Ensure that an affinity-constrained request is never routed to a node outside - its affinity list. +4. Ensure that a candidate-constrained request is never routed to a node + outside its candidate list. ## Design ### Terminology -**Affinity** is a set of storage-node DIDs supplied by an invoker of -`/blob/add`. It is a routing constraint, not a request that any particular -node be selected, unless the set contains just a single DID. +**Candidates** is a set of storage-node DIDs supplied by an invoker of +`/blob/add`. It is a routing constraint: Sprue selects exactly one node from +the set, so it is not a request that any particular node be selected, unless +the set contains just a single DID. **Storage node** is a Piri node registered with Sprue and eligible to receive blob allocations. ### `/blob/add` arguments -`/blob/add` gains an optional `affinity` field: +`/blob/add` gains an optional `candidates` field: ```ipldsch type BlobAddArguments struct { # Existing /blob/add arguments are omitted. - affinity optional [DID] + candidates optional [DID] } ``` -An invocation without `affinity` has the same routing semantics as it did +An invocation without `candidates` has the same routing semantics as it did before this change. -When present, `affinity` MUST contain one or more DIDs. Every DID in the list +When present, `candidates` MUST contain one or more DIDs. Every DID in the list MUST identify a storage node registered with Sprue. An invocation with an empty list or a DID that does not identify a registered storage node is invalid and Sprue MUST reject it. @@ -80,27 +82,27 @@ Sprue to select between them. ### Routing -For an invocation with `affinity`, Sprue MUST select a storage node from the +For an invocation with `candidates`, Sprue MUST select a storage node from the provided list. Sprue MUST NOT route that invocation to a storage node whose DID -is not in `affinity`. +is not in `candidates`. Sprue MAY use any of its normal routing considerations to choose among the provided nodes, including availability, capacity, and weights. It MAY choose -any node in the affinity list at its discretion; the order of the list has no +any node in the candidate list at its discretion; the order of the list has no meaning. -If none of the affinity nodes can serve the request, Sprue MUST fail the -invocation. It MUST NOT fall back to a storage node outside the affinity list. +If none of the candidate nodes can serve the request, Sprue MUST fail the +invocation. It MUST NOT fall back to a storage node outside the candidate list. -An affinity list constrains only the storage node selected for the current +A candidate list constrains only the storage node selected for the current `/blob/add` invocation. It does not guarantee that the selected node remains available, establish a durable placement policy, or request replication. ### Ingot use An Ingot instance SHOULD include the DID or DIDs of its co-located Piri nodes -as `affinity` on each `/blob/add` invocation it originates. Ingot MAY omit -`affinity` when it intentionally accepts Sprue's ordinary routing policy. +as `candidates` on each `/blob/add` invocation it originates. Ingot MAY omit +`candidates` when it intentionally accepts Sprue's ordinary routing policy. This preserves Sprue as the authority that selects the node while ensuring that an Ingot deployment does not cause a blob to be allocated to an unrelated @@ -119,13 +121,13 @@ subsequently change). Affinity would also have to be retrieved and considered when routing, adding an additional complexity to an otherwise relatively simple decision. -Conversely, dynamic affinity routing may be useful for use cases outside of an -S3 facade. A client may have a trusted set of providers and wish to manually +Conversely, dynamic routing candidates may be useful for use cases outside of +an S3 facade. A client may have a trusted set of providers and wish to manually route blobs to particular nodes. A space-level policy would make this routing choice unavailable to them or require additional per-client state in Sprue. -An additional benefit of dynamic affinity is that Ingot can add, remove, or -replace its local Piri nodes _immediately_ by changing the affinity included +An additional benefit of dynamic candidates is that Ingot can add, remove, or +replace its local Piri nodes _immediately_ by changing the candidates included with each write. A static space-level assignment instead requires a separate update before the change takes effect, and creates a window in which Sprue may route to an outdated node. From 61dff15650c7f354aef1f4e26f8f1b9706cbb2d6 Mon Sep 17 00:00:00 2001 From: ash Date: Tue, 8 Sep 2026 16:17:42 +0100 Subject: [PATCH 3/3] rfc: routing policies (#20) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit [📖 Preview](https://github.com/fil-one/RFC/blob/ash%2Frfc%2Frouting-policies/rfcs/2026-08-routing-policies.md) ## Summary Spaces are configured to use a **routing policy** which specifies the candidate nodes that can receive a blob. The policy can change over time but the reference to it does not. It is an alternative affinity routing proposal that allows blobs uploaded to spaces to be routed to a particular storage node _without having to specify said node in every invocation_. --- rfcs/2026-07-dynamic-routing-candidates.md | 6 + rfcs/2026-08-routing-policies.md | 265 +++++++++++++++++++++ 2 files changed, 271 insertions(+) create mode 100644 rfcs/2026-08-routing-policies.md diff --git a/rfcs/2026-07-dynamic-routing-candidates.md b/rfcs/2026-07-dynamic-routing-candidates.md index 84ad715..1fc0a3a 100644 --- a/rfcs/2026-07-dynamic-routing-candidates.md +++ b/rfcs/2026-07-dynamic-routing-candidates.md @@ -2,6 +2,12 @@ Status: Experimental +> **Note:** Implementation of this RFC is deferred for the time being. +> [Routing policies](2026-08-routing-policies.md), a space-level alternative +> to the per-invocation design described here, is being implemented instead. +> This RFC is retained as a record of the design and the trade-offs +> considered, and may be revisited. + ## Authors - [Alan Shaw](https://github.com/alanshaw) diff --git a/rfcs/2026-08-routing-policies.md b/rfcs/2026-08-routing-policies.md new file mode 100644 index 0000000..770982b --- /dev/null +++ b/rfcs/2026-08-routing-policies.md @@ -0,0 +1,265 @@ +# RFC: Routing policies + +Status: Proposed + +## Authors + +- [Alan Shaw](https://github.com/alanshaw) + +## Motivation + +[Dynamic routing candidates](2026-07-dynamic-routing-candidates.md) allows an +invoker of `/blob/add` to constrain Sprue's storage-node selection per +invocation. The constraint travels with each write, so it binds only the +invocations that carry it: any agent authorized to write to a space that omits +`candidates` — a misconfigured Ingot, a direct client, a future integration — +falls back to Sprue's default weighted random selection, and the space's data +lands on an arbitrary storage node. + +The [deployment proposal](2026-05-filone-forge-deployment-proposal.md#the-upload-path) +calls for a stronger property: every space has a home provider, and +allocations for that space go to the home provider's Piri nodes — regardless +of which agent invoked the write or from where. This keeps the data plane +in-region and future-proofs the case where a customer runs their own S3 +gateway on-prem and invokes `/blob/add` directly: the customer's gateway need +not know (or be trusted to supply) storage node DIDs. + +That property requires the constraint to be an attribute of the space, +enforced by Sprue for every write. Storing a candidate list per space would be +repetitive — every space in a region carries the same list, and a change to +the region's nodes becomes an update to every one of its spaces. This RFC +instead makes the candidate list a shared, first-class entity — a _routing +policy_ — that spaces reference by DID. + +## Hypothesis / Goals + +Routing policies give Sprue-enforced placement for all writes to a space, +while keeping the state Sprue stores minimal: one candidate list per policy +and one DID per space. Changing a region's storage nodes is a single policy +update that takes immediate effect for every space referencing it. + +The design has the following goals: + +1. Preserve existing routing behavior for spaces that reference no policy. +2. Allow all allocations for a space to be constrained to one or more + registered Piri nodes, without cooperation from writers. +3. Share one candidate list across many spaces, so node changes are a single + update. +4. Keep the final choice among the allowed nodes with Sprue. + +## Design + +### Terminology + +**Routing policy** is an entity identified by a DID that owns a set of +storage-node DIDs — its _candidates_. It is a routing constraint for the +spaces that reference it, not a request that any particular node be selected, +unless the set contains just a single DID. + +**Policy reference** is the association from a space to a routing policy, +stored by Sprue as an attribute of the space. + +**Storage node** is a Piri node registered with Sprue and eligible to receive +blob allocations. + +### Policy identity + +A routing policy MUST be a cryptographic key pair identified by a [`did:key`] +URI — the same pattern as a [space]. Authority over the policy is rooted in +its key: after creation, the policy key SHOULD issue a non-expiring delegation +of `/` to the principal that will manage it, after which the private key MAY +be discarded — mirroring the +[bucket key pattern](2026-06-forge-s3-tenant-management.md#bucket-creation). + +A policy requires no registration step: it exists in Sprue once its first +[`/routing/set`](#routingset) invocation succeeds. + +### `/routing/set` + +An authorized agent MAY invoke the `/routing/set` capability on a **routing +policy** subject to replace the policy's candidate set. + +```ipldsch +type RoutingSetArguments struct { + candidates { DID: Candidate } +} + +type Candidate struct {} +``` + +`candidates` MUST contain one or more entries. Every key MUST be a DID +identifying a storage node registered with Sprue. An invocation with an empty +map or a key that does not identify a registered storage node is invalid and +Sprue MUST reject it. + +The map is a candidate set, not a preference order — map encoding is +inherently unordered, and [dag-cbor] sorts map keys, keeping the encoding +deterministic. The `Candidate` value is an empty struct, reserved for future +per-node properties that influence routing decisions (e.g. weight). + +```jsonc +{ + "iss": "did:web:hilt.example.com", + "aud": "did:web:sprue.example.com", + "sub": "did:key:zPolicy", + "cmd": "/routing/set", + "args": { + "candidates": { + "did:key:zPiri1": {}, + "did:key:zPiri2": {} + } + }, + "prf": [{ "/": "bafy..dlgPolicy" }] + // ... +} +``` + +The success value is an empty object. + +A change to a policy's candidates applies to invocations routed after the +change takes effect. In-flight writes MAY be routed using the previous set. + +### `/routing/use` + +An authorized agent MAY invoke the `/routing/use` capability on a **space** +subject to set or clear the space's policy reference. + +```ipldsch +type RoutingUseArguments struct { + policy optional DID +} +``` + +When present, `policy` MUST identify a routing policy known to Sprue — one +with a stored candidate set. An invocation referencing an unknown policy is +invalid and Sprue MUST reject it. When absent, the space's policy reference is +cleared and the space returns to Sprue's default routing. + +The invocation MUST fail if the subject space is not provisioned with a +provider. + +```jsonc +{ + "iss": "did:plc:tenant", + "aud": "did:web:sprue.example.com", + "sub": "did:key:zBucket", + "cmd": "/routing/use", + "args": { + "policy": "did:key:zPolicy" + }, + "prf": [{ "/": "bafy..dlgBucketTenant" }] + // ... +} +``` + +The success value is an empty object. + +### `/routing/get` + +An authorized agent MAY invoke the `/routing/get` capability to inspect stored +routing state. It takes no arguments. On a **routing policy** subject the +success value is the policy's candidate set: + +```ipldsch +type RoutingGetPolicyOK struct { + candidates { DID: Candidate } +} +``` + +On a **space** subject the success value is the space's policy reference, +where `policy` is absent when the space references no policy: + +```ipldsch +type RoutingGetSpaceOK struct { + policy optional DID +} +``` + +### Routing + +For a `/blob/add` invocation on a space with a policy reference, Sprue MUST +resolve the referenced policy and select a storage node from its candidates. +Sprue MUST NOT route the invocation to a storage node outside the candidate +set, and MUST fail the invocation rather than fall back if no candidate can +serve it _(error name `CandidateUnavailable`)_. + +Sprue MAY use any of its normal routing considerations to choose among the +candidates, including availability, capacity, and weights. + +A `/blob/add` invocation on a space with no policy reference has the same +routing semantics as it did before this change. + +#### Replication + +How a policy applies to replica placement under the [replication protocol] is +undefined by this RFC. A future revision MAY specify it. + +### S3 deployment use + +[Hilt](2026-06-forge-s3-tenant-management.md) creates one routing policy per +region at setup and holds the policy's root delegation. It is configured with +the storage-node DIDs serving each region and maintains each policy's +candidates with `/routing/set`. + +On bucket creation, after provisioning the bucket (space) with Sprue, Hilt +invokes `/routing/use` — signed with the tenant key, whose authority flows +from the bucket's root delegation — pointing the bucket at its region's +policy. + +A provider adding, removing, or replacing a Piri node is then a single +`/routing/set` on the region's policy: no space is touched, and every bucket +in the region follows immediately. Ingot requires no knowledge of storage-node +DIDs at all. + +Policy management authority MAY in future be delegated to the regional +provider itself — the policy root delegation makes this a delegation, not a +protocol change — allowing providers to rotate their own nodes without +involving the network operator. + +## Trade-offs + +This design deliberately accepts costs identified in +[dynamic routing candidates](2026-07-dynamic-routing-candidates.md#static-affinity-routing): + +- Sprue stores state — a candidate set per policy and a reference per space — + and MUST resolve the reference on every routing decision for a constrained + space. +- Changes require a `/routing/set` or `/routing/use` invocation before taking + effect, creating a window in which Sprue may route using outdated state. +- A space is pinned to its policy's candidates; a future multi-region bucket + would need a policy spanning regions, weakening the locality property the + policy was created for. + +It buys server-side enforcement — no writer can, deliberately or by +misconfiguration, place a space's data outside its home provider — and the +policy indirection reduces the state and update costs to one list per region. + +## Alternatives Considered + +### Dynamic per-invocation candidates + +Specified in [dynamic routing candidates](2026-07-dynamic-routing-candidates.md), +to which this RFC is a mutually exclusive alternative. It requires no Sprue +state and changes take effect immediately, but the constraint binds only +invocations that carry it: enforcement depends on every writer supplying the +correct candidates on every write. + +### Per-space candidate list + +Store the candidate set directly as an attribute of each space (e.g. via a +`/routing/set` on the space subject). Rejected: every space in a region +duplicates the same list, and a change to the region's nodes becomes an update +to every one of its spaces — with a long window in which spaces route to +outdated nodes. + +### Extending `/provider/add` + +Provisioning is the moment a space is bound to a provider, so an optional +candidate list there would ride an existing flow. Rejected: it overloads +provisioning with policy management, requires awkward update semantics +(distinguishing an absent field from an empty list on re-invocation). + +[space]: https://github.com/fil-forge/ucan-protocol-specs/blob/main/blob.md#space +[replication protocol]: https://github.com/fil-forge/ucan-protocol-specs/blob/main/replication.md +[`did:key`]: https://w3c-ccg.github.io/did-key-spec/ +[dag-cbor]: https://ipld.io/specs/codecs/dag-cbor/spec/