diff --git a/rfcs/2026-07-dynamic-routing-candidates.md b/rfcs/2026-07-dynamic-routing-candidates.md new file mode 100644 index 0000000..1fc0a3a --- /dev/null +++ b/rfcs/2026-07-dynamic-routing-candidates.md @@ -0,0 +1,139 @@ +# RFC: Dynamic routing candidates + +Status: Experimental + +> **Note:** Implementation of this RFC is deferred for the time being. +> [Routing policies](2026-08-routing-policies.md), a space-level alternative +> to the per-invocation design described here, is being implemented instead. +> This RFC is retained as a record of the design and the trade-offs +> considered, and may be revisited. + +## Authors + +- [Alan Shaw](https://github.com/alanshaw) + +## Motivation + +[Ingot](2026-06-forge-s3-tenant-management.md#ingot---s3-api) is an S3 facade +that is normally co-located with a Forge Piri storage node. The default Sprue +routing policy selects an eligible storage provider using weighted random +selection among other routing factors. That policy can send an upload accepted +by Ingot to a _remote_ Piri, adding an avoidable data-plane hop and potentially +cross-region transfer. + +Ingot needs a way to express its affinity for its local Piri, that allows for +dynamic updates if needed, and that does not impose significant API changes or +state additions to Sprue. + +## Hypothesis / Goals + +Ingot should be able to dynamically choose a Piri node to write to, providing +opportunity to horizontally scale, manage downtime/maintenance and load balance +writes. + +Non-dynamic space-level affinity (referred herein as _static affinity_), where +ALL writes to a given space are routed to a specific storage node, imposes a +state management burden on Sprue, adds to routing complexity and necessitates an +additional UCAN command for affinity management. It will also disallow +multi-region buckets to exist in the future. + +Adding an optional candidate list to `/blob/add` will allow Ingot to constrain +Sprue's storage-node selection to its co-located Piri(s) without imposing +additional state management on Sprue, all via a minimal backward compatible +protocol change. + +The design has the following goals: + +1. Preserve existing `/blob/add` routing behavior when no candidate list is + given. +2. Allow an invoker to limit a request to one or more registered Piri nodes. +3. Keep the final choice among the allowed nodes with Sprue. +4. Ensure that a candidate-constrained request is never routed to a node + outside its candidate list. + +## Design + +### Terminology + +**Candidates** is a set of storage-node DIDs supplied by an invoker of +`/blob/add`. It is a routing constraint: Sprue selects exactly one node from +the set, so it is not a request that any particular node be selected, unless +the set contains just a single DID. + +**Storage node** is a Piri node registered with Sprue and eligible to receive +blob allocations. + +### `/blob/add` arguments + +`/blob/add` gains an optional `candidates` field: + +```ipldsch +type BlobAddArguments struct { + # Existing /blob/add arguments are omitted. + candidates optional [DID] +} +``` + +An invocation without `candidates` has the same routing semantics as it did +before this change. + +When present, `candidates` MUST contain one or more DIDs. Every DID in the list +MUST identify a storage node registered with Sprue. An invocation with an +empty list or a DID that does not identify a registered storage node is +invalid and Sprue MUST reject it. + +The list is a candidate set, not an ordered preference list. Clients MAY +include one DID to require a particular storage node or multiple DIDs to allow +Sprue to select between them. + +### Routing + +For an invocation with `candidates`, Sprue MUST select a storage node from the +provided list. Sprue MUST NOT route that invocation to a storage node whose DID +is not in `candidates`. + +Sprue MAY use any of its normal routing considerations to choose among the +provided nodes, including availability, capacity, and weights. It MAY choose +any node in the candidate list at its discretion; the order of the list has no +meaning. + +If none of the candidate nodes can serve the request, Sprue MUST fail the +invocation. It MUST NOT fall back to a storage node outside the candidate list. + +A candidate list constrains only the storage node selected for the current +`/blob/add` invocation. It does not guarantee that the selected node remains +available, establish a durable placement policy, or request replication. + +### Ingot use + +An Ingot instance SHOULD include the DID or DIDs of its co-located Piri nodes +as `candidates` on each `/blob/add` invocation it originates. Ingot MAY omit +`candidates` when it intentionally accepts Sprue's ordinary routing policy. + +This preserves Sprue as the authority that selects the node while ensuring that +an Ingot deployment does not cause a blob to be allocated to an unrelated +storage node. + +## Alternatives Considered + +### Static affinity routing + +AKA assigning a "home" node to each space. + +Sprue could store affinity as an attribute of a space as part of Ingot's space +creation procedure. This is additional state that _Sprue_ would have to store +as well as provide additional UCAN command for management (to initially set and +subsequently change). Affinity would also have to be retrieved and considered +when routing, adding an additional complexity to an otherwise relatively simple +decision. + +Conversely, dynamic routing candidates may be useful for use cases outside of +an S3 facade. A client may have a trusted set of providers and wish to manually +route blobs to particular nodes. A space-level policy would make this routing +choice unavailable to them or require additional per-client state in Sprue. + +An additional benefit of dynamic candidates is that Ingot can add, remove, or +replace its local Piri nodes _immediately_ by changing the candidates included +with each write. A static space-level assignment instead requires a separate +update before the change takes effect, and creates a window in which Sprue may +route to an outdated node. diff --git a/rfcs/2026-08-routing-policies.md b/rfcs/2026-08-routing-policies.md new file mode 100644 index 0000000..770982b --- /dev/null +++ b/rfcs/2026-08-routing-policies.md @@ -0,0 +1,265 @@ +# RFC: Routing policies + +Status: Proposed + +## Authors + +- [Alan Shaw](https://github.com/alanshaw) + +## Motivation + +[Dynamic routing candidates](2026-07-dynamic-routing-candidates.md) allows an +invoker of `/blob/add` to constrain Sprue's storage-node selection per +invocation. The constraint travels with each write, so it binds only the +invocations that carry it: any agent authorized to write to a space that omits +`candidates` — a misconfigured Ingot, a direct client, a future integration — +falls back to Sprue's default weighted random selection, and the space's data +lands on an arbitrary storage node. + +The [deployment proposal](2026-05-filone-forge-deployment-proposal.md#the-upload-path) +calls for a stronger property: every space has a home provider, and +allocations for that space go to the home provider's Piri nodes — regardless +of which agent invoked the write or from where. This keeps the data plane +in-region and future-proofs the case where a customer runs their own S3 +gateway on-prem and invokes `/blob/add` directly: the customer's gateway need +not know (or be trusted to supply) storage node DIDs. + +That property requires the constraint to be an attribute of the space, +enforced by Sprue for every write. Storing a candidate list per space would be +repetitive — every space in a region carries the same list, and a change to +the region's nodes becomes an update to every one of its spaces. This RFC +instead makes the candidate list a shared, first-class entity — a _routing +policy_ — that spaces reference by DID. + +## Hypothesis / Goals + +Routing policies give Sprue-enforced placement for all writes to a space, +while keeping the state Sprue stores minimal: one candidate list per policy +and one DID per space. Changing a region's storage nodes is a single policy +update that takes immediate effect for every space referencing it. + +The design has the following goals: + +1. Preserve existing routing behavior for spaces that reference no policy. +2. Allow all allocations for a space to be constrained to one or more + registered Piri nodes, without cooperation from writers. +3. Share one candidate list across many spaces, so node changes are a single + update. +4. Keep the final choice among the allowed nodes with Sprue. + +## Design + +### Terminology + +**Routing policy** is an entity identified by a DID that owns a set of +storage-node DIDs — its _candidates_. It is a routing constraint for the +spaces that reference it, not a request that any particular node be selected, +unless the set contains just a single DID. + +**Policy reference** is the association from a space to a routing policy, +stored by Sprue as an attribute of the space. + +**Storage node** is a Piri node registered with Sprue and eligible to receive +blob allocations. + +### Policy identity + +A routing policy MUST be a cryptographic key pair identified by a [`did:key`] +URI — the same pattern as a [space]. Authority over the policy is rooted in +its key: after creation, the policy key SHOULD issue a non-expiring delegation +of `/` to the principal that will manage it, after which the private key MAY +be discarded — mirroring the +[bucket key pattern](2026-06-forge-s3-tenant-management.md#bucket-creation). + +A policy requires no registration step: it exists in Sprue once its first +[`/routing/set`](#routingset) invocation succeeds. + +### `/routing/set` + +An authorized agent MAY invoke the `/routing/set` capability on a **routing +policy** subject to replace the policy's candidate set. + +```ipldsch +type RoutingSetArguments struct { + candidates { DID: Candidate } +} + +type Candidate struct {} +``` + +`candidates` MUST contain one or more entries. Every key MUST be a DID +identifying a storage node registered with Sprue. An invocation with an empty +map or a key that does not identify a registered storage node is invalid and +Sprue MUST reject it. + +The map is a candidate set, not a preference order — map encoding is +inherently unordered, and [dag-cbor] sorts map keys, keeping the encoding +deterministic. The `Candidate` value is an empty struct, reserved for future +per-node properties that influence routing decisions (e.g. weight). + +```jsonc +{ + "iss": "did:web:hilt.example.com", + "aud": "did:web:sprue.example.com", + "sub": "did:key:zPolicy", + "cmd": "/routing/set", + "args": { + "candidates": { + "did:key:zPiri1": {}, + "did:key:zPiri2": {} + } + }, + "prf": [{ "/": "bafy..dlgPolicy" }] + // ... +} +``` + +The success value is an empty object. + +A change to a policy's candidates applies to invocations routed after the +change takes effect. In-flight writes MAY be routed using the previous set. + +### `/routing/use` + +An authorized agent MAY invoke the `/routing/use` capability on a **space** +subject to set or clear the space's policy reference. + +```ipldsch +type RoutingUseArguments struct { + policy optional DID +} +``` + +When present, `policy` MUST identify a routing policy known to Sprue — one +with a stored candidate set. An invocation referencing an unknown policy is +invalid and Sprue MUST reject it. When absent, the space's policy reference is +cleared and the space returns to Sprue's default routing. + +The invocation MUST fail if the subject space is not provisioned with a +provider. + +```jsonc +{ + "iss": "did:plc:tenant", + "aud": "did:web:sprue.example.com", + "sub": "did:key:zBucket", + "cmd": "/routing/use", + "args": { + "policy": "did:key:zPolicy" + }, + "prf": [{ "/": "bafy..dlgBucketTenant" }] + // ... +} +``` + +The success value is an empty object. + +### `/routing/get` + +An authorized agent MAY invoke the `/routing/get` capability to inspect stored +routing state. It takes no arguments. On a **routing policy** subject the +success value is the policy's candidate set: + +```ipldsch +type RoutingGetPolicyOK struct { + candidates { DID: Candidate } +} +``` + +On a **space** subject the success value is the space's policy reference, +where `policy` is absent when the space references no policy: + +```ipldsch +type RoutingGetSpaceOK struct { + policy optional DID +} +``` + +### Routing + +For a `/blob/add` invocation on a space with a policy reference, Sprue MUST +resolve the referenced policy and select a storage node from its candidates. +Sprue MUST NOT route the invocation to a storage node outside the candidate +set, and MUST fail the invocation rather than fall back if no candidate can +serve it _(error name `CandidateUnavailable`)_. + +Sprue MAY use any of its normal routing considerations to choose among the +candidates, including availability, capacity, and weights. + +A `/blob/add` invocation on a space with no policy reference has the same +routing semantics as it did before this change. + +#### Replication + +How a policy applies to replica placement under the [replication protocol] is +undefined by this RFC. A future revision MAY specify it. + +### S3 deployment use + +[Hilt](2026-06-forge-s3-tenant-management.md) creates one routing policy per +region at setup and holds the policy's root delegation. It is configured with +the storage-node DIDs serving each region and maintains each policy's +candidates with `/routing/set`. + +On bucket creation, after provisioning the bucket (space) with Sprue, Hilt +invokes `/routing/use` — signed with the tenant key, whose authority flows +from the bucket's root delegation — pointing the bucket at its region's +policy. + +A provider adding, removing, or replacing a Piri node is then a single +`/routing/set` on the region's policy: no space is touched, and every bucket +in the region follows immediately. Ingot requires no knowledge of storage-node +DIDs at all. + +Policy management authority MAY in future be delegated to the regional +provider itself — the policy root delegation makes this a delegation, not a +protocol change — allowing providers to rotate their own nodes without +involving the network operator. + +## Trade-offs + +This design deliberately accepts costs identified in +[dynamic routing candidates](2026-07-dynamic-routing-candidates.md#static-affinity-routing): + +- Sprue stores state — a candidate set per policy and a reference per space — + and MUST resolve the reference on every routing decision for a constrained + space. +- Changes require a `/routing/set` or `/routing/use` invocation before taking + effect, creating a window in which Sprue may route using outdated state. +- A space is pinned to its policy's candidates; a future multi-region bucket + would need a policy spanning regions, weakening the locality property the + policy was created for. + +It buys server-side enforcement — no writer can, deliberately or by +misconfiguration, place a space's data outside its home provider — and the +policy indirection reduces the state and update costs to one list per region. + +## Alternatives Considered + +### Dynamic per-invocation candidates + +Specified in [dynamic routing candidates](2026-07-dynamic-routing-candidates.md), +to which this RFC is a mutually exclusive alternative. It requires no Sprue +state and changes take effect immediately, but the constraint binds only +invocations that carry it: enforcement depends on every writer supplying the +correct candidates on every write. + +### Per-space candidate list + +Store the candidate set directly as an attribute of each space (e.g. via a +`/routing/set` on the space subject). Rejected: every space in a region +duplicates the same list, and a change to the region's nodes becomes an update +to every one of its spaces — with a long window in which spaces route to +outdated nodes. + +### Extending `/provider/add` + +Provisioning is the moment a space is bound to a provider, so an optional +candidate list there would ride an existing flow. Rejected: it overloads +provisioning with policy management, requires awkward update semantics +(distinguishing an absent field from an empty list on re-invocation). + +[space]: https://github.com/fil-forge/ucan-protocol-specs/blob/main/blob.md#space +[replication protocol]: https://github.com/fil-forge/ucan-protocol-specs/blob/main/replication.md +[`did:key`]: https://w3c-ccg.github.io/did-key-spec/ +[dag-cbor]: https://ipld.io/specs/codecs/dag-cbor/spec/