Lease: add patterned subscriptions and paginated live inventory for fleet rebalancing

[smiggleworth]

Summary

Add client-facing patterned Lease inventory and patterned Lease subscriptions so fleet workers can observe the current set of workload leases in a namespace and rebalance cooperatively without weakening Lease ownership or fencing.

The target use case is:

lease://{realm}/{area}/{resource}

A worker should be able to:

1. list the current leases matching a selector such as lease://acme/renderers/*;
2. subscribe to the same selector for future matching inventory changes;
3. use that observed inventory to decide whether it should release one of its own leases or attempt a normal acquisition; and
4. periodically reconcile from LIST because live notifications remain ephemeral hints.

Patterned observation must never transfer, assume, revoke, renew, or release another session's lease. Only the existing exact-route ACQUIRE, EXTEND, and token-protected RELEASE operations may change ownership.

This feature has an explicit deployment boundary: every worker participating in one observed inventory MUST connect to the same Fitz broker process and RouteFamily. Separate broker processes are separate lease universes, even when their application-visible routes are identical. A broker restart creates a new, initially empty lease universe and requires every worker to reacquire, relist, and resubscribe.

Why

Lease already provides the correct ownership physics for a broker-local worker fleet: one live holder per concrete lease, TTL expiry, disconnect cleanup, FIFO waiting, and process-local fencing. What is missing is a client-facing way to observe the current ownership set at an area, realm, or family scope.

Without patterned inventory, every worker must already know every currently held concrete lease route and issue one QUERY/exact subscription per route. That does not work when the held set changes dynamically or when a fleet needs to rebalance as leases appear and disappear.

Lease inventory is not the authoritative catalog of work that could exist. A fleet manager still obtains candidate workload/resource routes from Portia, KV, Stream, or another application-owned source. Lease LIST answers "which matching routes are held now?"; it cannot enumerate an unowned route that has no live Lease state.

The existing admin lease search is not a substitute. It is an operator-facing projection, is not part of the client Lease protocol, and does not provide a race-aware bootstrap with the live subscription surface.

Fleet-management use case

Assume a fleet coordinates work through routes such as:

lease://acme/renderers/document-123
lease://acme/renderers/document-456
lease://acme/renderers/document-789

Each worker observes lease://acme/renderers/*. The observation is advisory input to client-owned rebalancing policy:

• an authoritative workload catalog supplies the exact routes that could be acquired;
• an underloaded worker may attempt ACQUIRE on an unowned exact route;
• an overloaded worker may RELEASE a lease it already owns using its current fencing token;
• a worker may do nothing when the observed distribution is already acceptable.

No worker may construct an owned Lease handle from an inventory item or notification. Seeing a route or holder does not grant authority over it. Competing workers still resolve ownership exclusively through the existing Lease state machine.

Current gap

• Lease SUBSCRIBE and UNSUBSCRIBE currently reject every wildcard.
• Lease has no client-facing LIST operation.
• LEASE_NOTIFY currently carries an empty change hint and is emitted for ownership removal paths, but not for new immediate acquisitions or successful queued grants.
• The authoritative routing and domain documents explicitly describe Lease watches as exact-only, so this is an intentional contract change rather than a local implementation tweak.

Required contract

1. One shared selector grammar

Lease LIST, SUBSCRIBE, and UNSUBSCRIBE MUST accept the same generic fixed-depth pattern grammar used by comparable three-segment domain operations.

The concrete Lease identity remains exactly:

lease://{realm}/{area}/{resource}

Supported selectors MUST include exact routes plus whole-segment * and ** patterns capable of matching that three-segment shape. This includes the complete literal-or-* matrix and valid generic ** compositions.

Examples that MUST be valid:

lease://acme/renderers/document-123
lease://acme/renderers/*
lease://acme/*/document-123
lease://acme/*/*
lease://*/renderers/*
lease://*/*/*
lease://acme/**
lease://**

Examples that MUST be rejected before retaining state or scanning inventory:

lease://acme/renderers
lease://acme/renderers/document-123/extra
lease://acme/renderers/doc*
lease://acme//document-123
notice://acme/renderers/*

The grammar, canonicalization, authorization containment, list planning, subscription retention, matching, and unsubscription path MUST share one selector classification or equivalent single source of truth. One layer accepting a selector that another narrows or interprets differently is a correctness defect.

Exact QUERY, ACQUIRE, EXTEND, and RELEASE MUST continue to reject wildcards.

2. Patterned live Lease inventory

Add a client-facing Lease LIST operation over the selector grammar above.

LIST MUST:

• return the current, non-expired held leases matching the selector in the caller's RouteFamily;
• treat an exact selector as a zero-or-one-item inventory read;
• return the exact concrete route for every item;
• return enough read-only state for rebalancing, including the application-supplied logical owner_id, an opaque live holder_incarnation, acquisition time, current expiry/remaining TTL, and renewal count;
• keep holder_incarnation stable for all leases held by one live session while distinguishing different live sessions that reuse the same logical owner_id;
• never expose a raw broker session ID as the public holder identity;
• exclude pending waiters from the held-lease inventory;
• process due expiry before an item is reported as held;
• never return an owned Lease handle or make an observed fencing token usable as an ownership capability;
• use a bounded default and maximum page size;
• expose a client iterator or language-equivalent pager rather than requiring an unbounded fully buffered result; and
• return an opaque continuation bound to the original selector, RouteFamily, broker lifetime/snapshot, and page ordering.

Pagination MUST have explicit consistency semantics. If the implementation promises a snapshot, concurrent acquire/release/expiry/renew activity MUST not cause duplicates or omissions within that snapshot. A cursor from another selector, family, broker lifetime, or incompatible snapshot MUST fail explicitly rather than silently restart or narrow the read.

If inventory or cursor state is unavailable, the operation MUST return a typed error rather than an empty result.

3. Patterned Lease subscriptions

Extend Lease SUBSCRIBE and UNSUBSCRIBE to accept the same selectors as LIST.

Subscription behavior MUST match the shared live-registration contract:

• subscriptions are session-scoped and are removed on disconnect;
• reconnect creates a new session and requires resubscription and a fresh list;
• duplicate (session, original selector string) subscriptions are idempotent and return the existing subscription identity;
• exact subscriptions do not count against the wildcard-registration cap;
• a session may retain at most 128 wildcard Lease subscriptions;
• the duplicate check occurs before enforcing that cap;
• distinct overlapping selectors remain independent registrations;
• a matching ownership change produces one notification for each matching registration, carrying that registration's subscription identity and the exact concrete lease route;
• exact registrations and wildcard registrations have equal matching semantics; neither has undocumented precedence;
• UNSUBSCRIBE uses the original selector identity and removes only that registration; and
• disconnect cleanup removes wildcard registrations, matching-cache references, and any other retained session state.

Wildcard matching MUST use the shared indexed matcher or an equivalent bounded index. Emitting a change for one concrete lease must scale with the matching registrations and delivery cost, not a scan of every Lease subscription.

4. Inventory-change notification semantics

Lease notifications are inventory invalidation hints, not replayable lifecycle events and not commands.

A matching notification MUST be emitted whenever the current held-lease set changes for a concrete route, including:

• successful immediate acquisition of a previously unowned lease;
• successful grant to a queued waiter;
• explicit release;
• TTL expiry;
• owner-session disconnect cleanup; and
• rollback/removal of an acquisition whose response cannot be delivered.

The notification MUST identify the matching subscription and exact concrete route. A client MUST be able to use the same selector with LIST to reconcile authoritative current state.

A successful renewal updates QUERY and subsequent LIST results but MUST NOT emit a fleet-membership notification by default because the held set and holder have not changed. Broadcasting every renewal to every fleet observer creates quadratic heartbeat fanout without changing the rebalancing decision. If renewal notifications are ever supported, they must be an explicit opt-in contract with separate fanout limits; they must not be silently added to the default membership watch.

Failed, fenced, rejected, or merely queued acquisition attempts MUST NOT appear as held-lease membership changes. Pending waiter changes are outside this inventory contract.

Notifications remain live, best-effort, non-durable, non-replayable, and unacknowledged. They do not replace LIST reconciliation.

5. Race-aware bootstrap and reconciliation

The public contract MUST document and support a bootstrap sequence without an unobservable list/subscribe window.

Every TypeScript, Go, .NET, Python, and Rust SDK MUST expose one safe, high-level inventory-observer operation that owns this sequence. The operation MUST NOT report the observed view as ready until bootstrap has completed. Application callers must not be required to hand-roll buffering, snapshot installation, invalidation draining, or relisting concurrency.

At minimum, clients must be able to:

1. establish the patterned subscription and wait for its acknowledgement;
2. begin buffering matching notifications;
3. drain a LIST snapshot for the same selector;
4. install that snapshot as the local observed view; and
5. process buffered invalidations and relist if the view may have changed.

An atomic broker snapshot-and-subscribe wire operation is acceptable, but it is not required if the ordinary broker operations provide an equivalent ordering contract. Regardless of the chosen wire shape, the high-level SDK observer behavior is required and MUST pass the same language-neutral conformance scenarios in all five clients.

Client documentation MUST state that:

• the observed inventory is advisory and can become stale;
• notification loss, reconnect, broker restart, slow-consumer failure, or handler overflow invalidates the local view;
• clients must resubscribe and relist after reconnect or broker lifetime change; and
• long-running fleet managers should periodically reconcile with LIST even while subscribed;
• the reconciliation interval must be configurable and jittered so a fleet does not poll in lockstep;
• reconnect, subscription failure, delivery overflow, and broker lifetime change trigger immediate reconciliation rather than waiting for the periodic interval; and
• documentation provides one recommended interval formula with explicit minimum and maximum bounds, related to the shortest expected lease TTL and supported by list-cost/performance evidence. SDKs that provide automatic reconciliation use the same documented default instead of inventing per-language heuristics.

No hidden replay cursor or durable subscription state may be implied.

6. Ownership and fencing safety

Patterned observation MUST NOT alter existing Lease ownership rules:

• only exact-route ACQUIRE may grant a lease;
• only the current owner with the current fencing token may EXTEND or RELEASE it;
• list items and notifications are read-only observations and MUST NOT create managed Lease handles;
• a worker cannot assume, transfer, revoke, renew, or release another worker's lease through LIST or SUBSCRIBE;
• rebalancing decisions remain application policy outside Fitz; and
• process-local fencing tokens remain non-durable and non-comparable across lease identities or broker restarts.

7. Authorization and isolation

• RouteFamily remains a hard isolation boundary for listing, matching, and delivery.
• realm remains an opaque application namespace and MUST NOT be inferred from or substituted with RouteFamily.
• A read grant for one pattern MUST cover the selector's complete concrete-route match set. Intersection is insufficient, and multiple partial grants MUST NOT be unioned silently.
• A wildcard-realm grant such as lease://*/renderers/*#read is explicit cross-realm observation authority. It can disclose matching route existence, logical owner identity, and live holder incarnation across every covered realm, so documentation MUST call out the information-disclosure scope and recommend least-privilege grants. realm remains opaque application terminology and must not be described as inherently equal to a tenant.
• Unauthorized selectors MUST fail before inventory is inspected or subscription state is retained.
• Results and notifications MUST never include a lease outside the authorized match set.
• Mutation permissions for exact ACQUIRE, EXTEND, and RELEASE remain separate from observation permissions.

8. Bounded work and observability

• List work must be bounded separately by matched identities, examined identities, returned items, encoded bytes, and execution slice.
• Pagination must make forward progress even when a page returns zero matches after examining bounded inventory.
• Route parsing/formatting should occur once per inventory item/source run rather than repeatedly per filter.
• Lease state, inventory, and subscription locks must not be held across client delivery or unrelated I/O.
• Slow consumers and delivery saturation must follow the existing explicit subscription failure/drop policy; they must not block the Lease actor.
• Metrics should expose active exact/wildcard Lease subscriptions, patterns and identities examined, results returned, notification fanout/drop counts, cursor resumes/mismatches, and work-budget exhaustion.
• Core Lease code remains synchronous; async stays at the transport edge.

9. Client and documentation parity

The canonical protocol, Rust broker, test fixtures, and TypeScript, Go, .NET, Python, and Rust clients must be updated as one coordinated contract change.

Each public client should expose:

• a typed Lease selector/list API;
• a closable iterator or pager for inventory;
• a typed Lease subscription object with unsubscribe/language equivalent;
• a safe high-level inventory observer that owns subscribe, buffering, snapshot installation, invalidation draining, relisting, and lifecycle recovery;
• concrete route and read-only holder/status fields for each observed item; and
• reconnect behavior that re-establishes the subscription only together with a fresh inventory bootstrap; and
• common configurable, jittered reconciliation behavior and documentation.

Public APIs must hide raw subscription identifiers and must not turn observed items into owned Lease objects.

Update at least:

• docs/development/architectural-laws.md;
• docs/development/domain-boundaries-spec.md;
• docs/development/routing-design.md;
• the canonical client operations/wire specifications;
• client requirements and acceptance fixtures;
• migration/release notes for the clean wire/API change; and
• admin/OpenAPI documentation only if the operator surface is intentionally aligned or reused.

Non-goals

• Cross-broker consensus or a distributed membership service.
• Durable lease history, audit replay, or recovery of missed notifications.
• Ownership transfer, forced takeover, or revocation through observation APIs.
• Server-owned fleet balancing policy.
• Discovery or persistence of the universe of unowned workload/resource routes.
• Exposing waiters as active fleet members.
• Making Lease ownership survive disconnect or broker restart.
• Allowing wildcard mutation operations.

Acceptance criteria

Pattern and authorization

• LIST, SUBSCRIBE, and UNSUBSCRIBE accept the same generic depth-three Lease selector grammar, including the complete literal/* matrix and valid ** forms.
• Wrong schemes, wrong depth, empty segments, partial wildcards, invalid ** forms, and over-limit route strings fail before state mutation or inventory work.
• Exact Lease mutation and query operations continue to reject wildcards.
• A selector is authorized only when one grant contains its complete concrete match set.
• Pattern matching and results are isolated by RouteFamily; wildcard realms do not weaken that isolation.
• Cross-realm read grants have explicit disclosure documentation and reveal no routes outside their contained match set.

Inventory

• Exact LIST returns zero or one current holder; wildcard LIST returns every matching current holder and no non-match.
• Expired leases and pending waiters are not returned as held leases.
• LIST does not invent or return unowned workload routes; candidate work discovery remains an explicit external composition.
• Every item includes its exact route, logical owner_id, opaque live holder_incarnation, and documented read-only status fields without exposing a raw session ID.
• One live holder incarnation groups that session's leases even when another session uses the same logical owner_id; a reconnect receives a new incarnation.
• Inventory items cannot be used to construct or synthesize an owned Lease handle.
• Pagination is bounded, deterministic under its documented snapshot contract, and has no duplicates or omissions within a snapshot.
• Continuations are opaque and selector/family/broker-snapshot bound; misuse returns a typed error.
• Backend/inventory failure is distinguishable from an empty result.

Subscription behavior

• Immediate acquisition and queued-waiter grant notify every matching exact/wildcard registration with the exact concrete route.
• Explicit release, expiry, disconnect cleanup, and undeliverable-acquire rollback notify every matching registration.
• Failed/fenced acquisition, queued waiter creation/removal, and successful renewal do not emit default fleet-membership notifications.
• Renewal changes are visible in subsequent exact query and patterned list results.
• Duplicate subscriptions are idempotent; overlapping selectors remain independently addressable and independently notified.
• The 128 wildcard-registration limit is enforced per session after duplicate detection; exact registrations do not consume it.
• Unsubscribe removes only the matching original registration, and disconnect removes all session-owned registrations.
• Notification fanout uses indexed matching and remains bounded under high-cardinality exact and wildcard registrations.

Bootstrap, lifecycle, and safety

• A subscribe-before-list bootstrap is tested with ownership changes before subscription acknowledgement, between acknowledgement and list, during pagination, and immediately after snapshot completion.
• Every public SDK exposes one safe observer operation that completes subscribe, buffering, snapshot installation, and invalidation draining before reporting the view ready; callers do not implement that sequence themselves.
• The same language-neutral bootstrap and lifecycle conformance scenarios pass in all five SDKs whether the broker uses an atomic wire operation or ordered ordinary operations.
• Clients can converge by applying buffered invalidations/relisting without any ownership transfer semantics.
• Disconnect, handler overflow, or broker restart invalidates the client view and requires a new subscription plus list bootstrap.
• Periodic list reconciliation is configurable and jittered; reconnect, subscription failure, delivery overflow, and broker lifetime change promptly schedule a fresh bootstrap with documented coalescing/backoff; one shared recommended interval formula has evidence-backed bounds.
• Tests prove that workers connected to separate broker processes or different RouteFamily values never observe one combined inventory, and that broker restart creates a new empty lease universe.
• Observing another lease never permits its extension, release, revocation, or takeover; existing owner and fencing-token validation remains covered by regression tests.
• Broker restart clears lease inventory, subscriptions, waiters, and fencing lineage without implying recovery.

Cross-surface validation

• Unit tests cover selector parsing, containment, indexed matching, inventory pagination, expiry filtering, limits, cleanup, and ownership safety.
• TCP and WebSocket end-to-end tests cover exact and wildcard list/subscription behavior, overlapping registrations, authorization, bootstrap races, reconnect, and broker restart.
• Performance tests demonstrate bounded list work and matching proportional to matches rather than total registrations.
• TypeScript, Go, .NET, Python, and Rust client conformance fixtures pass against the same broker contract.
• Architectural, routing, protocol, client, testing, and migration documentation are updated in the same delivery.

Architectural fit

This feature keeps Lease responsible for live, ephemeral ownership coordination. It exposes the broker's current Lease state and future invalidation hints but does not add durability, replay, consensus, ownership transfer, or fleet policy. Observation informs client policy; only the existing Lease state machine changes ownership.

[smiggleworth]

Implemented and validated across the broker, canonical docs, multi-architecture image, and all five public SDKs.

Broker and protocol:

• feat(lease): add patterned inventory observation #221 delivered the shared depth-three Lease selector grammar, paginated LIST, wildcard subscriptions, concrete-route notifications, read-only holder metadata, metrics, fixtures, and TCP/WebSocket coverage.
• build(deps): bump flate2 from 1.1.9 to 1.1.10 #222, build(deps): bump hyper from 1.11.0 to 1.11.1 #223, build(deps): bump uuid from 1.25.0 to 1.26.0 #224, and test: make hosted concurrency checks deterministic #225 closed the review findings around depth-aware authorization, authenticated malformed requests, wildcard registration quotas and indexed fanout, exact keyed lookup, atomic snapshots, candidate/item/byte retention limits, TLV page budgets, cursor session/selector/family/broker-lifetime binding and offset integrity, expiry cleanup, opaque keyed holder incarnations, strict Clippy, error classification, and test-module size.
• Bounded-work evidence includes the Lease exact-lookup/candidate/byte/snapshot/cursor/retention tests and the shared SubscriptionIndex tier-2 sparse/dense 10,000-registration benchmarks.

Client parity:

• .NET: feat(lease): add patterned inventory observation fitz-dotnet#11 and fix(lease): harden inventory observer recovery fitz-dotnet#12
• Go: feat(lease): add patterned inventory observation fitz-go#11 and fix(lease): harden inventory observer recovery fitz-go#12
• Python: feat(lease): add patterned inventory observation fitz-py#3 and fix(lease): harden inventory observer recovery fitz-py#4
• Rust: feat(lease): add patterned inventory observation fitz-rs#12 and fix(lease): harden inventory observer recovery fitz-rs#13
• TypeScript: feat(lease): add patterned inventory observation fitz-ts#23 and fix(lease): harden inventory observer recovery fitz-ts#28

All five SDKs now expose public typed LIST/pager and subscription surfaces plus a bounded, closable observer that owns subscribe-before-list buffering, snapshot installation, invalidation draining, reconnect/overflow recovery, prompt retry with bounded backoff, and configurable jittered periodic reconciliation using the shared clamp(shortest expected lease TTL / 2, 5 seconds, 60 seconds) guidance. Observation remains read-only and cannot construct ownership handles or extend/release another session lease.

Canonical contract and operations documentation landed in #226, including AC-LEASE-014, client conformance requirements, migration guidance, broker-lifetime terminology, and the Lease testing coverage contract.

Validation:

• Exact-head Fitz PR CI: https://github.com/cntryl/fitz/actions/runs/33530091970
• Post-merge Fitz main CI at b4629659f98bc86c444038679223ea1d4206fa15: https://github.com/cntryl/fitz/actions/runs/33531299428
• Rebuilt multi-arch image: https://github.com/cntryl/fitz/actions/runs/33518462989; ghcr.io/cntryl/fitz:main and :latest both resolve to sha256:f12f7466dec3af750df577712856553fd0800105a5148f121992af028c7e3ad3 for linux/amd64 and linux/arm64.
• Post-merge client main workflows are green: .NET, Go, Python, Rust, and TypeScript.
• The Fitz UI lockfile is on the current Askr package line from UI polish, Askr reconciliation, and test-suite audit #211; clean install and audit pass with zero vulnerabilities.