diff --git a/docs/clients/acceptance/queue-notice-rpc-lease.md b/docs/clients/acceptance/queue-notice-rpc-lease.md index ff425e48..d2d4d07e 100644 --- a/docs/clients/acceptance/queue-notice-rpc-lease.md +++ b/docs/clients/acceptance/queue-notice-rpc-lease.md @@ -526,4 +526,40 @@ match set, and zero or more currently held leases matching it - `LIST` items can never be turned into an owned Lease handle; only exact ACQUIRE/EXTEND/RELEASE change ownership +### AC-LEASE-014: Race-safe inventory observer lifecycle + +**MUST** expose the same safe high-level observer behavior in TypeScript, Go, +.NET, Python, and Rust +**Given:** A public SDK observer for one valid Lease selector +**When:** The client bootstraps, receives invalidations, reconnects, encounters +subscription delivery failure, reconciles periodically, and closes +**Then:** Each SDK's observer conformance suite covers all of these common +scenarios: + +1. `subscribe-before-list`: no LIST is sent before SUBSCRIBE acknowledgement; + a notification before/during LIST makes that candidate ineligible, a fresh + complete LIST is installed, and readiness becomes true only afterward. +2. `steady-state-invalidation`: a notification after readiness schedules a + coalesced full LIST and atomically replaces the complete view without + constructing an owned Lease handle. +3. `generation-change`: disconnect/reconnect or broker-lifetime change clears + readiness immediately; no LIST result spanning that boundary is installed; + a new/restored wire subscription plus a fresh complete LIST is required. +4. `subscription-failure`: handler overflow, delivery backpressure, or local + subscription termination clears readiness and triggers a replacement + SUBSCRIBE plus fresh LIST. A transient SUBSCRIBE or bootstrap LIST failure + retries with bounded backoff instead of silently degrading to polling-only. +5. `periodic-backstop`: a positive configurable interval causes a jittered + full LIST even without notifications. The documented recommendation is + `clamp(shortest expected lease TTL / 2, 5 seconds, 60 seconds)`, with ±20% + jitter and a 60-second default. +6. `bounded-close`: observer update delivery is bounded/coalesced; close is + idempotent, cancels recovery/reconciliation, unsubscribes, and leaves no + background work that can mutate the installed view. + +Tests may use a deterministic scripted transport for race placement and a +live broker for public-surface acquire/release convergence, but all five SDKs +MUST exercise the same six behaviors and MUST use both TCP and WebSocket in +their shared broker conformance matrix. + ## Schedule Domain diff --git a/docs/clients/client-requirements.md b/docs/clients/client-requirements.md index 29b68ebb..945affd9 100644 --- a/docs/clients/client-requirements.md +++ b/docs/clients/client-requirements.md @@ -144,7 +144,7 @@ The acceptance criteria in `client-acceptance-criteria.md` are the normative sou **REQ-API-004B (T0)** The `Lease.List` operation (LEASE_LIST 410) MUST be exposed, returning the current held-lease inventory matching a selector: exact route, the complete literal-or-`*` matrix over the three segments, or a `**` alias, using the same grammar and client-side validation as REQ-API-004. Each returned item MUST expose its exact route, logical `owner_id`, opaque `holder_incarnation` (never a raw session ID), `acquired_at`, remaining TTL, and renewal count. `Lease.List` MUST paginate per REQ-API-008 and MUST validate `limit` and any cursor offset as a non-negative value representable in an unsigned 32-bit integer before encoding it on the wire — silently wrapping or bit-coercing an out-of-range value (e.g. a `-1` limit becoming `4294967295`) is a REQ-API-004B violation, not acceptable input handling. -**REQ-API-004C (T0)** Every public client MUST expose one safe, high-level Lease inventory observer operation that owns the full subscribe-before-list bootstrap: (1) establish the patterned Subscribe and wait for its acknowledgement; (2) begin buffering matching LEASE_NOTIFY invalidations; (3) call `Lease.List` for the same selector; (4) install that result as the local observed view; (5) apply buffered invalidations accumulated during steps 1–4 and relist if the view may have changed. The observer MUST NOT report the view ready until this sequence completes, and callers MUST NOT be required to hand-roll it themselves. Reconnect MUST re-run the full bootstrap (a fresh Subscribe plus a fresh List) rather than resubscribing without refreshing inventory. See `docs/clients/spec/lease-schedule.md` and https://github.com/cntryl/fitz/issues/219 §5 for the wire-level sequencing this depends on and the required periodic-reconciliation and reconnect-recovery behavior; the same language-neutral bootstrap and lifecycle conformance scenarios MUST pass in all five SDKs. +**REQ-API-004C (T0)** Every public client MUST expose one safe, high-level Lease inventory observer operation that owns the full subscribe-before-list bootstrap: (1) establish the patterned Subscribe and wait for its acknowledgement; (2) begin buffering matching LEASE_NOTIFY invalidations; (3) call `Lease.List` for the same selector; (4) install that result as the local observed view; (5) apply buffered invalidations accumulated during steps 1–4 and relist if the view may have changed. The observer MUST NOT report the view ready until this sequence completes, and callers MUST NOT be required to hand-roll it themselves. Reconnect MUST re-run the full bootstrap (a fresh Subscribe plus a fresh List) rather than resubscribing without refreshing inventory. Every observer MUST also run a configurable, independently jittered periodic full-LIST backstop. The shared recommendation is `clamp(shortest expected lease TTL / 2, 5 seconds, 60 seconds)` with ±20% jitter and a 60-second default when the client has no workload-specific TTL. See `docs/clients/spec/lease-schedule.md` and AC-LEASE-014 in `docs/clients/acceptance/queue-notice-rpc-lease.md` for the sequencing, invalidation, retry, and common conformance scenarios that MUST pass in all five SDKs. **REQ-API-005 (T0)** The `Schedule.List` operation MUST support pagination (offset, limit) and return a total count alongside the results. diff --git a/docs/clients/spec/lease-schedule.md b/docs/clients/spec/lease-schedule.md index 5e895b35..ad48e5b4 100644 --- a/docs/clients/spec/lease-schedule.md +++ b/docs/clients/spec/lease-schedule.md @@ -305,6 +305,55 @@ selector, family, or broker lifetime it was issued from; 5012 (Invalid List Pattern) covers a pattern that fails the shared grammar on `LIST`. The related SUBSCRIBE/UNSUBSCRIBE operations use 5010 for that validation failure. +#### High-level Inventory Observer + +Every supported SDK exposes one high-level observer over a Lease selector. +The observer owns the race-sensitive ordering; applications do not compose a +bare `SUBSCRIBE` and `LIST` themselves: + +1. Send `SUBSCRIBE` and wait for its acknowledgement before starting `LIST`. +2. Buffer/coalesce every matching `LEASE_NOTIFY` invalidation from that point. +3. Drain one complete `LIST` snapshot for the identical selector. +4. Install the snapshot and report the view ready only if no invalidation, + reconnect, broker-lifetime change, subscription failure, or delivery + overflow crossed that LIST pass. +5. Otherwise discard the candidate and repeat a fresh full LIST while the + view remains not ready. + +In steady state, notifications schedule coalesced full-LIST reconciliation; +`QUERY` is insufficient because it cannot rebuild every LIST field. A +disconnect or broker restart invalidates readiness, and the normal reconnect +subscription restore counts as the required new wire subscription only when +it is followed immediately by a fresh complete LIST. Subscription termination +or delivery overflow removes the old registration and triggers a replacement +SUBSCRIBE plus fresh LIST. Transient replacement or LIST failures retry with +bounded exponential backoff and are coalesced so one observer has at most one +active recovery bootstrap. + +Each observer also performs a periodic full LIST as a backstop for lost +best-effort notifications. The interval is configurable, positive, and +independently jittered by ±20%. Use: + +```text +base_interval = clamp(shortest expected lease TTL / 2, 5 seconds, 60 seconds) +``` + +The 60-second default applies when the application has no workload-specific +TTL. The TTL/2 term targets two backstop passes during the shortest lease +lifetime; the five-second floor prevents one observer from scheduling more +than 0.2 bounded full scans per second; the 60-second ceiling bounds the normal +missed-notification window. LIST itself separately caps candidates examined, +returned items, encoded bytes, and retained snapshot state, so this formula +does not waive the requirement to narrow selectors whose bounded scan is +rejected. + +The installed view remains advisory and may become stale between +reconciliations. Observer output never contains an ownership capability and +cannot extend, release, transfer, or assume another session's lease. Observer +change/update queues must be bounded or coalesced; callers recover current +state by reading the installed view rather than treating updates as a durable +event log. + #### Response Types (Detailed) **Successful Responses:** diff --git a/docs/development/testing.md b/docs/development/testing.md index e8ea1bca..98e48a8e 100644 --- a/docs/development/testing.md +++ b/docs/development/testing.md @@ -84,7 +84,7 @@ Required coverage patterns: - KV: transaction scope, commit, rollback, stale transaction rejection, RouteFamily isolation, restart recovery of committed values only. - Queue: enqueue, reserve, complete, extend, redelivery, dead-letter handling, write-policy durability, invalid token rejection. - RPC: worker registration, request correlation, timeout, backpressure, streaming sequence, cleanup after disconnect or restart. -- Lease: single live holder, fencing token scope, renew/release token validation, wait ordering, restart loss. +- Lease: single live holder, fencing token scope, renew/release token validation, wait ordering, restart loss, selector grammar and authorization parity, exact keyed LIST, bounded wildcard snapshot work and retention, byte-bounded pagination, cursor misuse and expiry filtering, wildcard registration quotas and indexed fanout, notification transitions, RouteFamily isolation, and race-safe subscribe-before-list observers across reconnect, overflow, periodic reconciliation, and close on every supported client transport. - Schedule: persisted definitions, due-time handling, skip-forward overdue normalization, pending fire claims, live-only subscriptions. ## Determinism diff --git a/docs/operations/migration-guide.md b/docs/operations/migration-guide.md index 12b8a379..0fd88ebb 100644 --- a/docs/operations/migration-guide.md +++ b/docs/operations/migration-guide.md @@ -84,12 +84,20 @@ must surface the domain-specific validation and limit codes: KV 1012/1013, Stream 2010/2011, Notice 3002/3003, Queue 4010/4011, RPC 6012/6013, and Schedule 7006/7007. -Lease watches are exact-only. Rename Lease client request fields from `pattern` -to `route` and reject non-concrete `lease://realm/area/resource` input locally -when convenient; the broker returns 5010 authoritatively. The encoded string -payload is unchanged. Queue availability notifications now carry the concrete -three-segment Queue resource route rather than a synthetic `/ready` suffix. -Update Queue notification routing before upgrading the broker. +Lease `SUBSCRIBE` and `UNSUBSCRIBE` now accept the shared generic +three-segment selector grammar, including whole-segment `*` and valid +non-adjacent `**` forms. Existing exact subscriptions remain wire compatible; +new wildcard callers must use an updated SDK that applies the same grammar as +the broker and handles the 128-wildcard-registration session limit. `LIST` +(message 410) is a clean protocol addition with typed 5011/5012 +cursor/selector failures, and each supported SDK now provides a high-level +subscribe-before-list inventory observer. Upgrade broker and observer clients +together before enabling patterned fleet observation. ACQUIRE now rejects an +`owner_id` longer than 512 bytes so every legal holder can fit in a LIST item. + +Queue availability notifications now carry the concrete three-segment Queue +resource route rather than a synthetic `/ready` suffix. Update Queue +notification routing before upgrading the broker. ### Schedule cron day-field compatibility