Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions docs/clients/acceptance/queue-notice-rpc-lease.md
Original file line number Diff line number Diff line change
Expand Up @@ -526,4 +526,40 @@ match set, and zero or more currently held leases matching it
- `LIST` items can never be turned into an owned Lease handle; only exact
ACQUIRE/EXTEND/RELEASE change ownership

### AC-LEASE-014: Race-safe inventory observer lifecycle

**MUST** expose the same safe high-level observer behavior in TypeScript, Go,
.NET, Python, and Rust
**Given:** A public SDK observer for one valid Lease selector
**When:** The client bootstraps, receives invalidations, reconnects, encounters
subscription delivery failure, reconciles periodically, and closes
**Then:** Each SDK's observer conformance suite covers all of these common
scenarios:

1. `subscribe-before-list`: no LIST is sent before SUBSCRIBE acknowledgement;
a notification before/during LIST makes that candidate ineligible, a fresh
complete LIST is installed, and readiness becomes true only afterward.
2. `steady-state-invalidation`: a notification after readiness schedules a
coalesced full LIST and atomically replaces the complete view without
constructing an owned Lease handle.
3. `generation-change`: disconnect/reconnect or broker-lifetime change clears
readiness immediately; no LIST result spanning that boundary is installed;
a new/restored wire subscription plus a fresh complete LIST is required.
4. `subscription-failure`: handler overflow, delivery backpressure, or local
subscription termination clears readiness and triggers a replacement
SUBSCRIBE plus fresh LIST. A transient SUBSCRIBE or bootstrap LIST failure
retries with bounded backoff instead of silently degrading to polling-only.
5. `periodic-backstop`: a positive configurable interval causes a jittered
full LIST even without notifications. The documented recommendation is
`clamp(shortest expected lease TTL / 2, 5 seconds, 60 seconds)`, with ±20%
jitter and a 60-second default.
6. `bounded-close`: observer update delivery is bounded/coalesced; close is
idempotent, cancels recovery/reconciliation, unsubscribes, and leaves no
background work that can mutate the installed view.

Tests may use a deterministic scripted transport for race placement and a
live broker for public-surface acquire/release convergence, but all five SDKs
MUST exercise the same six behaviors and MUST use both TCP and WebSocket in
their shared broker conformance matrix.

## Schedule Domain
2 changes: 1 addition & 1 deletion docs/clients/client-requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ The acceptance criteria in `client-acceptance-criteria.md` are the normative sou

**REQ-API-004B (T0)** The `Lease.List` operation (LEASE_LIST 410) MUST be exposed, returning the current held-lease inventory matching a selector: exact route, the complete literal-or-`*` matrix over the three segments, or a `**` alias, using the same grammar and client-side validation as REQ-API-004. Each returned item MUST expose its exact route, logical `owner_id`, opaque `holder_incarnation` (never a raw session ID), `acquired_at`, remaining TTL, and renewal count. `Lease.List` MUST paginate per REQ-API-008 and MUST validate `limit` and any cursor offset as a non-negative value representable in an unsigned 32-bit integer before encoding it on the wire — silently wrapping or bit-coercing an out-of-range value (e.g. a `-1` limit becoming `4294967295`) is a REQ-API-004B violation, not acceptable input handling.

**REQ-API-004C (T0)** Every public client MUST expose one safe, high-level Lease inventory observer operation that owns the full subscribe-before-list bootstrap: (1) establish the patterned Subscribe and wait for its acknowledgement; (2) begin buffering matching LEASE_NOTIFY invalidations; (3) call `Lease.List` for the same selector; (4) install that result as the local observed view; (5) apply buffered invalidations accumulated during steps 1–4 and relist if the view may have changed. The observer MUST NOT report the view ready until this sequence completes, and callers MUST NOT be required to hand-roll it themselves. Reconnect MUST re-run the full bootstrap (a fresh Subscribe plus a fresh List) rather than resubscribing without refreshing inventory. See `docs/clients/spec/lease-schedule.md` and https://github.com/cntryl/fitz/issues/219 §5 for the wire-level sequencing this depends on and the required periodic-reconciliation and reconnect-recovery behavior; the same language-neutral bootstrap and lifecycle conformance scenarios MUST pass in all five SDKs.
**REQ-API-004C (T0)** Every public client MUST expose one safe, high-level Lease inventory observer operation that owns the full subscribe-before-list bootstrap: (1) establish the patterned Subscribe and wait for its acknowledgement; (2) begin buffering matching LEASE_NOTIFY invalidations; (3) call `Lease.List` for the same selector; (4) install that result as the local observed view; (5) apply buffered invalidations accumulated during steps 1–4 and relist if the view may have changed. The observer MUST NOT report the view ready until this sequence completes, and callers MUST NOT be required to hand-roll it themselves. Reconnect MUST re-run the full bootstrap (a fresh Subscribe plus a fresh List) rather than resubscribing without refreshing inventory. Every observer MUST also run a configurable, independently jittered periodic full-LIST backstop. The shared recommendation is `clamp(shortest expected lease TTL / 2, 5 seconds, 60 seconds)` with ±20% jitter and a 60-second default when the client has no workload-specific TTL. See `docs/clients/spec/lease-schedule.md` and AC-LEASE-014 in `docs/clients/acceptance/queue-notice-rpc-lease.md` for the sequencing, invalidation, retry, and common conformance scenarios that MUST pass in all five SDKs.

**REQ-API-005 (T0)** The `Schedule.List` operation MUST support pagination (offset, limit) and return a total count alongside the results.

Expand Down
49 changes: 49 additions & 0 deletions docs/clients/spec/lease-schedule.md
Original file line number Diff line number Diff line change
Expand Up @@ -305,6 +305,55 @@ selector, family, or broker lifetime it was issued from; 5012 (Invalid List
Pattern) covers a pattern that fails the shared grammar on `LIST`. The related
SUBSCRIBE/UNSUBSCRIBE operations use 5010 for that validation failure.

#### High-level Inventory Observer

Every supported SDK exposes one high-level observer over a Lease selector.
The observer owns the race-sensitive ordering; applications do not compose a
bare `SUBSCRIBE` and `LIST` themselves:

1. Send `SUBSCRIBE` and wait for its acknowledgement before starting `LIST`.
2. Buffer/coalesce every matching `LEASE_NOTIFY` invalidation from that point.
3. Drain one complete `LIST` snapshot for the identical selector.
4. Install the snapshot and report the view ready only if no invalidation,
reconnect, broker-lifetime change, subscription failure, or delivery
overflow crossed that LIST pass.
Comment thread
Copilot marked this conversation as resolved.
5. Otherwise discard the candidate and repeat a fresh full LIST while the
view remains not ready.

In steady state, notifications schedule coalesced full-LIST reconciliation;
`QUERY` is insufficient because it cannot rebuild every LIST field. A
disconnect or broker restart invalidates readiness, and the normal reconnect
subscription restore counts as the required new wire subscription only when
it is followed immediately by a fresh complete LIST. Subscription termination
or delivery overflow removes the old registration and triggers a replacement
SUBSCRIBE plus fresh LIST. Transient replacement or LIST failures retry with
bounded exponential backoff and are coalesced so one observer has at most one
active recovery bootstrap.

Each observer also performs a periodic full LIST as a backstop for lost
best-effort notifications. The interval is configurable, positive, and
independently jittered by ±20%. Use:

```text
base_interval = clamp(shortest expected lease TTL / 2, 5 seconds, 60 seconds)
```

The 60-second default applies when the application has no workload-specific
TTL. The TTL/2 term targets two backstop passes during the shortest lease
lifetime; the five-second floor prevents one observer from scheduling more
than 0.2 bounded full scans per second; the 60-second ceiling bounds the normal
missed-notification window. LIST itself separately caps candidates examined,
returned items, encoded bytes, and retained snapshot state, so this formula
does not waive the requirement to narrow selectors whose bounded scan is
rejected.

The installed view remains advisory and may become stale between
reconciliations. Observer output never contains an ownership capability and
cannot extend, release, transfer, or assume another session's lease. Observer
change/update queues must be bounded or coalesced; callers recover current
state by reading the installed view rather than treating updates as a durable
event log.

#### Response Types (Detailed)

**Successful Responses:**
Expand Down
2 changes: 1 addition & 1 deletion docs/development/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ Required coverage patterns:
- KV: transaction scope, commit, rollback, stale transaction rejection, RouteFamily isolation, restart recovery of committed values only.
- Queue: enqueue, reserve, complete, extend, redelivery, dead-letter handling, write-policy durability, invalid token rejection.
- RPC: worker registration, request correlation, timeout, backpressure, streaming sequence, cleanup after disconnect or restart.
- Lease: single live holder, fencing token scope, renew/release token validation, wait ordering, restart loss.
- Lease: single live holder, fencing token scope, renew/release token validation, wait ordering, restart loss, selector grammar and authorization parity, exact keyed LIST, bounded wildcard snapshot work and retention, byte-bounded pagination, cursor misuse and expiry filtering, wildcard registration quotas and indexed fanout, notification transitions, RouteFamily isolation, and race-safe subscribe-before-list observers across reconnect, overflow, periodic reconciliation, and close on every supported client transport.
- Schedule: persisted definitions, due-time handling, skip-forward overdue normalization, pending fire claims, live-only subscriptions.

## Determinism
Expand Down
20 changes: 14 additions & 6 deletions docs/operations/migration-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,12 +84,20 @@ must surface the domain-specific validation and limit codes: KV 1012/1013,
Stream 2010/2011, Notice 3002/3003, Queue 4010/4011, RPC 6012/6013, and Schedule
7006/7007.

Lease watches are exact-only. Rename Lease client request fields from `pattern`
to `route` and reject non-concrete `lease://realm/area/resource` input locally
when convenient; the broker returns 5010 authoritatively. The encoded string
payload is unchanged. Queue availability notifications now carry the concrete
three-segment Queue resource route rather than a synthetic `/ready` suffix.
Update Queue notification routing before upgrading the broker.
Lease `SUBSCRIBE` and `UNSUBSCRIBE` now accept the shared generic
three-segment selector grammar, including whole-segment `*` and valid
non-adjacent `**` forms. Existing exact subscriptions remain wire compatible;
new wildcard callers must use an updated SDK that applies the same grammar as
the broker and handles the 128-wildcard-registration session limit. `LIST`
(message 410) is a clean protocol addition with typed 5011/5012
cursor/selector failures, and each supported SDK now provides a high-level
subscribe-before-list inventory observer. Upgrade broker and observer clients
together before enabling patterned fleet observation. ACQUIRE now rejects an
`owner_id` longer than 512 bytes so every legal holder can fit in a LIST item.

Queue availability notifications now carry the concrete three-segment Queue
resource route rather than a synthetic `/ready` suffix. Update Queue
notification routing before upgrading the broker.

### Schedule cron day-field compatibility

Expand Down