diff --git a/admin-api/guide.md b/admin-api/guide.md index d40f77bb..9993f726 100644 --- a/admin-api/guide.md +++ b/admin-api/guide.md @@ -336,7 +336,7 @@ The admin server provides 21 webhook/event endpoints for real-time observability - **Delivery tracking**: list delivery attempts per subscription with status/date filters - **Event replay**: re-deliver historical events to a subscription - **Security config**: manage webhook URL SSRF protection (blocked CIDRs, HTTPS enforcement) -- **Tenant self-service**: tenants manage their own webhooks at `/v1/webhooks` for budget, reservation, and tenant events (27 of 47 registered event types, plus the additive `_via_tenant_cascade` fan-out events the reference admin server emits in those categories on tenant close — see [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics)) +- **Tenant self-service**: tenants manage their own webhooks at `/v1/webhooks` for budget, reservation, and tenant events (29 of 51 registered event types, including the `_via_tenant_cascade` fan-out events the admin server emits in those categories on tenant close — see [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics)) Events are emitted by admin controllers (tenant, budget, api-key, policy operations) and delivered asynchronously by the events service (`cycles-server-events`). See [Webhooks and Events](/concepts/webhooks-and-events) for architecture details. @@ -420,7 +420,7 @@ curl -X POST http://localhost:7979/v1/webhooks \ }' ``` -See [Webhook Event Delivery Protocol](/protocol/webhook-event-delivery-protocol) for the full 47-event-type reference and delivery specification. See [Webhook Integrations](/how-to/webhook-integrations) for PagerDuty, Slack, and ServiceNow examples. +See [Webhook Event Delivery Protocol](/protocol/webhook-event-delivery-protocol) for the full 51-event-type reference and delivery specification. See [Webhook Integrations](/how-to/webhook-integrations) for PagerDuty, Slack, and ServiceNow examples. ## List-endpoint features (v0.1.25.22+) diff --git a/blog/real-time-budget-alerts-for-ai-agents.md b/blog/real-time-budget-alerts-for-ai-agents.md index b769f9f8..66a266df 100644 --- a/blog/real-time-budget-alerts-for-ai-agents.md +++ b/blog/real-time-budget-alerts-for-ai-agents.md @@ -40,7 +40,7 @@ This is why we built a webhook event system into Cycles v0.1.25. ## 41 event types across 6 categories -> **As of post date.** The current Admin API `EventType` enum registers **47 event types across 7 categories** (the `webhook` category was added later). For the live count and per-category breakdown, see the [Event Payloads Reference](/protocol/event-payloads-reference). +> **As of post date.** The current Admin API `EventType` enum registers **51 event types across 7 categories** (the `webhook` category and the tenant-close cascade types were added later). For the live count and per-category breakdown, see the [Event Payloads Reference](/protocol/event-payloads-reference). Every observable state change in the system produces an event. We organized them into 6 categories covering the full lifecycle: @@ -172,7 +172,7 @@ The response includes a signing secret (returned once — store it). Your middle We have full integration guides with code examples for [PagerDuty, Slack, Datadog, Microsoft Teams, Opsgenie, and ServiceNow](/how-to/webhook-integrations), plus a [custom receiver pattern](/how-to/webhook-integrations#integration-custom-receiver-direct) with signature verification in Python, Node.js, and Go. -Tenants can also create their own webhook subscriptions via `/v1/webhooks` using their API key — restricted to budget, reservation, and tenant events (27 of 41 types as of post date; the live count is 27 of 47 — see the [Webhook Event Delivery Protocol](/protocol/webhook-event-delivery-protocol#tenant-accessible-events) for current category eligibility). Admin-only events (api_key, policy, webhook, system) require admin key access. +Tenants can also create their own webhook subscriptions via `/v1/webhooks` using their API key — restricted to budget, reservation, and tenant events (27 of 41 types as of post date; the live count is 29 of 51 — see the [Webhook Event Delivery Protocol](/protocol/webhook-event-delivery-protocol#tenant-accessible-events) for current category eligibility). Admin-only events (api_key, policy, webhook, system) require admin key access. Webhook URLs are validated at creation time with SSRF protection enabled by default: RFC 1918 private IP ranges, loopback, and link-local addresses are blocked, and HTTPS is required in production. These can be configured via `PUT /v1/admin/config/webhook-security` for environments that need internal endpoint access. diff --git a/blog/tenant-lifecycle-cascade-semantics-at-scale.md b/blog/tenant-lifecycle-cascade-semantics-at-scale.md index 855568ab..50af3a69 100644 --- a/blog/tenant-lifecycle-cascade-semantics-at-scale.md +++ b/blog/tenant-lifecycle-cascade-semantics-at-scale.md @@ -51,13 +51,15 @@ Any combination of these carries real operational and security risk: silent post The Cycles governance-admin spec addresses this directly with a two-rule contract, formalized in the `CASCADE SEMANTICS` section of the v0.1.25 governance-admin yaml. -**Rule 1 — Close Cascade.** When a tenant transitions to `CLOSED`, the server performs a strictly ordered sequence against every owned object: +**Rule 1 — Close Cascade.** When a tenant transitions to `CLOSED`, the server drives every owned object to its terminal state. In the atomic presentation (Mode A, below), the spec's recommended order within the single transaction is: 1. Drain open reservations (released with reason `tenant_closed`; no overage debt recorded). 2. Close budget ledgers (final balance snapshot preserved for audit; no new reservations accepted). 3. Disable webhook subscriptions and revoke API keys (either order). 4. Flip `tenant.status` to `CLOSED` last. +Mode B (below) inverts this by design — the tenant flip commits *first*, and the children converge afterward under the Rule 2 guard. Since the runcycles reference server implements Mode B, don't build tooling that depends on this ordering; depend on the terminal states and the guard. + Each mutated object produces a dedicated record: an Event row under a reserved dotted name — `budget.closed_via_tenant_cascade`, `api_key.revoked_via_tenant_cascade`, `reservation.released_via_tenant_cascade` (ledger-level), `webhook.disabled_via_tenant_cascade` — plus an audit row written as `operation="tenant_close_cascade"` with `resource_type`/`resource_id`. The event rows all share a server-composed `correlation_id` (`tenant_close_cascade::`) on the emitted event rows, so an auditor can reconstruct the cascade in a single events query (audit rows join via `request_id`/`trace_id`). **Rule 2 — Terminal-Owner Mutation Guard.** Every mutating endpoint on an owned object first checks the parent tenant's status. If the tenant is `CLOSED`, the endpoint returns `409 Conflict` with `error: "TENANT_CLOSED"`, regardless of the per-object terminal state. The guard applies across the budget, reservation, policy, API key, and webhook planes. `GET` endpoints remain available — closed-tenant state is readable for post-mortems and compliance evidence. @@ -77,7 +79,7 @@ How these rules are implemented can vary. A protocol spec that only accepted one - Convergence within a documented bound. - Observable reads of non-terminal children remain consistent until the cascade reaches them. -The important property is that both modes produce the same *client-observable* outcome: once the tenant is `CLOSED`, every mutation against any owned object returns a `409` with `error: "TENANT_CLOSED"`, regardless of which per-object row flipped first. The mode is an implementation detail the spec deliberately leaves open — a transactional SQL backend can deliver Mode A cleanly, while a Redis-backed admin can opt into Mode B as long as the guard activates at or before the flip's durability. +The important property is that both modes produce the same *client-observable* outcome: once the tenant is `CLOSED`, every admin-plane mutation against any owned object returns a `409` with `error: "TENANT_CLOSED"`, regardless of which per-object row flipped first — and the same guard holds post-auth on the runtime plane, where a revoked tenant key is rejected with `401` before the guard is ever consulted (the close walkthrough below shows both doors). The mode is an implementation detail the spec deliberately leaves open — a transactional SQL backend can deliver Mode A cleanly, while a Redis-backed admin can opt into Mode B as long as the guard activates at or before the flip's durability. ## Where operators actually trip @@ -116,10 +118,14 @@ curl -s -H "X-Admin-API-Key: $ADMIN_KEY" \ You'll see one record per owned object — one `budget.closed_via_tenant_cascade` per ledger, one `api_key.revoked_via_tenant_cascade` per key, one `webhook.disabled_via_tenant_cascade` per subscription — plus one **ledger-level** `reservation.released_via_tenant_cascade` per closed budget that had `reserved > 0`, carrying the aggregate `released_amount` (not one event per reservation). The corresponding event rows all share the server-composed cascade `correlation_id`, which is how an auditor reconstructs the cascade without having to cross-join on timestamp (audit rows join via the originating `request_id`). -A subsequent attempt to mutate an owned object under the closed tenant returns the terminal-owner guard's `409`. Reservation lifecycle lives on the runtime plane — the spec's Rule 2 explicitly scopes reservation create/commit/release/extend, so the `409 TENANT_CLOSED` below is the normative contract (note: the current runtime reference server's error enum does not yet include `TENANT_CLOSED`; today it surfaces closed tenants as `401`s from revoked keys or `BUDGET_CLOSED`): +A subsequent attempt to mutate an owned object under the closed tenant returns the terminal-owner guard's `409`. Reservation lifecycle lives on the runtime plane — the spec's Rule 2 explicitly scopes reservation create/commit/release/extend, so the `409 TENANT_CLOSED` below is the normative contract — implemented in `cycles-server` 0.1.25.47, which added `TENANT_CLOSED` to the runtime error enum per spec revision v0.1.25.13 (on 0.1.25.46 and earlier, the runtime plane surfaces closed tenants only as `401`s from revoked keys or `BUDGET_CLOSED`): ```bash # Mutation on a released reservation under a closed tenant +# (reachable with a not-yet-revoked tenant key in the post-flip window — +# once the cascade revokes the key, the 401 below wins first. Of the four +# guarded mutations only release also accepts an admin key on the runtime +# plane, so an admin-on-behalf-of release hits this 409 with no race.) curl -i -X POST \ -H "X-Cycles-API-Key: $TENANT_KEY" \ "http://localhost:7878/v1/reservations/res-xyz/commit" diff --git a/changelog.md b/changelog.md index 7aab8620..86e42d87 100644 --- a/changelog.md +++ b/changelog.md @@ -15,10 +15,10 @@ Since the initial v0.1.25 Events & Webhooks release, each component has shipped | Component | Version | Release date | |---|---|---| -| Protocol spec (runtime) | v0.1.25 (document revision v0.1.25.12) | 2026-05-22 | -| Governance spec (admin) | v0.1.25.34 | 2026-04-20 | -| `cycles-server` (runtime) | v0.1.25.46 | 2026-07-04 | -| `cycles-server-admin` | v0.1.25.48 | 2026-07-04 | +| Protocol spec (runtime) | v0.1.25 (document revision v0.1.25.13) | 2026-07-10 | +| Governance spec (admin) | v0.1.25.39 | 2026-07-11 | +| `cycles-server` (runtime) | v0.1.25.47 | 2026-07-10 | +| `cycles-server-admin` | v0.1.25.50 | 2026-07-10 | | `cycles-server-events` | v0.1.25.22 | 2026-07-04 | | `cycles-dashboard` | v0.1.25.67 | 2026-07-04 | @@ -40,12 +40,14 @@ Each client repo's AUDIT.md records the specific protocol revision that release ### Protocol spec suite (v0.1.26) +- **Runtime spec document revision v0.1.25.13** (2026-07-10) — adds `TENANT_CLOSED` to the runtime `ErrorCode` enum with a normative closed-tenant binding in §ERROR SEMANTICS: persisting reservation create/commit/release/extend MUST return `409 TENANT_CLOSED` on a closed owning tenant (taking precedence over reservation-state errors for non-replay attempts), while fresh `dry_run=true` / `/v1/decide` evaluations MUST return `200 decision=DENY reason_code=TENANT_CLOSED` (new `DecisionReasonCode` known value); present-but-malformed tenant records fail closed with `500 INTERNAL_ERROR`. Companion revisions: evidence spec 0.2.1 (declares `TENANT_CLOSED` in the evidence ErrorResponseMirror), protocol-extensions 0.1.27, governance 0.1.25.37 (adds `TENANT_CLOSED` to the `reservation.denied` event's documented `reason_code` values). Implemented by `cycles-server` 0.1.25.47. - The cycles-protocol repo added [`CONFORMANCE.md`](https://github.com/runcycles/cycles-protocol/blob/main/CONFORMANCE.md) — a formal MUST / SHOULD / MAY statement of what a conformant Cycles implementation has to do. The active v0.1.25 target requires 12 MUST operations: 4 core runtime reservation operations plus 8 cross-plane event / webhook / balance / auth-introspection operations. Four more runtime operations (`decide`, list/get reservations, direct-debit events) are SHOULD-level and implemented by the reference servers. - The README repositioned the spec suite around upcoming **v0.1.26** extensions (runtime base still v0.1.25). Action-kinds, action-quotas, observe mode, DenyDetail, and `ACTION_QUOTA_EXCEEDED` / `ACTION_KIND_DENIED` / `ACTION_KIND_NOT_ALLOWED` reason codes are SHOULD-level today and **not yet enforced** in runcycles' reference servers. They become MUST only when `CONFORMANCE.md` promotes v0.1.26 to the active target. - Spec-only trace_id alignment bumps on extension specs (`cycles-action-kinds`, `cycles-governance-extensions` to v0.1.27) — declare `trace_id` on `ErrorResponse` and `X-Cycles-Trace-Id` on `components.headers` for OpenAPI tooling consistency. Behavioral contract unchanged from what's documented in [Correlation and Tracing](/protocol/correlation-and-tracing-in-cycles). ### Runtime server (`cycles-server`) +- **v0.1.25.47** (2026-07-10) — **TENANT_CLOSED Rule 2 guard + webhook matcher parity.** The four reservation mutations (create/commit/release/extend) now reject with `409 TENANT_CLOSED` when the owning tenant's `CLOSED` flip is durable (governance CASCADE SEMANTICS Rule 2 / Mode B invariant (a); runtime spec v0.1.25.13) — checked inside the Lua scripts, atomic with the budget mutations, taking precedence over reservation-state errors for non-replay attempts. Fresh `dry_run` / `/v1/decide` evaluations on a closed tenant return `200 decision=DENY reason_code=TENANT_CLOSED`; malformed tenant records fail closed with `500`; deployments without tenant records are unaffected. Mutation-surface 409s emit `error` CyclesEvidence (create/commit/release). In practice tenant-key calls usually still `401` at the auth filter; the 409 surfaces mainly on admin-on-behalf-of release and the post-flip/pre-revocation race window. Also refreshes the webhook `scope_filter` dispatch matcher to be byte-identical to the admin plane's spec-conformant matcher: blank event scopes are treated as unscoped, and trailing-`/*` filters require a non-empty child segment. See [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics) and [Webhook Scope Filter Syntax](/protocol/webhook-scope-filter-syntax). - **v0.1.25.46** (2026-07-04) — **Public-endpoint rate limiting.** The unauthenticated `GET /v1/evidence/*` and CyclesEvidence JWKS endpoints are now rate-limited per client IP (default 300 requests/minute, `CYCLES_PUBLIC_RATE_LIMIT_REQUESTS_PER_MINUTE`; kill switch `CYCLES_PUBLIC_RATE_LIMIT_ENABLED`). Over-limit requests receive `429` with `error=LIMIT_EXCEEDED` (new runtime ErrorCode, spec v0.1.25.12). - **v0.1.25.45** (2026-06-27) — **Operational endpoints require the admin key.** `/actuator/prometheus`, `/actuator/info`, aggregate `/actuator/health`, and the API docs / Swagger endpoints now require `X-Admin-API-Key`; liveness/readiness probes and the protocol-public CyclesEvidence/JWKS endpoints stay unauthenticated. Also bounds the event-emission executor queue and adds correlation headers on auth-filter errors. - **v0.1.25.44** (2026-06-26) — Deployment hardening: production Compose disables tenant labels on Prometheus metrics and public SpringDoc/Swagger; `exec java` PID-1 entrypoint. No API change. @@ -73,6 +75,8 @@ Each client repo's AUDIT.md records the specific protocol revision that release ### Admin server (`cycles-server-admin`) +- **v0.1.25.50** (2026-07-10) — **SECURITY: tenant-plane webhook `event_categories` validated against the tenant-accessible boundary.** `POST /v1/webhooks` and `PATCH /v1/webhooks/{id}` validated `event_types` (budget/reservation/tenant only) but never `event_categories` — and delivery matching treats categories as an additive union with types, so a tenant key could subscribe to an admin-only category (`api_key`, `policy`, `webhook`, `system`) and receive admin event classes for its tenant. Both paths now reject an admin-only category with `400 INVALID_REQUEST` (governance spec revision v0.1.25.38). The update path also closes a legacy door where clearing both `event_types` and `event_categories` produced a delivery-side match-ALL subscription. **Upgrade past 0.1.25.49** and audit existing tenant subscriptions for admin-only categories and empty-both state — see the [release notes](https://github.com/runcycles/cycles-server-admin/releases/tag/v0.1.25.50) for the audit one-liner. (Category-only subscriptions — empty `event_types` with non-empty `event_categories` — remain valid on update per governance v0.1.25.39.) Residual event-provenance hardening tracked as cycles-server-admin#209. +- **v0.1.25.49** (2026-07-10) — **Spec-conformant webhook `scope_filter` matching + replay filtering.** BEHAVIOR CHANGE: the admin matcher (`WebhookRepository.matchesScope`) moves from literal prefix matching to the spec's exact-match-with-trailing-`*` semantics — bare-prefix filters must be rewritten as `…/*` to keep matching child scopes, "base + descendants" coverage now needs two subscriptions, and null/blank-scope events are excluded from every scope-filtered subscription. Webhook replay now applies the same matcher (previously it bypassed scope matching entirely). Converges with the runtime matcher shipped in `cycles-server` v0.1.25.47 — both planes now match identically and are pinned to the same test table. See the [migration notes](https://github.com/runcycles/cycles-server-admin/releases/tag/v0.1.25.49) and [Webhook Scope Filter Syntax](/protocol/webhook-scope-filter-syntax). - **v0.1.25.48** (2026-07-04) — Cascade event payloads now map to a typed `EventDataTenantCascade` class (the four `*_via_tenant_cascade` event types). Internal validation/registry only — no wire change. - **v0.1.25.47** (2026-06-26) — Admin image readiness healthcheck + `exec java $JAVA_OPTS` entrypoint; full-stack compose pins refreshed; README/OPERATIONS deployment doc refresh. - **v0.1.25.46** (2026-06-26) — Dependency currency: Jedis 7.5.0 → 7.5.2, springdoc-openapi 2.8.16 → 2.8.17. No code change. @@ -298,8 +302,8 @@ The default `commit_overage_policy` changed from **`REJECT`** to **`ALLOW_IF_AVA | `cycles-client-java-spring` | 0.2.5 | v0.1.23+, v0.1.24+, v0.1.25+ | | `@runcycles/mcp-server` | 0.2.4 | v0.1.23+, v0.1.24+, v0.1.25+ | | `@runcycles/openclaw-budget-guard` | 0.8.4 | v0.1.23+, v0.1.24+, v0.1.25+ | -| Cycles Server (runtime) | v0.1.25.46 | Protocol v0.1.25 plus CyclesEvidence v0.2 signer-authority layer | -| Cycles Admin Server | v0.1.25.48 | Governance spec v0.1.25.34 | +| Cycles Server (runtime) | v0.1.25.47 | Protocol v0.1.25 plus CyclesEvidence v0.2 signer-authority layer | +| Cycles Admin Server | v0.1.25.50 | Governance spec v0.1.25.39 | | Cycles Events Service | v0.1.25.22 | Shared Redis dispatch queue plus CyclesEvidence signing queue | | Cycles Dashboard | v0.1.25.67 | Admin v0.1.25.39+ for current governance views; runtime v0.1.25.37+ for reservation evidence links; events v0.1.25.14+ for signed evidence | @@ -311,7 +315,8 @@ All current SDK versions are backward-compatible with server v0.1.23. New v0.1.2 |---|---| | Dashboard Evidence viewer and reservation "View evidence" links | `cycles-dashboard` v0.1.25.63+; `cycles-server` v0.1.25.37+ for `include=evidence`; `cycles-server-events` v0.1.25.14+ to sign envelopes | | Unconfigured CyclesEvidence disabled mode (no queueing or dead-lettering when identity is blank) | `cycles-server` v0.1.25.38, `cycles-server-events` v0.1.25.15 | -| Tenant-close cascade + `TENANT_CLOSED` (409) error code + 4 `_via_tenant_cascade` event kinds | `cycles-server-admin` v0.1.25.35 (initial Mode B cascade) / v0.1.25.36 (full Rule 2 guard coverage); `cycles-dashboard` v0.1.25.43 (tombstone + cascade preview UI); governance-admin spec v0.1.25.29 / .30 / .31 | +| Tenant-close cascade + `TENANT_CLOSED` (409) on the admin plane + 4 `_via_tenant_cascade` event kinds | `cycles-server-admin` v0.1.25.35 (initial Mode B cascade) / v0.1.25.36 (full Rule 2 guard coverage); `cycles-dashboard` v0.1.25.43 (tombstone + cascade preview UI); governance-admin spec v0.1.25.29 / .30 / .31 | +| `TENANT_CLOSED` (409) runtime guard on reservation create/commit/release/extend + `reason_code=TENANT_CLOSED` on fresh dry-run / `/v1/decide` DENYs | `cycles-server` v0.1.25.47 (runtime spec v0.1.25.13) | | W3C Trace Context (`trace_id` on responses + audit/events filter) | `cycles-server` v0.1.25.14, `cycles-server-admin` v0.1.25.31, `cycles-server-events` v0.1.25.7, `cycles-dashboard` v0.1.25.39 | | Runtime audit-log retention TTL (`AUDIT_RETENTION_DAYS`) | `cycles-server` v0.1.25.15 | | Events service management port split (9980) | `cycles-server-events` v0.1.25.9 | diff --git a/concepts/webhooks-and-events.md b/concepts/webhooks-and-events.md index e3b5f43f..bb566702 100644 --- a/concepts/webhooks-and-events.md +++ b/concepts/webhooks-and-events.md @@ -45,21 +45,23 @@ A subscription defines which events to deliver and where: The events service is **optional**. If not deployed, events accumulate in Redis with TTL and are delivered when the service starts. -## 47 Registered Event Types +## 51 Registered Event Types | Category | Count | Examples | |---|---|---| -| budget | 16 | `budget.exhausted`, `budget.threshold_crossed`, `budget.over_limit_entered`, `budget.funded`, `budget.reset_spent` | -| reservation | 5 | `reservation.denied`, `reservation.commit_overage`, `reservation.expired` | +| budget | 17 | `budget.exhausted`, `budget.threshold_crossed`, `budget.over_limit_entered`, `budget.funded`, `budget.closed_via_tenant_cascade` | +| reservation | 6 | `reservation.denied`, `reservation.commit_overage`, `reservation.released_via_tenant_cascade` | | tenant | 6 | `tenant.created`, `tenant.suspended`, `tenant.closed` | -| api_key | 6 | `api_key.created`, `api_key.revoked`, `api_key.auth_failed` | +| api_key | 7 | `api_key.created`, `api_key.revoked`, `api_key.revoked_via_tenant_cascade` | | policy | 3 | `policy.created`, `policy.updated`, `policy.deleted` | -| webhook | 6 | `webhook.created`, `webhook.paused`, `webhook.disabled` | +| webhook | 7 | `webhook.created`, `webhook.paused`, `webhook.disabled_via_tenant_cascade` | | system | 5 | `system.store_connection_lost`, `system.webhook_delivery_failed` | +The four `*_via_tenant_cascade` types were added to the enum in governance spec revision v0.1.25.35. + ## Tenant Self-Service -Tenants can create their own webhook subscriptions via `/v1/webhooks` (requires `webhooks:write` permission). Tenant webhooks are restricted to budget, reservation, and tenant events: 27 of the 47 registered event types — plus the additive `_via_tenant_cascade` fan-out events that the reference admin server emits in those same categories during a tenant close (see [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics)). API key, policy, webhook lifecycle, and system events are admin-only. +Tenants can create their own webhook subscriptions via `/v1/webhooks` (requires `webhooks:write` permission). Tenant webhooks are restricted to budget, reservation, and tenant events: 29 of the 51 registered event types — including the `budget.*` and `reservation.*` `_via_tenant_cascade` fan-out events emitted during a tenant close (see [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics)). API key, policy, webhook lifecycle, and system events are admin-only. ## Security diff --git a/glossary.md b/glossary.md index 804fda66..ef9a9915 100644 --- a/glossary.md +++ b/glossary.md @@ -223,7 +223,7 @@ An HTTP POST callback triggered by a state change event. Cycles delivers webhook ### Event (Webhook) -An immutable record of a state change (e.g., `budget.exhausted`, `reservation.denied`). The v0.1.25 Admin API `EventType` enum registers 47 event types across 7 categories (budget: 16, reservation: 5, tenant: 6, api_key: 6, policy: 3, webhook: 6, system: 5). The `webhook` category covers six lifecycle events (`webhook.created` / `.updated` / `.paused` / `.resumed` / `.disabled` / `.deleted`) added in spec v0.1.25.33; the four `_via_tenant_cascade` event names are additive reference-server payloads emitted by the tenant-close cascade and not part of the registered enum. Events are stored in Redis with configurable TTL (default 90 days) and dispatched to matching webhook subscriptions. +An immutable record of a state change (e.g., `budget.exhausted`, `reservation.denied`). The v0.1.25 Admin API `EventType` enum registers 51 event types across 7 categories (budget: 17, reservation: 6, tenant: 6, api_key: 7, policy: 3, webhook: 7, system: 5). The `webhook` category covers six lifecycle events (`webhook.created` / `.updated` / `.paused` / `.resumed` / `.disabled` / `.deleted`) added in spec v0.1.25.33; the four `_via_tenant_cascade` event names emitted by the tenant-close cascade are part of the registered enum since governance revision v0.1.25.35. Events are stored in Redis with configurable TTL (default 90 days) and dispatched to matching webhook subscriptions. ### Signing Secret @@ -293,7 +293,7 @@ Two reserved `tenant_id` values on audit-log entries (v0.1.25.28+): `__admin__` ### TENANT_CLOSED -A `409` error code (introduced in v0.1.25.35; full coverage across every mutating admin-plane operation as of v0.1.25.36) returned when mutating an object whose owning tenant is `CLOSED`. Enforced by the "Rule 2 — Terminal-Owner Mutation Guard" half of the cascade contract; GET endpoints remain available for post-mortem audit reads. See [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics) and [Error Codes — TENANT_CLOSED](/protocol/error-codes-and-error-handling-in-cycles#tenant-closed-409). +A `409` error code returned when mutating an object whose owning tenant is `CLOSED`. Enforced by the "Rule 2 — Terminal-Owner Mutation Guard" half of the cascade contract: on the admin plane since `cycles-server-admin` v0.1.25.35 (full coverage across every mutating admin-plane operation as of v0.1.25.36), and on the runtime plane since `cycles-server` 0.1.25.47 (runtime spec v0.1.25.13) for persisting reservation create/commit/release/extend — where fresh dry-run and `/v1/decide` evaluations instead return `200 decision=DENY` with `reason_code=TENANT_CLOSED`. GET endpoints remain available for post-mortem audit reads. See [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics) and [Error Codes — TENANT_CLOSED](/protocol/error-codes-and-error-handling-in-cycles#tenant-closed-409). ### Tenant-Close Cascade diff --git a/how-to/managing-webhooks.md b/how-to/managing-webhooks.md index 69007b0d..60309b36 100644 --- a/how-to/managing-webhooks.md +++ b/how-to/managing-webhooks.md @@ -15,7 +15,7 @@ Every action in this guide — create, test, replay, pause/enable, reset failure ### Admin subscription -Required fields: `url` and `event_types` (at least one event type). Add `?tenant_id=acme-corp` to scope the subscription to a specific tenant; omit for system-wide subscriptions (all tenants). All other fields are optional — the server provides sensible defaults (`signing_secret` is auto-generated if omitted). +Required fields: `url` and `event_types` (at least one event type on create). Add `?tenant_id=acme-corp` to scope the subscription to a specific tenant; omit for system-wide subscriptions (all tenants). All other fields are optional — the server provides sensible defaults (`signing_secret` is auto-generated if omitted). On update (`PATCH`), `event_types` may be cleared to empty as long as `event_categories` is non-empty (a category-only subscription); the server rejects only the empty-both state. ```bash # Tenant-scoped subscription (receives events for acme-corp only) @@ -67,10 +67,10 @@ The generated secret (e.g., `whsec_dGVzdC1zZWNy...`) is in the response. Copy it ### Category-based subscriptions -Subscribe to **all events in a category** using `event_categories`. This is additive with `event_types` — if you specify both, you get the union. Note: `event_types` is always required (at least one), so include a representative type alongside the category wildcard. +Subscribe to **all events in a category** using `event_categories`. This is additive with `event_types` — if you specify both, you get the union. Note: on **create**, `event_types` must be non-empty, so include a representative type alongside the category wildcard. (A later `PATCH` may clear `event_types` to leave a category-only subscription; the server rejects only the state where both arrays are empty.) ```bash -# All budget events (16 types) + all reservation events (5 types) +# All budget events (17 types) + all reservation events (6 types) curl -X POST http://localhost:7979/v1/admin/webhooks \ -H "X-Admin-API-Key: $ADMIN_KEY" \ -H "Content-Type: application/json" \ @@ -275,6 +275,13 @@ curl -X PATCH http://localhost:7979/v1/admin/webhooks/whsub_abc123 \ -H "Content-Type: application/json" \ -d '{"event_types": ["budget.exhausted", "budget.threshold_crossed", "reservation.denied"]}' +# Switch to a category-only subscription: clear event_types, keep categories. +# Valid on update (unlike create); the server rejects only the empty-both state. +curl -X PATCH http://localhost:7979/v1/admin/webhooks/whsub_abc123 \ + -H "X-Admin-API-Key: $ADMIN_KEY" \ + -H "Content-Type: application/json" \ + -d '{"event_types": [], "event_categories": ["budget", "reservation"]}' + # Adjust retry policy curl -X PATCH http://localhost:7979/v1/admin/webhooks/whsub_abc123 \ -H "X-Admin-API-Key: $ADMIN_KEY" \ diff --git a/how-to/migrating-from-custom-rate-limiter-to-cycles.md b/how-to/migrating-from-custom-rate-limiter-to-cycles.md index 079cc62d..3fa04ffc 100644 --- a/how-to/migrating-from-custom-rate-limiter-to-cycles.md +++ b/how-to/migrating-from-custom-rate-limiter-to-cycles.md @@ -304,7 +304,7 @@ The migration is safe because: | Cross-provider budget | Manual per-provider tracking | Single scope hierarchy | | Retry deduplication | No | Idempotency keys on every operation | | Action-level risk control | No | RISK_POINTS budgets | -| Webhook alerts | Custom implementation | Built-in (47 registered event types across 7 categories, PagerDuty/Slack) | +| Webhook alerts | Custom implementation | Built-in (51 registered event types across 7 categories, PagerDuty/Slack) | | Multi-tenant isolation | Manual Redis key prefixing | Built-in scope hierarchy | | Delegation attenuation | No | Sub-budget carving for sub-agents | | Shadow mode validation | No | `decide()` endpoint for shadow evaluation | diff --git a/how-to/prometheus-metrics-reference.md b/how-to/prometheus-metrics-reference.md index 8e5edd33..781e5780 100644 --- a/how-to/prometheus-metrics-reference.md +++ b/how-to/prometheus-metrics-reference.md @@ -73,7 +73,7 @@ Introduced in v0.1.25.6. Mirrors the runtime's conventions: `cycles.webhook.*` r | Tag | Values | |---|---| -| `event_type` | Event kind from the [Event Payloads Reference](/protocol/event-payloads-reference) (e.g. `reservation.denied`, `budget.exhausted`, `webhook.disabled`). Up to 47 registered values, plus additive implementation events over time. | +| `event_type` | Event kind from the [Event Payloads Reference](/protocol/event-payloads-reference) (e.g. `reservation.denied`, `budget.exhausted`, `webhook.disabled`). Up to 51 registered values, plus additive implementation events over time. | | `status_code_family` | `2xx` (success bucket). Non-2xx responses land on `cycles_webhook_delivery_failed_total` with `reason` instead. | | `reason` (on `_failed_total`) | `event_not_found`, `subscription_not_found`, `subscription_inactive`, `http_4xx`, `http_5xx`, `transport_error` (timeouts, connection resets, DNS/SSL failures — status code 0), `ssrf_blocked`. | @@ -124,7 +124,7 @@ The `tenant` tag is the dominant cardinality driver. A deployment with 10,000 te 2. **Aggregate at scrape time** with `metric_relabel_configs` to drop the tag selectively on high-cardinality metrics while keeping it on the ones you still want tenant-sliced. 3. **Keep per-tenant on Timer, drop on Counters** if delivery-latency-per-tenant is the signal you care about most. -`event_type` is bounded by the spec (47 registered values today, additive over time). `reason` and `decision` are enum-bounded and safe. +`event_type` is bounded by the spec (51 registered values today, additive over time). `reason` and `decision` are enum-bounded and safe. ## Quick alert recipes diff --git a/how-to/tenant-creation-and-management-in-cycles.md b/how-to/tenant-creation-and-management-in-cycles.md index 916957fe..e9592132 100644 --- a/how-to/tenant-creation-and-management-in-cycles.md +++ b/how-to/tenant-creation-and-management-in-cycles.md @@ -298,7 +298,7 @@ Pre-v0.1.25.35, closing a tenant was a pure status flip — operators then had t | Open `Reservation` | → `RELEASED` (reason `tenant_closed`, no overage debt) | `reservation.released_via_tenant_cascade` | | `WebhookSubscription` | → `DISABLED` (re-enable blocked by Rule 2) | `webhook.disabled_via_tenant_cascade` | -The `*_via_tenant_cascade` identifiers are emitted as Event `event_type`s by the reference server (registered enum constants there, but absent from the published spec's `EventType` enum — so do not rely on cross-server `event_type=` filtering). The matching **audit rows** are written as `operation="tenant_close_cascade"` with `resource_type`/`resource_id`. Instead, all four cascade **event rows** share a server-composed `correlation_id` (`tenant_close_cascade::`; audit rows carry `request_id`/`trace_id`, not `correlation_id`) — you can find every side effect of a close with one events query: +The `*_via_tenant_cascade` identifiers are emitted as Event `event_type`s (declared in the governance spec's `EventType` enum since document revision v0.1.25.35, so `event_type=` filtering on them is spec-valid — though Event emission is SHOULD-level, so non-reference servers may not emit them). The matching **audit rows** are written as `operation="tenant_close_cascade"` with `resource_type`/`resource_id`. All four cascade **event rows** share a server-composed `correlation_id` (`tenant_close_cascade::`; audit rows carry `request_id`/`trace_id`, not `correlation_id`) — you can find every side effect of a close with one events query: ```bash # All cascade events for one close @@ -306,7 +306,7 @@ curl -s "http://localhost:7979/v1/admin/events?correlation_id=" \ -H "X-Admin-API-Key: $ADMIN_API_KEY" | jq '.events[] | {event_type, data}' ``` -After the cascade, mutating any owned object returns `409 TENANT_CLOSED` (Rule 2 — Terminal-Owner Mutation Guard). GET endpoints remain available for post-mortem audit reads. See [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics) for the full Rule 1 / Rule 2 contract and Mode A / Mode B semantics. +After the cascade, mutating any owned object returns `409 TENANT_CLOSED` (Rule 2 — Terminal-Owner Mutation Guard). GET endpoints remain available for post-mortem audit reads. On the runtime plane, `cycles-server` 0.1.25.47+ enforces the same guard on persisting reservation create/commit/release/extend (runtime spec v0.1.25.13) — fresh dry-run and `/v1/decide` evaluations return `200 decision=DENY reason_code=TENANT_CLOSED` instead of a 409; runtime 0.1.25.46 and earlier surface closure there only as `401`s from revoked keys or `BUDGET_CLOSED`. Even on 0.1.25.47+, tenant-key runtime calls usually still fail `401` at the auth filter before the guard is reached — in practice the runtime 409 surfaces on admin-on-behalf-of release and in the post-flip/pre-revocation race window (see the [observability note](/protocol/tenant-close-cascade-semantics#what-the-runtime-plane-sees)). See [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics) for the full Rule 1 / Rule 2 contract and Mode A / Mode B semantics. ::: warning Don't pre-freeze before closing On admin v0.1.25.35+, the cascade runs automatically and atomically from the operator's perspective (via Rule 2). **Do not** freeze budgets, revoke keys, or disable webhooks before closing — it's unnecessary, generates audit clutter, and (on future Mode A implementations) can cause cascades to roll back if a pre-freeze step fails. Just close the tenant; the cascade handles everything. @@ -322,7 +322,7 @@ Once you click close, the amber "Tenant closed — all owned objects are read-on | Your admin version | What happens | |---|---| -| **v0.1.25.36+** (recommended) | Rule 1 cascade runs; Rule 2 guard active on every mutation endpoint. Budgets, API keys, reservations, and webhooks all reach terminal state automatically. Any mutation attempt returns `409 TENANT_CLOSED`. | +| **v0.1.25.36+** (recommended) | Rule 1 cascade runs; Rule 2 guard active on every mutation endpoint. Budgets, API keys, reservations, and webhooks all reach terminal state automatically. Any admin-plane mutation attempt returns `409 TENANT_CLOSED`; runtime tenant-key mutations usually fail `401` at the auth filter first (see the runtime-plane caveat above). | | **v0.1.25.35** | Rule 1 cascade runs; Rule 2 guard covers budget operations plus webhook create/update only. Policy, API key, remaining webhook, and bulk-action-row mutations against the now-closed tenant slip through silently until you upgrade (all completed in .36). Cascade itself still completes correctly. | | **Pre-v0.1.25.35** | No cascade. Dashboard banner still renders (it's purely UI state), but owned objects stay in their pre-close state until you manually freeze / revoke / disable them. | @@ -822,6 +822,8 @@ curl -s -X PATCH http://localhost:7979/v1/admin/tenants/acme-corp \ The tenant has been permanently closed. This cannot be reversed. If you need a new tenant, create one with a different `tenant_id`. +Returned by every mutating admin-plane operation on the closed tenant's owned objects (admin v0.1.25.35+, full coverage v0.1.25.36+) and, since `cycles-server` 0.1.25.47, by persisting reservation create/commit/release/extend on the runtime plane (runtime spec v0.1.25.13). + ### 403 FORBIDDEN (tenant mismatch) The `subject.tenant` in your request does not match the effective tenant derived from the API key. diff --git a/how-to/webhook-integrations.md b/how-to/webhook-integrations.md index 23507a69..bfe20bf5 100644 --- a/how-to/webhook-integrations.md +++ b/how-to/webhook-integrations.md @@ -417,7 +417,7 @@ curl -X POST http://localhost:7979/v1/admin/webhooks \ }' ``` -> **Note:** `event_categories` is additive with `event_types`. If you specify `"event_categories": ["budget"]`, you receive **all** `budget.*` events (16 types including `budget.created`, `budget.debited`, etc.), not just the ones in `event_types`. Use `event_types` alone when you want precise control over which events trigger notifications. +> **Note:** `event_categories` is additive with `event_types`. If you specify `"event_categories": ["budget"]`, you receive **all** `budget.*` events (17 types including `budget.created`, `budget.debited`, `budget.closed_via_tenant_cascade`, etc.), not just the ones in `event_types`. Use `event_types` alone when you want precise control over which events trigger notifications. ### Middleware (Node.js) @@ -916,7 +916,7 @@ def handle(): ## Tenant Self-Service Webhooks -Tenants can manage their own webhooks (restricted to `budget.*`, `reservation.*`, `tenant.*` events — 27 of 47 registered types, plus the additive `_via_tenant_cascade` fan-out events the reference admin server emits in those categories on tenant close — see [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics)). Admin-only events (`api_key.*`, `policy.*`, `webhook.*`, `system.*`) are not available to tenants. +Tenants can manage their own webhooks (restricted to `budget.*`, `reservation.*`, `tenant.*` events — 29 of 51 registered types, including the `_via_tenant_cascade` fan-out events the admin server emits in those categories on tenant close — see [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics)). Admin-only events (`api_key.*`, `policy.*`, `webhook.*`, `system.*`) are not available to tenants. **Required API key permissions:** - `webhooks:write` — create, update, delete, and test subscriptions @@ -997,6 +997,6 @@ curl -X PUT http://localhost:7979/v1/admin/config/webhook-security \ ## Next steps - [Managing Webhooks](/how-to/managing-webhooks) — create, test, replay, and monitor webhook subscriptions -- [Webhook Event Delivery Protocol](/protocol/webhook-event-delivery-protocol) — full 47-event-type catalog, delivery headers, retry policy, and status lifecycle +- [Webhook Event Delivery Protocol](/protocol/webhook-event-delivery-protocol) — full 51-event-type catalog, delivery headers, retry policy, and status lifecycle - [Deploying the Events Service](/quickstart/deploying-the-events-service) — deploy the async webhook delivery service - [Security](/security#webhook-security) — SSRF protection, signing secret encryption, and deduplication diff --git a/protocol/api-reference-for-the-cycles-protocol.md b/protocol/api-reference-for-the-cycles-protocol.md index c5076967..47a9032f 100644 --- a/protocol/api-reference-for-the-cycles-protocol.md +++ b/protocol/api-reference-for-the-cycles-protocol.md @@ -206,9 +206,10 @@ curl -X POST http://localhost:7878/v1/reservations \ | 409 | `BUDGET_CLOSED` | Budget scope is permanently closed | | 409 | `OVERDRAFT_LIMIT_EXCEEDED` | Scope is over-limit | | 409 | `DEBT_OUTSTANDING` | Scope has unpaid debt (no overdraft limit configured) | +| 409 | `TENANT_CLOSED` | Owning tenant's status is `CLOSED` (persisting create only, `dry_run` absent or `false`; deployments with a governance plane — spec v0.1.25.13, cycles-server 0.1.25.47+) | | 409 | `IDEMPOTENCY_MISMATCH` | Same key, different payload | -**Dry run:** when `dry_run=true`, budget-state conditions (`BUDGET_EXCEEDED`, `BUDGET_FROZEN`, `BUDGET_CLOSED`, `OVERDRAFT_LIMIT_EXCEEDED`, `DEBT_OUTSTANDING`, and the 404 "no budget at any scope" case) surface as `200 OK` with `decision: DENY` and a `reason_code` field — `DecisionReasonCode` is an open string (as of v0.1.25); clients MUST handle unknown values gracefully — not as 4xx/409 errors. Request-validity errors (`INVALID_REQUEST`, `UNIT_MISMATCH`, `UNAUTHORIZED`, `FORBIDDEN`, `IDEMPOTENCY_MISMATCH`) are still returned as 4xx on dry-run. See [Decision reason codes](/protocol/error-codes-and-error-handling-in-cycles#decision-reason-codes). +**Dry run:** when `dry_run=true`, budget-state conditions (`BUDGET_EXCEEDED`, `BUDGET_FROZEN`, `BUDGET_CLOSED`, `OVERDRAFT_LIMIT_EXCEEDED`, `DEBT_OUTSTANDING`, `TENANT_CLOSED` on a closed owning tenant, and the 404 "no budget at any scope" case) surface as `200 OK` with `decision: DENY` and a `reason_code` field — `DecisionReasonCode` is an open string (as of v0.1.25); clients MUST handle unknown values gracefully — not as 4xx/409 errors. Request-validity errors (`INVALID_REQUEST`, `UNIT_MISMATCH`, `UNAUTHORIZED`, `FORBIDDEN`, `IDEMPOTENCY_MISMATCH`) are still returned as 4xx on dry-run. See [Decision reason codes](/protocol/error-codes-and-error-handling-in-cycles#decision-reason-codes). --- @@ -293,6 +294,7 @@ curl -X POST http://localhost:7878/v1/reservations/res-abc-123/commit \ | 409 | `BUDGET_CLOSED` | Budget scope is permanently closed | | 409 | `OVERDRAFT_LIMIT_EXCEEDED` | Debt would exceed limit (ALLOW_WITH_OVERDRAFT) | | 409 | `RESERVATION_FINALIZED` | Already committed or released | +| 409 | `TENANT_CLOSED` | Owning tenant's status is `CLOSED` (spec v0.1.25.13, cycles-server 0.1.25.47+); takes precedence over reservation-state errors for non-replay requests | | 409 | `IDEMPOTENCY_MISMATCH` | Same key, different payload | | 410 | `RESERVATION_EXPIRED` | TTL + grace period elapsed | @@ -352,6 +354,7 @@ curl -X POST http://localhost:7878/v1/reservations/res-abc-123/release \ | 403 | `FORBIDDEN` | Reservation owned by different tenant | | 404 | `NOT_FOUND` | Reservation does not exist | | 409 | `RESERVATION_FINALIZED` | Already committed or released | +| 409 | `TENANT_CLOSED` | Owning tenant's status is `CLOSED` (spec v0.1.25.13, cycles-server 0.1.25.47+); takes precedence over reservation-state errors for non-replay requests | | 409 | `IDEMPOTENCY_MISMATCH` | Same key, different payload | | 410 | `RESERVATION_EXPIRED` | TTL + grace period elapsed | @@ -406,6 +409,7 @@ curl -X POST http://localhost:7878/v1/reservations/res-abc-123/extend \ | 403 | `FORBIDDEN` | Reservation owned by different tenant | | 404 | `NOT_FOUND` | Reservation does not exist | | 409 | `RESERVATION_FINALIZED` | Already committed or released | +| 409 | `TENANT_CLOSED` | Owning tenant's status is `CLOSED` (spec v0.1.25.13, cycles-server 0.1.25.47+); takes precedence over reservation-state errors for non-replay requests | | 409 | `IDEMPOTENCY_MISMATCH` | Same key, different payload | | 409 | `MAX_EXTENSIONS_EXCEEDED` | Tenant `max_reservation_extensions` limit reached | | 410 | `RESERVATION_EXPIRED` | Past TTL (no grace period for extend) | @@ -573,7 +577,7 @@ Evaluate a budget decision without creating a reservation. Useful for preflight } ``` -The `reason_code` and `retry_after_ms` fields are present when the decision is `DENY`. `reason_code` is `DecisionReasonCode` — an open string (as of v0.1.25) with six documented known values: `BUDGET_EXCEEDED`, `BUDGET_FROZEN`, `BUDGET_CLOSED`, `BUDGET_NOT_FOUND`, `OVERDRAFT_LIMIT_EXCEEDED`, `DEBT_OUTSTANDING`. Clients MUST handle unknown values gracefully. See [Decision reason codes](/protocol/error-codes-and-error-handling-in-cycles#decision-reason-codes). +The `reason_code` and `retry_after_ms` fields are present when the decision is `DENY`. `reason_code` is `DecisionReasonCode` — an open string (as of v0.1.25) with seven documented known values: `BUDGET_EXCEEDED`, `BUDGET_FROZEN`, `BUDGET_CLOSED`, `BUDGET_NOT_FOUND`, `OVERDRAFT_LIMIT_EXCEEDED`, `DEBT_OUTSTANDING`, `TENANT_CLOSED` (added in spec v0.1.25.13 for closed owning tenants). Clients MUST handle unknown values gracefully. See [Decision reason codes](/protocol/error-codes-and-error-handling-in-cycles#decision-reason-codes). ### Example @@ -599,7 +603,7 @@ curl -X POST http://localhost:7878/v1/decide \ | 403 | `FORBIDDEN` | Tenant mismatch | | 409 | `IDEMPOTENCY_MISMATCH` | Same key, different payload | -Note: decide returns `200` with `decision: DENY` for all budget-state conditions — insufficient remaining, debt, overdraft, frozen, closed, and the "no budget exists at any scope" case — not a `409` or `404`. The specific reason is surfaced in the `reason_code` field. `DecisionReasonCode` is an open string (as of v0.1.25) with six documented known values: `BUDGET_EXCEEDED`, `BUDGET_FROZEN`, `BUDGET_CLOSED`, `BUDGET_NOT_FOUND`, `OVERDRAFT_LIMIT_EXCEEDED`, `DEBT_OUTSTANDING`. Clients MUST handle unknown values gracefully. See [Decision reason codes](/protocol/error-codes-and-error-handling-in-cycles#decision-reason-codes) for full semantics. Request-validity errors like `UNIT_MISMATCH` are still returned as `400`. +Note: decide returns `200` with `decision: DENY` for all budget-state conditions — insufficient remaining, debt, overdraft, frozen, closed, the "no budget exists at any scope" case, and a closed owning tenant — not a `409` or `404`. The specific reason is surfaced in the `reason_code` field. `DecisionReasonCode` is an open string (as of v0.1.25) with seven documented known values: `BUDGET_EXCEEDED`, `BUDGET_FROZEN`, `BUDGET_CLOSED`, `BUDGET_NOT_FOUND`, `OVERDRAFT_LIMIT_EXCEEDED`, `DEBT_OUTSTANDING`, `TENANT_CLOSED` (spec v0.1.25.13; fresh evaluations on a closed owning tenant — never `409 TENANT_CLOSED` on this endpoint, though a present-but-malformed tenant record fails closed with `500 INTERNAL_ERROR`). Clients MUST handle unknown values gracefully. See [Decision reason codes](/protocol/error-codes-and-error-handling-in-cycles#decision-reason-codes) for full semantics. Request-validity errors like `UNIT_MISMATCH` are still returned as `400`. --- diff --git a/protocol/dry-run-shadow-mode-evaluation-in-cycles.md b/protocol/dry-run-shadow-mode-evaluation-in-cycles.md index 8979522a..1092bb48 100644 --- a/protocol/dry-run-shadow-mode-evaluation-in-cycles.md +++ b/protocol/dry-run-shadow-mode-evaluation-in-cycles.md @@ -52,7 +52,7 @@ If the decision is ALLOW or DENY, `caps` is absent. When a dry run returns DENY, the `reason_code` field should be populated. This is the primary diagnostic signal for understanding why the dry run was denied. -`DecisionReasonCode` is an open string (as of v0.1.25) with six documented known values: `BUDGET_EXCEEDED`, `BUDGET_FROZEN`, `BUDGET_CLOSED`, `BUDGET_NOT_FOUND`, `OVERDRAFT_LIMIT_EXCEEDED`, `DEBT_OUTSTANDING`. Clients MUST handle unknown values gracefully — extension specs may add new reason codes additively. See [Decision reason codes](/protocol/error-codes-and-error-handling-in-cycles#decision-reason-codes) for full semantics. +`DecisionReasonCode` is an open string (as of v0.1.25) with seven documented known values: `BUDGET_EXCEEDED`, `BUDGET_FROZEN`, `BUDGET_CLOSED`, `BUDGET_NOT_FOUND`, `OVERDRAFT_LIMIT_EXCEEDED`, `DEBT_OUTSTANDING`, `TENANT_CLOSED` (added in spec v0.1.25.13 — a fresh dry run on a closed owning tenant returns `decision=DENY` with this reason, never `409 TENANT_CLOSED`; the persisting create surfaces the same condition as the 409). Clients MUST handle unknown values gracefully — extension specs may add new reason codes additively. See [Decision reason codes](/protocol/error-codes-and-error-handling-in-cycles#decision-reason-codes) for full semantics. The same field is used by `/v1/decide` responses. Note that on dry_run reserve, the `BUDGET_NOT_FOUND` reason code corresponds to a condition that non-dry reserve would surface as `HTTP 404` with `error=NOT_FOUND` — the wire shape is different, but the underlying "no budget at any derived scope" condition is the same. diff --git a/protocol/error-codes-and-error-handling-in-cycles.md b/protocol/error-codes-and-error-handling-in-cycles.md index d960d329..ae619472 100644 --- a/protocol/error-codes-and-error-handling-in-cycles.md +++ b/protocol/error-codes-and-error-handling-in-cycles.md @@ -33,7 +33,7 @@ Every response — error or success — also carries an `X-Cycles-Trace-Id` HTTP ## The error codes -The runtime protocol defines 16 wire error codes (the `ErrorCode` enum in the runtime OpenAPI spec — `LIMIT_EXCEEDED` was added in spec v0.1.25.12). This page documents those 16 plus the admin-plane `TENANT_CLOSED` lifecycle error — raised by the [tenant-close cascade](/protocol/tenant-close-cascade-semantics) against objects owned by closed tenants — totalling 17 codes covered here. The admin-plane error enum has additional codes that aren't part of the runtime wire contract; see the admin OpenAPI spec for the full set. +The runtime protocol defines 17 wire error codes (the `ErrorCode` enum in the runtime OpenAPI spec — `LIMIT_EXCEEDED` was added in spec v0.1.25.12, `TENANT_CLOSED` in spec v0.1.25.13). `TENANT_CLOSED` mirrors the governance/admin plane's lifecycle error of the same name — raised by the [tenant-close cascade](/protocol/tenant-close-cascade-semantics) against objects owned by closed tenants — so all 17 codes covered here are now part of the runtime wire contract. The admin-plane error enum has additional codes that aren't; see the admin OpenAPI spec for the full set. Each code has a specific HTTP status code and meaning. @@ -125,6 +125,8 @@ The owning tenant has been permanently closed. Every mutating admin-plane operat This error is issued by the **Rule 2 — Terminal-Owner Mutation Guard** half of the cascade contract (governance-admin spec v0.1.25.29, shipped in `cycles-server-admin` v0.1.25.35; full coverage v0.1.25.36). Rule 2's counterpart — **Rule 1 — Close Cascade** — runs at tenant-close time and automatically drives owned objects to terminal states (`BudgetLedger → CLOSED`, `ApiKey → REVOKED`, open reservations → `RELEASED`, `WebhookSubscription → DISABLED`), so by the time you see this error the owned objects are already terminal. There is no way to "undo" a close; this is not a race condition that will resolve on retry. +**On the runtime plane** (runtime spec v0.1.25.13, shipped in `cycles-server` 0.1.25.47): the persisting reservation mutations — create (`dry_run` absent or `false`), commit, release, extend — return `409 TENANT_CLOSED` once the owning tenant's `CLOSED` flip is durable. For non-replay mutations it takes precedence over the reservation-state errors (`RESERVATION_FINALIZED`, `RESERVATION_EXPIRED`); same-key replays of mutations that succeeded before the close return their original stored response. Fresh (non-replay) `dry_run=true` create and `POST /v1/decide` evaluations never 409 for this condition — they return `200` with `decision=DENY` and `reason_code=TENANT_CLOSED` (see [Decision reason codes](#decision-reason-codes)). A present-but-malformed tenant record (undecodable JSON, missing or unrecognized `status`) fails closed with `500 INTERNAL_ERROR` before any mutation; deployments with no tenant records at all are not guarded. A mutation-surface `409 TENANT_CLOSED` on the evidence endpoints (persisting create, commit, release) emits an `error` CyclesEvidence envelope and stamps `cycles_evidence` on the response. Note that the cascade also revokes the tenant's API keys and the runtime auth filter rejects CLOSED-tenant keys per request, so tenant-key calls usually fail with `401` first — the 409 surfaces mainly on admin-on-behalf-of **release** (the only guarded mutation the runtime plane exposes to `X-Admin-API-Key`; the admin dual-auth allowlist is reservation list/get/release, so create/commit/extend accept tenant keys only) and, for any of the four, in the post-flip/pre-revocation race window. `cycles-server` 0.1.25.46 and earlier surface closed tenants on the runtime plane only as `401`s (revoked/rejected keys) or budget-state errors such as `BUDGET_CLOSED`. + **What to do:** the tenant and its owned objects are read-only. Create a new tenant or escalate. **Not retryable against any object owned by this tenant** — unlike `BUDGET_FROZEN` (which an operator may unfreeze), `TENANT_CLOSED` is terminal. Implement no retry logic for this error. **For client-app developers.** If your users encounter `TENANT_CLOSED`, escalate to your platform operator — they control tenant lifecycle; a client cannot un-close a tenant. New workloads require a fresh active tenant. To proactively detect closed tenants and surface a friendlier message before mutation attempts, call `GET /v1/admin/tenants/{tenant_id}` (admin-scoped) or check the response of any `GET /v1/balances` / `GET /v1/reservations` call — reads against a closed tenant still succeed and return status metadata. @@ -222,6 +224,8 @@ The 429 response carries throttling headers alongside the standard correlation h An unexpected server error occurred. +Since `cycles-server` 0.1.25.47 this is also the deliberate fail-closed response when a tenant record exists but its status cannot be determined (undecodable JSON, non-object, missing or unrecognized `status`) — the closed-tenant guard refuses to treat a corrupt governance record as an open tenant, on the dry-run/decide surface too. That variant will not resolve on retry; the operator must repair the tenant record. + **What to do:** retry with exponential backoff. If the error persists, contact the Cycles server operator. ## Error handling by operation @@ -233,7 +237,7 @@ An unexpected server error occurred. | BUDGET_EXCEEDED | 409 | Insufficient budget | | BUDGET_FROZEN | 409 | Budget scope is frozen | | BUDGET_CLOSED | 409 | Budget scope is permanently closed | -| TENANT_CLOSED | 409 | Owning tenant is closed (v0.1.25.35+); every mutation on owned objects rejects | +| TENANT_CLOSED | 409 | Owning tenant is closed (cycles-server 0.1.25.47+, spec v0.1.25.13). Persisting create only — a fresh `dry_run=true` returns `200 decision=DENY reason_code=TENANT_CLOSED` instead | | OVERDRAFT_LIMIT_EXCEEDED | 409 | Scope is over-limit | | DEBT_OUTSTANDING | 409 | Scope has unresolved debt (no overdraft limit configured) | | IDEMPOTENCY_MISMATCH | 409 | Same key, different payload | @@ -253,7 +257,7 @@ An unexpected server error occurred. | FORBIDDEN | 403 | Tenant mismatch | | IDEMPOTENCY_MISMATCH | 409 | Same key, different payload | -Note: decide returns `200` with `decision: DENY` for budget-state conditions (insufficient remaining, debt, overdraft, and the "no budget exists at any scope" case — surfaced via `reason_code` from the [DecisionReasonCode enum](#decision-reason-codes)), not a `409` or `404` error. Request-validity errors like `UNIT_MISMATCH` are still returned as 400. The same applies to `POST /v1/reservations` when `dry_run=true`. +Note: decide returns `200` with `decision: DENY` for budget-state conditions (insufficient remaining, debt, overdraft, and the "no budget exists at any scope" case — surfaced via `reason_code` from the [DecisionReasonCode enum](#decision-reason-codes)), not a `409` or `404` error. The same holds for a closed owning tenant (cycles-server 0.1.25.47+, spec v0.1.25.13): a fresh (non-replay) evaluation returns `200 decision=DENY reason_code=TENANT_CLOSED`, never `409 TENANT_CLOSED` — though a present-but-malformed tenant record fails closed with `500 INTERNAL_ERROR`. Request-validity errors like `UNIT_MISMATCH` are still returned as 400. The same applies to `POST /v1/reservations` when `dry_run=true`. ### Commit errors @@ -262,7 +266,7 @@ Note: decide returns `200` with `decision: DENY` for budget-state conditions (in | BUDGET_EXCEEDED | 409 | Actual exceeds budget (REJECT only) | | BUDGET_FROZEN | 409 | Budget scope is frozen | | BUDGET_CLOSED | 409 | Budget scope is permanently closed | -| TENANT_CLOSED | 409 | Owning tenant is closed (v0.1.25.35+) | +| TENANT_CLOSED | 409 | Owning tenant is closed (cycles-server 0.1.25.47+, spec v0.1.25.13); takes precedence over reservation-state errors for non-replay requests | | OVERDRAFT_LIMIT_EXCEEDED | 409 | Debt would exceed limit (ALLOW_WITH_OVERDRAFT) | | RESERVATION_EXPIRED | 410 | Past TTL + grace period | | RESERVATION_FINALIZED | 409 | Already committed or released | @@ -276,7 +280,7 @@ Note: decide returns `200` with `decision: DENY` for budget-state conditions (in | Error | HTTP | Meaning | |---|---|---| -| TENANT_CLOSED | 409 | Owning tenant is closed (v0.1.25.35+) | +| TENANT_CLOSED | 409 | Owning tenant is closed (cycles-server 0.1.25.47+, spec v0.1.25.13); takes precedence over reservation-state errors for non-replay requests | | RESERVATION_EXPIRED | 410 | Past TTL + grace period | | RESERVATION_FINALIZED | 409 | Already committed or released | | IDEMPOTENCY_MISMATCH | 409 | Same key, different payload | @@ -289,7 +293,7 @@ Note: decide returns `200` with `decision: DENY` for budget-state conditions (in | Error | HTTP | Meaning | |---|---|---| | INVALID_REQUEST | 400 | Missing or invalid fields | -| TENANT_CLOSED | 409 | Owning tenant is closed (v0.1.25.35+) | +| TENANT_CLOSED | 409 | Owning tenant is closed (cycles-server 0.1.25.47+, spec v0.1.25.13); takes precedence over reservation-state errors for non-replay requests | | RESERVATION_EXPIRED | 410 | Past TTL (no grace period for extend) | | RESERVATION_FINALIZED | 409 | Already committed or released | | MAX_EXTENSIONS_EXCEEDED | 409 | Tenant max_reservation_extensions limit reached | @@ -305,7 +309,6 @@ Note: decide returns `200` with `decision: DENY` for budget-state conditions (in | BUDGET_EXCEEDED | 409 | Insufficient budget (REJECT only) | | BUDGET_FROZEN | 409 | Budget scope is frozen | | BUDGET_CLOSED | 409 | Budget scope is permanently closed | -| TENANT_CLOSED | 409 | Owning tenant is closed (v0.1.25.35+) | | OVERDRAFT_LIMIT_EXCEEDED | 409 | Debt would exceed limit (ALLOW_WITH_OVERDRAFT) | | NOT_FOUND | 404 | No budget exists at any derived scope in any unit (message: `"Budget not found for provided scope: ..."`) | | UNIT_MISMATCH | 400 | `actual.unit` does not match any budget at the target scope (budget exists in a different unit) | @@ -314,6 +317,8 @@ Note: decide returns `200` with `decision: DENY` for budget-state conditions (in | FORBIDDEN | 403 | Tenant mismatch | | IDEMPOTENCY_MISMATCH | 409 | Same key, different payload | +Note: `/v1/events` is not part of the closed-tenant mutation guard's surface — the runtime spec's `TENANT_CLOSED` binding covers the persisting reservation mutations (create/commit/release/extend) only. A closed owning tenant surfaces on `/v1/events` as `401` (the auth filter rejects revoked/CLOSED-tenant keys) or `BUDGET_CLOSED` (the cascade closes the tenant's budgets). + ## Decision reason codes Separately from the 4xx error code list, `POST /v1/decide` and `POST /v1/reservations` with `dry_run=true` may return `200 OK` with `decision: DENY` and a machine-readable `reason_code`. As of v0.1.25, `DecisionReasonCode` is an **open string** (was a closed enum in v0.1.24 and earlier — widened so future extension specs can add reason codes without a breaking protocol bump). Documented known values: @@ -326,6 +331,7 @@ Separately from the 4xx error code list, `POST /v1/decide` and `POST /v1/reserva | `BUDGET_NOT_FOUND` | No budget exists at any derived scope in the requested unit. On non-dry reserve and `/v1/events` paths this same underlying condition surfaces as `HTTP 404` with `error=NOT_FOUND` instead. | | `OVERDRAFT_LIMIT_EXCEEDED` | Either `debt + delta > overdraft_limit` on commit, OR the scope is in over-limit state (`is_over_limit=true`) and no new reservations are permitted until reconciled. | | `DEBT_OUTSTANDING` | A derived scope has `debt > 0` and `overdraft_limit == 0` (no policy permits further debt accrual). | +| `TENANT_CLOSED` | The owning tenant's status is `CLOSED` (deployments with a governance plane; added in spec v0.1.25.13, emitted by cycles-server 0.1.25.47+ on fresh dry-run/decide evaluations). The persisting mutation surface reports the same condition as `HTTP 409` with `error=TENANT_CLOSED` instead — see [TENANT_CLOSED (409)](#tenant-closed-409). | **Why this is a separate enum.** The 4xx error codes surface request-level failures in the `error` field. Decision reason codes surface budget-state outcomes in the `reason_code` field on successful HTTP responses. Some labels overlap (e.g. `BUDGET_EXCEEDED`) because the same underlying condition is reported differently depending on the endpoint: `/decide` and dry-run reserve surface it as a non-4xx DENY decision, while non-dry reserve surfaces it as a `409` error. diff --git a/protocol/event-payloads-reference.md b/protocol/event-payloads-reference.md index d6871642..9cea7eb0 100644 --- a/protocol/event-payloads-reference.md +++ b/protocol/event-payloads-reference.md @@ -8,21 +8,21 @@ description: "Complete payload reference for all Cycles webhook events — curre This page documents the payload structure for every webhook event Cycles can emit. Each event wraps a standard envelope with an event-specific `data` object. ::: info Currently Emitted Events -The v0.1.25 Admin API `EventType` enum registers **47 event types** total across seven categories (budget: 16, reservation: 5, tenant: 6, api_key: 6, policy: 3, webhook: 6, system: 5). Events marked as **Planned** below have their type registered in the protocol but are not yet emitted by any service. +The v0.1.25 Admin API `EventType` enum registers **51 event types** total across seven categories (budget: 17, reservation: 6, tenant: 6, api_key: 7, policy: 3, webhook: 7, system: 5). Events marked as **Planned** below have their type registered in the protocol but are not yet emitted by any service. -**Registered enum values currently emitted** (count toward the 47 total): +**Registered enum values currently emitted** (count toward the 51 total): - **Reservation:** `reservation.denied`, `reservation.expired`, `reservation.commit_overage` (runtime). - **Budget:** `budget.exhausted`, `budget.over_limit_entered`, `budget.debt_incurred`, `budget.reset_spent` (runtime + admin v0.1.25.18+); `budget.funded`, `budget.debited`, `budget.reset`, `budget.debt_repaid` (admin v0.1.25.38+). - **Tenant:** `tenant.suspended`, `tenant.reactivated`, `tenant.closed` (admin v0.1.25.38+, single-op + bulk-action paths). - **Webhook:** `webhook.created`, `webhook.updated`, `webhook.paused`, `webhook.resumed`, `webhook.deleted` (admin v0.1.25.39+); `webhook.disabled` (events service auto-disable v0.1.25.11+). All six webhook lifecycle types were added in spec v0.1.25.33 — see the [Webhook Lifecycle Events](#webhook-lifecycle-events) section below. +- **Tenant-close cascade fan-out:** `budget.closed_via_tenant_cascade`, `reservation.released_via_tenant_cascade`, `api_key.revoked_via_tenant_cascade`, `webhook.disabled_via_tenant_cascade` (admin v0.1.25.35+; declared in the governance spec's enum since revision v0.1.25.35) — see [Tenant-Close Cascade Events](#tenant-close-cascade-events-governance-spec-v0-1-25-35) below. - **API key, policy, system:** 0 registered enum values currently emitted; all planned. **Additive reference-server payloads** (observable in the reference implementation but not part of the published admin-openapi enum — consumers must ignore unrecognized event types gracefully): - Reservation lifecycle samples: `reservation.reserved`, `reservation.committed`, `reservation.released`, `reservation.extended`. - Runtime ledger application: `event.applied`. -- Tenant-close cascade fan-out: `budget.closed_via_tenant_cascade`, `reservation.released_via_tenant_cascade`, `api_key.revoked_via_tenant_cascade`, `webhook.disabled_via_tenant_cascade` (admin v0.1.25.35+) — see the [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics) contract. See the [Event Emission Summary](#event-emission-summary) at the bottom for the full per-category breakdown. ::: @@ -135,7 +135,7 @@ The `reservation.denied` event model defines 9 fields, but the current server em | Field | Type | Populated | Description | |---|---|---|---| | `scope` | string | Yes | Scope path that denied the reservation | -| `reason_code` | string | Yes | Why denied. Known values: `BUDGET_EXCEEDED`, `OVERDRAFT_LIMIT_EXCEEDED`, `DEBT_OUTSTANDING`, `BUDGET_FROZEN`, `BUDGET_CLOSED`. Open string — extensions (v0.1.26+) may emit additional values such as `ACTION_QUOTA_EXCEEDED`, `ACTION_KIND_DENIED`, `ACTION_KIND_NOT_ALLOWED`. | +| `reason_code` | string | Yes | Why denied. Known values: `BUDGET_EXCEEDED`, `OVERDRAFT_LIMIT_EXCEEDED`, `DEBT_OUTSTANDING`, `BUDGET_FROZEN`, `BUDGET_CLOSED`, and — from cycles-server 0.1.25.47 (spec v0.1.25.13) — `TENANT_CLOSED` on fresh dry-run/decide DENYs for a closed owning tenant. Open string — extensions (v0.1.26+) may emit additional values such as `ACTION_QUOTA_EXCEEDED`, `ACTION_KIND_DENIED`, `ACTION_KIND_NOT_ALLOWED`. | | `requested_amount` | number | Yes | Amount the reservation requested | | `unit` | string | Not yet | Budget unit (`USD_MICROCENTS`, `TOKENS`, `CREDITS`, `RISK_POINTS`) | | `remaining` | number | Not yet | Budget remaining at the scope that denied | @@ -394,14 +394,16 @@ The following budget events are defined in the protocol but not yet emitted. The --- -## Tenant-Close Cascade Events — Additive Reference-Server Payloads (v0.1.25.35+) +## Tenant-Close Cascade Events (governance spec v0.1.25.35+) Four event kinds are emitted by the reference admin server as side effects of a `* → CLOSED` tenant transition (Rule 1 — Close Cascade; see [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics) for the full contract). All four share the `_via_tenant_cascade` suffix and carry a server-composed `correlation_id` of the form `tenant_close_cascade::`, so subscribers can correlate cascade side effects to the operator action that triggered them (audit rows for the same operation join via `request_id`/`trace_id`). -These four event names are **absent from the published admin-openapi `EventType` enum** (they do not count toward the 47 registered types), but they are registered as first-class constants in the reference implementation's `EventType.java`. Consumers must ignore unrecognized event types gracefully and should not assume non-reference servers emit them. Tenant self-service subscriptions filter by category, so a tenant subscribed to `budget` or `reservation` events will receive the corresponding cascade events from the reference server in practice. +These four event names are **declared in the governance spec's `EventType` enum** since document revision v0.1.25.35 (raising the registered enum to 51 values), so cascade Events validate against `Event.event_type` and can be targeted with `event_type=` filters and webhook `event_types` lists like any other lifecycle event. Note the normative strength: the per-object cascade **audit entries** are a MUST (their `event_kind` values are RESERVED in the spec), while emitting the corresponding Event-stream records is a SHOULD — so non-reference servers may not emit them, and consumers should still ignore unrecognized event types gracefully. Tenant self-service subscriptions filter by category, so a tenant subscribed to `budget` or `reservation` events will receive the corresponding cascade events from the reference server in practice. Shipped in `cycles-server-admin` v0.1.25.35 (initial Mode B cascade) / v0.1.25.36 (full Rule 2 guard coverage). +All four kinds share one payload shape (`EventDataTenantCascade`, governance spec v0.1.25.35): exactly one of `ledger_id` / `subscription_id` / `key_id` identifies the transitioned object (matching the event's category), alongside `prior_status` / `new_status` and `cascade_reason: "tenant_closed"`. The reservation aggregate is the exception — it identifies the drained budget via `ledger_id` and carries `released_amount` instead of a status transition. + ### `budget.closed_via_tenant_cascade` Emitted once per owned `BudgetLedger` when the tenant closes. The per-budget `BudgetLedger.status` flips to `CLOSED` and `closed_at` is stamped; the final balance snapshot is preserved for audit. @@ -416,20 +418,15 @@ Emitted once per owned `BudgetLedger` when the tenant closes. The per-budget `Bu "scope": "tenant:acme-corp/workspace:prod", "source": "cycles-admin", "actor": { - "type": "api_key", - "key_id": "admin_key_...", - "source_ip": "..." + "type": "admin" }, "data": { "ledger_id": "led_...", "scope": "tenant:acme-corp/workspace:prod", "unit": "USD_MICROCENTS", - "final_allocated": 10000000, - "final_spent": 8234000, - "final_reserved": 0, - "final_debt": 0, - "closed_at": "2026-04-20T12:00:00Z", - "cascade_origin": "tenant.closed" + "prior_status": "ACTIVE", + "new_status": "CLOSED", + "cascade_reason": "tenant_closed" }, "correlation_id": "tenant_close_cascade:acme-corp:req_...", "trace_id": "" @@ -447,11 +444,11 @@ Emitted as a **ledger-level aggregate** when the tenant closes: one event per cl "category": "reservation", "tenant_id": "acme-corp", "data": { - "reservation_id": "rsv_...", - "scope": "tenant:acme-corp/...", - "reserved": { "amount": 1000, "unit": "TOKENS" }, - "release_reason": "tenant_closed", - "cascade_origin": "tenant.closed" + "ledger_id": "led_...", + "scope": "tenant:acme-corp/workspace:prod", + "unit": "USD_MICROCENTS", + "released_amount": 250000, + "cascade_reason": "tenant_closed" }, "correlation_id": "tenant_close_cascade:acme-corp:req_...", "trace_id": "" @@ -470,9 +467,10 @@ Emitted once per owned `ApiKey` when the tenant closes. The per-key `ApiKey.stat "tenant_id": "acme-corp", "data": { "key_id": "key_...", + "prior_status": "ACTIVE", + "new_status": "REVOKED", "name": "production", - "revoked_at": "2026-04-20T12:00:00Z", - "cascade_origin": "tenant.closed" + "cascade_reason": "tenant_closed" }, "correlation_id": "tenant_close_cascade:acme-corp:req_...", "trace_id": "" @@ -491,9 +489,10 @@ Emitted once per owned `WebhookSubscription` when the tenant closes. Status flip "tenant_id": "acme-corp", "data": { "subscription_id": "whsub_...", - "url": "https://...", - "disabled_at": "2026-04-20T12:00:00Z", - "cascade_origin": "tenant.closed" + "prior_status": "ACTIVE", + "new_status": "DISABLED", + "name": "ops-alerts", + "cascade_reason": "tenant_closed" }, "correlation_id": "tenant_close_cascade:acme-corp:req_...", "trace_id": "" @@ -504,7 +503,7 @@ Emitted once per owned `WebhookSubscription` when the tenant closes. Status flip The shared `correlation_id` is the primary join key — querying `GET /v1/admin/events?correlation_id=...` returns every event emitted by the cascade in one call. The dashboard (v0.1.25.43+) renders a "tenant cascade" chip on audit and event-timeline rows with these suffixes. See [Using the Cycles Dashboard](/how-to/using-the-cycles-dashboard#closed-tenant-tombstone-and-cascade-preview). -**Ordering guarantee.** The spec mandates emission order: reservations released → budgets closed → webhooks disabled + API keys revoked → `tenant.closed`. Subscribers that depend on ordered observation of these events can rely on this, modulo the usual at-least-once webhook-delivery duplicates and reordering risk. +**No emission-order guarantee.** The spec's ordering language covers the cascade's *mutations* (a SHOULD, and only within Mode A's single transaction — see [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics#the-two-rules)); it is silent on cascade *event emission* order. The reference implementation currently emits per budget — `budget.closed_via_tenant_cascade` then, for budgets with `reserved > 0`, the `reservation.released_via_tenant_cascade` aggregate, interleaved budget by budget — followed by webhook and API-key events, but this is implementation detail. Subscribers MUST NOT rely on arrival order (at-least-once webhook delivery can reorder and duplicate regardless — see [delivery mechanics](/protocol/webhook-event-delivery-protocol)); reconstruct the cascade by joining on the shared `correlation_id` instead. --- @@ -659,14 +658,14 @@ The tenant category is partially emitted as of admin v0.1.25.38. Policy, system, | Category | Total Defined | Currently Emitted | Notes | |---|---|---|---| -| Reservation | 5 | `reservation.denied`, `reservation.expired`, and `reservation.commit_overage` emitted by runtime paths; lifecycle and cascade examples above are additive reference-server payloads | Spike events still planned | -| Budget | 16 | `budget.exhausted`, `over_limit_entered`, `debt_incurred`, `reset_spent`, and admin funding events emitted by current services; legacy threshold aliases and cascade examples above are additive reference-server payloads | Remaining lifecycle/threshold types still planned | +| Reservation | 6 | `reservation.denied`, `reservation.expired`, and `reservation.commit_overage` emitted by runtime paths; the cascade aggregate (`reservation.released_via_tenant_cascade`, spec-declared since governance v0.1.25.35) emitted by the admin server on tenant close | Spike events still planned | +| Budget | 17 | `budget.exhausted`, `over_limit_entered`, `debt_incurred`, `reset_spent`, and admin funding events emitted by current services; `budget.closed_via_tenant_cascade` (spec-declared since v0.1.25.35) emitted on tenant close; legacy threshold aliases are additive reference-server payloads | Remaining lifecycle/threshold types still planned | | Tenant | 6 | `tenant.suspended`, `tenant.reactivated`, `tenant.closed` emitted by admin (single-op + bulk-action paths, bulk parity added in v0.1.25.38) | `tenant.created`, `.updated`, `.settings_changed` still planned | -| API Key | 6 | 0 registered enum values emitted directly | Lifecycle events still planned; cascade examples above are additive reference-server payloads | +| API Key | 7 | `api_key.revoked_via_tenant_cascade` (spec-declared since v0.1.25.35) emitted on tenant close; no other registered enum values emitted directly | Lifecycle events still planned | | Policy | 3 | 0 | All planned | -| Webhook | 6 | 6 lifecycle events (`webhook.created` / `.updated` / `.paused` / `.resumed` / `.disabled` / `.deleted`) from admin v0.1.25.39 + events v0.1.25.11 | All registered enum values emitted | +| Webhook | 7 | 6 lifecycle events (`webhook.created` / `.updated` / `.paused` / `.resumed` / `.disabled` / `.deleted`) from admin v0.1.25.39 + events v0.1.25.11, plus `webhook.disabled_via_tenant_cascade` (spec-declared since v0.1.25.35) on tenant close | All registered enum values emitted | | System | 5 | 0 | All planned | -| **Total** | **47** | See category rows above | — | +| **Total** | **51** | See category rows above | — | For webhook delivery mechanics, retry schedule, and signature verification, see the [Webhook Event Delivery Protocol](/protocol/webhook-event-delivery-protocol). diff --git a/protocol/how-decide-works-in-cycles-preflight-budget-checks-without-reservation.md b/protocol/how-decide-works-in-cycles-preflight-budget-checks-without-reservation.md index a53eaa04..ceb5c670 100644 --- a/protocol/how-decide-works-in-cycles-preflight-budget-checks-without-reservation.md +++ b/protocol/how-decide-works-in-cycles-preflight-budget-checks-without-reservation.md @@ -87,6 +87,7 @@ Only `decision` is required in the response — every other field is conditional | `BUDGET_NOT_FOUND` | No budget exists at any derived scope in the requested unit (on non-dry reserve and `/v1/events`, this same condition surfaces as `HTTP 404` with `error=NOT_FOUND`) | | `OVERDRAFT_LIMIT_EXCEEDED` | Either `debt + delta > overdraft_limit`, or the scope is in over-limit state (`is_over_limit=true`) | | `DEBT_OUTSTANDING` | A derived scope has `debt > 0` and `overdraft_limit == 0` | +| `TENANT_CLOSED` | The owning tenant's status is `CLOSED` (deployments with a governance plane; added in spec v0.1.25.13, emitted by cycles-server 0.1.25.47+ on fresh evaluations — the persisting reservation mutations surface the same condition as `HTTP 409` with `error=TENANT_CLOSED`) | `DecisionReasonCode` was widened from a closed enum to an open string in v0.1.25 so future extension specs can add new reason codes without a breaking protocol bump. **Clients MUST handle unknown values gracefully** (treat as DENY, log the raw string, do not crash on enum parsing). Known values above are stable; future values will always be additive. See [Decision reason codes](/protocol/error-codes-and-error-handling-in-cycles#decision-reason-codes) for full semantics. diff --git a/protocol/implement.md b/protocol/implement.md index e3a18805..89c49fd3 100644 --- a/protocol/implement.md +++ b/protocol/implement.md @@ -91,6 +91,7 @@ Implementations MUST return the exact HTTP status + `error` code pairs from `cyc - `NOT_FOUND` — 404 - `DEBT_OUTSTANDING` — 409 - `LIMIT_EXCEEDED` — 429 (server-side throttling / rate limiting, optional in v0 and never used for deterministic budget exhaustion; added to the ErrorCode enum in spec v0.1.25.12, revision 2026-07-04). 429 responses carry the `Retry-After` and `X-RateLimit-Reset` headers. +- `TENANT_CLOSED` — 409 (deployments with a governance plane only; added to the ErrorCode enum in spec v0.1.25.13, revision 2026-07-10, mirroring the governance code of the same name). Persisting reservation create/commit/release/extend MUST reject with it when the owning tenant's status is CLOSED and the flip is durable, taking precedence over `RESERVATION_FINALIZED`/`RESERVATION_EXPIRED` for non-replay attempts. Fresh `dry_run=true` and `/v1/decide` evaluations MUST NOT 409 for this — they return `200 decision=DENY reason_code=TENANT_CLOSED`; a present-but-malformed tenant record MUST fail closed with 500 `INTERNAL_ERROR`. Deployments with no tenant records anywhere are exempt (nothing to enforce). One `RESERVATION_EXPIRED` subtlety, clarified in the 2026-07-03 spec revision: `GET /v1/reservations/{reservation_id}` MUST return 410 `RESERVATION_EXPIRED` when the reservation exists but its status is EXPIRED — the pre-revision text enumerated only commit/release/extend, leaving the GET case ambiguous. EXPIRED reservations remain discoverable via `listReservations`, which returns them as normal 200 rows with `status=EXPIRED`; the 410 applies only to the single-resource GET. diff --git a/protocol/tenant-close-cascade-semantics.md b/protocol/tenant-close-cascade-semantics.md index 2a533abf..f1af1f01 100644 --- a/protocol/tenant-close-cascade-semantics.md +++ b/protocol/tenant-close-cascade-semantics.md @@ -29,6 +29,7 @@ The cascade contract, formalized at spec document revision 0.1.25.31 and shippin | Rule 1 cascade (budgets + reservations) | `cycles-server-admin` v0.1.25.35 | Closing a tenant cascades budgets → CLOSED and open reservations → RELEASED | | Rule 2 guard (budget operations, webhook create/update) | `cycles-server-admin` v0.1.25.35 | Admin-plane mutations against closed-tenant budgets, and webhook create/update, return `409 TENANT_CLOSED` | | Rule 2 full coverage (policies, api-keys, remaining webhook mutations) | `cycles-server-admin` v0.1.25.36 | All remaining admin-plane mutation endpoints also return `409 TENANT_CLOSED` | +| Rule 2 runtime guard (reservation create/commit/release/extend) | `cycles-server` v0.1.25.47 (runtime spec v0.1.25.13) | Persisting reservation mutations on a closed tenant return `409 TENANT_CLOSED`; fresh dry-run/decide evaluations return `200 decision=DENY reason_code=TENANT_CLOSED` | | Dashboard tombstone + cascade preview UI | `cycles-dashboard` v0.1.25.43 | Banner, CLOSE dialog preview, humanized errors, cascade audit/event chip | **Pre-v0.1.25.35 admin servers do not cascade** — operators must manually freeze budgets, revoke keys, and disable webhooks before or after closing the tenant. @@ -55,7 +56,7 @@ On any `* → CLOSED` tenant transition (via `PATCH /v1/admin/tenants/{id}` or ` Mode B (see below) inverts this by design — the tenant flip commits **first**, and children converge afterward under the Rule 2 guard. Since runcycles' reference server implements Mode B, do not rely on this ordering in practice. -**Audit and event emission.** One record per mutated owned object. The emitted **Event rows** share a server-composed `correlation_id` of the form `tenant_close_cascade::` — query `GET /v1/admin/events?correlation_id=...` to reconstruct the cascade. **Audit rows** carry `request_id`/`trace_id` (the AuditLogEntry schema has no correlation field); join them via the originating request's `request_id`. The dotted `*_via_tenant_cascade` names are emitted as Event `event_type`s (registered enum constants in the reference implementation, absent from the published spec enum); the matching audit rows are written as `operation="tenant_close_cascade"` with `resource_type`/`resource_id` identifying the mutated object. Reserved dotted names: +**Audit and event emission.** One record per mutated owned object. The emitted **Event rows** share a server-composed `correlation_id` of the form `tenant_close_cascade::` — query `GET /v1/admin/events?correlation_id=...` to reconstruct the cascade. **Audit rows** carry `request_id`/`trace_id` (the AuditLogEntry schema has no correlation field); join them via the originating request's `request_id`. The dotted `*_via_tenant_cascade` names are emitted as Event `event_type`s (declared in the governance spec's `EventType` enum since document revision v0.1.25.35, so cascade Events validate against `Event.event_type` and are filterable like any other lifecycle event; Event emission is SHOULD-level, while the audit entries below are a MUST); the matching audit rows are written as `operation="tenant_close_cascade"` with `resource_type`/`resource_id` identifying the mutated object. Reserved dotted names: - `budget.closed_via_tenant_cascade` - `webhook.disabled_via_tenant_cascade` @@ -127,16 +128,18 @@ The spec's Rule 2 scopes the guard to **every mutating admin-plane operation** w ### What the runtime plane sees -**Spec (normative):** Rule 2's scope explicitly includes runtime reservation mutations — "any reservation create/commit/release/extend" — so a conformant server MUST reject them with `409 TENANT_CLOSED` once the CLOSED flip is durable. +**Spec (normative):** Rule 2's scope explicitly includes runtime reservation mutations — "any reservation create/commit/release/extend" — so a conformant server MUST reject them with `409 TENANT_CLOSED` once the CLOSED flip is durable. Runtime spec revision v0.1.25.13 binds this directly on the runtime plane: `TENANT_CLOSED` is now part of the runtime `ErrorCode` enum, with a normative closed-tenant binding in the runtime spec's ERROR SEMANTICS. `cycles-server` 0.1.25.47 implements it — the reference-implementation gap this section previously documented is closed. Shipped behavior: -::: warning Reference implementation gap -The runtime reference server's `ErrorCode` enum does not yet include `TENANT_CLOSED` (it exists in the governance-admin spec's shared enum, whose normative scope covers these routes). Until the runtime server adopts it, a closed tenant is observable on the runtime plane only through the cascade's effects: -::: +- **Persisting mutations → `409 TENANT_CLOSED`.** Reservation create (`dry_run` absent or `false`), commit, release, and extend against a `CLOSED` owning tenant return `409` with `error=TENANT_CLOSED` once the flip is durable. The check runs inside the same Lua scripts as the budget mutations, so a post-flip request can never partially succeed, and it is not subject to any config-cache TTL. +- **Precedence.** For non-replay mutations, `TENANT_CLOSED` takes precedence over the reservation-state errors (`RESERVATION_FINALIZED`, `RESERVATION_EXPIRED`) — Rule 2 rejects regardless of the child's own current status. Same-key replays of mutations that succeeded before the close are the exception: they retain replay precedence and return the original stored response. +- **Non-persisting evaluations never 409.** A fresh (non-replay) `dry_run=true` create or `POST /v1/decide` on a closed tenant returns `200` with `decision=DENY` and `reason_code=TENANT_CLOSED` — dry-run and decide outcomes are attestations of what live execution would do (and may be captured as signed CyclesEvidence), so they reflect the closed tenant as-if-live instead of erroring. Same-key replays of pre-close evaluations return their original payload. +- **Fail-closed on malformed tenant records.** A tenant record that is present but whose status cannot be determined (undecodable JSON, non-object, missing or non-string `status`, unknown status string) returns `500 INTERNAL_ERROR` before any mutation — on the non-persisting surface too, because the server cannot attest against corrupt governance state. A subject tenant with **no** tenant record at all (runtime-only deployments without a governance plane) is not guarded. +- **Evidence receipts.** A mutation-surface `409 TENANT_CLOSED` on the evidence endpoints — persisting create, commit, release — emits an `error` CyclesEvidence envelope and stamps `cycles_evidence` on the response, like the other live denial codes (extend is not an evidence endpoint). +- **Reads unaffected.** `GET /v1/reservations` and `GET /v1/reservations/{id}` keep working on a closed tenant's reservations for post-close audit, mirroring Rule 2's read-access rule. -- **`401 UNAUTHORIZED`** — the cascade revokes the tenant's API keys, so calls authenticated with those keys fail authentication. -- **`BUDGET_CLOSED`** — the cascade closes the tenant's budgets, so any operation that reaches a cascaded budget is rejected as a closed-budget mutation. +**Observability note:** the close cascade revokes the tenant's API keys, and the runtime auth filter rejects CLOSED-tenant keys per request — so tenant-key calls usually still fail with `401 UNAUTHORIZED` before the guard is consulted. In practice `409 TENANT_CLOSED` surfaces in two places: **admin-on-behalf-of release** — of the four guarded mutations, release is the only one the runtime plane exposes to `X-Admin-API-Key` (the admin dual-auth allowlist covers reservation list/get/release only, so create/commit/extend are never admin-key reachable) — and the **post-flip/pre-revocation race window** for tenant-key requests on any of the four. -Client code handling runtime errors should therefore treat "tenant was closed" as a `401`/`BUDGET_CLOSED` scenario, not a `TENANT_CLOSED` one. +**Version scope:** `cycles-server` 0.1.25.46 and earlier surface closed tenants on the runtime plane only as `401`s (revoked/rejected keys) or budget-state errors (`BUDGET_CLOSED` on cascaded budgets). Client code on those versions should treat "tenant was closed" as a `401`/`BUDGET_CLOSED` scenario; on 0.1.25.47+ handle `409 TENANT_CLOSED` as well. ## Operator recipe — closing a tenant @@ -186,6 +189,7 @@ See [Using the Cycles Dashboard](/how-to/using-the-cycles-dashboard#closed-tenan - Pre-v0.1.25.35 servers do NOT return `409 TENANT_CLOSED` — they return the previous per-endpoint error (`409 BUDGET_FROZEN`, `403 FORBIDDEN`, etc.) or may accept mutations against orphaned objects. - Pre-v0.1.25.36 servers have partial Rule 2 coverage — `.35` guarded budget operations and webhook create/update; `.36` completed policies, api-keys, the remaining webhook mutations, and per-row bulk-action. - Pre-v0.1.25.43 dashboards render TENANT_CLOSED as a raw 409 error without the humanizer and without the cascade-preview dialog. +- `cycles-server` (runtime) 0.1.25.46 and earlier do NOT return `409 TENANT_CLOSED` on reservation mutations — closed tenants surface there only as `401`s (revoked/rejected keys) or `BUDGET_CLOSED`. The runtime guard ships in 0.1.25.47 (runtime spec v0.1.25.13). **Re-issuing close on an already-CLOSED tenant** is idempotent at the tenant level across all versions — it returns the current state and emits no duplicate audit entries for already-terminal children. Under Mode B it is not a pure no-op: a re-close completes any outstanding child transitions left by an interrupted cascade. diff --git a/protocol/webhook-event-delivery-protocol.md b/protocol/webhook-event-delivery-protocol.md index 2fcd9815..91625b52 100644 --- a/protocol/webhook-event-delivery-protocol.md +++ b/protocol/webhook-event-delivery-protocol.md @@ -1,6 +1,6 @@ --- title: "Webhook Event Delivery Protocol" -description: "Complete reference for Cycles webhook delivery: 47 registered event types, HTTP headers, payload format, HMAC-SHA256 signing, retry policy, delivery status lifecycle, and at-least-once guarantees." +description: "Complete reference for Cycles webhook delivery: 51 registered event types, HTTP headers, payload format, HMAC-SHA256 signing, retry policy, delivery status lifecycle, and at-least-once guarantees." --- # Webhook Event Delivery Protocol @@ -62,12 +62,12 @@ Fields `scope`, `actor`, `data`, `correlation_id`, `request_id`, `trace_id`, and **Correlation fields.** `request_id` narrows to one HTTP request; `trace_id` (32-hex W3C) narrows to one logical operation (may span many requests); `correlation_id` groups a family of related events — it is server-set in one of two shapes: a deterministic hash over `(tenant_id, scope, action_kind_or_risk_class, window, window_key)` for protocol event-stream clusters, or an explicit operation ID (e.g. `webhook_create:`, `webhook_bulk_action::`) for governance/admin operations. See [Correlation and Tracing](/protocol/correlation-and-tracing-in-cycles). -## Event types (47) +## Event types (51) -The current v0.1.25 Admin API `EventType` enum registers 47 event types across seven categories: budget (16), reservation (5), tenant (6), api_key (6), policy (3), webhook (6), and system (5). Implementations may add future event types, and consumers should ignore unrecognized values gracefully. +The current v0.1.25 Admin API `EventType` enum registers 51 event types across seven categories: budget (17), reservation (6), tenant (6), api_key (7), policy (3), webhook (7), and system (5). Implementations may add future event types, and consumers should ignore unrecognized values gracefully. The per-category tables below list the 47 non-cascade types; the four `*_via_tenant_cascade` types (one each in the budget, reservation, api_key, and webhook categories, added to the enum in governance revision v0.1.25.35) are covered in [Tenant-close cascade fan-out](#tenant-close-cascade-fan-out). ::: info Count note -The 47-type / 7-category count tracks the admin OpenAPI enum. The runtime spec's webhook-event guidance section in `cycles-protocol-v0.yaml` lists 35 event types across 6 categories — it predates the `webhook` lifecycle category and some later enum additions. +The 51-type / 7-category count tracks the admin OpenAPI enum. The runtime spec's webhook-event guidance section in `cycles-protocol-v0.yaml` lists 35 event types across 6 categories — it predates the `webhook` lifecycle category and some later enum additions. ::: ### Budget events (16) @@ -154,11 +154,11 @@ The 47-type / 7-category count tracks the admin OpenAPI enum. The runtime spec's ### Tenant-accessible events -Tenants creating self-service webhooks via `/v1/webhooks` can subscribe to budget, reservation, and tenant events: 27 of the 47 registered event types. They will also receive the additive `_via_tenant_cascade` fan-out events that the reference admin server emits in those same categories on tenant close — see the next section. API key, policy, webhook lifecycle, and system events are admin-only. +Tenants creating self-service webhooks via `/v1/webhooks` can subscribe to budget, reservation, and tenant events: 29 of the 51 registered event types — including the `budget.*` and `reservation.*` cascade fan-out events the admin server emits on tenant close (see the next section). API key, policy, webhook lifecycle, and system events are admin-only. The boundary is normative in governance spec revision v0.1.25.38: on `POST /v1/webhooks` and `PATCH /v1/webhooks/{id}`, both `event_types` **and** `event_categories` are validated against the tenant-accessible set, and an admin-only entry is rejected with `400 INVALID_REQUEST` — `event_categories` is additive with `event_types` in delivery matching, so validating types alone would leave the door open. cycles-server-admin **0.1.25.50** enforces the category check (and closes the legacy update path that left both arrays empty, matching every event class); **0.1.25.49 and earlier validated `event_types` only** — operators upgrading should audit existing tenant subscriptions for admin-only categories (see the [0.1.25.50 release notes](https://github.com/runcycles/cycles-server-admin/releases/tag/v0.1.25.50) for the audit one-liner). ### Tenant-close cascade fan-out -The reference implementation also emits cascade fan-out event names with the `_via_tenant_cascade` suffix as side effects of a `* → CLOSED` tenant transition (Rule 1 — Close Cascade). These names are absent from the published admin-openapi enum (they do not count toward the 47 registered types) but are registered as first-class constants in the reference implementation's `EventType.java`. Treat them as additive implementation events and ignore any unrecognized event type gracefully: +The admin server emits cascade fan-out events with the `_via_tenant_cascade` suffix as side effects of a `* → CLOSED` tenant transition (Rule 1 — Close Cascade). All four names are declared in the governance spec's `EventType` enum since document revision v0.1.25.35, so they count toward the 51 registered types and are filterable like any other lifecycle event. Emission is SHOULD-level in the spec (the matching per-object audit entries are a MUST), so non-reference servers may not emit them — keep ignoring unrecognized event types gracefully: - `budget.closed_via_tenant_cascade` — one per owned `BudgetLedger`. - `reservation.released_via_tenant_cascade` — a **ledger-level aggregate**: one per closed budget with `reserved > 0`, carrying `released_amount`. Reason `tenant_closed`; no overage debt. diff --git a/protocol/webhook-scope-filter-syntax.md b/protocol/webhook-scope-filter-syntax.md index f1c05948..71678ed3 100644 --- a/protocol/webhook-scope-filter-syntax.md +++ b/protocol/webhook-scope-filter-syntax.md @@ -7,18 +7,18 @@ description: "How to filter webhook events by scope using scope_filter on subscr Webhook subscriptions can filter events by scope path using the `scope_filter` field. When set, only events whose `scope` matches the filter are delivered to your endpoint. -::: danger Reference implementation divergence — cycles-server-admin 0.1.25.48 and earlier -**Scope of the divergence:** it applies only to **admin-plane-emitted events** (tenant, api_key, policy, webhook lifecycle, and admin-initiated budget events) plus **replay**. Runtime-emitted events (reservations, runtime budget events — the bulk of webhook volume) have always been matched by the runtime server's own matcher, which already implements the spec semantics below (exact match, trailing-`*` prefix, null scope excluded). In other words: on 0.1.25.48 and earlier, the *same filter* matched differently depending on which plane emitted the event. +::: warning Historical implementation divergence — cycles-server-admin 0.1.25.48 and earlier +**Resolved in current releases:** cycles-server-admin **0.1.25.49** (2026-07-10) ships the spec-conformant admin matcher (and applies the same filter on the replay path), and cycles-server **0.1.25.47** refined the runtime matcher's two edge cases to the same semantics (blank/whitespace-only event scopes are treated as unscoped and excluded from filtered subscriptions; trailing-`/*` filters require a non-empty child segment). The two matchers are pinned to the same table of (filter, scope, expected) test cases, so on admin 0.1.25.49+ / runtime 0.1.25.47+ both planes match identically per the spec and cannot drift. The rest of this callout is history for deployments on older versions. -**Update:** a spec-conformance fix is queued for the next cycles-server-admin release, after which both planes match identically per the spec (with one refinement on the admin side: a blank or null event scope is treated as unscoped, and the replay path applies the same filter). +**Scope of the divergence (admin 0.1.25.48 and earlier):** it applies only to **admin-plane-emitted events** (tenant, api_key, policy, webhook lifecycle, and admin-initiated budget events) plus **replay**. Runtime-emitted events (reservations, runtime budget events — the bulk of webhook volume) have always been matched by the runtime server's own matcher, which already implemented the spec semantics below (exact match, trailing-`*` prefix, null scope excluded; the two edge cases refined in 0.1.25.47 — see [Edge cases](#edge-cases)). In other words: on admin 0.1.25.48 and earlier, the *same filter* matched differently depending on which plane emitted the event. -The admin OpenAPI spec (normative, and described first below) defines exact-match semantics with an optional trailing `*` wildcard. The **admin server's** matcher (`WebhookRepository.matchesScope`, 0.1.25.48 and earlier) instead does **literal prefix matching**: a blank filter matches everything, a null event scope always matches, and otherwise the event scope must `startsWith(scope_filter)` — with a bare `"*"` filter special-cased to match everything. Three practical consequences for admin-plane events on those versions: +The admin OpenAPI spec (normative, and described first below) defines exact-match semantics with an optional trailing `*` wildcard. The **admin server's** matcher (`WebhookRepository.matchesScope`, 0.1.25.48 and earlier) instead did **literal prefix matching**: a blank filter matches everything, a null event scope always matches, and otherwise the event scope must `startsWith(scope_filter)` — with a bare `"*"` filter special-cased to match everything. Three practical consequences for admin-plane events on those versions: -1. **Trailing-`/*` filters match no admin-plane events.** The admin matcher compares the `*` literally, and real scopes never contain a `*` — so a spec-form filter delivers runtime events but silently misses admin-plane events until the fix ships. +1. **Trailing-`/*` filters match no admin-plane events.** The admin matcher compares the `*` literally, and real scopes never contain a `*` — so a spec-form filter delivers runtime events but silently misses admin-plane events until 0.1.25.49. 2. **A filter without `*` is a prefix, not an exact match.** `tenant:acme-corp/workspace:prod` also matches `tenant:acme-corp/workspace:prod/workflow:support` (and even `tenant:acme-corp/workspace:prod-eu`, since matching is character-wise). End the filter with `/` to bound it to child scopes. 3. **Events with a null scope ARE delivered** to scope-filtered subscriptions (a null scope matches every filter), rather than being excluded. -**Recommendation:** write filters in the spec's `/*` form. It matches runtime-emitted events (the bulk of webhook volume) today and becomes fully correct — both planes — once the conformance fix ships. There is no single filter form that matches child scopes on both planes on 0.1.25.48 and earlier: `/*` misses admin events, bare-prefix misses runtime events. If you must catch both before the fix, subscribe without a `scope_filter` and filter client-side on the envelope `scope`. +**Recommendation:** write filters in the spec's `/*` form. It matches runtime-emitted events on every version and both planes on admin 0.1.25.49+ / runtime 0.1.25.47+. There is no single filter form that matches child scopes on both planes on admin 0.1.25.48 and earlier: `/*` misses admin events, bare-prefix misses runtime events. If you must catch both on those older versions, subscribe without a `scope_filter` and filter client-side on the envelope `scope`. Upgrading to 0.1.25.49 is a **behavior change** for existing prefix-style filters — see the [release's migration notes](https://github.com/runcycles/cycles-server-admin/releases/tag/v0.1.25.49) (bare-prefix filters must be rewritten as `…/*`; "base + descendants" coverage now needs two subscriptions). ::: ## Matching rules (spec semantics — normative) @@ -66,14 +66,14 @@ If `scope_filter` is null, empty, or not provided, the subscription matches **al ## Syntax summary -| Filter | Spec semantics (normative; runtime plane today) | Admin plane, 0.1.25.48 and earlier (prefix match) | +| Filter | Spec semantics (normative; both planes on admin 0.1.25.49+ / runtime 0.1.25.47+) | Admin plane, 0.1.25.48 and earlier (prefix match) | |---|---|---| | `null` / empty / blank | All events | All events | | `tenant:acme-corp` | Only scope exactly `tenant:acme-corp` | Any scope starting with `tenant:acme-corp` (including `tenant:acme-corpX`) | | `tenant:acme-corp/` | Only scope exactly `tenant:acme-corp/` (unlikely to exist) | Any scope starting with `tenant:acme-corp/` | -| `tenant:acme-corp/*` | Scopes starting with `tenant:acme-corp/` | Nothing (literal `*` never appears in real scopes) | +| `tenant:acme-corp/*` | Scopes **under** `tenant:acme-corp/` — a non-empty child segment is required from cycles-server 0.1.25.47 onward (the degenerate empty-child scope `tenant:acme-corp/` matched on 0.1.25.46 and earlier) | Nothing (literal `*` never appears in real scopes) | | `tenant:acme-corp/workspace:prod` | Only that exact scope | That scope and anything starting with it | -| `*` | Undefined by spec; the runtime matcher treats it as an empty-prefix trailing wildcard — any non-null scope matches, including blank `""` | All events (including null-scope) | +| `*` | Undefined by spec; the runtime matcher treats it as an empty-prefix trailing wildcard — any non-blank scope matches from cycles-server 0.1.25.47 onward (on 0.1.25.46 and earlier a blank `""` scope also matched, via the empty-prefix comparison) | All events (including null-scope) | | *(any filter)* vs. null-scope event | Not delivered | Delivered | ## What's NOT supported @@ -88,17 +88,20 @@ Under spec semantics — and the runtime matcher — a `*` anywhere other than t ## Examples -The examples below use the spec `/*` form — correct for runtime-emitted events today and for both planes once the admin conformance fix ships. (On 0.1.25.48 and earlier, admin-plane events will not match these filters; see the callout above.) +The examples below use the spec `/*` form — correct for runtime-emitted events on every version, and for both planes on cycles-server-admin 0.1.25.49+ / cycles-server 0.1.25.47+. (On admin 0.1.25.48 and earlier, admin-plane events will not match these filters; see the callout above.) ### Subscribe to all events for one tenant +A subscription must match on at least one selector, so at least one of `event_types` / `event_categories` must be non-empty — the server rejects the empty-both state with `400 INVALID_REQUEST` (governance revision v0.1.25.39; enforced since cycles-server-admin 0.1.25.50). The two arrays are additive (union) in delivery matching. Note the create/update asymmetry: `POST /v1/admin/webhooks` (and `/v1/webhooks`) requires a non-empty `event_types` specifically, while `PATCH` may clear `event_types` to empty as long as `event_categories` is non-empty — a **category-only** subscription is valid on update. To cover whole categories on create, pair a representative type with the category list. See [Category-based subscriptions](/how-to/managing-webhooks#category-based-subscriptions). + ```bash curl -X POST http://localhost:7979/v1/admin/webhooks \ -H "X-Admin-API-Key: $ADMIN_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://ops.example.com/cycles-events", - "event_types": [], + "event_types": ["budget.exhausted"], + "event_categories": ["budget", "reservation", "tenant", "api_key", "policy", "webhook", "system"], "scope_filter": "tenant:acme-corp/*" }' ``` @@ -126,11 +129,12 @@ curl -X POST http://localhost:7979/v1/admin/webhooks \ -H "Content-Type: application/json" \ -d '{ "url": "https://ops.example.com/all-events", - "event_types": [] + "event_types": ["budget.exhausted"], + "event_categories": ["budget", "reservation", "tenant", "api_key", "policy", "webhook", "system"] }' ``` -Both `event_types` and `scope_filter` omitted — this subscription receives all events from all scopes. +`scope_filter` omitted — this subscription receives matching events from all scopes (including unscoped events). All-categories `event_categories` plus a representative `event_types` entry is the "everything" form on **create**, where `event_types` must be non-empty; on a later `PATCH` the type could be cleared, leaving the all-categories subscription category-only. The two arrays are a union in delivery matching. ### Combining event type filter with scope filter @@ -154,7 +158,7 @@ This delivers only `budget.exhausted` **or** `budget.over_limit_entered` events Some events may not have a `scope` field (null). The two semantics differ: - **Spec semantics (normative):** when `scope_filter` is set and an event has a null scope, the event is **not delivered** to that subscription. Use a separate subscription without a scope filter to capture unscoped events. -- **Per plane:** the runtime matcher follows the spec (null scope excluded from filtered subscriptions — `EventEmitterRepository.matchesScope`); the admin plane on 0.1.25.48 and earlier delivers null-scope events to every filter (fixed by the queued conformance change). +- **Per plane:** the runtime matcher follows the spec (null scope excluded from filtered subscriptions — `EventEmitterRepository.matchesScope`); the admin plane on 0.1.25.48 and earlier delivers null-scope events to every filter (fixed in cycles-server-admin 0.1.25.49). ::: tip Note on `reservation.commit_overage` As of cycles-server v0.1.25.46, `reservation.commit_overage` is emitted **with** the reservation's scope path on the envelope, so it participates in scope filtering like any other scoped event. (Earlier releases emitted it with a null envelope scope, in which case the null-scope rules above applied.) @@ -163,8 +167,9 @@ As of cycles-server v0.1.25.46, `reservation.commit_overage` is emitted **with** ## Edge cases - **Whitespace-only filter** (e.g., `" "`): Treated the same as null — matches all events (both semantics). -- **Filter `"*"` alone**: Under spec semantics this is undefined (arguably an exact match against a scope literally equal to `*`). The runtime matcher treats it as "match any non-null-scope event" (a blank `""` scope still matches, via the empty-prefix comparison); the queued admin fix excludes both null and blank scopes; only the admin plane on 0.1.25.48 and earlier matches **all** events including null-scope ones. Prefer omitting `scope_filter` entirely to mean "everything". -- **Events with some scope fields emitted as null**: On 0.1.25.48 and earlier, only `null` scope is checked — an empty string scope (`""`) is not treated as missing. From the conformance fix onward, blank and null scopes are both treated as unscoped (excluded from scope-filtered subscriptions). +- **Filter `"*"` alone**: Under spec semantics this is undefined (arguably an exact match against a scope literally equal to `*`). The runtime matcher treats it as "match any scoped event": from cycles-server 0.1.25.47 onward, blank/whitespace-only scopes are treated as unscoped and excluded — the same semantics as the admin matcher since cycles-server-admin 0.1.25.49; on runtime 0.1.25.46 and earlier, a blank `""` scope still matched via the empty-prefix comparison. Only the admin plane on 0.1.25.48 and earlier matches **all** events including null-scope ones. Prefer omitting `scope_filter` entirely to mean "everything". +- **Blank event scopes** (`""` or whitespace-only): From cycles-server 0.1.25.47 (runtime) and cycles-server-admin 0.1.25.49 onward, blank and null scopes are both treated as unscoped — excluded from every scope-filtered subscription. On runtime 0.1.25.46 and earlier, only `null` was checked (a blank scope could match the bare `*` filter); on admin 0.1.25.48 and earlier, null-scope events match every filter. +- **Empty-child scopes against trailing `/*`** (e.g., event scope `tenant:acme-corp/` against filter `tenant:acme-corp/*`): From cycles-server 0.1.25.47 onward, the runtime matcher requires a non-empty remainder after the prefix — `tenant:acme-corp/` no longer matches (the spec says "all scopes **under** acme-corp"). On 0.1.25.46 and earlier it matched via plain `startsWith`. Unchanged on both: the bare base scope `tenant:acme-corp` (no trailing slash) never matches a `…/*` filter on the runtime plane. ## Related diff --git a/quickstart/architecture-overview-how-cycles-fits-together.md b/quickstart/architecture-overview-how-cycles-fits-together.md index ad262943..a33b4a3e 100644 --- a/quickstart/architecture-overview-how-cycles-fits-together.md +++ b/quickstart/architecture-overview-how-cycles-fits-together.md @@ -20,7 +20,7 @@ This is a reference page. If you haven't set up Cycles yet, start with the [End- Your application talks to the **Cycles Server** (port 7878) at runtime. The **Cycles Admin Server** (port 7979) is the management plane where you create tenants, generate API keys, and configure budget ledgers. The **Cycles Events Service** is an outbound worker that delivers webhook notifications asynchronously and, when CyclesEvidence is enabled, signs evidence envelopes; its app port 7980 and management port 9980 should stay internal. All three services share the same Redis instance. ::: info Independent release cadences -Runtime, admin, events, and dashboard images ship patch releases independently. Latest tagged versions as of 2026-07-09: `cycles-server` 0.1.25.46, `cycles-server-admin` 0.1.25.48, `cycles-server-events` 0.1.25.22, `cycles-dashboard` 0.1.25.67. Older admin servers that predate newer query parameters (e.g., `sort_by`, `search`) ignore them rather than erroring — the APIs follow an additive-parameter guarantee. See the [changelog](/changelog) for the full matrix of minimum versions per feature. +Runtime, admin, events, and dashboard images ship patch releases independently. Latest tagged versions as of 2026-07-11: `cycles-server` 0.1.25.47, `cycles-server-admin` 0.1.25.50, `cycles-server-events` 0.1.25.22, `cycles-dashboard` 0.1.25.67. Older admin servers that predate newer query parameters (e.g., `sort_by`, `search`) ignore them rather than erroring — the APIs follow an additive-parameter guarantee. See the [changelog](/changelog) for the full matrix of minimum versions per feature. ::: ## Components