Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions admin-api/guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -336,7 +336,7 @@ The admin server provides 21 webhook/event endpoints for real-time observability
- **Delivery tracking**: list delivery attempts per subscription with status/date filters
- **Event replay**: re-deliver historical events to a subscription
- **Security config**: manage webhook URL SSRF protection (blocked CIDRs, HTTPS enforcement)
- **Tenant self-service**: tenants manage their own webhooks at `/v1/webhooks` for budget, reservation, and tenant events (27 of 47 registered event types, plus the additive `_via_tenant_cascade` fan-out events the reference admin server emits in those categories on tenant close — see [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics))
- **Tenant self-service**: tenants manage their own webhooks at `/v1/webhooks` for budget, reservation, and tenant events (29 of 51 registered event types, including the `_via_tenant_cascade` fan-out events the admin server emits in those categories on tenant close — see [Tenant-Close Cascade Semantics](/protocol/tenant-close-cascade-semantics))

Events are emitted by admin controllers (tenant, budget, api-key, policy operations) and delivered asynchronously by the events service (`cycles-server-events`). See [Webhooks and Events](/concepts/webhooks-and-events) for architecture details.

Expand Down Expand Up @@ -420,7 +420,7 @@ curl -X POST http://localhost:7979/v1/webhooks \
}'
```

See [Webhook Event Delivery Protocol](/protocol/webhook-event-delivery-protocol) for the full 47-event-type reference and delivery specification. See [Webhook Integrations](/how-to/webhook-integrations) for PagerDuty, Slack, and ServiceNow examples.
See [Webhook Event Delivery Protocol](/protocol/webhook-event-delivery-protocol) for the full 51-event-type reference and delivery specification. See [Webhook Integrations](/how-to/webhook-integrations) for PagerDuty, Slack, and ServiceNow examples.

## List-endpoint features (v0.1.25.22+)

Expand Down
4 changes: 2 additions & 2 deletions blog/real-time-budget-alerts-for-ai-agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ This is why we built a webhook event system into Cycles v0.1.25.

## 41 event types across 6 categories

> **As of post date.** The current Admin API `EventType` enum registers **47 event types across 7 categories** (the `webhook` category was added later). For the live count and per-category breakdown, see the [Event Payloads Reference](/protocol/event-payloads-reference).
> **As of post date.** The current Admin API `EventType` enum registers **51 event types across 7 categories** (the `webhook` category and the tenant-close cascade types were added later). For the live count and per-category breakdown, see the [Event Payloads Reference](/protocol/event-payloads-reference).

Every observable state change in the system produces an event. We organized them into 6 categories covering the full lifecycle:

Expand Down Expand Up @@ -172,7 +172,7 @@ The response includes a signing secret (returned once — store it). Your middle

We have full integration guides with code examples for [PagerDuty, Slack, Datadog, Microsoft Teams, Opsgenie, and ServiceNow](/how-to/webhook-integrations), plus a [custom receiver pattern](/how-to/webhook-integrations#integration-custom-receiver-direct) with signature verification in Python, Node.js, and Go.

Tenants can also create their own webhook subscriptions via `/v1/webhooks` using their API key — restricted to budget, reservation, and tenant events (27 of 41 types as of post date; the live count is 27 of 47 — see the [Webhook Event Delivery Protocol](/protocol/webhook-event-delivery-protocol#tenant-accessible-events) for current category eligibility). Admin-only events (api_key, policy, webhook, system) require admin key access.
Tenants can also create their own webhook subscriptions via `/v1/webhooks` using their API key — restricted to budget, reservation, and tenant events (27 of 41 types as of post date; the live count is 29 of 51 — see the [Webhook Event Delivery Protocol](/protocol/webhook-event-delivery-protocol#tenant-accessible-events) for current category eligibility). Admin-only events (api_key, policy, webhook, system) require admin key access.

Webhook URLs are validated at creation time with SSRF protection enabled by default: RFC 1918 private IP ranges, loopback, and link-local addresses are blocked, and HTTPS is required in production. These can be configured via `PUT /v1/admin/config/webhook-security` for environments that need internal endpoint access.

Expand Down
12 changes: 9 additions & 3 deletions blog/tenant-lifecycle-cascade-semantics-at-scale.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,13 +51,15 @@ Any combination of these carries real operational and security risk: silent post

The Cycles governance-admin spec addresses this directly with a two-rule contract, formalized in the `CASCADE SEMANTICS` section of the v0.1.25 governance-admin yaml.

**Rule 1 — Close Cascade.** When a tenant transitions to `CLOSED`, the server performs a strictly ordered sequence against every owned object:
**Rule 1 — Close Cascade.** When a tenant transitions to `CLOSED`, the server drives every owned object to its terminal state. In the atomic presentation (Mode A, below), the spec's recommended order within the single transaction is:

1. Drain open reservations (released with reason `tenant_closed`; no overage debt recorded).
2. Close budget ledgers (final balance snapshot preserved for audit; no new reservations accepted).
3. Disable webhook subscriptions and revoke API keys (either order).
4. Flip `tenant.status` to `CLOSED` last.

Mode B (below) inverts this by design — the tenant flip commits *first*, and the children converge afterward under the Rule 2 guard. Since the runcycles reference server implements Mode B, don't build tooling that depends on this ordering; depend on the terminal states and the guard.

Each mutated object produces a dedicated record: an Event row under a reserved dotted name — `budget.closed_via_tenant_cascade`, `api_key.revoked_via_tenant_cascade`, `reservation.released_via_tenant_cascade` (ledger-level), `webhook.disabled_via_tenant_cascade` — plus an audit row written as `operation="tenant_close_cascade"` with `resource_type`/`resource_id`. The event rows all share a server-composed `correlation_id` (`tenant_close_cascade:<tenant_id>:<request_id>`) on the emitted event rows, so an auditor can reconstruct the cascade in a single events query (audit rows join via `request_id`/`trace_id`).

**Rule 2 — Terminal-Owner Mutation Guard.** Every mutating endpoint on an owned object first checks the parent tenant's status. If the tenant is `CLOSED`, the endpoint returns `409 Conflict` with `error: "TENANT_CLOSED"`, regardless of the per-object terminal state. The guard applies across the budget, reservation, policy, API key, and webhook planes. `GET` endpoints remain available — closed-tenant state is readable for post-mortems and compliance evidence.
Expand All @@ -77,7 +79,7 @@ How these rules are implemented can vary. A protocol spec that only accepted one
- Convergence within a documented bound.
- Observable reads of non-terminal children remain consistent until the cascade reaches them.

The important property is that both modes produce the same *client-observable* outcome: once the tenant is `CLOSED`, every mutation against any owned object returns a `409` with `error: "TENANT_CLOSED"`, regardless of which per-object row flipped first. The mode is an implementation detail the spec deliberately leaves open — a transactional SQL backend can deliver Mode A cleanly, while a Redis-backed admin can opt into Mode B as long as the guard activates at or before the flip's durability.
The important property is that both modes produce the same *client-observable* outcome: once the tenant is `CLOSED`, every admin-plane mutation against any owned object returns a `409` with `error: "TENANT_CLOSED"`, regardless of which per-object row flipped first — and the same guard holds post-auth on the runtime plane, where a revoked tenant key is rejected with `401` before the guard is ever consulted (the close walkthrough below shows both doors). The mode is an implementation detail the spec deliberately leaves open — a transactional SQL backend can deliver Mode A cleanly, while a Redis-backed admin can opt into Mode B as long as the guard activates at or before the flip's durability.

## Where operators actually trip

Expand Down Expand Up @@ -116,10 +118,14 @@ curl -s -H "X-Admin-API-Key: $ADMIN_KEY" \

You'll see one record per owned object — one `budget.closed_via_tenant_cascade` per ledger, one `api_key.revoked_via_tenant_cascade` per key, one `webhook.disabled_via_tenant_cascade` per subscription — plus one **ledger-level** `reservation.released_via_tenant_cascade` per closed budget that had `reserved > 0`, carrying the aggregate `released_amount` (not one event per reservation). The corresponding event rows all share the server-composed cascade `correlation_id`, which is how an auditor reconstructs the cascade without having to cross-join on timestamp (audit rows join via the originating `request_id`).

A subsequent attempt to mutate an owned object under the closed tenant returns the terminal-owner guard's `409`. Reservation lifecycle lives on the runtime plane — the spec's Rule 2 explicitly scopes reservation create/commit/release/extend, so the `409 TENANT_CLOSED` below is the normative contract (note: the current runtime reference server's error enum does not yet include `TENANT_CLOSED`; today it surfaces closed tenants as `401`s from revoked keys or `BUDGET_CLOSED`):
A subsequent attempt to mutate an owned object under the closed tenant returns the terminal-owner guard's `409`. Reservation lifecycle lives on the runtime plane — the spec's Rule 2 explicitly scopes reservation create/commit/release/extend, so the `409 TENANT_CLOSED` below is the normative contract — implemented in `cycles-server` 0.1.25.47, which added `TENANT_CLOSED` to the runtime error enum per spec revision v0.1.25.13 (on 0.1.25.46 and earlier, the runtime plane surfaces closed tenants only as `401`s from revoked keys or `BUDGET_CLOSED`):

```bash
# Mutation on a released reservation under a closed tenant
# (reachable with a not-yet-revoked tenant key in the post-flip window —
# once the cascade revokes the key, the 401 below wins first. Of the four
# guarded mutations only release also accepts an admin key on the runtime
# plane, so an admin-on-behalf-of release hits this 409 with no race.)
curl -i -X POST \
-H "X-Cycles-API-Key: $TENANT_KEY" \
"http://localhost:7878/v1/reservations/res-xyz/commit"
Expand Down
Loading