Cycles Event Server — Event Delivery and Evidence Signing

Cycles Event Server is the asynchronous processing tier for the Cycles stack. It consumes Redis-backed work streams produced by cycles-server, persists operational records, and performs the out-of-band work that should not block the runtime reservation path: webhook fan-out and CyclesEvidence signing.

It currently runs two independent workers:

Webhook delivery — consumes Cycles events and delivers them to subscriber endpoints using signed, at-least-once HTTP delivery. Deliveries include HMAC-SHA256 signatures, retry with exponential backoff, delivery-attempt tracking, automatic subscription disablement after repeated failures, decryption of webhook signing secrets, and forwarding of administrator-configured custom headers. Custom-header persistence is owned by the admin producer and is not encrypted by this worker.
CyclesEvidence signing — consumes evidence source records emitted by cycles-server, validates every nested request/response against the bundled authoritative v0.2 schema, builds a cycles-evidence/v0.1 envelope, signs it with Ed25519, and stores its RFC 8785 canonical bytes by content address. cycles-server then serves the signed evidence at GET /v1/evidence/{id}. The evidence private signing key is isolated in this service; cycles-server does not need access to it. See CyclesEvidence signing and the identity enablement runbook.

The event server is not the runtime budget authority. It does not reserve, commit, release, or enforce budget. Its role is to turn Cycles state changes into durable, auditable delivery and identity artifacts: webhook deliveries, replayable event history, delivery diagnostics, trace-correlated records, and signed evidence envelopes.

Specs: cycles-governance-admin-v0.1.25.yaml for events, webhooks, delivery records, and replay · cycles-evidence-v0.2.yaml for the evidence envelope and signer authority model. The compatible wire discriminator remains cycles-evidence/v0.1.

Architecture

cycles-server-admin                    cycles-server (runtime)
    │ tenant/budget CRUD,                  │ reservation ops,
    │ api_key/policy lifecycle             │ budget thresholds,
    │                                      │ rate spike detection
    │                                      │
    └──────────┐              ┌────────────┘
               ▼              ▼
         EventService.emit() → save event + find matching subscriptions
         WebhookDispatchService → create PENDING delivery + LPUSH dispatch:pending
               │
               ▼
Redis ──BLMOVE──► cycles-server-events (DispatchLoop)
                    │
                    ├── DeliveryHandler: load delivery + event + subscription
                    ├── WebhookUrlGuard: delivery-time SSRF re-validation against
                    │     built-in baseline + config:webhook-security
                    │     (blocked → permanent FAIL, no retry)
                    ├── SubscriptionRepository: decrypt signing secret (AES-256-GCM)
                    ├── WebhookTransport: HTTP POST with HMAC-SHA256 signature
                    ├── On success: mark SUCCESS, reset consecutive failures
                    ├── On failure + retries left: exponential backoff → RETRYING
                    ├── On failure + retries exhausted: FAILED + increment consecutive failures
                    ├── Ack: LREM dispatch:processing only after state/retry is durable
                     └── When consecutive failures exceed threshold: subscription → DISABLED

Event sources (per spec source field): cycles-admin, cycles-server, cycles-events, expiry-sweeper, anomaly-detector

Why a separate service?

Concern	Admin / Runtime Servers	Events Service
Workload	Synchronous CRUD + reservation ops	Asynchronous delivery, variable latency
Scaling	Scale with API traffic	Scale with webhook volume
Failure isolation	Servers stay responsive during delivery backlog	Delivery retries don't block API
Concurrency	Multiple instances	Multiple instances safe (BLMOVE claim + ack)

CyclesEvidence signing

Alongside webhook delivery, this service is the signing tier for CyclesEvidence — tamper-evident, content-addressed audit envelopes for every budget decision (see the concept docs and cycles-evidence-v0.2.yaml).

cycles-server (producer)                       cycles-server-events (signer)
  emits {decide|reserve|commit|release|error}     EvidenceWorker (reliable queue: BLMOVE)
  source record + computes evidence_id ──LPUSH──►   ├── validate full nested payload against evidence v0.2
  (returns cycles_evidence on the response)         ├── CyclesEvidenceEnvelopeBuilder: build cycles-evidence/v0.1
                                                     ├── recompute evidence_id; CROSS-CHECK vs producer's
                                                    │     stamped id → retain in-flight + pause on drift
                                          evidence:pending ├── EnvelopeSigner: Ed25519 sign (PRIVATE KEY lives here)
                                                     └── EvidenceStore: SET canonical evidence:envelope:<id> ◄── cycles-server
                                                                                                        serves GET /v1/evidence/{id}

Why the signer is in this tier: the expensive work (JCS canonicalization + Ed25519 signing) is asynchronous and must not block a reservation, and the private signing key is isolated to this service — cycles-server only ever holds the public identity (it reproduces the evidence_id content hash synchronously, never signs). The worker recomputes the id and, on mismatch, retains the record in-flight and pauses new claims briefly. This fails closed without destroying evidence while operators reconcile producer/worker identity.

Configure it: the shared public identity (EVIDENCE_SERVER_ID, EVIDENCE_SIGNING_SIGNER_DID) plus this service's private key (EVIDENCE_SIGNING_PRIVATE_KEY_HEX) — full provisioning, coherence rules, and verification steps are in docs/evidence-identity-enablement.md. If EVIDENCE_SERVER_ID is blank, the evidence signer is disabled and does not claim or dead-letter source records.

Quick Start

Full stack (with admin + runtime server)

# From cycles-server-admin directory
docker compose -f docker-compose.full-stack.yml up

Services: Redis (6379), Admin (7979), Runtime Server (7878), Events worker (9980 management; 7980 internal/no API)

Standalone (requires existing Redis)

REDIS_HOST=localhost REDIS_PORT=6379 \
  WEBHOOK_SECRET_ENCRYPTION_KEY="<persistent-base64-encoded-32-byte-key>" \
  java -jar target/cycles-server-events-*.jar

Generate the key once with openssl rand -base64 32, store it in the deployment secret manager, and configure the same value in cycles-server-admin. For local development only, plaintext storage requires an explicit opt-out:

REDIS_HOST=localhost REDIS_PORT=6379 WEBHOOK_SECRET_ENCRYPTION_KEY="" \
  WEBHOOK_SECRET_ALLOW_PLAINTEXT=true \
  java -jar target/cycles-server-events-*.jar

Configuration

Variable	Default	Description
`REDIS_HOST`	localhost	Redis hostname
`REDIS_PORT`	6379	Redis port
`REDIS_PASSWORD`	(empty)	Redis password
`REDIS_USERNAME`	(empty)	Redis ACL username
`REDIS_TLS_ENABLED`	false	Enable TLS for Redis connections
`REDIS_CONNECT_TIMEOUT_MS` / `REDIS_SOCKET_TIMEOUT_MS`	2000 / 5000	Positive Redis connection and non-blocking command timeouts
`REDIS_BLOCKING_SOCKET_TIMEOUT_MS`	10000	Finite socket timeout for blocking Redis claims. Must exceed the longest configured BLMOVE timeout.
`WEBHOOK_SECRET_ENCRYPTION_KEY`	(none; startup fails)	AES-256-GCM key for signing secret encryption (base64-encoded 32 bytes). Existing plaintext values remain readable after a key is added; new writes use `enc:`.
`WEBHOOK_SECRET_ALLOW_PLAINTEXT`	false	Local/development compatibility escape hatch. When true and the key is empty, secrets are stored unencrypted and startup emits a prominent warning. Never enable in production.
`WEBHOOK_URL_GUARD_ALLOW_PRIVATE_NETWORKS`	false	Local/development escape hatch that disables the built-in private-network SSRF baseline. Unspecified addresses and admin-configured CIDRs remain blocked. Never enable in production.
`dispatch.pending.timeout-seconds`	5	BLMOVE blocking timeout (seconds)
`DISPATCH_LOOP_DELAY_MS`	25	Delay between dispatch-loop invocations after a claim completes or times out.
`DISPATCH_PROCESSING_RECOVERY_IDLE_MS`	180000	Minimum age before an in-flight `dispatch:processing` delivery is considered stale and requeued. Startup requires it to exceed both `DISPATCH_ORDERING_LEASE_MS` and the pending-plus-HTTP timeout duration.
`DISPATCH_PROCESSING_RECOVERY_INTERVAL_MS`	30000	Periodic stale-processing recovery interval; prevents quick-restart orphans from remaining stranded
`DISPATCH_FAILED_MAX_LEN`	10000	Maximum newest corrupt-delivery quarantine entries retained in `dispatch:failed`.
`DISPATCH_ORDERING_LEASE_MS`	120000	Fleet-wide claim/send lease that preserves initial-claim FIFO order. Must exceed BLMOVE plus HTTP timeouts; replicas provide failover, not webhook throughput.
`DISPATCH_ORDERING_CONTENTION_BACKOFF_MS`	500	Bounded randomized delay after a global dispatch-lease miss; prevents idle replicas from hammering Redis.
`DISPATCH_EVENT_OUTBOX_POLL_INTERVAL_MS` / `DISPATCH_EVENT_OUTBOX_BATCH_SIZE`	1000 / 25	Poll interval and bounded batch for durable dispatcher-generated meta-events.
`DISPATCH_EVENT_OUTBOX_CLAIM_LEASE_MS` / `DISPATCH_EVENT_OUTBOX_RETRY_DELAY_MS`	30000 / 5000	Per-task publication lease and retry deferral for dispatcher meta-events.
`DISPATCH_EVENT_OUTBOX_MAX_ATTEMPTS` / `DISPATCH_EVENT_OUTBOX_FAILED_MAX_LEN`	100 / 10000	Attempts before a poison meta-event task moves to the bounded `dispatcher:event-outbox:failed` DLQ, and the maximum retained DLQ entries.
`dispatch.retry.poll-interval-ms`	5000	Retry queue poll interval (ms)
`RETRY_BATCH_SIZE`	100	Max retries to requeue per poll cycle
`dispatch.http.timeout-seconds`	30	HTTP request timeout for webhook delivery
`dispatch.http.connect-timeout-seconds`	5	HTTP connect timeout
`MAX_DELIVERY_AGE_MS`	86400000	Maximum delivery age (ms). Deliveries older than this after events service outage are auto-failed instead of delivered stale. Default: 24 hours.
`EVENT_TTL_DAYS`	90	Redis TTL for `event:{id}` keys (days). Spec: "90 days hot."
`DELIVERY_TTL_DAYS`	14	Redis TTL for `delivery:{id}` keys (days).
`RETENTION_CLEANUP_INTERVAL_MS`	3600000	How often to trim expired ZSET index entries (ms). Default: 1 hour.
`RETENTION_LOCK_LEASE_MS`	300000	Cross-replica retention-cleanup lease; keep above the expected cleanup duration
`CYCLES_METRICS_TENANT_TAG_ENABLED`	false	Include tenant IDs as Prometheus metric labels. Keep false in production unless per-tenant drill-down is worth the cardinality.
`SCHEDULING_POOL_SIZE`	5	Scheduled-task executor size. Do not lower below 5 when evidence signing is enabled.
`JAVA_OPTS`	(empty)	Extra JVM options appended after image defaults by the Docker entrypoint.
`EVIDENCE_SERVER_ID`	(empty)	CyclesEvidence `server_id`. Blank disables the evidence signer. When set, it must be an absolute URI and byte-identical to `cycles-server`'s value (incl. the `/v1` base), or the worker retains the mismatched record in-flight and pauses new claims. See the enablement runbook.
`EVIDENCE_SIGNING_SIGNER_DID`	(empty)	Public Ed25519 key (64 hex), the public half of `EVIDENCE_SIGNING_PRIVATE_KEY_HEX`, identical to `cycles-server`'s value.
`EVIDENCE_SIGNING_PRIVATE_KEY_HEX`	(empty)	Secret — Ed25519 seed (64 hex) this service signs with; lives only here. Required when `EVIDENCE_SERVER_ID` is set unless ephemeral dev mode is explicitly allowed; setting only one of the signing pair fails startup.
`EVIDENCE_ALLOW_EPHEMERAL_SIGNING_KEY`	false	Development-only escape hatch. When `true` and evidence is enabled without a signing pair, generates an ephemeral key that will not verify across restarts. Keep `false` in production.
`EVIDENCE_PENDING_KEY` / `EVIDENCE_PROCESSING_KEY` / `EVIDENCE_FAILED_KEY`	evidence:pending / evidence:processing / evidence:failed	Source, in-flight, and bounded poison-record queue keys.
`EVIDENCE_POP_TIMEOUT_SECONDS`	5	Evidence BLMOVE blocking timeout; keep below `REDIS_BLOCKING_SOCKET_TIMEOUT_MS`.
`EVIDENCE_RECOVERY_IDLE_MS` / `EVIDENCE_RECOVERY_INTERVAL_MS`	120000 / 30000	Age threshold and polling interval for recovering evidence work orphaned in `evidence:processing`
`EVIDENCE_RECOVERY_BATCH_SIZE`	100	Maximum processing records inspected per recovery pass
`EVIDENCE_FAILED_MAX_LEN`	10000	Positive maximum length of the evidence poison-record DLQ
`EVIDENCE_LOOP_DELAY_MS` / `EVIDENCE_QUEUE_FAILURE_BACKOFF_MS`	25 / 1000	Scheduler delay and Redis claim/ack outage backoff; prevents tight retry/log loops.
`EVIDENCE_INFRASTRUCTURE_BACKOFF_MS`	30000	Claim pause after identity, signing, or store failures; limits in-flight growth during systemic outages.
`EVIDENCE_STORE_BACKEND` / `EVIDENCE_STORE_KEY_PREFIX` / `EVIDENCE_STORE_TTL_SECONDS`	redis / evidence:envelope: / 0	Content-addressed evidence storage backend, Redis key prefix, and archival TTL (`0` means no expiry).

The delivery-time URL guard always blocks loopback, RFC 1918, IPv4 and IPv6 link-local, carrier-grade NAT, and IPv6 unique-local ranges, including the cloud metadata address 169.254.169.254. Every admin-authored range in config:webhook-security is additive; an empty stored list does not replace the built-in baseline. Unspecified addresses are rejected semantically. The development-only WEBHOOK_URL_GUARD_ALLOW_PRIVATE_NETWORKS=true escape hatch disables only the baseline, logs a warning, and does not override admin ranges or the unspecified-address check. A malformed configured CIDR is treated as indeterminate rather than silently skipped: the delivery remains in-flight for recovery and cycles_webhook_security_config_indeterminate_total increments. Host resolution is checked immediately before delivery, but the JDK HTTP client performs its own connection-time DNS lookup; deployments requiring DNS-rebinding resistance must enforce the same egress policy in a proxy or network firewall.

The Redis client sets the connection name cycles-server-events; a least-privilege Redis ACL therefore needs +client|setname in addition to the commands and key patterns used by this worker.

Generating the encryption key

openssl rand -base64 32

The same key must be configured in both cycles-server-admin and cycles-server-events. Admin encrypts secrets on write; events decrypts on read. Missing key material now fails events-service startup unless WEBHOOK_SECRET_ALLOW_PLAINTEXT=true is explicitly set.

Signing Secret Lifecycle

1. Client creates subscription (POST /v1/admin/webhooks)
   └── optionally provides signing_secret, or admin auto-generates one (whsec_...)

2. Admin stores encrypted secret in Redis
   └── webhook:secret:{subscriptionId} = AES-256-GCM(secret, WEBHOOK_SECRET_ENCRYPTION_KEY)
   └── Returns plaintext secret to client ONCE in WebhookCreateResponse (never again)

3. Events service reads + decrypts secret on each delivery
   └── CryptoService.decrypt(redis.get("webhook:secret:{id}"))
   └── Backward compatible: plaintext secrets (no "enc:" prefix) returned as-is
   └── Missing secrets are treated as transient and left for stale-claim recovery
   └── Fail closed: missing or undecryptable secrets are never delivered unsigned

4. PayloadSigner computes HMAC-SHA256(JSON payload, decrypted secret)
   └── Sent as X-Cycles-Signature: sha256=<hex> header

5. Webhook receiver verifies signature using their copy of the secret

HMAC-SHA256 Signature Verification

The X-Cycles-Signature header contains sha256=<hex> where <hex> is the HMAC-SHA256 of the raw JSON request body using the subscription's signing secret as the key.

Why HMAC? Without it, anyone who discovers a webhook URL can send fake events. HMAC proves both identity (shared secret) and integrity (body hash). Same standard used by GitHub, Stripe, and Slack webhooks.

Verification example (Python):

import hmac, hashlib

def verify(body: bytes, secret: str, signature: str) -> bool:
    expected = "sha256=" + hmac.new(
        secret.encode(), body, hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, signature)

Webhook Delivery Headers

Header	Value	Description
`Content-Type`	`application/json`	Always JSON
`X-Cycles-Signature`	`sha256=<hex>`	HMAC-SHA256 of body; a delivery is deferred or failed closed if the secret cannot be loaded
`X-Cycles-Event-Id`	`evt_abc123...`	For deduplication (at-least-once delivery)
`X-Cycles-Event-Type`	`budget.exhausted`	Event type for routing
`User-Agent`	`cycles-server-events/<build-version>`	Service identifier
`X-Cycles-Trace-Id`	`<32-hex-lowercase>`	W3C trace-id (spec v0.1.25.27) — always present
`traceparent`	`00-<trace-id>-<16-hex-span>-<flags>`	W3C Trace Context v00 — always present. `<flags>` preserves upstream sampling when `WebhookDelivery.traceparent_inbound_valid=true` (spec v0.1.25.28), else `01`
`X-Request-Id`	`<request-id>`	Originating HTTP request id — present when `event.request_id` is populated
Custom headers	Per subscription	From `WebhookSubscription.headers` map

Custom headers are additive only. Reserved delivery headers such as Content-Type, User-Agent, X-Cycles-*, X-Request-Id, and traceparent are ignored if a subscription tries to set them.

Retry Policy

Default: 5 retries with exponential backoff (1s, 2s, 4s, 8s, 16s), capped at 60s max delay.

Setting	Default	Range	Description
`max_retries`	5	0-10	Retries after initial failure (6 total attempts)
`initial_delay_ms`	1000	100-60000	Delay before first retry
`backoff_multiplier`	2.0	1.0-10.0	Multiplier per retry
`max_delay_ms`	60000	1000-3600000	Maximum delay cap

Auto-disable: after disable_after_failures (default 10) consecutive delivery failures, the subscription status is set to DISABLED. Reset to 0 on any successful delivery.

Delivery Status Lifecycle

PENDING ──HTTP 2xx──► SUCCESS (reset consecutive_failures)
    │
    └──non-2xx──► RETRYING ──retry──► SUCCESS
                      │
                      └──max retries exceeded──► FAILED
                                                    │
                                                    └──consecutive > threshold──► subscription DISABLED

Redis Keys (shared with cycles-server-admin)

Key	Type	Written By	Read By	Description
`dispatch:pending`	LIST	Admin (LPUSH), Events retry scheduler (LPUSH)	Events (BLMOVE)	Delivery IDs awaiting processing
`dispatch:processing`	LIST	Events (BLMOVE)	Events (LREM/recovery)	In-flight delivery IDs claimed but not yet acked
`dispatch:processing:claimed_at`	ZSET	Events (ZADD/ZREM)	Events recovery	Claim timestamps used to recover only stale in-flight deliveries
`dispatch:processing:claim_owner`	HASH	Events claim Lua	Events ack/recovery Lua	Per-delivery claim generation; a stale worker cannot ack a successor's claim
`dispatch:retry`	ZSET	Events (ZADD)	Events (ZRANGEBYSCORE)	Retry queue (score = timestamp)
`dispatch:failed`	LIST	Events quarantine Lua	Operators	Bounded wrappers for corrupt delivery records; source `delivery:{id}` remains available for repair
`evidence:processing`	LIST	Events evidence claim Lua	Events evidence ack/recovery Lua	Unique evidence claim IDs; raw payloads are not used as processing identities
`evidence:processing:claimed_at`	ZSET	Events evidence claim Lua	Events evidence recovery Lua	Evidence claim timestamps keyed by unique claim ID
`evidence:processing:claim_owner`	HASH	Events evidence claim Lua	Events evidence ack/DLQ/recovery Lua	Evidence claim generations used to fence stale workers
`evidence:processing:payload`	HASH	Events evidence claim Lua	Events evidence ack/DLQ/recovery Lua	Raw source payload retained by claim ID so identical records remain independent
`delivery:{id}`	STRING	Admin (SET)	Events (GET/SET)	Delivery record JSON
`event:{id}`	STRING	Admin (SET)	Events (GET)	Event record JSON
`webhook:{id}`	STRING	Admin (SET)	Events (GET/SET)	Subscription JSON
`webhook:secret:{id}`	STRING	Admin (SET, encrypted)	Events (GET, decrypts)	AES-256-GCM encrypted signing secret
`dispatcher:event-outbox:pending`	ZSET	Events terminal-state Lua	Events outbox publisher	Due dispatcher meta-event task IDs (score = next attempt time)
`dispatcher:event-outbox:task:{id}`	STRING	Events terminal-state Lua	Events outbox publisher	Durable `webhook.disabled` or `system.webhook_delivery_failed` publication task
`dispatcher:event-outbox:lock:{id}`	STRING	Events outbox publisher	Events outbox publisher	Short owner-token publication lease; removed on ack or expiry
`dispatcher:event-outbox:attempts`	HASH	Events outbox publisher	Events outbox publisher	Publish-failure count used for bounded poison-task retries
`dispatcher:event-outbox:failed`	LIST	Events outbox publisher	Operators	Bounded DLQ of `{task_id,payload}` wrappers for malformed or repeatedly unpublishable tasks

Concurrent safety

Multiple events service instances can safely share dispatch:pending. One global Redis lease deliberately serializes the complete claim/send section across the fleet, preventing simultaneous sends while preserving FIFO order for initial claims. This caps webhook throughput at one in-flight HTTP request (about two deliveries/minute when every endpoint consumes the 30-second timeout); extra replicas provide failover, not throughput. Delayed retries can still complete after later events, as expected for at-least-once delivery with backoff. Each claim receives a unique owner token, and every delivery state write, retry enqueue, and ack verifies that token atomically, so a worker waking after stale recovery cannot alter or erase its successor's work. Recovery is age-gated and periodic: only entries older than DISPATCH_PROCESSING_RECOVERY_IDLE_MS are moved back to pending, and startup validates that threshold is above both the ordering lease and the worst-case claim/send duration.

Final retry exhaustion uses one Redis transaction to persist the failed delivery, increment/disable the subscription, and stage both required dispatcher meta-events. A background publisher uses deterministic event IDs and per-task leases, so a crash or Redis error can replay safely without losing or multiplying the logical event. Meta-events are persisted for observation but are not recursively fanned out as webhook deliveries.

TTL and retention

Key	TTL	Cleanup
`event:{id}`	90 days (configurable)	Auto-expire via Redis EXPIRE
`delivery:{id}`	14 days (configurable)	Auto-expire via Redis EXPIRE
`events:{tenantId}`, `events:_all`	N/A (ZSET)	Hourly trim via RetentionCleanupService
`deliveries:{subId}`	N/A (ZSET)	Hourly trim via RetentionCleanupService
`dispatch:pending`	Self-draining	Claimed by BLMOVE into `dispatch:processing`
`dispatch:processing`	Self-draining	Acked by LREM; stale entries recovered to pending after the idle window
`dispatch:retry`	Self-draining	Entries move to pending when ready
`dispatch:failed`	Bounded to 10000 by default	Operators inspect, repair the retained `delivery:{id}`, and explicitly replay or discard
`dispatcher:event-outbox:pending`, `task:`, `lock:`, `attempts`	Self-draining	Deterministic tasks remain until durably saved, acknowledged, or moved to the bounded DLQ after retry exhaustion
`dispatcher:event-outbox:failed`	Bounded to 10000 by default	Operators inspect and replay or discard poison meta-event tasks

Resilience: events service down

If cycles-server-events is not running or not deployed:

Admin and runtime servers are unaffected — event emission is fire-and-forget, wrapped in try-catch, never blocks the API response
Events and deliveries accumulate in Redis — event:{id} keys (90-day TTL), delivery:{id} keys (14-day TTL), dispatch:pending list grows
Redis memory is bounded — TTLs ensure keys auto-expire even if never consumed
When the events service restarts:
- Stale deliveries (older than MAX_DELIVERY_AGE_MS, default 24h) are immediately marked FAILED — they won't be delivered late
- Orphaned in-flight deliveries in dispatch:processing are recovered to dispatch:pending after DISPATCH_PROCESSING_RECOVERY_IDLE_MS
- Fresh deliveries are processed normally via BLMOVE claim + ack
- RetentionCleanupService trims orphaned ZSET index entries hourly
No data loss for events — event records persist in Redis for 90 days regardless of delivery status

Admin dual-auth on tenant webhook endpoints (informational)

As of admin-spec v0.1.25.16, six tenant-scoped webhook REST endpoints on cycles-server-admin accept both ApiKeyAuth and AdminKeyAuth: GET /v1/webhooks, GET/PATCH/DELETE /v1/webhooks/{id}, POST /v1/webhooks/{id}/test, GET /v1/webhooks/{id}/deliveries. Admin-initiated updates record actor_type=admin_on_behalf_of in audit metadata (vs api_key for tenant-initiated).

No functional impact on this service — cycles-server-events reads subscriptions from Redis directly and does not call those admin HTTP endpoints. Noted here for observability and ops awareness.

Event Types (51)

Category	Count	Types
`budget`	17	created, updated, funded, debited, reset, reset_spent, debt_repaid, frozen, unfrozen, closed, closed_via_tenant_cascade, threshold_crossed, exhausted, over_limit_entered, over_limit_exited, debt_incurred, burn_rate_anomaly
`reservation`	6	denied, denial_rate_spike, expired, expiry_rate_spike, commit_overage, released_via_tenant_cascade
`tenant`	6	created, updated, suspended, reactivated, closed, settings_changed
`api_key`	7	created, revoked, revoked_via_tenant_cascade, expired, permissions_changed, auth_failed, auth_failure_rate_spike
`policy`	3	created, updated, deleted
`system`	5	store_connection_lost, store_connection_restored, high_latency, webhook_delivery_failed, webhook_test
`webhook`	7	created, updated, paused, resumed, deleted, disabled, disabled_via_tenant_cascade

budget.reset_spent (v0.1.25.6, admin-spec v0.1.25.18) is emitted for billing-period rollovers and is distinct from budget.reset (which is a ceiling resize that preserves spent). Consumers can route these separately. The payload's spent_override_provided flag indicates whether spent was explicitly supplied (migration / proration / correction) vs defaulted to 0 (routine rollover).

Transport Layer

Pluggable transport interface. Currently implements webhook (HTTP POST).

public interface Transport {
    String type();
    TransportResult deliver(Event event, Subscription subscription, String signingSecret, Delivery delivery);
}

Monitoring

Spring Actuator endpoints run on a separate management port (9980). This worker has no public HTTP API; do not publish 7980 on an ingress. Keep 9980 on an internal-only ClusterIP / network and scrape from there.

Endpoint	Description
`GET :9980/actuator/health/liveness`	Process liveness check
`GET :9980/actuator/health/readiness`	Readiness check, including Redis `PING`
`GET :9980/actuator/health`	Aggregate health
`GET :9980/actuator/info`	Build info (version, artifact)
`GET :9980/actuator/prometheus`	Prometheus-format metrics for scraping

Prometheus scrape config example:

scrape_configs:
  - job_name: cycles-server-events
    metrics_path: /actuator/prometheus
    static_configs:
      - targets: ['localhost:9980']

Override the management port via the MANAGEMENT_PORT env var if 9980 collides.

In addition to Spring Boot's auto-emitted http_server_requests_seconds, the service exposes webhook delivery meters under cycles_webhook_* and evidence lifecycle counters under cycles_evidence_*. Operators can alert on delivery failures, stale backlogs, subscription auto-disables, invalid payloads, evidence DLQ growth, and infrastructure-deferred evidence retries without grepping logs.

Full metric inventory, tag semantics, ready-to-paste Prometheus alert rules, SLO definitions, dashboard queries, and an incident playbook live in OPERATIONS.md.

Webhook Payload Example

The webhook POST body is the full event JSON. Null fields are omitted.

{
  "event_id": "evt_abc123",
  "event_type": "budget.exhausted",
  "category": "budget",
  "timestamp": "2026-04-01T12:00:00Z",
  "tenant_id": "t_xyz789",
  "source": "runtime",
  "data": {
    "budget_id": "bdg_001",
    "current_balance": 0,
    "limit": 10000
  },
  "actor": {
    "type": "api_key",
    "key_id": "key_abc",
    "source_ip": "10.0.1.42"
  },
  "correlation_id": "req_def456"
}

Build & Test

# Run the fast unit suite (515 tests in the current suite)
mvn test

# Run all 540 tests including integration; enforces 95% line / 95% branch coverage
# (requires Docker for Testcontainers Redis)
mvn verify -Pintegration-tests

# Run
REDIS_HOST=localhost REDIS_PORT=6379 java -jar target/cycles-server-events-*.jar

Documentation

CHANGELOG.md — release notes for downstream consumers (Docker / JAR / operators)
OPERATIONS.md — operator runbook: metrics inventory, alert recipes, SLOs, incident playbook
AUDIT.md — engineering history, audit posture, and cross-repo drift notes
Sibling services (same conventions, dashboards carry over):
- cycles-server — runtime reservation + budget authority
- cycles-server-admin — admin plane (tenants, budgets, webhooks, API keys)

License

Apache License 2.0

Name		Name	Last commit message	Last commit date
Latest commit History 242 Commits
.github		.github
docs		docs
src		src
tools		tools
.dockerignore		.dockerignore
.gitignore		.gitignore
AUDIT.md		AUDIT.md
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
Dockerfile		Dockerfile
LICENSE		LICENSE
MAINTAINERS.md		MAINTAINERS.md
OPERATIONS.md		OPERATIONS.md
README.md		README.md
lombok.config		lombok.config
pom.xml		pom.xml

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

Cycles Event Server — Event Delivery and Evidence Signing

Architecture

Why a separate service?

CyclesEvidence signing

Quick Start

Full stack (with admin + runtime server)

Standalone (requires existing Redis)

Configuration

Generating the encryption key

Signing Secret Lifecycle

HMAC-SHA256 Signature Verification

Webhook Delivery Headers

Retry Policy

Delivery Status Lifecycle

Redis Keys (shared with cycles-server-admin)

Concurrent safety

TTL and retention

Resilience: events service down

Admin dual-auth on tenant webhook endpoints (informational)

Event Types (51)

Transport Layer

Monitoring

Webhook Payload Example

Build & Test

Documentation

License

About

Releases

Packages

Used by

Contributors

Languages

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

Cycles Event Server — Event Delivery and Evidence Signing

Architecture

Why a separate service?

CyclesEvidence signing

Quick Start

Full stack (with admin + runtime server)

Standalone (requires existing Redis)

Configuration

Generating the encryption key

Signing Secret Lifecycle

HMAC-SHA256 Signature Verification

Webhook Delivery Headers

Retry Policy

Delivery Status Lifecycle

Redis Keys (shared with cycles-server-admin)

Concurrent safety

TTL and retention

Resilience: events service down

Admin dual-auth on tenant webhook endpoints (informational)

Event Types (51)

Transport Layer

Monitoring

Webhook Payload Example

Build & Test

Documentation

License

About

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages