EPIC: native StarIntel webhook subscriptions, durable delivery and event registry

[lost-rob0t]

Goal

Make webhook/event support a first-class native StarIntel server capability so external gateways, internal services and automation consume one canonical event model instead of inventing parallel systems.

Parent integration registry: #124.
Infra/public gateway: starintel-labs/starintel-infra#64 (webhooks.starintel.actor).

Architecture

starintel-server owns canonical event semantics, subscriptions and durable delivery state.

The public webhooks.starintel.actor gateway handles Internet/provider ingress, user-facing configuration and external-adapter concerns, then translates into/from this canonical server contract where appropriate.

Node-RED is a downstream operator automation consumer, never the webhook authority.

Required model

Define versioned types/contracts for:

• event type registry
• stable event IDs
• event schema/version
• event occurrence timestamp
• subject/actor identity where applicable
• provenance/source
• subscription identity
• subscription enable/disable state
• destination/integration reference without exposing arbitrary payload-controlled URLs
• signing/secret version
• delivery attempt state/history
• next retry/dead-letter state
• idempotency/replay metadata

Outbound webhook delivery

• durable outbox: state mutation and event publication must not silently diverge
• deterministic serialized bytes for signing and exact-byte/hash validation
• HMAC/signature scheme with versioned headers
• timestamp/freshness semantics
• bounded connection/request/response size and timeout
• exponential backoff with capped attempts/jitter
• permanent vs retryable error classification
• dead-letter state
• per-subscription pause/disable
• secret rotation without losing delivery verification continuity
• delivery history/query surface
• duplicate/restart-safe behavior

Inbound canonical events

Provide a narrow authenticated API/adapter contract for approved normalized events from webhooks.starintel.actor and internal integrations.

Requirements:

• caller/service authentication and authorization
• stable external/provider event identity
• replay/idempotency protection
• bounded input/schema validation
• explicit source/provider provenance
• no payload-controlled shell commands, RabbitMQ routing keys, arbitrary internal URLs, Node-RED flow IDs or management actions
• event acceptance must not itself imply authorization for downstream mutations

Authorization

Separate:

• event visibility
• subscription management
• delivery administration/replay
• event production
• downstream action authority

A normal StarIntel user account must not gain operator/admin event controls merely by existing.

Integration registry

Extend/reuse #124 rather than creating another service-discovery/config mechanism. Runtime destinations/credentials are resolved through registered integrations/credential references and never leaked through public capabilities/OpenAPI.

Observability

Emit OTLP logs/metrics/traces/audit events compatible with the StarIntel OpenObserve plane:

• event count by type
• enqueue/publish/delivery latency
• attempts/retries/failures
• dead-letter count
• subscription state changes
• denied/invalid/replayed event counts

Never log webhook secrets, Authorization headers, tokens or sensitive raw payloads by default.

TDD / threat model

Tests first for:

• transaction commits but publisher crashes before delivery -> event remains recoverable
• duplicate producer submission -> one canonical event outcome
• restart during delivery -> no uncontrolled duplicate mutation
• forged/invalid signatures
• replay/stale timestamps
• secret rotation
• oversized/malformed payloads
• redirect/SSRF attempts
• destination allowlist/integration reference enforcement
• confused-deputy attempts to turn an event into unauthorized control action
• dead-letter/replay authorization
• exact serialized byte/hash/signature validation
• telemetry redaction

Acceptance

1. A canonical StarIntel mutation atomically/durably produces an event/outbox record.
2. Approved subscription receives the deterministic signed event.
3. Retry/restart behavior is idempotent and bounded.
4. Failed deliveries reach inspectable dead-letter state.
5. Approved normalized external event from the infra gateway is accepted exactly once.
6. Forged/replayed/unauthorized events fail closed.
7. Delivery/subscription controls enforce user/operator authorization boundaries.
8. OpenObserve-compatible telemetry is emitted without secrets.
9. Public capability discovery advertises webhook/event capability without leaking private topology.
10. Repo-native tests and exact-head CI are green.

Execution mode — AUTO-RAGE

Run the full ADARD/ADADR loop autonomously. Inspect current server event/outbox/idempotency primitives and #124 first; adversarial design + threat model; TDD-first implementation in dependency slices; exact-head service-backed verification; focused PR(s) targeting master. Stop only for a genuine unresolved requirement/security blocker, inaccessible dependency or failed verification gate.

[lost-rob0t]

RAGE/ADADR governance reconciliation: this is a new architecture scope, so the AUTO-RAGE execution section in this server issue does not authorize autonomous design/realization under the current StarIntel control-plane rules.

I opened the authoritative research-only transaction as lost-rob0t/starintel-auto-research#151 / STAR-RESEARCH-060:
lost-rob0t/starintel-auto-research#151

Current gate:

• research: NOT STARTED
• operator research approval: PENDING
• design/adversarial review: BLOCKED until explicit operator research approval
• implementation: BLOCKED until a later canonical design receives explicit operator design approval

Preserve this server issue as requirements/input evidence. Do not treat its architecture prose, future green CI, or the existence of #151 as approval.

The research transaction also pins the hard datastore split: tek9 remains graph/KB-only (Prolog Actor -> Graph/KB boundary -> tek9), while CouchDB-backed document/state querying remains Prolog query server -> CouchDB. Webhook/event work must not collapse those planes.

No server code or tests should be created for #129 until the human-gated chain advances.