Skip to content

integrations.md

Latest commit

 

History

History
97 lines (72 loc) · 5.23 KB

integrations.md

File metadata and controls

97 lines (72 loc) · 5.23 KB
Raw

Integrations

Kafka (event streaming)

MIDAS publishes governance events to Kafka via a transactional outbox. This is optional — all API functionality works with or without the dispatcher running.

How it works

  1. Every evaluation writes an outbox row atomically in the same Postgres transaction as the domain state.
  2. The outbox dispatcher (background goroutine) polls the outbox table for unpublished rows.
  3. Each row is published to Kafka and marked published_at after broker acknowledgement.

At-least-once delivery: if MIDAS crashes after publishing but before marking the row published, the message will be re-published on the next poll cycle. Consumers must be idempotent.

Configuration

Variable Default Description
MIDAS_DISPATCHER_ENABLED false Set to true to start the outbox dispatcher
MIDAS_DISPATCHER_PUBLISHER none Publisher backend. Only valid value when enabled: kafka
MIDAS_DISPATCHER_BATCH_SIZE 100 Maximum outbox rows claimed per poll cycle; this is not broker publish batching
MIDAS_DISPATCHER_POLL_INTERVAL 2s Active runtime config for sleep between empty-queue poll cycles (Go duration string)
MIDAS_DISPATCHER_MAX_BACKOFF 30s Active runtime config for maximum backoff on consecutive poll errors
MIDAS_KAFKA_BROKERS (none) Comma-separated host:port broker addresses. Required when publisher is kafka
MIDAS_KAFKA_CLIENT_ID midas Client identifier sent to the broker for observability
MIDAS_KAFKA_REQUIRED_ACKS -1 Acknowledgement level: -1 = all in-sync replicas, 0 = none, 1 = leader only
MIDAS_KAFKA_WRITE_TIMEOUT (none) Per-message publish timeout. Zero means no timeout. Go duration string
MIDAS_KAFKA_TLS_ENABLED false Enable TLS for broker connections. Required for Azure Event Hubs Kafka endpoints
MIDAS_KAFKA_SASL_MECHANISM (none) SASL mechanism. Supported value: plain
MIDAS_KAFKA_SASL_USERNAME (none) SASL username. For Azure Event Hubs, use $ConnectionString
MIDAS_KAFKA_SASL_PASSWORD (none) SASL password. For Azure Event Hubs, use the send connection string from a secret

Example:

export MIDAS_DISPATCHER_ENABLED=true
export MIDAS_DISPATCHER_PUBLISHER=kafka
export MIDAS_KAFKA_BROKERS=broker1:9092,broker2:9092

Azure Event Hubs Kafka-compatible example:

export MIDAS_DISPATCHER_ENABLED=true
export MIDAS_DISPATCHER_PUBLISHER=kafka
export MIDAS_KAFKA_BROKERS=evh-midas-test.servicebus.windows.net:9093
export MIDAS_KAFKA_TLS_ENABLED=true
export MIDAS_KAFKA_SASL_MECHANISM=plain
export MIDAS_KAFKA_SASL_USERNAME='$ConnectionString'
export MIDAS_KAFKA_SASL_PASSWORD='<secretref-or-secret-value>'

Startup validation:

  • MIDAS_DISPATCHER_ENABLED=false — all publisher and Kafka fields are ignored.
  • MIDAS_DISPATCHER_ENABLED=true + publisher=kafka + empty MIDAS_KAFKA_BROKERS — MIDAS fails at startup.
  • SASL username and password must be configured together; unsupported SASL mechanisms fail at startup.

D40f-b publishes each claimed topic group with one Kafka batch write and marks successfully acknowledged rows with one batch database update. Delivery remains at-least-once: failed or unconfirmed messages stay unpublished for retry, and a crash after broker acknowledgement but before published_at is written may produce duplicates. MIDAS_DISPATCHER_BATCH_SIZE now has direct throughput impact because it bounds the publish/mark batch size. Durable claim leases are still deferred to D40f-c, and bounded dispatcher concurrency is deferred to D40f-d. MIDAS_KAFKA_REQUIRED_ACKS=-1 remains the recommended durable default.

Events published

Event When
decision.completed Every evaluation
decision.escalated Evaluation produces escalate outcome
decision.review_resolved Reviewer submits a decision via POST /v1/reviews
surface.approved Surface transitions from review to active
surface.deprecated Surface transitions to deprecated

Events sharing the same event_key are published to the same Kafka partition and are ordered within that partition. For decision events, event_key is {request_source}:{request_id}.

Limitations

  • The only supported broker backend is Kafka. The Publisher interface is extensible but no additional broker implementations are included in v1.
  • Consumer offset management is the consumer's responsibility.
  • MIDAS does not provide a replay API. Consumers needing replay must replay from their own store or from the Kafka topic retention window.

For full event payload schemas, see docs/operations/events.md.

SSO / OIDC authentication

MIDAS supports OIDC-based authentication for the Explorer and platform console. The implementation is provider-agnostic and works with any OIDC-compliant identity provider. Configuration examples are provided for Microsoft Entra ID and Google Workspace.

OIDC is independent of the /v1/* API authentication surface. Configuring SSO for the Explorer does not affect how API clients authenticate.

See docs/guides/authentication.md for configuration reference, provider-specific notes, role mapping, and deployment patterns.