Skip to content

Add regions to kafka - #2417

Open
Goyo (korgan00) wants to merge 13 commits into
mainfrom
kafka-regions
Open

Add regions to kafka#2417
Goyo (korgan00) wants to merge 13 commits into
mainfrom
kafka-regions

Conversation

@korgan00

@korgan00 Goyo (korgan00) commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Multi-Region Event Streams Routing

Summary

Implement region-aware event publishing for Qiskit Functions. The service is global but usage events must be published to regional Kafka buses. This change makes KafkaEventStreamsClient hold multiple regional producers and route each job's usage events to the correct bus based on the region encoded in the job's instance CRN.

Details and comments

Core

  • kafka_event_streams_client.py: Changed from single producer to registry of regional producers. Auto-discovers regions from EVENT_STREAMS_BOOTSTRAP_SERVERS_<REGION> env vars. Routes events by extracting region from CRN (6th colon-delimited segment).

Tests (9 new cases)

  • Default region from unsuffixed vars (back-compat)
  • Suffixed vars discovered by scan
  • EVENT_STREAMS_DEFAULT_REGION respected
  • Correct producer selected by region
  • Unconfigured region raises error
  • Null/malformed CRN falls back to default
  • Missing API key raises at init
  • Startup log lists regions

Environment:

EVENT_STREAMS_ENABLED=true
EVENT_STREAMS_DEFAULT_REGION=us-east
ENVIRONMENT=production

Checklist

  • I have added tests to cover my changes.
  • I have updated the documentation accordingly.
  • I have added a release note for any user-facing change (new feature, bug fix, deprecation, or breaking change), or this PR has no user-facing changes.

@korgan00
Goyo (korgan00) marked this pull request as ready for review August 31, 2026 10:45
@korgan00
Goyo (korgan00) requested a review from a team as a code owner August 31, 2026 10:45
Goyo (korgan00) and others added 5 commits September 1, 2026 12:51
Co-authored-by: Ismael Rozas <80251943+ismaelRozasRamallal@users.noreply.github.com>
Resolves conflict in test_event_streams_client.py by combining both:
- Multi-region tests from kafka-regions branch
- test_filler_job_publishes_nothing from main
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants