Cluster-aware Subscription notification delivery

[smunini]

The Subscriptions engine tracks connected WebSocket clients in process memory (crates/subscriptions/src/channels/ws_manager.rs), so in a multi-instance deployment only the instance holding a client's connection can deliver its notifications.

Subscription resources themselves are database-backed; what's needed is a shared delivery channel (e.g., a message bus or pub/sub fan-out) so resource events on any instance reach clients connected to any other.

Until then, WebSocket subscription delivery is single-instance / sticky-session only (rest-hook channels are unaffected).

Scope

• Introduce a shared delivery channel (message bus / pub-sub fan-out) for subscription notifications
• Ensure resource events on any instance reach WebSocket clients connected to any other instance
• rest-hook channels are already cluster-safe and out of scope

Tracked on the ROADMAP under Next → FHIR Server Capabilities.

[aacruzgon]

Scope note: the in-process gap is broader than ws_manager.rs alone

While investigating this, the same in-process DashMap pattern shows up across crates/subscriptions, all constructed with empty in-memory maps in SubscriptionEngine::new() (engine/mod.rs:80–88) and held behind an Arc within a single process — with no shared/distributed backing and nothing persisted. Behind a load balancer this produces four related gaps, not one. Fixing only the WS client registry won't make WebSocket subscriptions cluster-correct, so suggesting we expand the scope accordingly.

#	Component	Location	Multi-instance impact
1	WS client registry	channels/ws_manager.rs (clients: DashMap<(tenant, sub_id), …>)	The triggering resource write can land on any instance, but the WS client is connected to one specific instance. The writing instance's ws_manager has no entry for that client → no delivery. (The originally-reported issue.)
2	WS binding token store	channels/ws_token.rs (tokens: DashMap<…>, opaque UUID, consumed-on-first-use)	$get-ws-binding-token stores the token in instance A's map; the subsequent WS upgrade connect may land on instance B, whose map has no such token → validate_and_consume returns None → bind rejected.
3	Event counter + runtime state	manager/mod.rs — increment_event_count (:331) bumps an in-memory DashMap entry; events_since_start is never persisted or reloaded	Writes for a subscription's topic can be handled by different instances, each bumping its own counter from its own start → clients see non-monotonic / duplicated eventNumber and eventsSinceSubscriptionStart, violating the spec's monotonic-numbering requirement.
4	$events recovery + event history	subscriptions.rs:63 (subscription_events_handler)	The route exists but is a status stub: it returns only a query-status SubscriptionStatus echoing the in-memory counter, ignores eventsSinceNumber/eventsUntilNumber, and there is no persisted event/notification history to replay. The spec's intended recovery mechanism is therefore non-functional today.

Note: Subscription resources are DB-backed as described above, but the manager's runtime state is in-memory.

Sticky sessions are not a sufficient workaround. Session affinity can pin a WS client to one instance (making the token store and connection registry work for that client), but it does not address #1, #3, or #4, because those break on the write path — the resource write that triggers a notification can originate from any source and land on any instance, regardless of which instance holds the WS connection. So sticky sessions mitigate #2 only.

Spec-driven constraints / backstop for the eventual design

The FHIR Subscriptions Framework is silent on deployment topology (clustering is infra-level, not a conformance requirement), but a few spec mechanisms shape the solution:

• Monotonic event numbering (SubscriptionStatus.eventsSinceSubscriptionStart, per-event eventNumber) must be coordinated across instances — i.e. a shared/atomic counter rather than the current in-memory increment.
• $events is the spec's intended reliability backstop, but is not functional as one today. The route exists (subscription_events_handler), but the handler returns only a query-status SubscriptionStatus echoing the in-memory counter — it ignores eventsSinceNumber/eventsUntilNumber and there is no persisted event/notification history to replay. Standing up a persisted event log (and honoring the range params) is a prerequisite for at-least-once delivery + $events recovery, which is the conformant alternative to exactly-once fan-out under the framework's best-effort baseline.
• $get-ws-binding-token — the spec states token security is implementation-dependent, so we're free to make the token stateless (e.g. signed JWT) or back it with a shared store so it validates on whichever instance the LB routes the client to.

Proposed scope update

1. Shared delivery channel / pub-sub fan-out for notifications (original ask).
2. Cluster-valid WS binding tokens (stateless or shared store).
3. Shared/atomic per-subscription event counter + runtime state.
4. Persisted event log + functional $events (range params) as the at-least-once recovery backstop.

Relates to the design discussion in #59 (Open Questions #4 — first-party messaging channels / ChannelDispatcher, and #5 — subscription state store).

[smunini]

Instead of these approaches, what if we have a single hfs instance nominated as "the notifier" and the others would route their notifications through it. This would be controlled by env variables.

That's a real simplification, and it works — it collapses three of the four items by making the database the coordination point instead of new pub/sub infrastructure. The trick is that "one notifier" only helps if that node is also the only place the trigger, the counter, and the WS connections live. Here's the model and how it maps to the four gaps.

The model

Designate one instance as the notifier. It is the only node that evaluates subscription matches, increments counters, persists the event log, and holds WS connections. Every other node is a plain write node that just persists resources.

The one question that decides whether this works: how does a write on node B trigger a notification owned by the notifier? It can't be the current in-process path (today each node evaluates its own writes against its own in-memory subscription set — engine/mod.rs). Under a single notifier you need the write to reach the notifier without an RPC mesh. The clean, backend-agnostic way is an outbox: each resource write inserts a row into a queue/outbox table in the same transaction, and the notifier polls that table by cursor, runs matching, and delivers. The database you already have becomes the delivery channel — no Redis/NATS/broker.

How it maps to the four gaps

┌─────┬────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ # │ Original proposal │ Under a single notifier │
├─────┼────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 1 │ Shared pub-sub fan-out bus │ Removed. Outbox table → notifier polls → notifier delivers to its own local WebSocketManager. The DB is the bus. │
├─────┼────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 2 │ Cluster-valid binding tokens │ Removed. Route $get-ws-binding-token and the WS upgrade to the notifier via an LB rule. Both hit one node, so │
│ │ (JWT or shared store) │ the current DashMap token store (ws_token.rs) just works — no JWT, no shared store. │
├─────┼────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 3 │ Shared atomic counter (Redis │ Downgraded to a single in-memory counter on the notifier — naturally monotonic because there's one writer. Just │
│ │ INCR / DB sequence) │ persist it so failover/restart doesn't reset numbering. No distributed atomic op. │
├─────┼────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 4 │ Persisted event log + │ Still required — and more important. It's now your failover/restart recovery net: anything missed while the │
│ │ functional $events │ notifier was down is recovered via $events. Single writer, so the log is trivial to write consistently. │
└─────┴────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

So the simplification deletes the message bus, the stateless/JWT token work, and the distributed counter. You keep the event log (which you wanted anyway), and you pay for two new, smaller things:

What it costs instead

1. WS traffic must pin to the notifier. An LB routing rule: subscription WS paths (and $get-ws-binding-token) go to the notifier instance. This is the single-node-delivery assumption made explicit, not sticky-sessions-per-client.
2. Failover. The notifier is a single point of failure. Two options, pick by how much you want now:

• Static config (simplest v1): one instance flagged HFS_SUBSCRIPTION_NOTIFIER=true. If it dies, ops promotes another / it restarts. No election code at all. Item feat: first iteration of bindings #4 covers the recovery gap.
• DB-backed lease (auto-failover): a lease row the notifier renews on a TTL; if it lapses, another node grabs it and becomes notifier. ~50 lines, no consensus library, and it's the one piece of genuine distributed-systems logic you take on — far less than a pub/sub mesh + shared-state abstraction.

3. Throughput ceiling. One node does all matching and holds all WS connections. For typical FHIR subscription cardinality that's fine; if you outgrow it you can reintroduce fan-out later — and the outbox + event log you built remain the right foundation for that.

My recommendation

Go with the single notifier, static config for v1, outbox-driven trigger, persisted counter, and a real $events log as the recovery backstop. It's strictly less infrastructure than the four-part proposal, and it degrades gracefully: when you later need horizontal WS scale, you swap "notifier delivers locally" for "notifier publishes to a bus" without touching the outbox, counter, or event-log work. The honest caveat to write down is that it trades a scaling bottleneck + a failover story for the elimination of three distributed components — a good trade until subscription volume says otherwise.

One thing to confirm before committing: whether every storage backend (/test-hfs lists SQLite, PG, ES, Mongo) can support a transactional outbox write alongside the resource write. SQLite/PG yes; for Mongo and the composite/ES paths it's worth checking the write path can append the outbox row atomically, or you fall back to a _lastUpdated cursor poll (at-least-once, which the event log already tolerates).

Want me to write this up as a comment on #170 — framed as an alternative/simpler design to the four-part scope — or sketch the outbox + notifier-lease design in more detail first?