Skip to content

[Question][scalable-topics] How are consumer assignments ordered across controller reconnects? #26273

Description

@FlorentinDUBOIS

Context

PIP-468 allows consumer membership changes to recompute assignments without changing the segment layout. Different ScalableConsumerAssignment messages can therefore carry the same layout_epoch.

The current protocol carries the layout epoch and segment list, but no assignment generation, controller term, or session incarnation. The Java V5 client rejects only epoch < currentEpoch, so changed equal-epoch assignments are deliberately accepted.

Within one live ClientCnx, wire order gives clients an ordering. Across controller reconnect or leader failover, I could not find a protocol-level fencing rule.

There is another ordering detail in the current broker path: registerConsumer can push an assignment update before ServerCnx writes the subscribe response, so the response does not appear to be an obvious synchronization barrier.

Questions

  1. Within one controller connection, should clients apply changed equal-epoch assignments strictly in receive order?
  2. Is CommandScalableTopicSubscribeResponse a baseline that precedes all pushed updates, or may a push received before the response represent newer state?
  3. After reconnect or controller leader failover, can a callback or frame from the previous connection race the new connection's initial assignment?
  4. Should clients locally fence all callbacks by connection incarnation?
  5. Is an assignment generation or controller/session term planned for the wire protocol?

Concrete example: assignments A and B both have layout_epoch = 7, but differ because a consumer joined. If A comes from the old connection and B from the new connection, is there any protocol-supported way to determine which is authoritative?

References: PIP-468, SubscriptionCoordinator.computeAssignment, ScalableConsumerClient.onAssignmentUpdate, and ServerCnx.handleCommandScalableTopicSubscribe on v5.0.0-M1 and current master.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status
    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions