Skip to content

coordinator: resent RunningError advances backoff multiple times during one restart #6082

Description

@3AceShowHand

What did you do?

Run a changefeed and trigger a retryable error in a DispatcherManager.

After receiving its first error, DispatcherManager sends the corresponding RunningError immediately and resends the same message every five seconds until the dispatcher manager is closed:

// resend message until the event dispatcher manager is closed
ticker := time.NewTicker(time.Second * 5)

These repeated reports can reach the coordinator while the same changefeed restart is still in progress.

What did you expect to see?

Repeated delivery of the same RunningError during one restart attempt should advance the backoff once.

A later failed restart attempt may advance the next backoff step.

What did you see instead?

Each resent RunningError reaches Backoff.CheckStatus.

When isRestarting is already true, CheckStatus logs that the error is ignored, then continues into HandleError. Each call to HandleError invokes NextBackOff and updates nextRetryTime.

One observed sequence was:

10:59:33.189  nextRetryTime = 11:02:14
10:59:33.190  already in restarting progress
10:59:33.190  nextRetryTime = 11:05:18
10:59:33.190  already in restarting progress
10:59:33.190  nextRetryTime = 11:08:45

All three reports contained the same error timestamp, source node, error code, and message.

A single failed restart attempt can therefore consume multiple exponential backoff steps, delaying recovery and potentially exhausting the configured backoff duration prematurely.

Versions of the cluster

Upstream TiDB cluster version:

N/A — this behavior is independent of the upstream TiDB version.

Upstream TiKV version:

N/A — this behavior is independent of the upstream TiKV version.

TiCDC version:

v26.3.4
Git commit: 65002945d670fea61492e35545326a0a289853c5

Metadata

Metadata

Assignees

No one assigned

    Labels

    type/enhancementThe issue or PR belongs to an enhancement.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions