maintainer: balance scheduler spins on rejected quiescing‑move operations and generates verbose logs at 34 MB/s

[asddongmen]

Risk: Excessive log generation may cause disk full.
Workaround: Restart the problematic CDC server.

What did you do?

Run a TiCDC new-architecture failover/scale test with 3 CDC captures, 4
MySQL-sink changefeeds, and continuous DML. During a maintainer handoff, scale
the CDC cluster out so a new empty capture joins while the old maintainer is
being removed.

The failure was observed with this sequence:

1. A new capture (97382844-bec6-4a89-bef1-7187d5e84578) joined the cluster.
2. The coordinator started moving default/p0-scale-04 from capture
   63658e62-2196-4bb1-b580-a121c869d1f8 to the new capture.
3. The old maintainer received the remove request, entered removing mode, and
   its operator controller became quiescing.
4. The old maintainer's still-running balance scheduler selected a dispatcher
   move to the new capture.

This was reproduced during the second scale-out of a 55-minute chaos run. The
test included ordinary non-race binaries only.

What did you expect to see?

The old maintainer should finish its handoff without scheduling ordinary move
operators. If an operator admission attempt is rejected, the balance round
must make progress or return; it must not retry the same unchanged candidate
in a tight loop.

What did you see instead?

The following log sequence shows the new capture, maintainer handoff, and the
first rejected dispatcher move (timestamps are CST):

[2026/08/21 17:46:00.628 +08:00] [INFO] [operator_controller.go:537] ["add operator to running queue"] [role=coordinator] [operator="move maintainer operator: default/p0-scale-04, origin:63658e62-2196-4bb1-b580-a121c869d1f8, dest:97382844-bec6-4a89-bef1-7187d5e84578"]
[2026/08/21 17:46:00.652 +08:00] [INFO] [maintainer_manager_maintainers.go:355] ["received remove maintainer request"] [changefeedID=default/p0-scale-04]
[2026/08/21 17:46:01.252 +08:00] [INFO] [maintainer.go:585] ["changefeed maintainer closed"] [changefeedID=default/p0-scale-04] [checkpointTs=468531428390338831] [removed=true]
[2026/08/21 17:46:01.269 +08:00] [INFO] [operator_controller.go:237] ["add operator failed, controller is quiescing"] [role=maintainer] [changefeedID=default/p0-scale-04] [dispatcherID=155117822891070323075970390249676308648] [operator="move dispatcher operator: 155117822891070323075970390249676308648, origin:63658e62-2196-4bb1-b580-a121c869d1f8, dest:97382844-bec6-4a89-bef1-7187d5e84578"]

The final line was then emitted continuously for about 21 minutes, until CDC
8300 was restarted. The last archived occurrence was at
2026/08/21 18:07:22.193 +08:00.

During that interval CDC 8300 produced 166 rotated log files totaling 48.6
GiB. A representative 300 MiB file contained 800,467 lines and covered only
8.738 seconds (17:46:05.540 through 17:46:14.278), approximately 91,600
lines/s and 34.3 MiB/s. The process did not recover after workload stop,
changefeed catch-up, or checksum; restarting CDC 8300 stopped the loop.

There was no observed data-plane corruption: all 4 changefeeds caught up,
there was no checkpoint regression or duplicate active maintainer metric, and
checksum passed for all 64 tables. The impact was a control-plane livelock plus
high CPU, log, and disk consumption.

Root cause analysis

This is a non-progress loop spanning the maintainer lifecycle and generic
balance helper:

1. onRemoveMaintainer
   sets removing=true and calls EnterRemovingMode.
2. EnterRemovingMode
   quiesces operator admission, but it does not stop the ordinary scheduler task
   handles. Those handles are canceled only by
   Controller.Stop.
3. The balance scheduler's
   doMove
   forwards the boolean returned by AddOperator.
4. AddOperator
   correctly rejects the move while quiescing, logs at INFO, and returns false.
   The move operator is never admitted.
5. In pkg/scheduler.Balance,
   target load, victim load, victim task list, and movedSize are updated only
   when move(...) returns true. On false, the exact unchanged target and
   victim are reinserted into their heaps and the unbounded for loop
   immediately selects the same pair again.

Cancellation cannot stop this loop after Execute has entered it:
TaskHandle.Cancel
only sets the cancellation state, and the thread pool checks it only before
calling
task.Execute().
It cannot preempt an Execute call already spinning inside Balance.

The direct liveness defect is therefore the missing false-return progress/exit
rule in Balance. Removing mode leaving normal schedulers runnable makes this
path reachable, and the unthrottled INFO message amplifies it into a log storm.

Related: #5403 introduced the quiescing admission gate and protects against
unsafe control-plane work during handoff, but the scheduler/rejection
interaction above remains unbounded.

Suggested fix and regression coverage

• Make Balance bounded when the move callback returns false (for example,
  return from the current round or track attempted candidates so an unchanged
  pair cannot be selected forever).
• Stop ordinary scheduler task handles when a maintainer enters removing mode,
  while preserving only the DDL-trigger close path required by the handoff.
• Rate-limit or downgrade repeated quiescing rejection logs as defense in
  depth.
• Add a unit test where an always-false move callback must return promptly with
  zero moves.
• Add a lifecycle test for removing/quiescing maintainer plus node scale-out;
  it must neither admit ordinary moves nor leave a spinning scheduler task.

Versions of the cluster

Upstream TiDB:

Release Version: v8.5.5
Git Commit Hash: 1fa258b833ff113883beeba40bc130be7ce66610

Upstream TiKV:

Release Version: 8.5.5
Git Commit Hash: a0234aad114258b60e0ec23c17486c79845299be

TiCDC:

Release Version: v8.5.4-release
Git Commit Hash: 799216ba3cddd4611b48d4790f1554193349f858
Source branch: upstream/release-8.5
Failpoint Build: false
Kernel Type: Classic

[asddongmen]

Trigger conditions and probability estimate

This is not triggered by every scale-out. It requires an ordering race
between two by-design schedulers:

• the coordinator balances changefeed maintainers across CDC nodes; and
• each maintainer balances its dispatcher spans across CDC nodes.

For example, with 4 changefeeds and a 2-to-3 CDC scale-out, the maintainer
distribution changes from 2/2 to 2/2/0. The coordinator is expected to move
one maintainer to obtain 2/1/1. The bug occurs only if the selected old
maintainer enters quiescing while it still has an ordinary dispatcher-balance
round that can run against the new empty node.

Scenario / ordering	Maintainer handoff	Dispatcher imbalance still present when the old scheduler runs	Result
Add a node, but coordinator balance is currently blocked by an in-flight operator or absent changefeed	No (yet)	Irrelevant	Does not trigger
Scale out; dispatchers finish rebalancing before the selected maintainer starts handoff	Yes, later	No	Does not trigger
Move a maintainer while the node set is already balanced for that changefeed	Yes	No	Does not trigger
Scale out; the same changefeed starts maintainer handoff before its next dispatcher-balance tick	Yes	Yes	Can reach the bug
pkg/scheduler.Balance calls move, and AddOperator returns false because the controller is quiescing	Yes	Yes	Deterministic tight loop until process termination

Concrete examples from this run

Scale-out 1 did not trigger:

• 17:25:59.875: the new capture became visible to p0-scale-02.
• 17:26:01.823 to 17:26:03.904: its dispatcher moves were admitted and
  completed.
• 17:26:14.390: only then did the coordinator start moving that changefeed's
  maintainer.

The dispatcher imbalance had already been removed before quiescing, so the old
maintainer had no ordinary move to reject.

Scale-out 2 triggered:

• 17:46:00.480: the new capture became visible to p0-scale-04.
• 17:46:00.628: the coordinator started the maintainer move.
• 17:46:00.652: the old maintainer received remove and entered quiescing.
• 17:46:01.252: the old maintainer reported closed.
• 17:46:01.269: its dispatcher balance attempted a move and AddOperator
  returned false; the unchanged balance candidate was then retried forever.

Simple probability calculation

There are two different probabilities:

1. After the rejecting callback has been reached: effectively 100% on
   the tested commit. The false branch does not change the victim, target,
   task list, or moved count, and has no break/backoff.
2. Probability that one eligible handoff enters that callback: not measured
   reliably yet. We observed 1 incident in 2 scale-outs, but 1/2 is not a
   defensible production rate because the samples are not independent and had
   different ordering, as shown above.

A timing-only estimate can show the order of magnitude. In the triggering run,
the remove/close handoff lasted about 0.6 s, while the maintainer balance task
runs on an approximately 5 s cadence. If the task phase were uniformly random
and independent:

P(tick overlaps the vulnerable handoff) ~= 0.6 / 5 = 12%

This 12% is only a rough conditional estimate for an eligible handoff, not
a measured probability for every scale-out. The real per-scale-out probability
is lower when dispatcher rebalance has already completed or scheduling is
blocked, and scheduler phases may be correlated rather than random.

For repeated independent eligible handoffs, the same illustrative model gives:

5 handoffs:  1 - (1 - 0.12)^5  ~= 47%
10 handoffs: 1 - (1 - 0.12)^10 ~= 72%

This explains why the issue may look intermittent in a single operational
scale-out but becomes much easier to hit in repeated scale-in/scale-out and
failover chaos. DML throughput and MySQL sink load are not direct trigger
conditions; they mainly affect how much work is in flight.

Relevant code:

• Coordinator maintainer balance:
  

  ticdc/coordinator/scheduler/balance.go

  Lines 91 to 112 in 799216b

  	if s.operatorController.OperatorSize() > 0 || s.changefeedDB.GetAbsentSize() > 0 {
  	// not in stable schedule state, skip balance
  	return now.Add(s.checkBalanceInterval)
  	}
  	// check the balance status
  	activeNodes := s.nodeManager.GetAliveNodes()
  	activeNodes = filterSchedulableAliveNodes(activeNodes, s.liveness)
  	moveSize := pkgScheduler.CheckBalanceStatus(s.changefeedDB.GetTaskSizePerNode(), activeNodes)
  	if moveSize <= 0 {
  	// fast check the balance status, no need to do the balance,skip
  	return now.Add(s.checkBalanceInterval)
  	}
  	// balance changefeeds among the active nodes
  	movedSize := pkgScheduler.Balance(s.batchSize, s.random, activeNodes, s.changefeedDB.GetReplicating(),
  	func(cf *changefeed.Changefeed, nodeID node.ID) bool {
  	return s.operatorController.AddOperator(operator.NewMoveMaintainerOperator(s.changefeedDB, cf, cf.GetNodeID(), nodeID))
  	})
  	s.forceBalance = movedSize >= s.batchSize
  	s.lastRebalanceTime = time.Now()
  	return now.Add(s.checkBalanceInterval)

• Maintainer dispatcher balance:
  

  ticdc/maintainer/scheduler/balance.go

  Lines 91 to 112 in 799216b

  	// TODO: consider to ignore split tables' dispatcher basic schedule operator to decide whether we can make balance schedule
  	if s.operatorController.OperatorSize() > 0 || s.spanController.GetAbsentSize() > 0 {
  	// not in stable schedule state, skip balance
  	return time.Now().Add(time.Second * 5)
  	}
  	// 1. check whether we have spans in defaultGroupID need to be splitted.
  	// we only consider the not splitted span here.
  	checkResults := s.spanController.CheckByGroup(pkgReplica.DefaultGroupID, s.batchSize)
  	count := s.doSplit(checkResults)
  	// to many split operators, do move operator later
  	if count >= s.batchSize {
  	return time.Now().Add(time.Second * 5)
  	}
  	moveBudget := s.batchSize - count
  	// 2. do balance for the spans in defaultGroupID
  	s.schedulerDefaultGroup(moveBudget, state)
  	return time.Now().Add(time.Second * 5)

• Non-progress false branch:
  

  ticdc/pkg/scheduler/balance.go

  Lines 118 to 141 in 799216b

  	movedSize = 0
  	for {
  	target, _ := minPriorityQueue.PeekTop()
  	if target.Load >= lowerLimitPerCapture {
  	// the minimum workload has reached the lower limit
  	break
  	}
  	victim, _ := maxPriorityQueue.PeekTop()
  	task := victim.Tasks[0]
  	if move(task, target.Node) {
  	// update the task size priority queue
  	target.Load++
  	victim.Load--
  	clear(victim.Tasks[:1])
  	victim.Tasks = victim.Tasks[1:]
  	movedSize++
  	if movedSize >= batchSize || movedSize >= totalMoveSize {
  	break
  	}
  	}
  	minPriorityQueue.AddOrUpdate(target)
  	maxPriorityQueue.AddOrUpdate(victim)
  	}