unstable integration test ddl_for_split_tables_with_random_merge_and_split

[wk989898]

What did you do?

https://prow.tidb.net/jenkins/job/pingcap/job/ticdc/job/pull_cdc_mysql_integration_heavy_next_gen_legacy_safepoint/186/stages/

What did you expect to see?

No response

What did you see instead?

Data inconsistencies

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

master(next-gen)

[wk989898]

Timeline

Time	Event
00:37:24.528	The maintainer creates a split operator for redo dispatcher 159454700166960836582477485010614754944. Table 19 is split into five spans.
00:37:24.563	The split operator sends a remove request for the old dispatcher with mode=1.
00:37:25.754	The old redo dispatcher reports that it has stopped and is ready for cleanup. Its final checkpoint and resolved-ts are both 468537898321576064.
00:37:26.074	Another terminal status for the same old dispatcher is produced during cleanup.
00:37:26.083	The split operator processes the terminal status and marks the removal phase complete. SplitDispatcherOperator.PostFinish begins.
00:37:26.087	ReplaceReplicaSet removes the old dispatcher from the scheduler and creates the five replacement spans.
00:37:26.097	Split post-processing finishes. The split operator is marked finished and removed from the operator controller.
00:37:26.098	The concurrent status handler continues processing the old Stopped status. It still holds the old SpanReplication, but now finds no operator for the dispatcher.
00:37:26.098	The status handler calls MarkSpanAbsent for the obsolete dispatcher. The scheduler logs scheduler: marking replica absent.
00:37:26.576	The scheduler tries to create an Add operator for the ghost span. Admission fails because the span controller no longer contains the dispatcher: add operator failed, span not found.
Every approximately 500 ms afterward	The scheduler retries the same Add operation and receives the same span not found failure. The retries continue until the test ends.

Conclusion

The replication failure is caused by a race between terminal dispatcher
status handling and split-operator finalization in the maintainer. The race
leaves an obsolete dispatcher in the scheduler's Absent set even though the
dispatcher has already been removed from the span controller.

This creates a ghost Absent span:

• the scheduler believes that the old dispatcher must be recreated;
• the span controller no longer contains that dispatcher;
• every scheduling attempt fails with add operator failed, span not found;
• the ghost span's old checkpoint remains in the non-replicating checkpoint
  tracker and prevents the global checkpoint from advancing.

The affected dispatcher in this run was a redo dispatcher (mode=1). However,
the root cause is not redo-specific. Normal and redo dispatchers use the same
status-handling, split-operator, and span-controller lifecycle logic. The same
interleaving can therefore create a ghost span for a normal dispatcher
(mode=0).

Redo makes the impact broader: when the redo watermark stalls, normal sink
dispatchers cache events whose commit timestamps are greater than the redo
global resolved-ts. A ghost redo span can therefore stop both redo progress
and normal downstream replication.