What did you do?
- TiDB cluster with TiCDC redo log
consistent.level=eventual to S3 (minio), mysql sink.
- Changefeed filter
test_ticdc.*; workload writes continuously to 4 tables (t0..t3, ~15k rows/s, 4-byte payloads) via sustained concurrent writers; big_t0/big_t1 are static.
- Create the changefeed (or pause + resume it after a downtime so a large backlog accumulates) with
start_ts at current time, then observe.
What did you expect to see?
The changefeed should bootstrap, initialize the region subscriptions for all tables, reach Working on all dispatchers, advance the checkpoint, and sync DML to the downstream sink while writing redo logs.
What did you see instead?
The changefeed stays normal with error: null but is functionally stuck:
- Checkpoint frozen at the resume point (never advances).
- Downstream sink receives nothing (
ticdc_sink_* counters flat), no DML written.
- Redo writer initially writes a burst then also stops.
- 4 of 12 event-store subscriptions initialize (the static
big_t0/big_t1 tables); the 8 subscriptions for t0..t3 (each covering ~12–13 regions, 25 regions/table) stay Initialized:false for 12+ minutes:
span_registry.go:333 "subscription client initializes a region too slow" / :337 "subscription client finds a uninitialized slow region", e.g. RegionID:2265, Initialized:false, Created: 09:11:38.
- Add-dispatcher operators stay in the maintainer running queue:
operator_controller.go:406 "operator is still in running queue" ... timeSinceCreated=8m36s (grows unbounded).
- Sink / redo / event-service goroutines are all healthy in goroutine dumps (no deadlock, no blocked mutex) — the pipeline is simply idle because no dispatcher ever becomes
Working.
Key metrics (all from the stuck changefeed):
ticdc_event_service_scan_task_count 263
ticdc_event_service_interrupt_scan_count 256
ticdc_event_service_pending_scan_task_count 0
ticdc_event_service_scan_window_interval ~103.37 (s) # adaptive recovery interval kept growing 5s -> 103s
ticdc_event_service_scan_window_usage_ema 0.21
ticdc_event_service_scan_window_pressure_score 0
ticdc_event_store_subscription_num 12
# scan_window_base_ts frozen, no advance over 60s:
ticdc_event_service_scan_window_base_ts <frozen>
scan_window_base_ts sampled 60s apart was identical (frozen at a timestamp ~38 min behind current time), proving the recovery scan makes zero forward progress while scans are continuously interrupted.
Versions of the cluster
Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):
(paste TiDB cluster version here)
Upstream TiKV version (execute tikv-server --version):
(paste TiKV version here)
TiCDC version (execute cdc version):
What did you do?
consistent.level=eventualto S3 (minio), mysql sink.test_ticdc.*; workload writes continuously to 4 tables (t0..t3, ~15k rows/s, 4-byte payloads) via sustained concurrent writers;big_t0/big_t1are static.start_tsat current time, then observe.What did you expect to see?
The changefeed should bootstrap, initialize the region subscriptions for all tables, reach
Workingon all dispatchers, advance the checkpoint, and sync DML to the downstream sink while writing redo logs.What did you see instead?
The changefeed stays
normalwitherror: nullbut is functionally stuck:ticdc_sink_*counters flat), no DML written.big_t0/big_t1tables); the 8 subscriptions fort0..t3(each covering ~12–13 regions, 25 regions/table) stayInitialized:falsefor 12+ minutes:span_registry.go:333 "subscription client initializes a region too slow"/:337 "subscription client finds a uninitialized slow region", e.g.RegionID:2265, Initialized:false, Created: 09:11:38.operator_controller.go:406 "operator is still in running queue" ... timeSinceCreated=8m36s(grows unbounded).Working.Key metrics (all from the stuck changefeed):
scan_window_base_tssampled 60s apart was identical (frozen at a timestamp ~38 min behind current time), proving the recovery scan makes zero forward progress while scans are continuously interrupted.Versions of the cluster
Upstream TiDB cluster version (execute
SELECT tidb_version();in a MySQL client):(paste TiDB cluster version here)Upstream TiKV version (execute
tikv-server --version):(paste TiKV version here)TiCDC version (execute
cdc version):PR #5956