Skip to content

TiCDC changefeed stalls during recovery scan: scan_window base_ts freezes, regions never initialize #6020

Description

@wk989898

What did you do?

  1. TiDB cluster with TiCDC redo log consistent.level=eventual to S3 (minio), mysql sink.
  2. Changefeed filter test_ticdc.*; workload writes continuously to 4 tables (t0..t3, ~15k rows/s, 4-byte payloads) via sustained concurrent writers; big_t0/big_t1 are static.
  3. Create the changefeed (or pause + resume it after a downtime so a large backlog accumulates) with start_ts at current time, then observe.

What did you expect to see?

The changefeed should bootstrap, initialize the region subscriptions for all tables, reach Working on all dispatchers, advance the checkpoint, and sync DML to the downstream sink while writing redo logs.

What did you see instead?

The changefeed stays normal with error: null but is functionally stuck:

  • Checkpoint frozen at the resume point (never advances).
  • Downstream sink receives nothing (ticdc_sink_* counters flat), no DML written.
  • Redo writer initially writes a burst then also stops.
  • 4 of 12 event-store subscriptions initialize (the static big_t0/big_t1 tables); the 8 subscriptions for t0..t3 (each covering ~12–13 regions, 25 regions/table) stay Initialized:false for 12+ minutes:
    • span_registry.go:333 "subscription client initializes a region too slow" / :337 "subscription client finds a uninitialized slow region", e.g. RegionID:2265, Initialized:false, Created: 09:11:38.
  • Add-dispatcher operators stay in the maintainer running queue:
    • operator_controller.go:406 "operator is still in running queue" ... timeSinceCreated=8m36s (grows unbounded).
  • Sink / redo / event-service goroutines are all healthy in goroutine dumps (no deadlock, no blocked mutex) — the pipeline is simply idle because no dispatcher ever becomes Working.

Key metrics (all from the stuck changefeed):

ticdc_event_service_scan_task_count               263
ticdc_event_service_interrupt_scan_count          256
ticdc_event_service_pending_scan_task_count       0
ticdc_event_service_scan_window_interval          ~103.37 (s)   # adaptive recovery interval kept growing 5s -> 103s
ticdc_event_service_scan_window_usage_ema         0.21
ticdc_event_service_scan_window_pressure_score    0
ticdc_event_store_subscription_num                12
# scan_window_base_ts frozen, no advance over 60s:
ticdc_event_service_scan_window_base_ts           <frozen>

scan_window_base_ts sampled 60s apart was identical (frozen at a timestamp ~38 min behind current time), proving the recovery scan makes zero forward progress while scans are continuously interrupted.

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

PR #5956

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions