TiCDC new arch: checkpoint advances past TSO but MySQL sink remains inconsistent after network loss

[Lily2025]

What did you do?

Ran the cdc_network_loss integration test on a TiCDC new-architecture cluster
with a MySQL-compatible downstream sink.

Reproduction steps:

1. Create changefeed cdc-network-loss-sync to the downstream MySQL sink.
2. Pause the changefeed, prepare 100 sysbench tables with 500,000 rows each,
   then resume the changefeed.
3. Run sysbench oltp_update_non_index with 100 threads for one hour.
4. During a two-hour period, inject NetworkChaos every 30 minutes:
   • randomly select one upstream TiCDC peer;
   • packet loss: 100%;
   • correlation: 25%;
   • duration: 10s.
5. After the workload and chaos period finish, get the current upstream TSO and
   wait until ticdc_owner_checkpoint_ts for the changefeed is greater than it.
6. Compare all upstream and downstream tables using CRC32.

Timeline (UTC):

• 16:10:17: resumed changefeed and started the one-hour workload.
• 16:40:17-27, 17:10:17-27, 17:40:17-27, 18:10:17-27:
  four 10-second network-loss injections.
• 17:10:18: workload finished.
• 18:10:28: captured the upstream TSO and started waiting for checkpoint.
• 18:10:58: checkpoint exceeded the target TSO; CRC comparison started.

Note: the current test creates NetworkChaos without an explicit target.
Because TiDB Cloud components use host networking, the framework warns that the
fault can affect the selected host's whole network scope rather than a single
TiCDC-to-peer link.

Internal execution: https://tcms.pingcap.net/dashboard/executions/plan/8226703

What did you expect to see?

After network connectivity recovers, TiCDC should reconnect and replicate all
transactions up to the checkpoint. Once the changefeed checkpoint exceeds the
captured upstream TSO, CRC32 results between upstream and downstream should be
identical.

What did you see instead?

The changefeed checkpoint exceeded the target TSO within 30 seconds, while the
workload had already stopped for one hour. However, CRC32 comparison remained
inconsistent for the full 90-minute retry window:

• 18:11:19: workload.sbtest33, source 1909784911, sink 4228885997
• 18:41:40: workload.sbtest90, source 3273849050, sink 1173175951
• 19:12:01: workload.sbtest90, same mismatch as above
• 19:42:22: workload.sbtest8, source 3724601131, sink 2572728669

The checker reports only the first mismatched table in each run; the table name
can vary because the comparison iterates over a Go map and stops at the first
mismatch. Therefore, the changing table name does not show that earlier tables
recovered.

The changefeed remained in normal state in the available logs. This indicates
that TiCDC's reported checkpoint can advance beyond the target TSO even though
the downstream data is not fully consistent.

Versions of the cluster

• TiCDC: v26.3.3-9-g6500294
• TiCDC git hash: 65002945d670fea61492e35545326a0a289853c5
• TiCDC build time: 2026-08-19 00:50:55 UTC
• TiCDC kernel type: Next Generation
• TiDBX test plan: v26.3.99-tidbx-release-20260820-hpmwh
• Downstream: MySQL-compatible sink provisioned by the TiDBX testbed

[Lily2025]

/severity critical
/assign hongyunyan

[Lily2025]

/type bug