Kafka changefeed checkpoint lag exceeds threshold during concurrent workloads

[Lily2025]

What did you do?

Ran a TiCDC next-generation Kafka/Avro changefeed with two concurrent upstream workloads.

Changefeed configuration:

• Kafka sink with Avro protocol and Schema Registry
• partition-num=2
• replication-factor=2
• avro-enable-watermark=true
• dispatcher topic template: tidb_{schema}_{table}
• integrity check level: correctness

The workloads were:

• ycsb_bigtable, one table, TPS 10000, 32 threads
• go-ycsb betting workload, 2000 tables, 32 threads

The test first captured an upstream TSO and confirmed that the changefeed checkpoint had advanced beyond it. It then continuously checked that checkpoint lag remained below 120 seconds, allowing at most 3 minutes of continuous violation.

All timestamps below are Asia/Shanghai (UTC+8).

What did you expect to see?

The Kafka changefeed should remain healthy under the workload and keep checkpoint lag below the configured threshold.

If Kafka temporarily cannot serve a dynamically-dispatched topic or partition, the changefeed should either recover after the Kafka metadata/topic becomes available, or transition promptly to a clearly observable failed state with the sink error exposed to the test/operator.

What did you see instead?

The changefeed repeatedly entered retry because Kafka message delivery failed.

• 2026-08-24 10:39:47: checkpoint-message delivery first failed with kafka: Failed to deliver messages.
• 2026-08-24 10:46:11: TiCDC reported CDC:ErrKafkaAsyncSendMessage; Kafka returned Request was for a topic or partition that does not exist on this broker.
• 2026-08-24 10:48:59: the test captured the upstream TSO.
• 2026-08-24 10:50:14: the checkpoint successfully advanced beyond that captured TSO.
• 2026-08-24 10:55:01 onward: Kafka unknown-topic/partition errors recurred. The changefeed checkpoint remained at TSO 468592865798783218 (physical time 10:52:03).
• 2026-08-24 10:58:49: another TiCDC node reported the same Kafka error. During dispatcher recovery, TiCDC logs showed a checkpoint/resolved-ts gap of about 6 minutes 40 seconds.
• 2026-08-24 10:59:33: the changefeed again entered backoff; retries were scheduled beyond the lag-check window.
• 2026-08-24 11:00:21: checkpoint-lag checking began. The lag stayed above 120 seconds for more than the 3-minute tolerance, causing the assertion to fail.

The test framework surfaced only a Ginkgo panic from the lag assertion, rather than the sampled lag value and the sink error:

step occurred panic, step: CDC checkpoint lag check,
threshold: 120, duration: 1h0m0s, toleration duration: 3m0s

Suspected: the Kafka sink error and retry behavior coincided with the checkpoint stall. The available artifacts do not establish why Kafka considered the topic or partition absent, so this issue does not attribute that underlying Kafka condition to TiCDC.

Versions of the cluster

• TiCDC: v26.3.3-9-g6500294
• TiCDC commit: 65002945d670fea61492e35545326a0a289853c5
• TiCDC build time: 2026-08-19 00:50:55 UTC
• Architecture: TiCDC Next Generation
• Sink: Kafka with Avro protocol and Schema Registry

[3AceShowHand]

This is a scale-sensitive topic visibility problem.

Failure scenario

The changefeed uses tidb_{schema}_{table} as its dispatcher topic template. Each table maps to an independent Kafka topic.

The workload contains thousands of tables. During changefeed initialization and recovery, TiCDC creates or queries a large number of topics within a short period.

Kafka topic creation has a metadata visibility window. Kafka documents that after CreateTopics reports success, other brokers may need several seconds to learn about the new
topic. During this interval, DescribeTopics may return no metadata for the topic. See the [Kafka Admin API documentation](https://kafka.apache.org/39/javadoc/org/apache/kafka/
clients/admin/Admin.html#createTopics(java.util.Collection,org.apache.kafka.clients.admin.CreateTopicsOptions)).

The failure follows this sequence:

1. TiCDC creates a dynamically dispatched topic.
2. Kafka accepts the creation request.
3. TiCDC immediately queries the topic metadata.
4. Kafka temporarily returns UnknownTopicOrPartition.
5. GetTopicsMeta suppresses the error even when the caller sets ignoreTopicError=false.
6. GetTopicsMeta returns an empty metadata map with a nil error.
7. The affected waitUntilTopicVisible implementation reads the missing map entry and gets the Go zero value, producing partitionNumber=0.
8. TiCDC treats the topic as visible and starts sending messages.
9. The producer receives UnknownTopicOrPartition, and the sink exits.
10. Repeated changefeed recovery and backoff prevent checkpoint progress until the lag check fails.

The logs contain 971 topic found records. Among them, 598 report partitionNumber=0. This value comes from a missing metadata map entry. The configured topic partition count
remains 2.

Impact of topic scale

Every dynamically created topic introduces a create-and-describe visibility window. With thousands of topics, TiCDC performs this sequence thousands of times. The probability that
at least one topic encounters the visibility window increases with the topic count.

A single affected topic can cause the async producer to return an error and terminate the sink. Changefeed recovery rebuilds many dispatchers and repeats the metadata operations,
creating a feedback loop:

1. A topic encounters the visibility window.
2. The sink exits.
3. The changefeed enters backoff.
4. Recovery rebuilds dispatchers and queries many topics again.
5. Another topic can encounter the same window.

This explains why the checkpoint initially advanced and later remained stalled while resolved-ts continued moving forward.

Why the issue became easier to reproduce

The test had previously succeeded with Bitnami Kafka 3.1. Test-infra later changed the deployment to Apache Kafka 3.9.0.

Kafka already permits the temporary metadata visibility condition described above. The Kafka 3.9.0 deployment changed the timing and distribution of topic metadata propagation
observed by this workload. Combined with thousands of dynamically created topics, this increased the probability of reaching the vulnerable TiCDC code path.

The TiCDC behavior converts a recoverable topic visibility condition into a sink failure, repeated changefeed backoff, and checkpoint stagnation.