kafka: improve stability when creating many topics with Kafka v4

[3AceShowHand]

What problem does this PR solve?

Issue Number: close #6076

What is changed and how it works?

• Make GetTopicsMeta return topic-level errors, including UnknownTopicOrPartition, when ignoreTopicError is false.
• Preserve the existing discovery behavior when ignoreTopicError is true.
• Classify temporary Kafka topic, broker, network, and controller metadata errors as retryable. Configuration and authorization errors return immediately.
• Write the topic manager cache only after the topic is present in the returned metadata map.

Check List

Tests

• Unit test
  • GOTOOLCHAIN=go1.25.12 GOMAXPROCS=2 go test -p 1 -vet=off ./pkg/sink/kafka ./downstreamadapter/sink/topicmanager -count=1

Questions

Will it cause performance regression or break compatibility?

No public API, configuration, or Kafka protocol behavior changes. Metadata retries remain bounded by the existing six-attempt limit, and deterministic errors stop after the first attempt.

Do you need to update user documentation, design documentation or monitoring documentation?

No.

Release note

Improve the robustness of Kafka topic metadata visibility handling during large-scale topic creation.

Summary by CodeRabbit

• Bug Fixes
  • Improved Kafka topic creation reliability by retrying temporary visibility and metadata errors.
  • Newly created topics are cached only after Kafka confirms they are visible.
  • Missing topics, delayed broker responses, and transient controller or network issues are handled more consistently.
  • Unknown-topic errors are no longer silently ignored unless configured to be ignored.
  • Configuration, authorization, and other non-retryable errors now stop promptly and report the failure.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6993d2de-e0a6-44c8-b4a0-2799bfd5b214

📥 Commits

Reviewing files that changed from the base of the PR and between 4d3bab1 and f6a8200.

📒 Files selected for processing (4)

• downstreamadapter/sink/topicmanager/kafka_topic_manager.go
• downstreamadapter/sink/topicmanager/kafka_topic_manager_test.go
• pkg/sink/kafka/admin.go
• pkg/sink/kafka/sarama_admin_test.go

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

GetTopicsMeta now applies its regular error policy to unknown-topic metadata errors and classifies unretryable Kafka metadata errors. Topic creation retries other metadata errors and caches partition metadata only after visibility succeeds.

Changes

Kafka topic management

Layer / File(s)	Summary
Apply Kafka metadata error policies pkg/sink/kafka/admin.go, pkg/sink/kafka/sarama_admin_test.go	Unknown-topic errors now follow ignoreTopicError, authorization checks, and error wrapping. IsUnretryableTopicMetadataError classifies authorization, configuration, SASL, version, request, and related errors. Tests cover ignored, propagated, retryable, and unretryable errors.
Wait for visible topic metadata downstreamadapter/sink/topicmanager/kafka_topic_manager.go	Topic visibility checks retry metadata errors that are not classified as unretryable. The manager updates the topic cache with the created partition count only after visibility succeeds.
Validate retries and cache timing downstreamadapter/sink/topicmanager/kafka_topic_manager_test.go	Tests cover unknown-topic responses, post-create retries, cache timing, non-retryable errors, and topic metadata construction.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to f6a82

This change improves Kafka topic metadata error handling and retry behavior without identified merge-blocking risk; it is merge-ready after normal checks and review.

Sequence Diagram(s)

sequenceDiagram
  participant TopicManager
  participant KafkaAdmin
  participant TopicCache
  TopicManager->>KafkaAdmin: Create topic
  loop Until metadata is visible
    TopicManager->>KafkaAdmin: Get topic metadata
    KafkaAdmin-->>TopicManager: Metadata or retryable error
  end
  TopicManager->>TopicCache: Cache partition count

Loading

Suggested reviewers: lidezhu, wk989898

Poem

A rabbit checks each Kafka call,

Unknown topics follow the error path.
Retry metadata when permitted,
Cache partitions when visible.
The topic appears, and tests agree.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 18.18% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 11 functions across 4 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: improved Kafka topic creation stability under Kafka v4 and large-scale topic creation.
Description check	✅ Passed	The description includes the required issue reference, change summary, test details, compatibility and documentation answers, and a release note.
Linked Issues check	✅ Passed	The changes address issue #6076 by retrying temporary Kafka metadata failures, stopping promptly on configuration or authorization errors, returning sink errors, and updating the topic cache only afte…
Out of Scope Changes check	✅ Passed	The code and test changes are limited to Kafka topic metadata handling, retry classification, topic visibility, and cache timing. These changes directly support the linked issue objectives.

Full details: Linked Issues check

Explanation

The changes address issue #6076 by retrying temporary Kafka metadata failures, stopping promptly on configuration or authorization errors, returning sink errors, and updating the topic cache only after metadata confirms topic visibility.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[3AceShowHand]

/test all

[Copilot]

Pull request overview

This PR improves Kafka topic-creation robustness in TiCDC’s Kafka sink by treating newly-created topics as potentially not yet visible in metadata, retrying transient Kafka metadata errors, and only caching a topic after Kafka confirms it is visible.

Changes:

• Update GetTopicsMeta to return topic-level errors (including UnknownTopicOrPartition) when topic errors are not ignored, while preserving the previous “best-effort discovery” behavior when errors are ignored.
• Introduce IsRetryableTopicMetadataError to classify transient Kafka metadata errors as retryable and use it to drive bounded retries when waiting for topic visibility.
• Ensure the topic manager’s cache is only updated after the topic becomes visible (and add/adjust unit tests to validate caching + retry behavior).

Validation:

• Not run here (code review only; no CI execution available in this environment).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
pkg/sink/kafka/admin.go	Makes topic metadata error handling stricter when not ignoring topic errors; adds retryable-metadata error classifier.
pkg/sink/kafka/sarama_admin_test.go	Adds unit coverage for the stricter unknown-topic error behavior and for retryable metadata error classification.
downstreamadapter/sink/topicmanager/kafka_topic_manager.go	Adds retryable-error classification to topic visibility wait; caches topic only after visibility.
downstreamadapter/sink/topicmanager/kafka_topic_manager_test.go	Adds tests ensuring cache isn’t written before visibility and retry behavior honors cancellation / stops on non-retryable errors.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[3AceShowHand]

/hold

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lidezhu, wk989898

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [lidezhu,wk989898]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-08-25 12:47:34.790766637 +0000 UTC m=+592289.961860747: ☑️ agreed by lidezhu.
• 2026-08-26 03:13:52.941972575 +0000 UTC m=+644268.113066716: ☑️ agreed by wk989898.