server, dispatcher: improve node liveness self fence (#5106)

[ti-chi-bot]

This is an automated cherry-pick of #5106

What problem does this PR solve?

Issue Number: close #5202

When a TiCDC capture loses its etcd session or lease, it can no longer prove that it still owns local dispatcher work. The previous shutdown path did not immediately fence local write paths, so a stale capture could still accept maintainer requests or continue writing downstream while failover was already in progress. This created a short but unsafe window for duplicate or out-of-date downstream writes before normal cleanup finished.

What is changed and how it works?

This PR adds a local fence path for session-done and lease-expired events. The server watches the etcd session, triggers local fencing before shutdown, and the dispatcher orchestrator stops accepting new maintainer requests. Dispatcher managers cancel local write paths immediately, close local dispatchers asynchronously, and continue cleanup without waiting for progress draining. The redo cleanup path is also guarded so partially initialized managers do not panic when fencing. New integration coverage simulates capture session loss and verifies the local fence behavior and downstream consistency.

Check List

Tests

• Unit test
• Integration test
• Manual test: dev-machine MySQL integration cases and GitHub /test all

Questions

Will it cause performance regression or break compatibility?

No. The new path only runs when a local capture loses its session or is being fenced.

Do you need to update user documentation, design documentation or monitoring documentation?

No.

Release note

Locally fences stale captures after etcd session loss to avoid unsafe downstream writes.

Summary by CodeRabbit

• New Features

  • Added server-side etcd session watchdog and explicit local-fence controls to quickly close local write paths across orchestrator and managers.

• Bug Fixes

  • Prevented races during dispatcher/redo creation, merge and shutdown; avoid registering partially-initialized dispatchers and short-circuit operations when the write path is fenced.
  • Improved close lifecycle to ensure single close launch and reliable finalization.

• Tests

  • Added unit and integration tests covering fencing, shutdown, and session-watchdog scenarios.

[ti-chi-bot]

@asddongmen This PR has conflicts, I have hold it.
Please resolve them or ask others to resolve them, then comment /unhold to remove the hold label.

[ti-chi-bot]

@ti-chi-bot: ## If you want to know how to resolve it, please read the guide in TiDB Dev Guide.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 690a56cb-327e-479a-a720-0b4454969670

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[ti-chi-bot]

Cherry-pick conflicts appear resolved; removing the do-not-merge/hold label.

[ti-chi-bot]

Cherry-pick conflicts appear resolved; removing the do-not-merge/hold label.

[asddongmen]

/unhold

[asddongmen]

/test all

[asddongmen]

/retest

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: asddongmen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [asddongmen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[asddongmen]

/test pull-cdc-mysql-integration-light