api/etcdserverpb/rpc.proto: add leader_id response header, populate defragment API response header by gyuho · Pull Request #22327 · etcd-io/etcd

gyuho · 2026-08-24T15:58:30Z

Adds leader_id (field 5) to the v3 ResponseHeader and fills it at every server-side header site from the already-wired RaftStatusGetter.Leader(). The field is optional: 0 means "no hint," and it is not a fencing token. Old clients ignore it; new clients treat 0 from old servers as "unknown leader." Cost is at most ~11 bytes per response for a value the server already holds in memory.

Today a client must poll Status to learn the leader; this field puts that leader member ID in every Put/Get/Watch response, which is what Phase 2's opt-in leader-aware balancer builds on.

A new common-suite test (TestResponseHeaderLeaderId, integration and e2e) verifies leader_id is nonzero once a leader is elected, matches each member's status.leader, agrees across members, and appears on Put and Get, with mixed-version clusters checked per current-version endpoint.

c.f., #22268

kubernetes-prow · 2026-08-24T15:58:55Z

Hi @gyuho. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

fuweid · 2026-08-24T20:37:57Z

/ok-to-test

fuweid · 2026-08-24T20:38:45Z

The commit requires author to signoff by git commit -s

codecov · 2026-08-24T21:07:33Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.03%. Comparing base (fd8581e) to head (9b60402).
⚠️ Report is 9 commits behind head on main.

Additional details and impacted files

Files with missing lines	Coverage Δ
etcdctl/ctlv3/command/printer_json.go	`83.18% <100.00%> (+5.81%)`	⬆️
server/etcdserver/apply/backend.go	`88.19% <100.00%> (+2.16%)`	⬆️

... and 443 files with indirect coverage changes

@@            Coverage Diff             @@
##             main   #22327      +/-   ##
==========================================
+ Coverage   69.69%   73.03%   +3.34%     
==========================================
  Files         448      448              
  Lines       38074    31585    -6489     
==========================================
- Hits        26534    23067    -3467     
+ Misses      10117     8515    -1602     
+ Partials     1423        3    -1420

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fd8581e...9b60402. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ahrtr · 2026-09-08T14:51:05Z

I think the change is useful by itself. One example in etcd-operator, we need to know the leaderId so that we can move leader before we recreate a POD.

For completeness, let's also populate the header for DefragmentResponse?

etcd/server/etcdserver/api/v3rpc/maintenance.go

Lines 119 to 130 in fd8581e

    
           func (ms *maintenanceServer) Defragment(ctx context.Context, sr *pb.DefragmentRequest) (*pb.DefragmentResponse, error) { 
        
           	ms.lg.Info("starting defragment") 
        
           	ms.healthNotifier.defragStarted() 
        
           	defer ms.healthNotifier.defragFinished() 
        
           	err := ms.defrag.Defragment() 
        
           	if err != nil { 
        
           		ms.lg.Warn("failed to defragment", zap.Error(err)) 
        
           		return nil, togRPCError(err) 
        
           	} 
        
           	ms.lg.Info("finished defragment") 
        
           	return &pb.DefragmentResponse{}, nil 
        
           }

Defragment was the only maintenance RPC that returned an empty response header, so a client defragmenting through the v3 API got no member_id, raft_term, or leader_id back. Fill the header after the defrag returns: a defrag can run long enough for leadership to move, and callers (e.g. etcd-operator moving leadership before recreating a pod) want the current leader, not the one from when the call started. ref. etcd-io#22327 Signed-off-by: Gyuho Lee <gyuhol@nvidia.com>

Add leader_id (field 5) to the ResponseHeader proto and populate it at every server-side fill site from RaftStatusGetter.Leader(). That interface is already wired at each site. The field is advisory: 0 means the responding member does not know who the leader is. It is not a fencing token and is not atomic with raft_term. Old clients ignore the field; new clients treat 0 as "no hint." Covers all v3 RPC responses through fillWithoutRevision / fill, newResponseHeader (watch), ClusterServer.header, EtcdServer.newHeader, and applierV3backend.newHeader. ref. etcd-io#22268 Signed-off-by: Gyuho Lee <gyuhol@nvidia.com>

Defragment was the only maintenance RPC that returned an empty response header, so a client defragmenting through the v3 API got no member_id, raft_term, or leader_id back. Fill the header after the defrag returns: a defrag can run long enough for leadership to move, and callers (e.g. etcd-operator moving leadership before recreating a pod) want the current leader, not the one from when the call started. Signed-off-by: Gyuho Lee <gyuhol@nvidia.com>

gyuho · 2026-09-09T15:28:28Z

/retest

TestCurlV3MaintenanceDefragment asserted the POST /v3/maintenance/defragment response body is exactly "{}". Now that Defragment populates the response header (cluster_id, member_id, revision, raft_term, leader_id), the gateway returns a non-empty body and the expectation times out. Assert on the leader_id field instead, which is the field this change introduces. Signed-off-by: Gyuho Lee <gyuhol@nvidia.com>

gyuho · 2026-09-09T16:09:28Z

@ahrtr PTAL.

ahrtr · 2026-09-09T19:44:21Z

LGTM, please also add a changelog item for 3.8

https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.8.md#etcd-server

Signed-off-by: Gyuho Lee <gyuhol@nvidia.com>

gyuho · 2026-09-10T04:35:49Z

/retest-required

gyuho · 2026-09-10T14:50:29Z

/retest-required

ahrtr · 2026-09-10T15:38:52Z

Please signoff the last commit, either your sigature or mine (below),

Signed-off-by: Benjamin Wang <benjamin.ahrtr@gmail.com>

Co-authored-by: Benjamin Wang <benjamin.ahrtr@gmail.com> Signed-off-by: Gyuho Lee <gyuhol@nvidia.com>

gyuho · 2026-09-10T16:59:33Z

@ahrtr PTAL

ahrtr

LGTM & thx

kubernetes-prow · 2026-09-10T17:41:50Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahrtr, gyuho

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [ahrtr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ahrtr · 2026-09-10T20:24:34Z

cc @fuweid @ivanvc @serathius

fuweid · 2026-09-10T20:43:27Z

  - [Remove flag `--max-snapshots` and `--v2-deprecation`](https://github.com/etcd-io/etcd/pull/22306)
  - [Cleanup the legacy v2 snapshot files on bootstrap](https://github.com/etcd-io/etcd/pull/22336)
  - [Cleanup the legacy v2 snapshot source code and cleanup orphaned defragmentation files on bootstrap](https://github.com/etcd-io/etcd/pull/22341)
+- [Add `LeaderId` (`leader_id`) to `ResponseHeader` and populate it in every response, including `DefragmentResponse`](https://github.com/etcd-io/etcd/pull/22327)


Do we need to support Snapshot/MoveLeader as well?

fuweid · 2026-09-10T20:46:23Z

+  // leader_id is the member ID of the raft leader that the responding
+  // member believes is the current leader, or 0 if the responding member
+  // does not know who the leader is.
+  uint64 leader_id = 5 [(versionpb.etcd_version_field)="3.8"];


We need to update this file as well -

etcd/etcdctl/ctlv3/command/printer_json.go

Line 38 in e703769

func (h *HexResponseHeader) MarshalJSON() ([]byte, error) {

fuweid

Left two comments and overall it looks good to me.

Signed-off-by: Gyuho Lee <gyuhol@nvidia.com>

kubernetes-prow · 2026-09-11T01:10:57Z

@gyuho: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-etcd-verify	`9b60402`	link	true	`/test pull-etcd-verify`
pull-etcd-grpcproxy-e2e-amd64	`9b60402`	link	true	`/test pull-etcd-grpcproxy-e2e-amd64`
pull-etcd-grpcproxy-e2e-arm64	`9b60402`	link	true	`/test pull-etcd-grpcproxy-e2e-arm64`
pull-etcd-e2e-386	`9b60402`	link	true	`/test pull-etcd-e2e-386`
pull-etcd-e2e-amd64	`9b60402`	link	true	`/test pull-etcd-e2e-amd64`
pull-etcd-e2e-arm64	`9b60402`	link	true	`/test pull-etcd-e2e-arm64`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

kubernetes-prow Bot added area/clientv3 area/documentation area/testing size/L needs-ok-to-test labels Aug 24, 2026

gyuho mentioned this pull request Aug 24, 2026

client v3: implement leader-aware balancer (via gRPC) #22328

Draft

kubernetes-prow Bot added ok-to-test and removed needs-ok-to-test labels Aug 24, 2026

gyuho force-pushed the leader-id-response-header branch from 43ef676 to 575d780 Compare August 25, 2026 04:07

gyuho force-pushed the leader-id-response-header branch from 575d780 to 28140e3 Compare September 8, 2026 15:52

gyuho added 2 commits September 8, 2026 23:55

gyuho force-pushed the leader-id-response-header branch from 28140e3 to a930242 Compare September 8, 2026 15:55

gyuho changed the title ~~api/etcdserverpb/rpc.proto: add leader_id response header~~ api/etcdserverpb/rpc.proto: add leader_id response header, populate defragment response header Sep 8, 2026

gyuho changed the title ~~api/etcdserverpb/rpc.proto: add leader_id response header, populate defragment response header~~ api/etcdserverpb/rpc.proto: add leader_id response header, populate defragment API response header Sep 8, 2026

changelog: add 3.8 entry for leader_id response header

8646a70

Signed-off-by: Gyuho Lee <gyuhol@nvidia.com>

ahrtr reviewed Sep 10, 2026

View reviewed changes

Comment thread CHANGELOG/CHANGELOG-3.8.md Outdated

Update CHANGELOG/CHANGELOG-3.8.md

04590e1

Co-authored-by: Benjamin Wang <benjamin.ahrtr@gmail.com> Signed-off-by: Gyuho Lee <gyuhol@nvidia.com>

gyuho force-pushed the leader-id-response-header branch from 7e77eb3 to 04590e1 Compare September 10, 2026 16:59

ahrtr approved these changes Sep 10, 2026

View reviewed changes

kubernetes-prow Bot added the approved label Sep 10, 2026

fuweid reviewed Sep 10, 2026

View reviewed changes

server,etcdctl: complete leader ID response header support

9b60402

Signed-off-by: Gyuho Lee <gyuhol@nvidia.com>

kubernetes-prow Bot added the area/etcdctl label Sep 11, 2026

Uh oh!

Conversation

gyuho commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kubernetes-prow Bot commented Aug 24, 2026

Uh oh!

fuweid commented Aug 24, 2026

Uh oh!

fuweid commented Aug 24, 2026

Uh oh!

codecov Bot commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ahrtr commented Sep 8, 2026

Uh oh!

gyuho commented Sep 9, 2026

Uh oh!

gyuho commented Sep 9, 2026

Uh oh!

ahrtr commented Sep 9, 2026

Uh oh!

gyuho commented Sep 10, 2026

Uh oh!

Uh oh!

gyuho commented Sep 10, 2026

Uh oh!

ahrtr commented Sep 10, 2026

Uh oh!

gyuho commented Sep 10, 2026

Uh oh!

ahrtr left a comment

Choose a reason for hiding this comment

Uh oh!

kubernetes-prow Bot commented Sep 10, 2026

Uh oh!

ahrtr commented Sep 10, 2026

Uh oh!

fuweid Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

fuweid Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

fuweid left a comment

Choose a reason for hiding this comment

Uh oh!

kubernetes-prow Bot commented Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

gyuho commented Aug 24, 2026 •

edited

Loading

codecov Bot commented Aug 24, 2026 •

edited

Loading