[release-3.6] fix: deflake TestIssue20271 - #22406
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted filessee 25 files with indirect coverage changes @@ Coverage Diff @@
## release-3.6 #22406 +/- ##
===============================================
- Coverage 68.84% 68.81% -0.04%
===============================================
Files 420 420
Lines 35900 35900
===============================================
- Hits 24716 24705 -11
- Misses 9736 9751 +15
+ Partials 1448 1444 -4 Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
0d3efaf to
0e8a10b
Compare
0e8a10b to
ed1e62d
Compare
|
can you show us the PR indeed fix the flaky test, such as run the test 100 times on main and your dev branch separately? |
fuweid
left a comment
fuweid
left a comment
There was a problem hiding this comment.
LGTM thanks for working on this.
| // member and replayed on resume. Writing more entries than the leader's | ||
| // inflight window can hold makes it stop replicating to that member, so it | ||
| // cannot catch up by itself and the new leader is forced to send a snapshot. | ||
| keyCount = 1024 |
There was a problem hiding this comment.
The change looks good to me. I run that 100 times without any error.
However, it's to use etcdctl for each operation. In my local, it takes 70 seconds for one round.
Maybe we can use client instead of etcdctl.
There was a problem hiding this comment.
Thanks for the help! I am going to run 100 times on current release-3.6 branch, and expect to see flaky errors.
I have a quick question and would love to get your opinion: Is it necessary to put 1024 keys in Step1? If I understand correctly, the key point is the write count in Step3. So the changes are as follows:
- put small amount of data in Step1, via etcdctl
- put 1024 keys in Step3, via etcd client
Maybe we can use client instead of etcdctl.
Done
There was a problem hiding this comment.
I have run go test ./e2e -run '^TestIssue20271$' -count=100 -v --failfast -timeout=600m > test-flaky.log 2>&1 on current release-3.6, and the test failed at 16th round.
I think the two experiments show that this PR is a valid fix. cc @ahrtr
reproduce_20271_test.go:79: Step 5: After opening snapshot file from new leader, invoke defragment\n
to override boltdb file. So, for the following changes, the third member will commit them into deleted boltdb file.
reproduce_20271_test.go:81: context done before matching log found: context deadline exceeded
logger.go:146: 2026-09-10T12:02:23.428+0800 INFO closing test cluster...
logger.go:146: 2026-09-10T12:02:23.428+0800 INFO stopping server... {"name": "TestIssue20271-test-0"}
logger.go:146: 2026-09-10T12:02:23.548+0800 INFO stopped server. {"name": "TestIssue20271-test-0"}
logger.go:146: 2026-09-10T12:02:23.548+0800 INFO stopping server... {"name": "TestIssue20271-test-1"}
logger.go:146: 2026-09-10T12:02:24.220+0800 INFO stopped server. {"name": "TestIssue20271-test-1"}
logger.go:146: 2026-09-10T12:02:24.220+0800 INFO stopping server... {"name": "TestIssue20271-test-2"}
logger.go:146: 2026-09-10T12:02:24.244+0800 INFO stopped server. {"name": "TestIssue20271-test-2"}
logger.go:146: 2026-09-10T12:02:24.244+0800 INFO closing server... {"name": "TestIssue20271-test-0"}
logger.go:146: 2026-09-10T12:02:24.244+0800 INFO closing server... {"name": "TestIssue20271-test-1"}
logger.go:146: 2026-09-10T12:02:24.244+0800 INFO closing server... {"name": "TestIssue20271-test-2"}
logger.go:146: 2026-09-10T12:02:24.244+0800 INFO closed test cluster.
--- FAIL: TestIssue20271 (52.07s)
FAIL
FAIL go.etcd.io/etcd/tests/v3/e2e 560.347s
FAIL
Signed-off-by: shenmu.wy <shenmu.wy@antfin.com>
ed1e62d to
f637d17
Compare
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ahrtr, fuweid, silentred The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/cherry-pick release-3.7 |
|
/cherry-pick release-3.6 |
|
@ahrtr: I cannot cherry-pick the present PR on top of its base branch ( DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/cherry-pick release-3.5 |
|
@ahrtr: #22406 failed to apply on top of branch "release-3.7": DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@ahrtr: #22406 failed to apply on top of branch "release-3.5": DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@silentred can you manually backport the fix to 3.7, 3.6 and 3.5 ? thx |
Sure, I will issue PRs. |
fuweid
changed the title
4 participants
Failed tests
Error logs:
Fail point:
Root cause: SIGSTOP doesn't drop packets
The mechanism. EtcdServerProcess.Pause() (tests/framework/e2e/etcd_process.go:280-288) only sends SIGSTOP —
it freezes the process but leaves the network stack alive, so the kernel keeps accepting and queueing inbound
data. Meanwhile the graceful restart path (Restart() → Stop() → SIGTERM → Close()) makes the peer see a TCP
FIN, not an RST. Under Linux semantics FIN does not discard data already sitting in the receiver's kernel
queue; only RST does.
The causal chain.
MsgApp(commit=88) is already written into test-2's kernel socket receive buffer.
impossible with only one healthy member), so the cluster has no quorum during that window.
That's far too fast for a new election (≥1s election timeout), which proves the entries came from data
already delivered to kernel buffers.
The consequence.
The new leader's raftLog has firstIndex=89 (snapshot@88), so it should have been forced to
send MsgSnap. But member 3 had already self-recovered to 88, so no snapshot transfer ever happens — and
"applySnapshot: opened snapshot backend" (server/etcdserver/server.go:1057), the log the test blocks on, is
never printed. AssertProcessLogs times out at 30s → t.Fatal.
Supporting evidence.
All snapshot-transfer log lines (sending/receiving/applied database snapshot, opened
snapshot backend, etc.) appear 0 times in the whole test block. skip compaction since there is an inflight
snapshot is also 0, meaning the compaction path itself was healthy — the fault isn't in compaction. The
asserted string and the applyAfterOpenSnapshot failpoint do exist in 3.6 code, so the assertion isn't simply
wrong.
/cc @ahrtr