Skip to content

[release-3.6] fix: deflake TestIssue20271 #22406

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ahrtr merged 1 commit into from
Sep 10, 2026
+17 −9
Merged

[release-3.6] fix: deflake TestIssue20271 #22406

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 17 additions & 9 deletions tests/e2e/reproduce_20271_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,21 @@ import (

// TestIssue20271 reproduces the issue: https://github.com/etcd-io/etcd/issues/20271.
func TestIssue20271(t *testing.T) {
const (
snapCount = 10
// keyCount must be greater than maxInflightMsgs (server/etcdserver/raft.go).
// SIGSTOP only freezes the process, the kernel keeps accepting packets for
// it, so every MsgApp the leader sends in Step 3 is buffered for the paused
// member and replayed on resume. Writing more entries than the leader's
// inflight window can hold makes it stop replicating to that member, so it
// cannot catch up by itself and the new leader is forced to send a snapshot.
keyCount = 1024

@fuweid fuweid Sep 10, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks good to me. I run that 100 times without any error.
However, it's to use etcdctl for each operation. In my local, it takes 70 seconds for one round.
Maybe we can use client instead of etcdctl.

@silentred silentred Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the help! I am going to run 100 times on current release-3.6 branch, and expect to see flaky errors.

I have a quick question and would love to get your opinion: Is it necessary to put 1024 keys in Step1? If I understand correctly, the key point is the write count in Step3. So the changes are as follows:

  • put small amount of data in Step1, via etcdctl
  • put 1024 keys in Step3, via etcd client

Maybe we can use client instead of etcdctl.

Done

@silentred silentred Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have run go test ./e2e -run '^TestIssue20271$' -count=100 -v --failfast -timeout=600m > test-flaky.log 2>&1 on current release-3.6, and the test failed at 16th round.
I think the two experiments show that this PR is a valid fix. cc @ahrtr

    reproduce_20271_test.go:79: Step 5: After opening snapshot file from new leader, invoke defragment\n
        to override boltdb file. So, for the following changes, the third member will commit them into deleted boltdb file.
    reproduce_20271_test.go:81: context done before matching log found: context deadline exceeded
    logger.go:146: 2026-09-10T12:02:23.428+0800 INFO    closing test cluster...
    logger.go:146: 2026-09-10T12:02:23.428+0800 INFO    stopping server...      {"name": "TestIssue20271-test-0"}
    logger.go:146: 2026-09-10T12:02:23.548+0800 INFO    stopped server. {"name": "TestIssue20271-test-0"}
    logger.go:146: 2026-09-10T12:02:23.548+0800 INFO    stopping server...      {"name": "TestIssue20271-test-1"}
    logger.go:146: 2026-09-10T12:02:24.220+0800 INFO    stopped server. {"name": "TestIssue20271-test-1"}
    logger.go:146: 2026-09-10T12:02:24.220+0800 INFO    stopping server...      {"name": "TestIssue20271-test-2"}
    logger.go:146: 2026-09-10T12:02:24.244+0800 INFO    stopped server. {"name": "TestIssue20271-test-2"}
    logger.go:146: 2026-09-10T12:02:24.244+0800 INFO    closing server...       {"name": "TestIssue20271-test-0"}
    logger.go:146: 2026-09-10T12:02:24.244+0800 INFO    closing server...       {"name": "TestIssue20271-test-1"}
    logger.go:146: 2026-09-10T12:02:24.244+0800 INFO    closing server...       {"name": "TestIssue20271-test-2"}
    logger.go:146: 2026-09-10T12:02:24.244+0800 INFO    closed test cluster.
--- FAIL: TestIssue20271 (52.07s)
FAIL
FAIL    go.etcd.io/etcd/tests/v3/e2e    560.347s
FAIL

)

e2e.BeforeTest(t)

ctx := t.Context()

snapCount := 10

cfg := e2e.NewConfig(
e2e.WithSnapshotCount(uint64(snapCount)),
e2e.WithSnapshotCatchUpEntries(uint64(snapCount)),
Expand All @@ -51,22 +60,21 @@ func TestIssue20271(t *testing.T) {
defer func() {
require.NoError(t, epc.Close())
}()
cli := newClient(t, epc.Procs[0].EndpointsGRPC(), e2e.ClientConfig{})

t.Log("Step 1: Write some data to the cluster")
for i := 0; i < snapCount*5; i++ {
require.NoError(t, epc.Procs[0].Etcdctl().Put(ctx,
fmt.Sprintf("foo%d", i),
strings.Repeat("Oops", 1024),
config.PutOptions{}))
for i := 0; i < keyCount; i++ {
_, err = cli.Put(ctx, fmt.Sprintf("foo%d", i), strings.Repeat("Oops", 1024))
require.NoError(t, err)
}

t.Log(`Step 2: Config the third member to sleep 15s after OpenSnapshotBackend and use SIGSTOP to pause it.`)
require.NoError(t, epc.Procs[2].Failpoints().SetupHTTP(ctx, "applyAfterOpenSnapshot", `sleep("15s")`))
epc.Procs[2].Pause()

t.Log("Step 3: Delete some key values to trigger new snapshot on the first two members")
for i := 0; i < snapCount+20; i++ {
_, err = epc.Procs[0].Etcdctl().Delete(ctx, fmt.Sprintf("foo%d", i), config.DeleteOptions{})
for i := 0; i < keyCount; i++ {
_, err = cli.Delete(ctx, fmt.Sprintf("foo%d", i))
require.NoError(t, err)
}

Expand Down