Skip to content

Set a tlsHandshakeTimeout for tlsListener - #22130

Merged
ahrtr merged 1 commit into
etcd-io:mainfrom
ahrtr:20260721_listener_deadline
Jul 21, 2026
Merged

Set a tlsHandshakeTimeout for tlsListener#22130
ahrtr merged 1 commit into
etcd-io:mainfrom
ahrtr:20260721_listener_deadline

Conversation

@ahrtr

@ahrtr ahrtr commented Jul 21, 2026

Copy link
Copy Markdown
Member

Set a tlsHandshakeTimeout for tlsListener to resolve a security issue (see below).

A network attacker who can reach an etcd TLS listener can open many TCP connections and never send a ClientHello. Each connection spawns a goroutine in the etcd server process that blocks indefinitely inside tls.Conn.Handshake(), and each is tracked in the pending map. Unbounded goroutine and map growth exhausts memory in the etcd process, causing loss of availability for the etcd cluster (and, when etcd backs Kubernetes, the control plane).

cc @fuweid @ivanvc @serathius

Signed-off-by: Benjamin Wang <benjamin.ahrtr@gmail.com>
@ahrtr

ahrtr commented Jul 21, 2026

Copy link
Copy Markdown
Member Author

/retest

@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.71%. Comparing base (81e09b0) to head (8e4dd06).

Additional details and impacted files
Files with missing lines Coverage Δ
client/pkg/transport/listener_tls.go 63.85% <100.00%> (+0.44%) ⬆️

... and 18 files with indirect coverage changes

@@            Coverage Diff             @@
##             main   #22130      +/-   ##
==========================================
+ Coverage   69.66%   69.71%   +0.05%     
==========================================
  Files         449      449              
  Lines       38177    38179       +2     
==========================================
+ Hits        26597    26618      +21     
+ Misses      10148    10132      -16     
+ Partials     1432     1429       -3     

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 81e09b0...8e4dd06. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

l.handshakeFailure(tlsConn, herr)
return
}
_ = tlsConn.SetDeadline(time.Time{})

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
_ = tlsConn.SetDeadline(time.Time{})
if err := tlsConn.SetDeadline(time.Time{}); err != nil {
l.handshakeFailure(tlsConn, err)
return
}

we can fast-fail here

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I intentionally ignore the err, to avoid breaking any existing behaviour. Even SetDeadline fails, we still continue to execute the check

@kubernetes-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahrtr, fuweid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ahrtr
ahrtr merged commit 8d2f46d into etcd-io:main Jul 21, 2026
34 checks passed
@ahrtr

ahrtr commented Jul 21, 2026

Copy link
Copy Markdown
Member Author

/cherry-pick release-3.7

@ahrtr

ahrtr commented Jul 21, 2026

Copy link
Copy Markdown
Member Author

/cherry-pick release-3.6

@k8s-infra-cherrypick-robot

Copy link
Copy Markdown

@ahrtr: new pull request created: #22141

Details

In response to this:

/cherry-pick release-3.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-infra-cherrypick-robot

Copy link
Copy Markdown

@ahrtr: new pull request created: #22142

Details

In response to this:

/cherry-pick release-3.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ahrtr

ahrtr commented Jul 22, 2026

Copy link
Copy Markdown
Member Author

/cherry-pick release-3.5

@k8s-infra-cherrypick-robot

Copy link
Copy Markdown

@ahrtr: new pull request created: #22160

Details

In response to this:

/cherry-pick release-3.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@brandond

brandond commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Is there a security advisory for this? Unless I'm misunderstanding the description above, it sounds like this is a remote DOS vector even when mutual TLS is enabled?

A network attacker who can reach an etcd TLS listener can open many TCP connections and never send a ClientHello. (...) Unbounded goroutine and map growth exhausts memory in the etcd process, causing loss of availability for the etcd cluster.

Since the connection is tracked before the client even sends a hello, it sounds like enabling mtls would not be a mitigation?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

4 participants