🐛 Share OpenStack HTTP transports by TLS trust config to stop connection leaks - #3255
🐛 Share OpenStack HTTP transports by TLS trust config to stop connection leaks#3255bartekle wants to merge 1 commit into
Conversation
…n leaks pkg/scope cached one *http.Transport per credential (via the scope LRU cache), but evicting or refreshing a scope never closed it. net/http has no finalizer on Transport, so an evicted scope's idle keep-alive connections were never released - only accumulated - for the lifetime of the manager process. With one credential per cluster and a small scope cache, this showed up as tens of thousands of stale established connections across a fleet of clusters. Key transports by TLS trust config (CA bundle + verify mode) instead of by credential: that dimension is low-cardinality and stable across a fleet against the same cloud(s), so it can be cached in a small, explicitly-evicting LRU (closing idle connections on overflow) without reintroducing the leak. The credential-level scope cache can now evict freely since it no longer owns an exclusive connection pool. Also raises MaxIdleConnsPerHost (2 -> 25, matching client-go's tlsTransportCache) since this transport is now shared by every credential in the fleet against a given host, and adds --transport-cache-max-size to make the new cache's bound configurable.
kubernetes-prow
Bot
added
the
do-not-merge/hold
✅ Deploy Preview for kubernetes-sigs-cluster-api-openstack ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
|
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Welcome @bartekle! |
kubernetes-prow
Bot
added
the
size/L
|
Hi @bartekle. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
kubernetes-prow
Bot
added
cncf-cla: no
|
i tried this version on my staging cluster with ~80 clusters and it seems like it helps with number of connections ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn
11 TIME-WAIT
6 ESTAB
3 LISTEN |
|
@bartekle could you please sign the cla as documented here: https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/main/CONTRIBUTING.md#sign-the-cla and also update PR title with the correct emoji |
bartekle
changed the title
bartekle
changed the title
|
i will try to deal with CLA asap |
kubernetes-prow
Bot
added
cncf-cla: yes
|
/ok-to-test |
kubernetes-prow
Bot
added
ok-to-test
lentzi90
left a comment
lentzi90
left a comment
There was a problem hiding this comment.
Thanks for the PR!
I am a bit concerned with the complexity of this. Package level state (cache), mutex, changes to exported package signatures (NewFactory) and a new flag.
Do we need all this to fix the connection leak? Mainly I wonder if we could simply set IdleConnTimeout on the existing transport and skip all the caching changes/additions? The caching seems like a separate concern to me and should not be strictly needed for fixing the bug.
|
|
||
| provider.HTTPClient.Transport = &http.Transport{Proxy: http.ProxyFromEnvironment, TLSClientConfig: config} |
There was a problem hiding this comment.
What if we simply set IdleConnTimeout here?
provider.HTTPClient.Transport = &http.Transport{
Proxy: http.ProxyFromEnvironment, TLSClientConfig: config
IdleConnTimeout: 90 * time.Second
}
Maybe also MaxIdleConnsPerHost?
| // None of these tests use t.Parallel(): they share the package-level, | ||
| // size-bounded transportCache, and running them concurrently would let one | ||
| // test's insertions evict entries another test is mid-assertion on. |
There was a problem hiding this comment.
I do not like the sound of this. I think we need to consider alternatives. Could the cache be stored in the providerScopeFactory together with the clientCache instead?
3 participants
pkg/scope cached one *http.Transport per credential (via the scope LRU cache), but evicting or refreshing a scope never closed it. net/http has no finalizer on Transport, so an evicted scope's idle keep-alive connections were never released - only accumulated - for the lifetime of the manager process. With one credential per cluster and a small scope cache, this showed up as tens of thousands of stale established connections across a fleet of clusters.
Key transports by TLS trust config (CA bundle + verify mode) instead of by credential: that dimension is low-cardinality and stable across a fleet against the same cloud(s), so it can be cached in a small, explicitly-evicting LRU (closing idle connections on overflow) without reintroducing the leak. The credential-level scope cache can now evict freely since it no longer owns an exclusive connection pool.
Also raises MaxIdleConnsPerHost to 25 since this transport is now shared by every credential in the fleet against a given host, and adds --transport-cache-max-size to make the new cache's bound configurable.
What this PR does / why we need it:
Which issue(s) this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when PR gets merged):Fixes #3227
Special notes for your reviewer:
TODOs:
/hold