Registry connections are not pooled across operations, and idle sockets are never released

[mzihlmann]

Kaniko builds a fresh http.Transport for every logical registry operation, so almost nothing shares a connection pool. A build against one registry opens dozens of TCP connections where a handful would do, and the sockets are never released before the process exits.

pkg/util/transport_util.go:82 is the only place a transport is made:

var tr http.RoundTripper = http.DefaultTransport.(*http.Transport).Clone()

Clone() returns a transport with a private, empty connection pool. Callers:

• pkg/image/remote/remote.go:145 via remoteOptions, once per base image pull, and again on every retry because remoteOptions is called inside retryFunc (lines 67 and 98)
• pkg/cache/cache.go:71, once per distinct cache key looked up in the cache repo
• pkg/executor/push.go:125, once per destination repo in the push permission precheck
• pkg/executor/push.go:285, once per destination in DoPush, which every cache layer push also goes through via pushLayerToCache

Two secondary effects follow from the same line. The clone inherits Go's MaxIdleConnsPerHost default of 2, while go-containerregistry sets 50 in its own DefaultTransport (remote/options.go:127) precisely for this workload, and kaniko discards that by passing remote.WithTransport. And each new transport means a new go-containerregistry fetcher, so transport.NewWithContext (remote/fetcher.go:69) repeats the /v2/ ping and token exchange every time.

Nothing ever calls CloseIdleConnections, so idle sockets sit until IdleConnTimeout (90s), which is longer than most builds.

Measurement

A counting TCP proxy in front of a local registry, logging every accept, every HTTP request line and every close. Kaniko ran in a container with --net=host against a 5 layer base image and a 5 step Dockerfile, cache enabled, one destination and one cache repo.

Cold cache, 4.65s of registry activity:

metric	value
TCP connections opened	29
HTTP requests carried	90
peak concurrent sockets	18
max requests on a single connection	8
sockets still open at process exit	16 of 19
longest socket held idle	4609 ms
total idle socket time	18.1 socket-seconds

Warm cache, all 5 steps hit, 278ms of registry activity:

metric	value
TCP connections opened	19
HTTP requests carried	42
peak concurrent sockets	11
requests per connection	2 to 6
sockets still open at process exit	9 of 11

Connection reuse does work inside a single transport. The base image pull put 8 requests and 59MB on one socket. The problem is that each of the 10 logical operations in the cold build got its own pool and its own sockets.

10 of the 29 connections in the cold run carried no HTTP request at all: 1519 bytes up, 103 bytes down, closed by the registry. That is go-containerregistry pinging https:// first and falling back to http://, one wasted connection per transport. Signature confirmed against the same registry with curl -k https://, which produces the same 103 byte reply. This part is per fetcher rather than per transport, so it needs a shared puller to fix, not just a shared transport.

Idle hold, cold run, times relative to the proxy start:

id  accept  close  idle_held  reqs
3   32695   37344  4609       3
7   32754   37345  4580       2
5   32724   37345  2229       8     <- the 59MB base image pull
10  36061   37345  1206       3
9   36056   37344  1175       6
12  36326   37344  924        5
13  36334   37343  891        4

Every one of those closes at 37343 to 37345, when the process exits. They were done doing work seconds earlier.

Suggested direction

Memoize the transport per registry so operations against the same host share one pool, keep go-containerregistry's MaxIdleConnsPerHost of 50, hoist remoteOptions out of retryFunc so retries reuse the pool, and reuse a remote.Puller per registry to collapse the repeated pings and token exchanges. Close idle connections at the end of the build.

Expected impact is largest on cache heavy builds against a TLS registry, where every avoided connection is also an avoided TLS handshake, and on registries that rate limit by connection.

Measurement harness is a throwaway counting proxy, reproducible on request. Numbers above are from a single host, so they show shape rather than a portable benchmark.

[mzihlmann]

Two follow-ups worth folding into the implementation.

Instrument this with OTel rather than a proxy. httptrace.ClientTrace exposes the facts the counting proxy had to infer: GotConn carries Reused, WasIdle and IdleTime, and ConnectStart/ConnectDone/TLSHandshakeStart/TLSHandshakeDone give the dial and handshake cost. Hooking that onto the RoundTripper returned by MakeTransport puts reused=false and handshake duration on every registry span, which works against real TLS registries where a proxy would have to terminate TLS. It also makes the fix self-verifying.

Re-measure FF_KANIKO_DISABLE_HTTP2 once connections are pooled. HTTP/1.1 currently beats HTTP/2 on the GitLab registry, and the MaxIdleConnsPerHost=2 problem only affects the HTTP/1.1 path, so the HTTP/2 penalty is the larger effect today. A transport per operation means each HTTP/2 connection carries a handful of requests and is then abandoned, so the one-connection-many-streams benefit never materialises while per-stream flow control and TCP head-of-line blocking still apply. Pooling helps HTTP/2 disproportionately, so the gap may narrow or flip and the flag guidance may need updating.