Terminated tunnels end

README.md

@@ -474,10 +474,10 @@ What the check leaves alone: allowlist entries that **are** IP literals (allow-l
   - **mTLS upstreams** - only the in-sandbox client holds the client certificate, so the proxy cannot re-originate the connection on its behalf.
   - **Certificate-pinning clients** - clients that verify the upstream's identity themselves (custom CAs, SAN pinning) and reject the MITM certificate.
 - `network.tlsTerminate.extraCaCertPaths` - Paths to PEM CA certificate files appended to that trust bundle, after the MITM CA and the host's regular roots. Excluded (non-terminated) hosts are verified by the client inside the sandbox, and the trust env vars SRT sets (`SSL_CERT_FILE`, `GIT_SSL_CAINFO`, ...) _replace_ each tool's own trust configuration, so a site-local root (e.g. an internal mTLS CA) must be in the bundle or those hosts can never be verified. Only the `CERTIFICATE` blocks of each file are copied into the bundle (anything else, e.g. a private key in a combined PEM, is never exposed to the sandbox); files that are missing, unreadable, or contain no PEM `CERTIFICATE` block are skipped, so it is safe to list paths that exist on only some hosts.
-- `network.tlsTerminate.maxTunnels` - At most this many CONNECT tunnels are TLS-terminated at once (integer, 1 to 65536, default 256). Past it a CONNECT is answered `503` with `X-Proxy-Error: too-many-tunnels`. A tunnel holds its slot from its CONNECT until it closes, so idle keep-alive tunnels count. The slot is taken before the client's first bytes are seen, so at the cap a non-TLS CONNECT (such as SSH) to a host that would be terminated is refused too; one that turns out not to carry TLS frees its slot. Hosts in `excludeDomains` do not count.
+- `network.tlsTerminate.maxTunnels` - At most this many CONNECT tunnels are TLS-terminated at once (integer, 1 to 65536, default 256). Past it a CONNECT is answered `503` with `X-Proxy-Error: too-many-tunnels`, and the refusal is recorded as a violation, once per command until half the slots are free again. A tunnel holds its slot from its CONNECT until it closes, so idle keep-alive tunnels count until they are [closed](#tls-termination). The slot is taken before the client's first bytes are seen, so at the cap a non-TLS CONNECT (such as SSH) to a host that would be terminated is refused too; one that turns out not to carry TLS frees its slot. Hosts in `excludeDomains` do not count.
 - `network.tlsTerminate.handshakeTimeoutMs` - A tunnel to be terminated must finish its TLS handshake within this many milliseconds of its CONNECT (integer, 100 to 600000, default 10000), or it is closed and its slot freed. A CONNECT whose client sends nothing is closed at the deadline too. A tunnel past its handshake is never timed out by it.
 
-These two are read when the proxy starts; `updateConfig` does not change them.
+These two are read at each CONNECT, so `updateConfig` changes them for the tunnels that follow.
 
 ```json
 {
@@ -858,14 +858,15 @@ tlsTerminate needs Node, or Bun 1.4 or later (this is Bun 1.3.13)
 
 A program built with `bun build --compile` runs on the Bun that built it, so build it with Bun 1.4 or later. Without `tlsTerminate`, nothing changes on any runtime. Should a tunnel still reach the proxy on such a runtime, it is closed rather than left hanging.
 
-**Limits.** At most `maxTunnels` tunnels (default 256) are terminated at once, and each must finish its handshake within `handshakeTimeoutMs` (default 10 s) of its CONNECT; see [Network Configuration](#network-configuration). Leaf certificates share one key per CA, and the proxy keeps those of the 256 most recently used host names.
+**Limits.** At most `maxTunnels` tunnels (default 256) are terminated at once, and each must finish its handshake within `handshakeTimeoutMs` (default 10 s) of its CONNECT; see [Network Configuration](#network-configuration). Past its handshake, an idle tunnel (no request on it, nothing queued for its client) is closed and its slot freed: 10 s after its ClientHello if no whole request head has come, and 6 s after a response if none has followed, which is 1 s later than the `Keep-Alive: timeout=5` that responses carry; a client that does not answer the close is cut off as long again later. A tunnel whose request is not all in or is held back for an upstream that takes no more of it, or whose client does not take what is queued for it, is closed 300 to 360 s after anything last moved. A client that goes on reading, slowly or in bursts, loses nothing; that it reads shows as the kernel's send buffer frees up, up to a megabyte or two at a time, so one that takes less than about 5 KB/s of a response larger than the kernel's buffers hold counts as one that has stopped. A whole request may wait for its response without limit. Leaf certificates share one key per CA, and the proxy keeps those of the 256 most recently used host names.
 
 **Memory held for slow peers.** A response to a client that is not reading waits once about 1 MiB is queued for that client, or once all tunnels together have 256 MiB queued, on Node and Bun alike. A request body to an upstream that is not reading is held back by the runtime under Node. Under Bun, the proxy does it itself (it pauses a tunnel once 4 MiB of its request body is buffered, or once the shared 256 MiB is used), but that only works on a connection the proxy is handed by its host. The proxy `SandboxManager` runs accepts its own connections, and Bun keeps reading such a tunnel while it is paused. So under Bun an upload to an upstream that has stopped reading is buffered in the SRT process without bound. This is a limitation of the runtime. Use Node where a sandboxed process may upload more than the host can hold.
 
 **Changes a `tlsTerminate` user can observe.** On a supported runtime, compared with the per-tunnel listener:
 
 - The tunnel cap: past `maxTunnels`, a CONNECT is answered `503` (`X-Proxy-Error: too-many-tunnels`), whether or not it would have carried TLS.
 - The handshake deadline: a tunnel that has not finished its TLS handshake within `handshakeTimeoutMs` of its CONNECT is closed.
+- Idle tunnels are closed 6 s after a response, as before, and 10 s after a ClientHello that no request follows (before: about 12 s under Bun, 90 s under Node). An upload is cut once it has not moved for 300 to 360 s, no longer 300 s after it began; so is a response of which the client takes nothing (before: never).
 - The server name passed to `filterRequest` (`sni`) is lower-cased.
 - With `requireHostMatch`, the leaf certificate is always the CONNECT target's, whatever server name the client asks for (a name that does not match gets `421`, as before).
 - A ClientHello whose server name cannot be read as a plain DNS name (letters, digits and inner hyphens in dot-separated labels, with no trailing dot; for example one with an underscore) gets the CONNECT target's leaf certificate instead of one minted for that name. With `requireHostMatch`, its requests are answered `421`.

docs/srt-proxy.md

@@ -284,17 +284,19 @@ addresses and the private ranges (`10/8`, `172.16/12`, `192.168/16`,
 ### Limits and backpressure
 
 The memory and CPU a client can make the proxy spend are bounded by the
-limits below. Slots are not: there is no request-head deadline and no idle
-deadline, so once its TLS handshake is done a client can hold a tunnel open
-indefinitely by sending nothing or half a request head, and it can do the
-same on a plain connection; plain connections are not capped in number. The
-proxy serves a single sandboxed client, so a client that exhausts its slots
-denies service only to itself.
+limits below, and so is the time a tunnel holds its slot with no request
+moving on it. A plain connection has no request-head deadline and no idle
+deadline, so a client can hold one open indefinitely by sending nothing or
+half a request head; plain connections are not capped in number. The proxy
+serves a single sandboxed client, so a client that exhausts its slots denies
+service only to itself.
 
 | Resource                                                           | Limit                                                                                                       |
 | ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------- |
 | TLS-terminated tunnels at once                                     | 256; past it, 503 before the 200                                                                            |
 | Time to finish a tunnel's TLS handshake                            | 10 s after the CONNECT; then the tunnel is closed and its slot freed                                        |
+| A tunnel with no request on it and nothing queued for its client   | closed 10 s after its ClientHello, or 6 s after its last response (`Keep-Alive: timeout=5`, and 1 s)        |
+| A request not all in or held back, a response its client leaves    | the tunnel is closed 300 to 360 s after anything last moved. A whole request may wait for its response      |
 | Request head                                                       | 2 MiB                                                                                                       |
 | Request body the decider sees (`need_body`)                        | `max`, at most 32 MiB                                                                                       |
 | Request bodies held for the decider at once                        | 64 MiB, counting each at its Content-Length or else at `max`; past it, 503 `decider_unavailable`            |
@@ -303,8 +305,8 @@ denies service only to itself.
 | Per connection                                                     | 1 MiB queued for the client; under Bun, 4 MiB of request body (Node's own backpressure bounds it otherwise) |
 | Leaf certificates                                                  | the 256 most recently used names; all share one key, so a new name costs a signature, not a key generation  |
 
-The tunnel cap and the handshake deadline are fixed in srt-proxy. SRT's full
-sandbox mode sets them with `network.tlsTerminate.maxTunnels` and
+These are fixed in srt-proxy. SRT's full sandbox mode sets the tunnel cap and
+the handshake deadline with `network.tlsTerminate.maxTunnels` and
 `handshakeTimeoutMs` (see the [README](../README.md#tls-termination)).
 
 A slow upstream holds back the client's upload, and a slow client holds back

eslint.config.js

@@ -21,6 +21,7 @@ export default [
         projectService: {
           allowDefaultProject: [
             'eslint.config.js',
+            'test/fixtures/tunnel-ends.mjs',
             'test/utils/which-node-test.mjs',
             'vendor/build-common.ts',
             'vendor/seccomp/build.ts',

src/sandbox/emitted-connection.ts

@@ -197,12 +197,15 @@ export function gateSocketOnBufferedBody(
   }
   const release = (): void => {
     socket.off('data', check)
+    socket.off('close', release)
     if (budget) adjustByteBudget(budget, -counted)
     counted = 0
   }
   socket.on('data', check)
   req.once('end', release)
   req.once('close', release)
+  // Not every runtime closes the request of a socket that closes.
+  socket.once('close', release)
 }
 
 const endsGracefully = new WeakSet<object>()

src/sandbox/http-proxy.ts

@@ -40,6 +40,7 @@ const ALLOWLIST_DENY = [
 import {
   peekForClientHello,
   terminateAndForward,
+  type TunnelTimeouts,
 } from './tls-terminate-proxy.js'
 import {
   prepareBodySubstitution,
@@ -341,19 +342,33 @@ export interface HttpProxyServerOptions {
    * terminated is accepted, before its first bytes are seen, and held until
    * the tunnel closes (an idle keep-alive tunnel holds one too), or until
    * the tunnel turns out not to carry TLS. So at the cap, a CONNECT that
-   * would have been tunnelled opaquely as non-TLS is refused too.
+   * would have been tunnelled opaquely as non-TLS is refused too. The
+   * refusal is logged and passed to onFilterRequestDenied with method CONNECT,
+   * once per command until half the slots are free again.
    * SandboxManager sets it from network.tlsTerminate.maxTunnels.
    */
   maxTerminatedTunnels?: number
 
   /**
    * A tunnel to be TLS-terminated must finish its TLS handshake this long
    * after its CONNECT (default 10 s), or it is closed and its slot freed.
-   * A tunnel that finished its handshake is not timed out afterwards.
+   * Past its handshake, tunnelTimeouts apply.
    * SandboxManager sets it from network.tlsTerminate.handshakeTimeoutMs.
    */
   tlsHandshakeTimeoutMs?: number
 
+  /**
+   * When a TLS-terminated tunnel is closed, and its slot freed, past its
+   * handshake. Idle, with nothing queued for its client: with no whole
+   * request head yet (default 10 s), or none since its last response (6 s:
+   * the 5 s that `Keep-Alive: timeout` tells the client, and a margin of
+   * 1 s). Under way, with a request that is not all in or is held back for
+   * its upstream, or bytes queued for its client: once nothing has moved for
+   * 300 s. A whole request that waits for its response is not timed out.
+   * Primarily a test seam.
+   */
+  tunnelTimeouts?: Partial<TunnelTimeouts>
+
   /**
    * Bytes all of this proxy's connections together may hold for slow
    * parties: request bodies waiting for an upstream, and responses waiting
@@ -395,6 +410,8 @@ export function createHttpProxyServer(options: HttpProxyServerOptions): Server {
       : {},
   )
   let terminatedTunnels = 0
+  /** The commands told of the tunnel cap in this burst of refusals. */
+  const toldOfCap = new Set<string | undefined>()
   const byteBudget =
     options.byteBudget ??
     createByteBudget(options.maxBufferedBytes ?? 256 << 20)
@@ -709,7 +726,29 @@ export function createHttpProxyServer(options: HttpProxyServerOptions): Server {
         // only terminate if it is one. Non-TLS falls through to the opaque
         // tunnel below — i.e. base-sandbox behaviour, hostname-allowlisted
         // but not content-inspected (same as the SOCKS path).
-        if (terminatedTunnels >= (options.maxTerminatedTunnels ?? 256)) {
+        const maxTunnels = options.maxTerminatedTunnels ?? 256
+        if (terminatedTunnels >= maxTunnels) {
+          // Of a refused CONNECT clients show the status at most, so that
+          // the limit is the sandbox's own is said where refusals are.
+          if (!toldOfCap.has(auth.encodedCommand)) {
+            // The client picks the name, so the names kept are bounded.
+            if (toldOfCap.size >= 100) toldOfCap.clear()
+            toldOfCap.add(auth.encodedCommand)
+            const reason =
+              `the sandbox's limit of ${maxTunnels} TLS-terminated tunnels at ` +
+              `once is reached; the limit is network.tlsTerminate.maxTunnels ` +
+              `in the sandbox's configuration`
+            logForDebugging(
+              `[proxy] CONNECT ${hostname}:${port} refused: ${reason}`,
+              { level: 'warn' },
+            )
+            options.onFilterRequestDenied?.({
+              method: 'CONNECT',
+              url: formatAuthority(hostname, port, 0),
+              reason,
+              encodedCommand: auth.encodedCommand,
+            })
+          }
           const mark = markFor('too_many_tunnels', 'too-many-tunnels')!
           const [name, value] =
             typeof mark === 'string'
@@ -728,6 +767,7 @@ export function createHttpProxyServer(options: HttpProxyServerOptions): Server {
           if (!slotHeld) return
           slotHeld = false
           terminatedTunnels--
+          if (terminatedTunnels * 2 <= maxTunnels) toldOfCap.clear()
         }
         let handshakeDone = (): boolean => false
         const deadline = setTimeout(() => {
@@ -776,6 +816,7 @@ export function createHttpProxyServer(options: HttpProxyServerOptions): Server {
               denyHeader: options.denyHeader,
               requestTargetAsSpelled: options.requestTargetAsSpelled,
               byteBudget,
+              timeouts: options.tunnelTimeouts,
               onEstablished: () => clearTimeout(deadline),
               onHandshakeProbe: isDone => {
                 handshakeDone = isDone

src/sandbox/mux-proxy.ts

@@ -7,6 +7,7 @@ import { unlink } from 'node:fs/promises'
 import { logForDebugging } from '../utils/debug.js'
 import { getPlatform } from '../utils/platform.js'
 import { listenInRange } from './listen-in-range.js'
+import { closeAfterFlush } from './tls-terminate-proxy.js'
 
 /**
  * First-byte values that select the SOCKS handler. SOCKS5's greeting is
@@ -138,7 +139,8 @@ export function createMuxProxyServer(opts: MuxProxyOptions): MuxProxyServer {
     })
     client.on('error', () => upstream.destroy())
     client.once('close', () => upstream.destroy())
-    upstream.once('close', () => client.destroy())
+    // What is still queued for the client then is the end of an answer.
+    upstream.once('close', () => closeAfterFlush(client, () => {}))
     client.pipe(upstream)
     upstream.pipe(client)
   }

src/sandbox/sandbox-config.ts

@@ -879,13 +879,16 @@ export const NetworkConfigSchema = z.object({
         .describe(
           'At most this many CONNECT tunnels are TLS-terminated at once ' +
             '(default 256); past it a CONNECT is answered 503 with ' +
-            '"X-Proxy-Error: too-many-tunnels". A tunnel holds its slot from ' +
-            'its CONNECT until it closes, so idle keep-alive tunnels count; ' +
+            '"X-Proxy-Error: too-many-tunnels", and the refusal is recorded ' +
+            'as a violation, once per command until half the slots are free ' +
+            'again. A tunnel holds its slot from its CONNECT until it ' +
+            'closes, so idle keep-alive tunnels count until they are closed ' +
+            '(6 s after a response); ' +
             'one that turns out not to carry TLS frees it. The slot is taken ' +
             'before the first bytes are seen, so at the cap a non-TLS CONNECT ' +
             '(SSH, say) to a host that would be terminated is refused too. ' +
-            'Hosts in excludeDomains do not count. Read when the proxy ' +
-            'starts; updateConfig does not change it.',
+            'Hosts in excludeDomains do not count. Read at each CONNECT, so ' +
+            'updateConfig changes it for the tunnels that follow.',
         ),
       handshakeTimeoutMs: z
         .number()
@@ -897,8 +900,8 @@ export const NetworkConfigSchema = z.object({
           'A tunnel to be TLS-terminated must finish its TLS handshake ' +
             'within this many milliseconds of its CONNECT (default 10000), ' +
             'or it is closed and its slot freed. A tunnel past its handshake ' +
-            'is never timed out by it. Read when the proxy starts; ' +
-            'updateConfig does not change it.',
+            'is never timed out by it. Read at each CONNECT, so ' +
+            'updateConfig changes it for the tunnels that follow.',
         ),
     })
     .refine(o => !o.caCertPath === !o.caKeyPath, {

src/sandbox/sandbox-manager.ts

@@ -744,11 +744,18 @@ async function startMuxProxyServer(
     stripResponseHeaders: config?.network.stripResponseHeaders,
     refuseOpaqueTunnels: config?.network.refuseOpaqueTunnels,
     requireHostMatch: config?.network.requireHostMatch,
-    maxTerminatedTunnels: config?.network.tlsTerminate?.maxTunnels,
-    tlsHandshakeTimeoutMs: config?.network.tlsTerminate?.handshakeTimeoutMs,
+    // Read at each CONNECT, so that updateConfig changes them.
+    get maxTerminatedTunnels() {
+      return config?.network.tlsTerminate?.maxTunnels
+    },
+    get tlsHandshakeTimeoutMs() {
+      return config?.network.tlsTerminate?.handshakeTimeoutMs
+    },
     onFilterRequestDenied: ({ method, url, reason, encodedCommand }) => {
+      // A refused CONNECT comes with its authority (`host:port`), not a URL.
+      const target = method === 'CONNECT' ? url : redactUrlForViolation(url)
       recordProxyViolation(
-        `deny http-request ${method} ${redactUrlForViolation(url)} (${reason})`,
+        `deny http-request ${method} ${target} (${reason})`,
         encodedCommand,
       )
     },