Skip to content

Terminated tunnels end - #700

Merged
dylan-conway merged 3 commits into
mainfrom
fix/terminated-tunnels-end
Oct 13, 2026
Merged

dylan-conway merged 3 commits into
mainfrom
fix/terminated-tunnels-end

Conversation

@ronleizrowice-ant

@ronleizrowice-ant ronleizrowice-ant commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Since #665 a TLS-terminated tunnel's HTTP server never listens: it is handed the connection. The runtime then applies a server's connection timeouts in part (Node: keepAliveTimeout alone) or not at all (Bun). So a tunnel can hold its slot, three descriptors and up to 4 MiB for as long as its client or its upstream leaves it so. At maxTunnels (256) every further CONNECT gets 503. In no release.

state of a tunnel 0.0.79 main, Bun main, Node here, all
idle after a response; curl -X HEAD 6 s never 6 s 6 s
8 MiB announced, 64 KiB sent, answered 413 6 s never 6 s 6 s
handshake, then nothing; half a head 12 s (Bun), 90 s (Node) never never 10 s
half a body, then nothing 300 to 330 s never never 300 to 360 s
a response its client never reads not in 440 s 300 to 360 s

And an upload answered early can hold its slot for good. Past 4 MiB buffered the tunnel is paused. Once the upstream has answered (413, 401) or failed (reset, port closed, bad certificate), nothing reads on, so not even the client's FIN is seen. Whether that shows depends on the runtime build: some drop the rest of such a request themselves. On one that does not, with a cap of 4, eight uploads of 16 MiB answered 413: four of them, and every later CONNECT, get 503. Here all pass.

What changes

boundTunnel(), called once per tunnel:

  • Idle (no response owed, nothing queued for the client): ended 6 s after a response, which is 1 s later than Keep-Alive: timeout=5 says, as in 0.0.79, or 10 s after the ClientHello. Destroyed if it is still there, and idle, as long again.
  • What is queued for the client is the client's to take. No one layer counts it: a response is done, and Bun's TLS socket empty and finished, with a megabyte in the stream under it. So three queues are summed, and a timer that finds anything in them starts over.
  • Once a response is out, the rest of its request is dropped. One line covers relayed answers, the proxy's own 502s and its early refusals.
  • Looked at every 60 s: destroyed once nothing has moved for 300 s, be it a request not all in or held back, or bytes queued for the client.
  • A whole request that waits for its response is left alone, so long polls and event streams are not cut.

Also:

  • When its backend closes first, the mux closes the client after the flush. Under Node a stalled reader lost up to 31 KB, on main and in 0.0.79 too.
  • closeAfterFlush waits 300 s for the flush, where it waited 30 s. The kernel takes more a megabyte or two at a time, so below about 50 KB/s a Connection: close response could lose its last bytes. No progress is counted there: a socket's writableLength falls in one step, when the whole batch has gone (measured on three runtimes).
  • The 503 at the cap is logged and recorded as a violation that names the limit and the option, once per command until half the slots are free again.
  • maxTunnels and handshakeTimeoutMs are read at each CONNECT, so updateConfig changes them.
  • The buffering gate gives its bytes back when the socket closes.
  • A refused CONNECT's violation line names host:port (it read null443).

Testing

  • 84 end-to-end tests, and one of closeAfterFlush's time, run as children under Node and Bun. CI's Bun drops the rest of an early-answered request by itself, so those tests switch that off, and then main fails them on every runtime. 87 mutations: 86 fail a test, and one changes no answer.
  • Proxy, upstream and clients as processes of their own, direct and through two socat relays, on four runtime builds, against 0.0.79 and main: the table above, and:
responses cut 0.0.79 main here, three runtimes
curl --limit-rate 50k to 400k, 24 MiB, both ways 6 of 20 0 of 20 0 of 60
reads nothing for 15 s, then all; 61 sizes, 1 to 16 MiB, both ways 5 of 122 0 of 122 0 of 366
Connection: close, 15k to 45k, 14 MiB, both ways 30 of 32 0 of 96
40 tunnels: ask, wait, ask 0.0.79 here
4.0, 5.0, 5.5, 5.9 s all served all served
6.0 s closed, or the request lost the same
  • Not cut either: an 18 s event stream, an answer after 13 s, 16 MiB up at full speed and at 1 MiB/s (checksums equal).

Not covered

  • That a client reads is seen only when the kernel takes more, a megabyte or two at a time. So a steady reader below about 5 KB/s, of a response larger than the kernel's buffers hold, counts as stopped and is cut at 360 s. So is a reader that pauses for more than 300 s.
  • A client that takes nothing of a Connection: close response holds its slot for 300 s, where it held it for 30 s.
  • A live client that waits for an answer keeps its slot, as in 0.0.79.
  • One counter for all commands. A CONNECT that stays silent holds a slot for the handshake deadline. A non-TLS CONNECT counts until it is seen to be one.
  • Plain connections in srt proxy have no timers. Opaque tunnels still destroy the client when the upstream closes.
  • Uploads through a terminated tunnel take 1.6 to 2.4 times as long as on 0.0.79 under Bun, at about twice the CPU. Unchanged here.

Merge order

Meets #698 in one list of options (sandbox-manager.ts). Whichever merges second needs main merged in.

A TLS-terminated tunnel's HTTP server never listens, so the runtime applies a
server's connection timeouts in part (Node) or not at all (Bun): a tunnel
could hold its slot for as long as its client or its upstream left it so, and
an upload answered early could leave it paused with its client's FIN unread.

boundTunnel closes a tunnel that owes no response 5 s after a response or
10 s after its ClientHello, drops the rest of a request once its response is
out, and closes a tunnel whose request has not moved for 300 to 600 s. A whole
request that waits for its response is left alone.

The 503 at the cap is logged and recorded as a violation, and updateConfig
changes maxTunnels and handshakeTimeoutMs.
…nd keeps a margin

The idle timer was armed when a response closed, with up to a megabyte still
queued for the client, and what was left when it fired twice was thrown away:
a slow reader lost the tail of a large response. No one layer counts what is
queued, so three are summed. A timer that finds anything there, when it is
armed or when it fires, starts over.

The idle end comes 1 s later than Keep-Alive: timeout says, as in 0.0.79, for
a client that takes it at its word.

A tunnel with something under way is looked at five times in the time
allowed, so it is closed 300 to 360 s after anything last moved. That now
holds for a response its client never reads as well.

When its backend closes first, the mux closes the client after the flush.
It gave up after a fixed 30 s. The kernel takes more a megabyte or two at a
time, so below about 50 KB/s a response that ends its connection could lose
its last bytes. It now waits the 300 s a tunnel may stall. No progress is
counted: a socket's writableLength falls in one step, when the whole batch has
gone, so a slow reader cannot be told from one that has stopped.

The README's figure for the slowest reader that is not cut: about 5 KB/s.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants