perf: cache TLS stores, pool clients, and optimize response bodies - #208
Draft
sqdshguy wants to merge 17 commits into
Draft
perf: cache TLS stores, pool clients, and optimize response bodies#208sqdshguy wants to merge 17 commits into
sqdshguy wants to merge 17 commits into
Conversation
build_cert_store() parsed ~150 Mozilla roots plus the system store into a fresh BoringSSL X509_STORE on every call. Explicit transports and the ephemeral client cache only paid that once, but requests that pass a string sessionId with cookieMode 'session' build a client per request, so every one of them rebuilt the store (~1 ms of X509_NAME_cmp per request, ~860 req/s on loopback). CertStore is an Arc, so cache one per TrustStoreMode and clone it. Loopback sessionId GETs go from ~860 to ~14,900 req/s on Bun and Node.
Requests that name a session by id without an explicit transport fell through to the 'fresh client' branch of make_request, which built and dropped a full wreq client per request: a new TCP connection every time, no keep-alive, and (before the previous commit) a trust store rebuild as well. On loopback that path ran at ~860 req/s while createSession() sessions did ~100k. Cache those clients in TransportManager keyed by (session id, config) with the same 300s idle timeout as the session cache, so sessions still never share connections or TLS state, and per-request overrides such as a different browser still get their own client. drop_session() also evicts a session's implicit clients. Loopback sessionId GETs now match sessions: ~111k req/s on Bun and ~99k on Node, with 35 sockets in TIME_WAIT after 660k requests instead of one per request. Adds a spec that counts server connections.
Bodies were only inlined when the response carried a Content-Length of at most 2 MiB. Chunked responses never qualify, and neither does any compressed response, because tower-http's decoder drops the length hint when it strips Content-Encoding. On real sites that is nearly every response: a probe of six hosts saw 10 of 12 responses (including a 559-byte example.com page) take the body-handle path, which costs a moka insert, a mutex, a tokio spawn, and a second FFI round trip via readBodyAll(). For unknown-length bodies, collect frames that arrive within a 10ms window. If the body completes under the inline limit it is inlined exactly like a known-length body, with contentLength set to the decoded size. Otherwise the collected prefix is replayed ahead of the live stream through the usual handle, so streaming responses keep working with at most 10ms of extra first-byte latency. text/event-stream is exempt and always streams immediately. Same real-site probe now inlines 9 of 12; the rest are large bodies that legitimately span more than 10ms. Loopback 4 KB chunked/gzip GETs: Bun 81k -> 96k req/s, Node 65k -> 93k req/s, within 8% of a content-length body. Adds /stream/slow and /sse test routes and specs for inlining, stalled streams, and event streams.
Replace the 10ms wait from 28459b2 with the approach Bun's fetch uses: a body is inlined only if it has fully arrived by the time the response is handed to JS. Frames hyper has already delivered are drained without waiting on the network; if that completes the body under the inline limit it is inlined, otherwise the prefix is replayed ahead of the live stream through a body handle. Nothing sits behind a timer any more, so the text/event-stream exemption and the tokio 'time' feature go away. One scheduler yield is taken before deciding: hyper resolves the response as soon as headers are parsed while its connection task is still delivering frames it already holds, and the chunked decoder only reports end-of-body on a later poll. Without the yield, 4 KB chunked GETs on loopback stayed at the pre-fix 83k/63k req/s (Bun/Node); with it they reach 105k/93k, the same as a Content-Length body. gzip 4 KB is 97k/93k either way. Real-site probe: 5-6 of 12 responses inline (everything up to ~35 KB); the 10ms window also caught a 120 KB and a 1.3 MB page, which now stream as before. Adds a /chunked fixture whose data and terminating chunk leave in one flush; write()+end() would split them and miss.
Session cookie jars sat in a moka cache with a 300s time-to-idle, and a lookup miss silently created an empty jar. A createSession() that logged in, went quiet for five minutes, and fetched again sent no cookies and raised no error. Nothing documented the expiry. Native session state now lives in a plain map until dropSession(), the way tls-client keeps its cffi session table until destroySession(). For Session objects that are never closed, a FinalizationRegistry drops the jar and the owned transport on garbage collection, the pattern undici uses for collected Response bodies and persona-http uses via Drop on its napi classes. close() marks the state released and unregisters it so the two never race. Sessions addressed only by a string sessionId have no JS object to finalize and persist until dropped, as before. Adds a Rust test that a jar survives lookups until dropped and a JS lifetime spec, including one that forces GC (the test harness now runs node with --expose-gc) and observes the jar disappear through the string-id path. Documents the lifetime under session.close().
The main thread is the throughput ceiling for small requests, and the profile put most of it in napi calls on either side of the request: - request() probed an options object with napi_get_property for 24 fields, most of them absent on the pooled-session hot path. - The settle callback built headers as an array of [name, value] arrays (an allocation plus three element stores per header), stored an empty cookies array, and set explicit nulls for bodyHandle, bodyBytes, contentLength and diagnostics. request() now takes positional arguments fetched with a single napi_get_cb_info: url, method, sessionId, requestId, a flags bitfield (ephemeral/insecure/compress/redirect/... the way Node's fs binding takes open flags), timeout, a flat headers array, body, transportId, and an optional extras object for browser/os/emulation/proxy/trustStore /onRequestEvent that is only present, and only probed, when one of those is set. Responses carry headers and cookies as a flat [name, value, ...] array, the shape of IncomingMessage.rawHeaders, converted to Headers only on access; cookies and absent fields are omitted rather than set to null. Back-to-back A/B on loopback, 16 concurrent small GETs on a session: Node 96.8k -> 116.8k req/s with main-thread CPU per request down ~24%; Bun 99.5k -> 103.4k, its main thread was only 65% busy either way.
…eads Streaming a body cost one FFI round trip per hyper frame: each readBodyChunk() spawned onto the runtime, awaited a single frame, settled a promise and allocated a Buffer. hyper's body channel is mpsc::channel(0) and the connection only reads more once the body is polled, so nothing ever accumulated between reads. An 8 MB body in 16 KB chunked frames streamed at ~470 req/s against ~1,400 for the same bytes with a Content-Length, whose frames are ~400 KB. Each stored body now gets a pump task that pulls frames off the connection into a buffer bounded at 256 KiB by a semaphore, the way Node's Readable reads ahead to its highWaterMark and Bun's fetch fills its native response buffer between JS reads. readBodyChunk() waits for one frame, then merges whatever is already buffered up to 256 KiB before settling. An error taken off the buffer mid-merge is held and returned by the next read. Dropping the handle closes the buffer, which stops the pump and releases the connection as before. Loopback, 16 concurrent 8 MB chunked downloads streamed with for-await: Bun 513 -> 1,242 req/s, Node 457 -> 1,084, within 10-27% of the Content-Length path (1,365 / 1,495). Adds a /chunked/many test route and a spec that streams 512 frames and checks bytes, order and that reads were coalesced.
Every body reached JS through napi_create_buffer_copy: a fresh V8 buffer, zero-filled on Node, then a memcpy. In the 8 MB profile those two were the largest settle-callback costs, and the copy is why a ~128 MB working set became a 750 MB footprint in the memory soak. Bodies and stream chunks of 64 KiB or more are now wrapped with napi_create_external_buffer, which lets the Buffer own the Rust allocation and free it from a finalizer; below that a copy is cheaper than the finalizer bookkeeping. The symbol is resolved from the host with libloading and probed once at module load, so a runtime built with V8's sandbox that answers napi_no_external_buffers_allowed falls back to copies (neon's own JsBuffer::external would abort there); napi-rs does the same. Bytes -> Vec is zero-copy when the Bytes is uniquely owned, which the inline and coalesced paths guarantee. Node, 16 concurrent, no forced GC: 64 KB 73.5k -> 86.0k req/s, 1 MB 14.9k -> 15.7k, 8 MB streamed 1,622 -> 1,894, peak footprint after a 1 MB + 8 MB soak 926 MB -> 523 MB. Bun keeps copies by default. JSC cannot see external bytes, so under load it collected them late and the same soak peaked at 343 MB against 109 MB with copies, for +2-44% throughput; reporting the sizes through napi_adjust_external_memory pushed its collection threshold out further and peaked in the gigabytes. WREQ_EXTERNAL_BUFFERS=1|0 overrides the per-runtime default. Adds large-body specs covering the threshold.
A Transport from createTransport() that was never closed kept its native client, and every pooled connection in it, for the life of the process. Register each Transport in a FinalizationRegistry that calls dropTransport() on collection, mirroring the Session finalizer; close() marks the state released and unregisters first so the two never race. Adds a lifetime spec that watches the server side: a transport's keep-alive connection closes when close() is called, and closes on its own once an unreferenced Transport is collected. Documents the lifetime under transport.close().
…used Cancelling or dropping a response body with data still unread left the HTTP/1.1 connection mid-message, so hyper closed it and the next request opened a fresh connection. A scraper that reads status and headers of many small responses and moves on churned one connection per request, which on this host exhausted the ephemeral port range. The body pump now, when nobody will read the body any more, keeps reading and discards up to its 256 KiB read-ahead before letting the stream drop, so the connection returns to the pool; past that limit, or when a known Content-Length leaves more than that outstanding, it drops the body and lets hyper close the connection rather than pull megabytes just to throw them away. undici's dump() draws the same line at 128 KiB. Abandonment is now signalled with an explicit CancellationToken on the stored body instead of the mpsc sender closing, since the cache releases evicted entries from its housekeeper rather than inline. Loopback, small chunked body cancelled after the first frame: 8.4k -> 32.4k req/s, connections in TIME_WAIT after the run 16,352 -> 583. Large bodies are unchanged (dropped, connection closed). Adds a spec covering both the drained-and-reused and dropped-and-closed sides.
Isolated fetch() calls share one cached ephemeral client for speed. That client's TLS session cache carried session tickets from one call to the next, so the second connection resumed the first and the server could see isServerReused() == true, linking two requests that are meant to be independent. That is a fingerprint/linkability leak for a library whose purpose is to look like separate browser visits. Give the ephemeral client a session cache that stores nothing and returns nothing, so those connections never resume. It leaves pre_shared_key and therefore the ClientHello untouched: a browser opening its first connection has no session to resume either, so the wire fingerprint is unchanged. Sessions and explicit transports keep wreq's real LRU cache, where resumption within one context is wanted. Promotes the previously-untracked tls-resumption spec: isolated fetches now show no resumption across six requests, explicit transports still resume. Full suite is green.
sqdshguy
force-pushed
the
perf/client-pooling-and-body-streaming
branch
from
September 5, 2026 18:05
3ad94e6 to
54648d7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Repeated
sessionIdrequests rebuilt clients and trust stores, fragmented responses crossed N-API once per frame, and abandoned small bodies discarded reusable connections. This PR caches and pools that work, inlines already-received bodies, coalesces streamed reads with bounded buffering, and uses external Node buffers for bodies/chunks ≥256 KiB. It also fixes session/transport lifetime cleanup and preserves TLS isolation for default fetches.Measured benefit of each change
91cc67c)540ae2d)28459b2,83f2c92)439bfe2)60c53cc)6c7bd5f)a196e02)a9c8118)8206720)da073f1)a4b4be6)2697cbb)4291cd1)b2250f2)The 64 KiB memory cost has a concrete cause: external buffers retain 1.8× payload length in Rust Vec capacity on average, versus 1× for 8 MiB full bodies. Tracked native allocations return to zero after forced GC, while allocator-retained RSS can remain high.
Evidence and limits
Apple M4 Pro / Node 24.19.0, separate loopback HTTP/1.1/HTTPS server, release builds. Historical commits are compared with their immediate parents using six alternating pairs; ambiguous cases use ten longer pairs. Follow-up thresholds/pump variants use six rounds, including concurrency-1 checks. Memory runs hold request counts and rates equal for 20 seconds.
These are workload-specific benefits, not a universal net score. Parent deltas cannot be added or multiplied. Connection-heavy comparisons use capped batches to avoid macOS port exhaustion. Intervals use paired log throughput ratios and Student t critical values. HTTP/2, WAN, and other-platform performance are not established.
Before the final threshold adjustment, the complete 11-commit stack versus
74b92femeasured 50.6× string-sessionId HTTP, +13.7% chunked, +19.8% gzip, 4.1× fragmented streams, and +62.5% cancellation throughput. Pooled small requests were ~1% slower and continuous 8 MiB streams ~3.4% slower. The threshold-only candidate was separately checked againstda073f1; the full matrix has not been rerun with that last adjustment.Bun and native profiling follow-up
Bun 1.4.2 built-in CPU profiles and macOS native stack sampling of the release Rust addon identified a redundant JS forwarding stream and Neon's zero-fill immediately before copying response bytes. Post-change profiles no longer contain the forwarding pull on native streams, and zero-fill disappears from the native full-body hot-stack summary. Remaining costs include buffer copies, N-API calls, and socket/runtime work; sample percentages are not throughput gains.
Each new change was measured separately, with six alternating before/after pairs per runtime/workload, fresh client and separate server processes, 200 ms warm-up, 1.5 s measurement, and concurrency 16. All 192 new measured runs completed without errors. These are local HTTP/1.1 results; combined gains are not inferred by adding separate percentages. Profilers and builds did not overlap throughput measurements. Reports and raw samples are kept outside the PR.
Isolated-fetch setup and identity checks
Bare fetch retains its independent-connection policy. Trust-root insertion and ephemeral client construction now coalesce concurrent cache misses, avoiding duplicate cold builds. Eight alternating pairs per runtime/protocol/burst shape measured cold batches; six paired warmed controls per runtime/protocol found no statistically clear sustained throughput regression. The HTTPS fixture uses a configured CA with verification enabled. These 240 new client runs completed without errors; this change does not claim the ~10× HTTPS gain associated with connection reuse.
A new HTTP/2 test verifies two 16-request bursts use 32 distinct connections, never resume TLS, and do not carry cookies or authorization headers between calls. A Rust concurrency test checks matching configurations share one build and different configurations remain separate. Docs clarify that a shared Transport permits connection-level linkage and that isolation does not conceal caller IPs or caller-supplied identifiers.
Validation
npm run buildnpm test: 223 tests passed locally on Node 20 and 24; the prior 222-test head also passed with external buffers disabled on Node 24npm run typechecknpm run checkDraft for review; package versions are unchanged.