HttpArena: static files over HTTP/2 and production-stack below the field #102

Activity

1. xuckless commented on Oct 23, 2026

   

   xuckless

   on Oct 23, 2026

   OwnerAuthor

   More actions

   Notes from my performance audit (2026-10-23), to keep in mind while fixing this. Measured on Anole 480fd09 (0.3.0) with the zio fork at 88f6d69, using the same HttpArena entry built against that commit (the published entry is still on v0.1.2). Labels: [Measured] reproduced locally (a 4-core VM with the load generator on the same box, so only ratios, CPU per request and I/O operations per request count, not absolute rps; SQE/req = io_uring submissions per request read from the rings' SqTail, mmap/req from strace -f -c); [Code] checked in the code; [Hypothesis] fits the arena data but needs a 64-core run or a profile.

   K7: static files are re-opened, re-stat'ed and re-read on every request [Measured + Code]

   /static/app.js, .br variant 47 KB:

   	SQE/req	CPU/req
   JSON over TLS (h1), for comparison	2.0	16.4 µs
   static over TLS (h1)	13.0	61 µs
   static over h2/TLS (20-file rotation)	8.1	80 µs

   Static.serve (src/http/core/server/static.zig:100-160): sendFile(plain) = openat + fstat; for each precompressed candidate sendFile(path.br) = another openat + fstat, then close the first; sendFileBody reads the file through the Io in file_piece chunks (response.zig:2033-2065); close. Each is an io_uring operation with a park/resume and a loop tick, and the ETag string is formatted per request. Over TLS the bytes are then copied file → write buffer → SSL_write → BIO pending → socket. On h3 the sink buffer is capped at 4 KiB, so a 47 KB file takes 12 read round trips (h3/server.zig:2941).

   Leaders (fib, agensio, ioxide, cnetmod, vanilla) serve precompressed bytes from memory; vanilla snapshots files and re-checks them every 100 ms, with kTLS + sendfile for bodies ≥ 16 KiB and the header sent MSG_MORE; zix prewarms its static cache at startup.

   Fixes:

   1. A static cache inside staticDir: path → {bytes per encoding, content type, ETag, Last-Modified, length}, bounded by total size, revalidated by statx at most every N ms (or inotify). The body becomes a borrowed slice: one write and zero file syscalls per hit. Serve from memory up to X MB; keep sendfile for large plain-TCP files.
   2. Pre-render the response head per (file, encoding) and pre-format the ETag.
   3. Over TLS, pass the cached slice to SSL_write in 16 KiB chunks; kTLS later (BoringSSL doesn't do kTLS natively, but the traffic secrets can be exported).
   4. h3: lift the 4 KiB sink cap for static bodies or hand nghttp3 the cached slice with a no-copy data reader.

   Also relevant here

   • static-h2 (225-325K, 2.7 GiB at 1024 connections): the allocator findings (K2, on HttpArena: unary gRPC at 76k req/s with p99 5 s and ~3 cores busy (leaders ~7M) #99) apply: the h2 per-stream 32 KiB out_queue_len + 16 KiB body_writer_len (h2.zig:91-95) put one streamed response's arena over SmpAllocator's 32 KiB mmap line, and BodyBuffer.nextSegment doubles segments up to 64 KiB in the stream arena.
   • json-tls p99 1.88 s and the production stack: probably connection imbalance under churn (K5, on HttpArena: json-comp 80k req/s with 14k reconnects (Bun 461k, Fiber 290k) #101), not crypto. JSON over TLS on h1 is 16.4 µs/req locally, which is reasonable; BoringSSL record handling is not the problem.
   • gateway-64 / production-stack are behind nginx with keep-alive upstreams; the HTTP/1.1 per-response flush (K3, on Performance follow-ups: gRPC transcoded routes and no_wait, HPACK indexing of Content-Length, QUIC handshake config #50) and the per-request timers and atomics (K9, K10, on Performance follow-ups: gRPC transcoded routes and no_wait, HPACK indexing of Content-Length, QUIC handshake config #50) apply to every request there.
2. xuckless commented on Oct 28, 2026

   

   xuckless

   on Oct 28, 2026

   OwnerAuthor

   More actions

   Reference for K7 from kaitk/goose-http at 6b566a7 and plod's entry. The board figures are HttpArena results on its 3995WX harness, not my measurements.

   goose-http's file cache (decisions.md#L1124-L1166, file.goose#L176-L280):

   • it is opt-in per route, with a byte budget;
   • it holds files up to 64 KiB, per worker;
   • on every request it stats each candidate, plain and precompressed, and compares size, mtime to the nanosecond, and inode.

   With it, plod went from 408K to 938K on static-h2 and from 438K to 918K on static-tls (plod tasks/047-httparena.md#L92-L100).

   What changes in K7 (S2):

   • Freshness: a stat per request (size, ns mtime and inode, for every candidate including .gz/.br) instead of a revalidation interval. The rules say the next response after a replacement must carry the new bytes. inotify on Linux comes later as an optimisation, with stat as the fallback.
   • Shape: one process-wide cache, not a copy per executor. It is built on the atomic-pointer + quiescent-state helper that the S1.5 safety retrofit builds first, so K7 lands after that helper. It is on by default, with its budget stated and printed at startup.
   • resolve_beneath: it becomes the default for static directories (Static files: turn resolve_beneath on by default #133), in S1.5 or with K7, whichever comes first. Today symlinks are followed unless it is set.
   • kTLS + sendfile: a candidate to measure, not struck. goose measured +10% on small responses and -44% at 16 KB with user-space sends (goose-tls decisions.md#L25-L40). aegon, whose HttpArena description lists kTLS TX offload, is ahead of this entry on every TLS profile.

   My audit note above put json-tls's 1.88 s p99 on K5. K5 doesn't apply (#101). HTTP/1.1-over-TLS CPU now has its own issue: #130.
3. xuckless commented on Oct 31, 2026

   

   xuckless

   on Oct 31, 2026

   OwnerAuthor

   More actions

   These are numbers from the S2 flame-graph atlas, run on v0.4.1 (c36cca0) on my x86 box.

   • Load: HttpArena's 20 static URIs with Accept-Encoding: br;q=1, gzip;q=0.8, over TLS h2 at c64 m32. One executor on one core, epoll and io_uring.
   • Caveat: one pass each, so these are leads for K7, not an A/B.
   • Write-up: README.

   h2-static	epoll	io_uring
   req/s (one core)	34,995	47,370
   instructions:u / :k per request	81.5k / 104.8k	78.5k / 38.3k
   context switches per request	11.1	0.3
   memcpy share of CPU	8.0%	13.9%
   arena + malloc allocated per request	53.9 KB + 29.5 KB	54.2 KB + 27.6 KB
   peak RSS	208 MB	211 MB

   What this confirms from the audit

   • K7: files are re-opened and re-read on every request.

     • On epoll, every open and read goes through the blocking pool (ev.thread_pool.ThreadPool.submit). Its threads' futex wait is 55% of all off-CPU time, at 11 context switches per request, and the kernel spends 105k instructions per request.
     • io_uring avoids the handoff and serves 35% more on the same core.
     • The same handoff shows on h3-static: 19 context switches per request on epoll, which fits the 4 KiB h3 sink cap's extra read round trips.

   • The response body is copied at least three times on its way to the socket (the "3-4 times" in K6 on HttpArena: unary gRPC at 76k req/s with p99 5 s and ~3 cores busy (leaders ~7M) #99):

     copy	where	share of CPU
     into the stream's byte queue	StreamSink.put → Io.TypeErasedQueue.putLocked	4.2%
     out to nghttp2	provideData → Queue.getUncancelable	3.0%
     into the TLS BIO	dusty_tls.bioWrite under BIO_write_ex	3.0%

     The body also lands in the request arena first, at 41 KB per request in Response.write, from StreamSink.flushFn and .body.

   • BoringSSL allocates per write. bssl::tls_write_app_data mallocs 10.5 KB per request.

   • Memory per stream: 208 MB peak for 2,048 streams (64 × 32) is about 100 KB per open stream. That fits the 2.7 GiB the entry used at 1024 connections.

   What changes

   • Nothing in the plan. These are the K7 baseline on the box.
   • A cache hit should remove the blocking-pool handoff on epoll and the first arena copy on both backends. The other two copies go with the takeover's zero-copy DATA (K6 fix 4).
   • The after run is bench/atlas/atlas.sh h2-static h3-static.
4. xuckless commented on Nov 1, 2026

   

   xuckless

   on Nov 1, 2026

   Last edited by xuckless

   OwnerAuthor

   More actions

   Update from the v0.5.0 run on HttpArena's machine (MDA2AV/HttpArena#1692, saved): static-h2 is 352k / 284k req/s at 256 / 1024 connections (+9% / +26% against v0.1.2) and production-stack 109k / 108k. Still well below the field: plod serves ~915-938k on static-h2 and 207-253k on production-stack, zix 1.65-1.69M on static-h2. static-h2 also uses 1.3-3.0 GiB. Keeping this open for the static cache (audit K7), which serves static files from memory as the leaders do; production-stack needs its own profile (JWT, cache lookup and Postgres in one request).