From bc40b10deef345ef4234af1f9f598cf2534a0f82 Mon Sep 17 00:00:00 2001 From: Nir Soffer Date: Sat, 25 Jul 2026 21:50:00 +0300 Subject: [PATCH] Limit ENOBUFS retries to avoid head-of-line blocking MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When the kernel's vmnet buffer is full, the helper retries sendmsg_x() with a 50 microsecond sleep between attempts. Previously the retries were unbounded, causing head-of-line blocking: a high-priority small packet (e.g. an HTTP/2 PING or TCP ACK) queued behind large data packets is delayed without bound while the helper waits for buffer space. If the delay exceeds the protocol's timeout, the connection breaks even though the network is otherwise healthy. Limit retries to 10 attempts (500µs maximum wait) shared across the entire batch. The fast path retries sendmsg_x() within this budget, then falls back to the slow path which writes remaining packets one at a time, also retrying within the same budget. Packets are dropped only after the budget is exhausted or on non-ENOBUFS errors. Before (ENOBUFS in fast path): sendmsg_x(batch) → ENOBUFS → sleep 50µs → retry → ENOBUFS → sleep 50µs → retry → ... (unbounded retries, blocks all packets in the batch) After (ENOBUFS in fast path): sendmsg_x(batch) → ENOBUFS → sleep 50µs → retry (up to 10 times, shared budget) → fall back to slow path → write(packet₁) → ENOBUFS → retry (same budget) → write(packet₁) → success → write(packet₂) → ENOBUFS → budget exhausted → drop → write(packet₃) → success (bounded wait across both paths, then drop) The retry limit is per batch, not per sendmsg_x call, ensuring the total wait is bounded regardless of partial successes between retries. Dropping packets is correct because the helper implements an ethernet link between the VM and the host. Like a physical network link under congestion, packets may be lost. TCP retransmits lost segments, and UDP applications are expected to tolerate loss. Bounded retries are preferable to dropping immediately because TCP retransmission is far more expensive than a 50µs sleep. Dropping causes TCP to halve its congestion window and wait for retransmit, taking milliseconds to seconds to recover. In testing, dropping immediately on ENOBUFS caused throughput to collapse from 26 Gbps to 20 Gbps with 4 parallel streams as TCP repeatedly backed off. ENOBUFS occurs when the VM cannot drain the vmnet buffer fast enough. With a fast VMM (krunkit) and --busy-poll, the VM processes packets fast enough that ENOBUFS never occurs even at 36 Gbps: krunkit, --busy-poll, 4 CPUs, 8 MiB buffer: TX (Gbps) drops fast calls/sec before (retry) 36.4 0 14.5k after (drop) 36.4 0 14.5k With a slower VMM (vfkit), the VM cannot keep up and ENOBUFS occurs. The bounded retry mechanism absorbs most ENOBUFS pressure, but some packets are dropped. The previous unbounded retry behavior achieved slightly higher throughput for vfkit by naturally throttling the sending rate: vfkit, --busy-poll, 4 CPUs, 4 MiB buffer: TX (Gbps) drops fast calls/sec before (retry) 9.8 0 15-28k after (drop) 9.4 780 ~28k The slight throughput regression for slow VMMs is acceptable because head-of-line blocking is a correctness issue, not a performance issue. Without bounded retries, a single blocked batch can stall all traffic for an unbounded duration. Based on #266 with the following changes: - Tighter retry budget: 10 retries (500µs) vs 100 (5ms). Benchmarks show retries rarely exceed a few attempts. - Change error logging in sendmsg_x() and write() from ERROR to DEBUG for ENOBUFS (normal backpressure signal) and WARN for other errors (unexpected but recoverable). - Drops tracked via stats counters instead of a separate rate-limited warning log. Thanks: Leo Lännenmäki Fixes: #267 --- programs/helper.c | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/programs/helper.c b/programs/helper.c index 68c4d0b..147d885 100644 --- a/programs/helper.c +++ b/programs/helper.c @@ -54,6 +54,9 @@ // 13 1 | #define VM_RETRY_DELAY (50 * MICROSECOND) +// Maximum number of retries per batch, shared by fast and slow paths. +#define VM_RETRY_MAX 10 + #define ARRAY_SIZE(a) (sizeof(a) / sizeof(a[0])) static const uintptr_t SHUTDOWN_EVENT = 1; @@ -850,6 +853,7 @@ static void write_to_vm(int count) int packets = 0; size_t bytes = 0; uint64_t fast = 0; + int retries = 0; // Fast path. @@ -858,12 +862,19 @@ static void write_to_vm(int count) fast++; ssize_t n = sendmsg_x(options.fd, &host.msgs[packets], count-packets, 0); if (n == -1) { - if (errno == ENOBUFS) { + if (errno == ENOBUFS && retries < VM_RETRY_MAX) { + retries++; wait_for_buffer_space(); continue; } - ERRORF("[host->vm] sendmsg_x: %s", strerror(errno)); + // ENOBUFS after retry budget exhausted, or another error. Fall + // back to slow path for completing this batch. + if (errno == ENOBUFS) { + DEBUGF("[host->vm] sendmsg_x: %s", strerror(errno)); + } else { + WARNF("[host->vm] sendmsg_x: %s", strerror(errno)); + } bytes = host_packets_size(packets); break; } @@ -890,32 +901,32 @@ static void write_to_vm(int count) for (int i = packets; i < count; i++) { struct vmpktdesc *packet = &host.packets[i]; ssize_t len; - uint64_t retries = 0; while (1) { slow++; len = write(options.fd, packet->vm_pkt_iov[0].iov_base, packet->vm_pkt_size); - if (len == -1 && errno == ENOBUFS) { - wait_for_buffer_space(); + if (len == -1 && errno == ENOBUFS && retries < VM_RETRY_MAX) { retries++; + wait_for_buffer_space(); continue; } break; } if (len < 0) { - // TODO: like socket_vmnet we drop the packet and continue. Maybe trigger shutdown? - ERRORF("[host->vm] write: %s", strerror(errno)); + // ENOBUFS after retry budget exhausted, or another error. + if (errno == ENOBUFS) { + DEBUGF("[host->vm] write: %s", strerror(errno)); + } else { + WARNF("[host->vm] write: %s", strerror(errno)); + } drops++; continue; } packets++; bytes += packet->vm_pkt_size; - if (retries > 0) { - DEBUGF("[host->vm] write completed after %lld retries", retries); - } // Partial write should not be possible with datagram socket. assert((size_t)len == packet->vm_pkt_size);