Limit ENOBUFS retries to avoid head-of-line blocking

[nirs]

When the kernel's vmnet buffer is full, the helper retries sendmsg_x() with a 50 microsecond sleep between attempts. Previously the retries were unbounded, causing head-of-line blocking: a high-priority small packet (e.g. an HTTP/2 PING or TCP ACK) queued behind large data packets is delayed without bound while the helper waits for buffer space. If the delay exceeds the protocol's timeout, the connection breaks even though the network is otherwise healthy.

Limit retries to 10 attempts (500µs maximum wait) shared across the entire batch. The fast path retries sendmsg_x() within this budget, then falls back to the slow path which writes remaining packets one at a time, also retrying within the same budget. Packets are dropped only after the budget is exhausted or on non-ENOBUFS errors.

Before (ENOBUFS in fast path):

sendmsg_x(batch) → ENOBUFS
→ sleep 50µs → retry → ENOBUFS
→ sleep 50µs → retry → ...
(unbounded retries, blocks all packets in the batch)

After (ENOBUFS in fast path):

sendmsg_x(batch) → ENOBUFS
→ sleep 50µs → retry (up to 10 times, shared budget)
→ fall back to slow path
→ write(packet₁) → ENOBUFS → retry (same budget)
→ write(packet₁) → success
→ write(packet₂) → ENOBUFS → budget exhausted → drop
→ write(packet₃) → success
(bounded wait across both paths, then drop)

The retry limit is per batch, not per sendmsg_x call, ensuring the total wait is bounded regardless of partial successes between retries.

Dropping packets is correct because the helper implements an ethernet link between the VM and the host. Like a physical network link under congestion, packets may be lost. TCP retransmits lost segments, and UDP applications are expected to tolerate loss.

Bounded retries are preferable to dropping immediately because TCP retransmission is far more expensive than a 50µs sleep. Dropping causes TCP to halve its congestion window and wait for retransmit, taking milliseconds to seconds to recover. In testing, dropping immediately on ENOBUFS caused throughput to collapse from 26 Gbps to 20 Gbps with 4 parallel streams as TCP repeatedly backed off.

ENOBUFS occurs when the VM cannot drain the vmnet buffer fast enough. With a fast VMM (krunkit) and --busy-poll, the VM processes packets fast enough that ENOBUFS never occurs even at 36 Gbps:

krunkit, --busy-poll, 4 CPUs, 8 MiB buffer:

                 TX (Gbps)  drops   fast calls/sec
before (retry)       36.4       0          14.5k
after (drop)         36.4       0          14.5k

With a slower VMM (vfkit), the VM cannot keep up and ENOBUFS occurs. The bounded retry mechanism absorbs most ENOBUFS pressure, but some packets are dropped. The previous unbounded retry behavior achieved slightly higher throughput for vfkit by naturally throttling the sending rate:

vfkit, --busy-poll, 4 CPUs, 4 MiB buffer:

                 TX (Gbps)  drops   fast calls/sec
before (retry)        9.8       0          15-28k
after (drop)          9.4     780          ~28k

The slight throughput regression for slow VMMs is acceptable because head-of-line blocking is a correctness issue, not a performance issue. Without bounded retries, a single blocked batch can stall all traffic for an unbounded duration.

Based on #266 with the following changes:

• Tighter retry budget: 10 retries (500µs) vs 100 (5ms). Benchmarks show retries rarely exceed a few attempts.
• Change error logging in sendmsg_x() and write() from ERROR to DEBUG for ENOBUFS (normal backpressure signal) and WARN for other errors (unexpected but recoverable).
• Drops tracked via stats counters instead of a separate rate-limited warning log.

Thanks: Leo Lännenmäki leo.lannenmaki@avrea.com
Fixes: #267

[nirs]

Benchmark: bounded retries vs unbounded retries

Test setup: macOS (freshly rebooted), iperf3 from host to VM with 4
parallel streams, no rate limit, 600 seconds. 4 CPUs, --busy-poll enabled,
Ubuntu 26.04 (kernel 7.0). Before = unbounded retries (previous commit), After
= bounded retries (this commit, VM_RETRY_MAX=10, 500µs budget per batch).

krunkit (offloading enabled, ~39 Gbps)

Metric (host→vm)	Before	After	Change
Bitrate	39.0 Gbps	38.8 Gbps	—
Packets	~100k/s	~101k/s	—
Fast calls	~15.4k/s	~15.5k/s	—
Slow calls	0/s	0.1/s	—
Drops (total/600s)	0	23	—

With offloading, the helper sends large TSO/GSO segments, so the VM
drains the vmnet buffer fast enough that ENOBUFS rarely occurs.
Over 600 seconds at ~39 Gbps, only 23 packets were dropped — a
negligible loss rate that TCP handles transparently.

Bitrate and drops

Efficiency (fast and slow calls/sec)

vfkit (no offloading, ~12 Gbps)

Metric (host→vm)	Before	After	Change
Bitrate	12.4 Gbps	12.2 Gbps	-2%
Packets	~1,022k/s	~1,010k/s	—
Fast calls	~27k/s	~30k/s	+11%
Slow calls	0/s	0/s	—
Drops (total/600s)	0	0	—

Without offloading, the helper sends individual MTU-sized packets.
After a fresh reboot, vfkit achieves ~12 Gbps. At this throughput, ENOBUFS does
not occur and no packets are dropped. Performance is essentially identical.

The higher fast calls/sec in the "after" run reflects more helper
calls per second to achieve the same throughput, likely due to minor
system state differences between runs.

Bitrate and drops

Efficiency (fast and slow calls/sec)

Summary

With offloading (krunkit): no measurable difference at ~39 Gbps over
600 seconds. ENOBUFS is rare (23 drops total), handled transparently
by TCP.

Without offloading (vfkit): no measurable difference at ~12 Gbps.
After a fresh reboot, ENOBUFS does not occur at this throughput.

The key improvement is that no batch can block for more than ~500µs,
eliminating head-of-line blocking for time-sensitive packets. The
bounded retry mechanism has negligible impact on throughput in both
configurations.