helper: Drop packets when the peer stops reading from the socket

[leolannenmaki]

When sendmsg_x() or write() fail with ENOBUFS, write_to_vm() retries every
50 microseconds with no bound. The vm socket is a datagram socket, so ENOBUFS
means the peer receive buffer is full: the client (vfkit, krunkit) is not
draining the socket as fast as the host sends. A client that stops reading,
or reads too slowly, blocks write_to_vm() until the client catches up.

All host->vm forwarding runs on one serial queue, so one slow client delays
every host->vm packet behind it, for all flows on the interface, including
latency sensitive frames such as TCP acks and HTTP/2 pings.

A stalled client also prevents the helper from stopping: on SIGTERM the
atexit handler stops the vmnet interface on the same blocked queue, so the
helper never exits and has to be killed.

We hit this with vfkit VMs. During large concurrent downloads into a guest,
the guest drains slowly and the helper holds the queue full for seconds at a time.
HTTP/2 clients inside the guest time out their connection health pings and
close the connection, killing all in-flight streams with "http2: client connection lost".

Fix

Bound the wait to VM_MAX_RETRIES (5 ms), shared by the whole batch. When the
budget is exhausted every remaining packet gets one last write() before being dropped,
since a smaller packet may still fit in the peer receive buffer. Dropping is safe: ethernet
does not guarantee delivery, the guest transport protocols recover from the loss, and
TCP backs off, which keeps the queue short. Dropped packets are counted and logged
at most once per second.

Example

WARN  [host->vm] dropped 11 packets (11 total): peer is not reading from the socket

The first commit adds a stalled client test: it fills its own receive buffer by sending pings
to the gateway without reading the replies, waits for the dropped packets warning in the
helper log, and verifies that forwarding recovers once the client reads again. It fails
without the fix, since the helper never gives up and blocks until the client reads,
and passes with it.

Benchmarked with ./bench (vfkit, shared mode, iperf3): no measurable change.
The bound is never reached at line rate and no packets were dropped.

AI Disclaimer

As you can probably see from the description text above, LLMs were used
in generating this PR description. They were also used in debugging the problem in
the first place and in coming up with the failing test and actual fix.

I'm not that interested getting this exact PR merged as I'm in letting the maintainers
know of the issue we ran into and the, AI generated, workaround we found in
hopes of getting some fix landed.

Happy to provide more debugging assistance if need be.

[nirs]

@leolannenmaki Thanks for the PR!

When sendmsg_x() or write() fail with ENOBUFS, write_to_vm() retries every 50 microseconds with no bound. The vm socket is a datagram socket, so ENOBUFS means the peer receive buffer is full: the client (vfkit, krunkit) is not draining the socket as fast as the host sends. A client that stops reading, or reads too slowly, blocks write_to_vm() until the client catches up.

All host->vm forwarding runs on one serial queue, so one slow client delays every host->vm packet behind it, for all flows on the interface, including latency sensitive frames such as TCP acks and HTTP/2 pings.

I agree the current behaviour is simplistic, but I never seen an issue in real use. libkrun uses a similar solution - packets are never dropped.

A stalled client also prevents the helper from stopping: on SIGTERM the atexit handler stops the vmnet interface on the same blocked queue, so the helper never exits and has to be killed.

This is easy to fix, we can check if signal was received in the wait loop.

We hit this with vfkit VMs. During large concurrent downloads into a guest, the guest drains slowly and the helper holds the queue full for seconds at a time. HTTP/2 clients inside the guest time out their connection health pings and close the connection, killing all in-flight streams with "http2: client connection lost".

This is the most important thing - can you open an issue and explain how to reproduce this and which real world use cases are affected?

Fix

Bound the wait to VM_MAX_RETRIES (5 ms), shared by the whole batch. When the budget is exhausted every remaining packet gets one last write() before being dropped, since a smaller packet may still fit in the peer receive buffer. Dropping is safe: ethernet does not guarantee delivery, the guest transport protocols recover from the loss, and TCP backs off, which keeps the queue short. Dropped packets are counted and logged at most once per second.

Makes sense to limit the time we wait. Do you know how real switches handle the same? how much time a packet is waiting in the switch buffers when the receiver is not fast enough?
vmnet-helper should behave in a similar way by default, and maybe have a configurable delay.

[nirs]

Fixed in #278