Fix high CPU usage on UDP GRO receive path - #2069
Conversation
|
The one-line blocking change looks good and makes the GRO path consistent with the ordinary UDP receive path. One correction is needed in the commit message and PR description: I verified the Linux GRO build and exercised normal, reverse, bidirectional, parallel, and stalled-sender timeout cases successfully. The code change is sound, but the safety explanation should reflect the actual cancellation path. |
The UDP GRO receive path used recvmsg() with MSG_DONTWAIT, causing the receiver thread to busy-poll and waste CPU without throughput benefit. Make the GRO receive path blocking instead. Blocking is safe because the server receives in a worker thread with deferred cancellation, while --rcv-timeout is enforced by the main progress loop. On timeout, cleanup_server() cancels and joins the worker; because recvmsg() is a cancellation point, a blocked receive is interrupted and the server exits cleanly. Signed-off-by: Kusuma Vasana <kusuma.vasana@amd.com>
|
Thanks for the review and for verifying the cases — and for the correction. Agreed; updated the commit message and PR description accordingly. |
The UDP GRO receive path used recvmsg() with MSG_DONTWAIT, causing the receiver thread to busy-poll and waste CPU without throughput benefit. Make the GRO receive path blocking instead.
Blocking is safe because the server receives in a worker thread with deferred cancellation, while --rcv-timeout is enforced by the main progress loop. On timeout, cleanup_server() cancels and joins the worker; because recvmsg() is a cancellation point, a blocked receive is interrupted and the server exits cleanly.
PLEASE NOTE the following text from the iperf3 license. Submitting a
pull request to the iperf3 repository constitutes "[making]
Enhancements available...publicly":
The complete iperf3 license is available in the
LICENSEfile in thetop directory of the iperf3 source tree.
Version of iperf3 (or development branch, such as
masteror3.1-STABLE) to which this pull request applies: masterIssues fixed (if any): Follow up to discussion High CPU utilization on RX side with UDP GRO (--gsro) #2065
Brief description of code changes (suitable for use as a commit message):
Make the UDP GRO receive path blocking. The GRO receive (recv_msg_gro in src/net.c) used recvmsg() with MSG_DONTWAIT, causing the receiver thread to busy-poll and saturate a CPU core with no throughput benefit. Blocking is safe because --rcv-timeout (active by default) sets SO_RCVTIMEO, so recvmsg() returns EAGAIN on timeout and Nread_gro() already maps that to 0, letting the server exit cleanly rather than block forever.
Results (representative setup: ARM host, single UDP stream, -b 10G -l 1472, receiver pinned to one core): the pinned core went from ~0.9% idle (saturated) to ~28% idle at the same 9.57 Gbps, with no loss regression. Magnitude of savings varies by environment; always an improvement or neutral across the platforms/versions tested, no throughput or loss regression.
Tested (all pass, no hangs): normal, --rcv-timeout + stalled sender (server reports "idle timeout for receiving data" and recovers), Ctrl-C interrupt, reverse (-R), bidirectional (--bidir), and multiple streams (-P).