Skip to content

Fix high CPU usage on UDP GRO receive path - #2069

Open
kvasana wants to merge 1 commit into
esnet:masterfrom
kvasana:gsro-cpu-fix
Open

Fix high CPU usage on UDP GRO receive path#2069
kvasana wants to merge 1 commit into
esnet:masterfrom
kvasana:gsro-cpu-fix

Conversation

@kvasana

@kvasana kvasana commented Aug 17, 2026

Copy link
Copy Markdown

The UDP GRO receive path used recvmsg() with MSG_DONTWAIT, causing the receiver thread to busy-poll and waste CPU without throughput benefit. Make the GRO receive path blocking instead.

Blocking is safe because the server receives in a worker thread with deferred cancellation, while --rcv-timeout is enforced by the main progress loop. On timeout, cleanup_server() cancels and joins the worker; because recvmsg() is a cancellation point, a blocked receive is interrupted and the server exits cleanly.

PLEASE NOTE the following text from the iperf3 license. Submitting a
pull request to the iperf3 repository constitutes "[making]
Enhancements available...publicly":

You are under no obligation whatsoever to provide any bug fixes, patches, or
upgrades to the features, functionality or performance of the source code
("Enhancements") to anyone; however, if you choose to make your Enhancements
available either publicly, or directly to Lawrence Berkeley National
Laboratory, without imposing a separate written license agreement for such
Enhancements, then you hereby grant the following license: a non-exclusive,
royalty-free perpetual license to install, use, modify, prepare derivative
works, incorporate into other computer software, distribute, and sublicense
such enhancements or derivative works thereof, in binary and source code form.

The complete iperf3 license is available in the LICENSE file in the
top directory of the iperf3 source tree.

  • Version of iperf3 (or development branch, such as master or
    3.1-STABLE) to which this pull request applies: master

  • Issues fixed (if any): Follow up to discussion High CPU utilization on RX side with UDP GRO (--gsro) #2065

  • Brief description of code changes (suitable for use as a commit message):
    Make the UDP GRO receive path blocking. The GRO receive (recv_msg_gro in src/net.c) used recvmsg() with MSG_DONTWAIT, causing the receiver thread to busy-poll and saturate a CPU core with no throughput benefit. Blocking is safe because --rcv-timeout (active by default) sets SO_RCVTIMEO, so recvmsg() returns EAGAIN on timeout and Nread_gro() already maps that to 0, letting the server exit cleanly rather than block forever.

Results (representative setup: ARM host, single UDP stream, -b 10G -l 1472, receiver pinned to one core): the pinned core went from ~0.9% idle (saturated) to ~28% idle at the same 9.57 Gbps, with no loss regression. Magnitude of savings varies by environment; always an improvement or neutral across the platforms/versions tested, no throughput or loss regression.

Tested (all pass, no hangs): normal, --rcv-timeout + stalled sender (server reports "idle timeout for receiving data" and recovers), Ctrl-C interrupt, reverse (-R), bidirectional (--bidir), and multiple streams (-P).

@gegles

gegles commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

The one-line blocking change looks good and makes the GRO path consistent with the ordinary UDP receive path.

One correction is needed in the commit message and PR description: --rcv-timeout does not set SO_RCVTIMEO on the accepted server UDP socket. The only SO_RCVTIMEO here is the fixed 30-second timeout in iperf_udp_connect(). On the server, --rcv-timeout is enforced by the main progress loop; when it expires, cleanup_server() cancels and joins the blocked receiver worker.

I verified the Linux GRO build and exercised normal, reverse, bidirectional, parallel, and stalled-sender timeout cases successfully. The code change is sound, but the safety explanation should reflect the actual cancellation path.

The UDP GRO receive path used recvmsg() with MSG_DONTWAIT, causing the
receiver thread to busy-poll and waste CPU without throughput benefit.
Make the GRO receive path blocking instead.

Blocking is safe because the server receives in a worker thread with
deferred cancellation, while --rcv-timeout is enforced by the main
progress loop. On timeout, cleanup_server() cancels and joins the worker;
because recvmsg() is a cancellation point, a blocked receive is
interrupted and the server exits cleanly.

Signed-off-by: Kusuma Vasana <kusuma.vasana@amd.com>
@kvasana

kvasana commented Aug 18, 2026

Copy link
Copy Markdown
Author

Thanks for the review and for verifying the cases — and for the correction. Agreed; updated the commit message and PR description accordingly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants