EAGAIN write-backoff busy-spins at 100% CPU — $maxwrite is always 0 in _send_bytes/_optimal_sleep

[pReya]

Summary

When a syswrite on the IMAP socket returns EAGAIN (send buffer temporarily full), _send_bytes() is supposed to wait a little and retry, using an adaptive backoff in _optimal_sleep(). Instead, the wait time collapses toward zero and the retry loop turns into a CPU busy-spin — thousands of syswrite/select iterations per second while a message is being sent.

The adaptive backoff never actually works, because the value it tunes against ($maxwrite) is always 0.

Root cause

Two bugs in the same mechanism:

1. $maxwrite is initialized to 0 and never updated.

In _send_bytes():

sub _send_bytes($) {
    my ( $self, $byteref ) = @_;
    my ( $total, $temperrs, $maxwrite ) = ( 0, 0, 0 );   # $maxwrite = 0 ...
    ...
    while ( $total < length $$byteref ) {
        my $written = syswrite( $socket, $$byteref, length($$byteref) - $total, $total );
        if ( defined $written ) {
            $temperrs = 0;
            $total += $written;                          # ... and never reassigned
            next;
        }
        if ( $! == EAGAIN ) {
            ...
            $waittime = $self->_optimal_sleep( $maxwrite, $waittime, \@previous_writes );
            next;
        }
        ...
    }
}

$maxwrite is passed to _optimal_sleep() on every EAGAIN, but it's always 0.

2. _optimal_sleep() compares against 0, so the wait halves every time.

sub _optimal_sleep($$$) {
    my ( $self, $maxwrite, $waittime, $last5writes ) = @_;

    push @$last5writes, $waittime;                        # pushes the WAIT TIME ...
    shift @$last5writes if @$last5writes > 5;

    my $bufferavail = ( sum @$last5writes ) / @$last5writes;  # ... so this is avg wait, not bytes

    if ( $bufferavail < .4 * $maxwrite ) {   # .4 * 0 = 0  -> false
        $waittime *= 1.3;
    }
    elsif ( $bufferavail > .9 * $maxwrite ) { # .9 * 0 = 0  -> always TRUE
        $waittime *= .5;                      # -> waittime halved on every EAGAIN
    }

    CORE::select( undef, undef, undef, $waittime );
    $waittime;
}

With $maxwrite == 0, the first if can never be true and the elsif is always true, so $waittime is halved on every call:

0.02, 0.01, 0.005, 0.0025, ... -> ~0

select(undef, undef, undef, ~0) returns immediately, so _send_bytes retries syswrite as fast as the CPU allows.

There's also a secondary defect: @last5writes is meant to hold recent write sizes (so $bufferavail can be compared to $maxwrite), but _optimal_sleep pushes the wait time into it. So even if $maxwrite were nonzero, $bufferavail would be comparing wait-times against byte-counts. (Seeding the array with realistic write sizes actually makes $waittime explode by ×1.3 per call, i.e. multi-second select() sleeps — the same broken logic in the other direction.)

Impact

• 100% CPU busy-spin whenever the send buffer backs up (slow/tls destination, large messages, APPEND of big bodies).
• The documented adaptive backoff in _optimal_sleep never engages.
• On macOS this busy-spin is especially damaging: each spin iteration reads $! after the failed syswrite, and macOS's locale-aware strerror_l() allocates on every read under a UTF-8 LC_MESSAGES. Millions of allocate/free cycles per second outrun the allocator's scavenger and RSS balloons into the GBs. This is the underlying cause of imapsync issue #312 (200GB+ memory usage on Mac M1 imapsync/imapsync#312), where imapsync's --darwinfix01/02 options work around it by injecting 1 ms sleeps — i.e. by throttling this exact loop.

Reproduction

Minimal, no server needed (Linux/macOS). Fills a non-blocking socket so every write returns EAGAIN, then drives the real _send_bytes bounded by Maxtemperrors and measures the retry rate:

use strict; use warnings;
use Mail::IMAPClient;
use Socket; use Fcntl;
use Time::HiRes qw(time);

socketpair(my $w, my $r, AF_UNIX, SOCK_STREAM, PF_UNSPEC) or die "socketpair: $!";
my $fl = fcntl($w, F_GETFL, 0); fcntl($w, F_SETFL, $fl | O_NONBLOCK);
1 while defined syswrite($w, "x" x 65536);        # fill the send buffer

my $imap = Mail::IMAPClient->new;
$imap->RawSocket($w);
$imap->Maxtemperrors(100);                         # bound the loop so it returns

my $payload = "y" x 1000;
my $t0 = time;
$imap->_send_bytes(\$payload);                      # returns undef after 100 EAGAINs
my $dt = time - $t0;
printf "100 EAGAIN retries in %.4fs => %.0f retries/sec\n", $dt, 100/$dt;

Observed on 3.43:

100 EAGAIN retries in 0.0250s => 4003 retries/sec

i.e. ~4000 retries/second — the retry loop is spinning, not waiting.

Disclaimer: This bug report was generated with an LLM. I have however manually looked at the code and confirmed the suspicion to the best oft my abilities.