From 356647522c21beffbd57d5f9ae75ee89c2861016 Mon Sep 17 00:00:00 2001 From: Vlad Tudose Date: Thu, 6 Aug 2026 23:30:30 +0000 Subject: [PATCH] backend: initialize comp_time before the rate checks read it do_io() declared comp_time uninitialized inside the IO loop. It is only stamped by wait_for_completions() (called when the queue is full or when polling) and by io_queue_event()'s inline-completion path. An async engine on a block device with a rate limit takes neither: submissions return FIO_Q_QUEUED and completions are reaped during the rate-throttle sleep via io_u_quiesce(), which does not touch comp_time. Once bytes_done is non-zero, check_min_rate() (enabled by rate_iops_min / ratemin) computes mtime_since(&td->start, &comp_time) from whatever bytes happen to be on the stack, and when the garbage tv_nsec is out of range rel_time_since() aborts: fio: gettime.c:530: rel_time_since: Assertion `0 <= nsec && nsec < 1000ULL * 1000 * 1000' failed. This crashed every run of a rate-limited randwrite latency workload (libaio, iodepth=16, rate_iops=500, rate_iops_min=500, write_lat_log) ~100 ms after start on our EC2 test fleet, deterministically per host: the stack leftovers at that slot depend on the preceding code paths, so the same binary can crash on one fleet and run clean on another. When the garbage is numerically benign there is no crash, but the min-rate settle arithmetic is still computed from garbage. Hoist comp_time to function scope and seed it with fio_gettime() at do_io() entry (the previous declaration was loop-scoped, so it was fresh garbage on every iteration). A seed at loop start is safe: until the first real completion stamp the settle check just stays inside its 2-second settle window, which is the intended behavior for the start of a job anyway. handle_thinktime() only ever writes the passed time, so check_min_rate() is the sole reader. Tested with the workload above against a loop device: runs complete cleanly with correct sample counts (5000 lat samples for a 10s run at 500 IOPS), and seeding comp_time with the exact garbage values recovered from a production core dump no longer has any code path to reach the rate checks. Fixes: https://github.com/axboe/fio/issues/2128 Signed-off-by: Vlad Tudose --- backend.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/backend.c b/backend.c index 35f522e597..45d18107ca 100644 --- a/backend.c +++ b/backend.c @@ -1155,6 +1155,9 @@ static void do_io(struct thread_data *td, uint64_t *bytes_done) unsigned int i; int ret = 0; uint64_t total_bytes, bytes_issued = 0; + struct timespec comp_time; + + fio_gettime(&comp_time, NULL); for (i = 0; i < DDIR_RWDIR_CNT; i++) bytes_done[i] = td->bytes_done[i]; @@ -1197,7 +1200,6 @@ static void do_io(struct thread_data *td, uint64_t *bytes_done) while ((td->o.read_iolog_file && !flist_empty(&td->io_log_list)) || (!flist_empty(&td->trim_list)) || !io_issue_bytes_exceeded(td) || td->o.time_based) { - struct timespec comp_time; struct io_u *io_u; int full; enum fio_ddir ddir;