Skip to content

Pin CI's Rust toolchain and fix three intermittent Windows failures - #162

Merged
JunjoSick merged 4 commits into
mainfrom
fix/ci-reliability
Oct 21, 2026
Merged

JunjoSick merged 4 commits into
mainfrom
fix/ci-reliability

Conversation

@JunjoSick

Copy link
Copy Markdown
Owner

Summary

Four CI-reliability fixes from the 2026-10-20 triage: one prevents a repeat of the Oct 2–20 outage, and three remove intermittent Windows failures, one of which was a real product bug.

  1. ci: pin the Rust toolchain (F10). Every stable job installed dtolnay/rust-toolchain's floating stable while the workflow sets RUSTFLAGS="-D warnings". When Rust 1.99 deprecated a std method, all of CI went red for 18 days without any code change. The six non-MSRV jobs now install the exact version in RUST_TOOLCHAIN (1.99.1) through the master ref the msrv job already pins, so raising it is a deliberate PR. CONTRIBUTING documents running the local gate on that version. msrv stays on 1.88.0.
  2. test(llm): decode-bisection fixture (F7). invalid_json_endpoint read each request with one read() and closed right after answering. On Windows, closing a socket with unread input sends RST, which can discard the reply before the client reads it. The decode error under test then becomes a transport error, so there's no bisection ([2, 2] instead of [2, 2, 1, 1]), or the test hangs forever on server.await. The fixture now reads the whole request head, half-closes, and drains (bounded); both tests bound server.await at 30 s.
  3. test(audio): oversized-response test (F8). Its one-request server was created outside retry_transient_transport, so every retry hit a closed port, and a 5 s client timeout could expire on a busy machine. The server is now created per attempt, as the ElevenLabs tests already do, with a 60 s client timeout.
  4. fix(control): resuming a paused job on Windows. Resume rewrites the worker's control file while the paused worker polls it. atomic_replace used MoveFileExW(MOVEFILE_REPLACE_EXISTING), which fails with ERROR_ACCESS_DENIED whenever another process has the destination open, so bookforge resume intermittently failed with "Access is denied (os error 5)". This was the lifecycle flake cli_pause_during_inflight_qa_request_parks_before_more_finalize_work. std::fs::rename uses POSIX rename semantics on Windows and succeeds in that case. The replace stays atomic and failure-preserving, and std falls back to MoveFileExW where POSIX renames are unsupported. CHANGELOG entry added.

Validation

All on Windows (GNU), Rust 1.99.1 with -D warnings:

  • Root cause of (4) shown directly: a standalone probe holds a file open and replaces it. MoveFileExW fails with Access is denied. (os error 5); std::fs::rename succeeds. Same result on Rust 1.88.0 and 1.99.1. A new #[cfg(windows)] regression test does the same through atomic_replace; the 26 control:: tests pass.
  • (2): the bisection test passed 30/30 alone and the batch::tests module 10/10, against 26/30 and 4/10 on main before.
  • (3): 15/15 alone, and the full bookforge-audio lib passes (123).
  • (4): the pause/resume lifecycle test passed 10/10, and the full lifecycle suite passed 56/56.
  • (1): every non-MSRV toolchain step resolves to ${{ env.RUST_TOOLCHAIN }}. The diff touches only those steps, and validate-release-security.sh passes.
  • The local runs above used the tree before build(deps): bundle dependabot updates #153 #155 #156 #157 #158 #161 (lockfile bumps) merged. This PR's CI covers the combined tree.

During the rebase over #161 I found and fixed a mistake in my own first version of (1): a replace-all had dropped the space after uses: on four steps, which would have made the workflow invalid. That version was never part of an open PR.

🤖 Generated with Claude Code

JunjoSick and others added 4 commits October 21, 2026 16:17
Every stable job installed dtolnay/rust-toolchain's floating `stable`
while the workflow sets RUSTFLAGS="-D warnings". When the runners moved
to Rust 1.99 on 2026-10-02, a new std deprecation failed every CI run
until 2026-10-20 without any change to the code.

The six stable jobs now install the exact version in RUST_TOOLCHAIN
(1.99.1, what main last passed on) through the master ref the msrv job
already pins. A Rust release can no longer change CI's verdict on its
own; raising the pin is a deliberate PR. CONTRIBUTING documents running
the local gate on the same version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
invalid_json_endpoint read each request with a single read() and closed
the socket right after answering. On Windows a socket closed with unread
input sends RST, which can discard the response before the client reads
it: the decode failure under test then surfaces as a transport error, so
the batch never bisects (request counts [2, 2] instead of [2, 2, 1, 1]),
or the fixture waits forever for a request that never comes.

The fixture now reads the whole request head, shuts down its write side
and drains until the client hangs up (bounded). Both tests bound
server.await at 30 s so a missing request fails instead of hanging CI.

Windows GNU, Rust 1.99.1: the bisection test passed 30/30 alone and the
batch::tests module 10/10, against 26/30 and 4/10 on main before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test created its one-request server outside retry_transient_transport,
so any retried transient error reached a closed listener and failed with
an unrelated connection error. It also used a 5 s client timeout, which a
busy test machine can exceed. The server is now created per attempt, as
the ElevenLabs tests already do, and the client timeout is 60 s; the
content-length limit under test does not depend on either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ndows

Resuming a paused job rewrites the worker's control file while the
paused worker polls it with fs::read_to_string. atomic_replace installed
the new file with MoveFileExW(MOVEFILE_REPLACE_EXISTING), which fails
with ERROR_ACCESS_DENIED whenever another process has the destination
open, so `bookforge resume` intermittently failed with "Access is denied
(os error 5)".

std::fs::rename uses POSIX rename semantics on Windows and succeeds in
that case; a standalone probe on Rust 1.88 and 1.99.1 showed MoveFileExW
failing and fs::rename succeeding over a file held open by a reader. The
replace stays atomic and failure-preserving, and std falls back to
MoveFileExW where POSIX renames are unsupported. A Windows regression
test holds the file open across the replace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JunjoSick
JunjoSick merged commit 12a42b8 into main Oct 21, 2026
15 checks passed
@JunjoSick
JunjoSick deleted the fix/ci-reliability branch October 21, 2026 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant