Skip to content

fix(fleet): create the staging worktree before the recut can use it - #442

Merged
swackhamer merged 1 commit into
mainfrom
fix/merger-staging-worktree-ordering
Aug 5, 2026
Merged

fix(fleet): create the staging worktree before the recut can use it#442
swackhamer merged 1 commit into
mainfrom
fix/merger-staging-worktree-ordering

Conversation

@swackhamer

Copy link
Copy Markdown
Collaborator

Problem

poll_once ran its staleness re-cut check before ensure_staging_worktree. recut_squad_branch's first act is checkout_detached(staging_path, ...), which hands staging_path to subprocess as cwd — so on a host where ~/.oxidex/worktrees/squad-staging/<squad> doesn't exist yet (a fresh machine, or after stop_parallel_fix.py reaped the worktrees), the recut raises FileNotFoundError before the one function that would have created that worktree ever runs.

Measured 2026-08-04 launching a 20-worker fleet: all 14 mergers died within one second. Every squad/<name> branch was stale against a main that had moved 249 commits, so should_recut fired for all of them simultaneously and the whole merger tier went down together — the configuration where workers keep producing fixes with no publish path at all.

File "scripts/squad_merge_loop.py", line 1565, in poll_once
    recut_result = recut_squad_branch(
File "scripts/squad_merge_loop.py", line 1497, in recut_squad_branch
    checkout_detached(staging_path, origin_ref)
FileNotFoundError: [Errno 2] No such file or directory:
    PosixPath('/Users/allen/.oxidex/worktrees/squad-staging/tail')

Fix

Hoist ensure_staging_worktree above the recut check. It is idempotent — reuse-in-place on an existing directory (reset --hard + clean -fd + checkout), worktree add otherwise — so the move costs one cheap reset/checkout on the common path and makes the recut survive a cold start.

Why it went undetected

Every pre-existing poll_once test passes check_recut=False, so the crashing path was never exercised. RecutSquadBranchTests calls recut_squad_branch directly with a staging worktree already built.

The new test drives poll_once with check_recut=True against an absent staging directory and reproduced the production traceback line-for-line before the fix.

Verification

  • New test fails with the exact production FileNotFoundError before the change, passes after
  • python -m unittest test_squad_merge_loop116/116 pass
  • Full just pre-commit gate (fmt-check, cbindgen-check, lint, test) green

🤖 Generated with Claude Code

poll_once ran its staleness re-cut check BEFORE ensure_staging_worktree.
recut_squad_branch's first act is checkout_detached(staging_path, ...),
which hands staging_path to subprocess as cwd -- so on a host where
~/.oxidex/worktrees/squad-staging/<squad> does not exist yet (a fresh
machine, or after stop_parallel_fix.py reaped the worktrees) the recut
raised FileNotFoundError before the one function that would have CREATED
that worktree ever ran.

Measured 2026-08-04: all 14 mergers died within one second of launch.
Every squad/<name> branch was stale against a main that had moved 249
commits, so should_recut fired for all of them simultaneously and the
whole merger tier went down together -- the failure mode where workers
keep producing fixes with no publish path at all.

ensure_staging_worktree is idempotent (reuse-in-place on an existing
directory), so hoisting it above the recut check costs one cheap
reset/checkout and makes the recut path survive a cold start.

Every pre-existing poll_once test passes check_recut=False, which is why
this ordering survived: the crashing path was never exercised. The new
test drives poll_once with check_recut=True against an absent staging
directory and reproduces the exact production traceback.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@swackhamer
swackhamer merged commit 84bbed7 into main Aug 5, 2026
6 of 7 checks passed
@swackhamer
swackhamer deleted the fix/merger-staging-worktree-ordering branch August 5, 2026 02:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant