fix(fleet): create the staging worktree before the recut can use it - #442
Merged
Conversation
poll_once ran its staleness re-cut check BEFORE ensure_staging_worktree. recut_squad_branch's first act is checkout_detached(staging_path, ...), which hands staging_path to subprocess as cwd -- so on a host where ~/.oxidex/worktrees/squad-staging/<squad> does not exist yet (a fresh machine, or after stop_parallel_fix.py reaped the worktrees) the recut raised FileNotFoundError before the one function that would have CREATED that worktree ever ran. Measured 2026-08-04: all 14 mergers died within one second of launch. Every squad/<name> branch was stale against a main that had moved 249 commits, so should_recut fired for all of them simultaneously and the whole merger tier went down together -- the failure mode where workers keep producing fixes with no publish path at all. ensure_staging_worktree is idempotent (reuse-in-place on an existing directory), so hoisting it above the recut check costs one cheap reset/checkout and makes the recut path survive a cold start. Every pre-existing poll_once test passes check_recut=False, which is why this ordering survived: the crashing path was never exercised. The new test drives poll_once with check_recut=True against an absent staging directory and reproduces the exact production traceback. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
poll_onceran its staleness re-cut check beforeensure_staging_worktree.recut_squad_branch's first act ischeckout_detached(staging_path, ...), which handsstaging_pathtosubprocessascwd— so on a host where~/.oxidex/worktrees/squad-staging/<squad>doesn't exist yet (a fresh machine, or afterstop_parallel_fix.pyreaped the worktrees), the recut raisesFileNotFoundErrorbefore the one function that would have created that worktree ever runs.Measured 2026-08-04 launching a 20-worker fleet: all 14 mergers died within one second. Every
squad/<name>branch was stale against amainthat had moved 249 commits, soshould_recutfired for all of them simultaneously and the whole merger tier went down together — the configuration where workers keep producing fixes with no publish path at all.Fix
Hoist
ensure_staging_worktreeabove the recut check. It is idempotent — reuse-in-place on an existing directory (reset --hard+clean -fd+checkout),worktree addotherwise — so the move costs one cheap reset/checkout on the common path and makes the recut survive a cold start.Why it went undetected
Every pre-existing
poll_oncetest passescheck_recut=False, so the crashing path was never exercised.RecutSquadBranchTestscallsrecut_squad_branchdirectly with a staging worktree already built.The new test drives
poll_oncewithcheck_recut=Trueagainst an absent staging directory and reproduced the production traceback line-for-line before the fix.Verification
FileNotFoundErrorbefore the change, passes afterpython -m unittest test_squad_merge_loop— 116/116 passjust pre-commitgate (fmt-check, cbindgen-check, lint, test) green🤖 Generated with Claude Code