fix(windows): clean stale install locks + fix db-cleanup race between M cells#6
Open
Tejeshyewale wants to merge 1 commit into
Open
Conversation
… M cells - provision.sh: remove stale engine/cache/install.lock and uv-*.lock left by an interrupted previous run, which otherwise makes the engine's dependency bootstrap hang forever on 'Waiting for another installation to complete...' - concurrent-processing/run.py: _clean_dbs() now retries with backoff and verifies deletion instead of silently ignoring errors. On Windows, a just-torn-down warm pool's sqlite handles can still be releasing when the next M cell starts, so the rmtree can silently no-op and leave the previous M's rows behind (observed: M=8's 72 rows + M=16's 80 rows == 152, reported as status=check).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two Windows-specific fixes found while reproducing the concurrent-work benchmarks:
1. Stale lock files hang the engine's dependency bootstrap forever
If a previous
start_engine.shrun is interrupted (Ctrl-C, crash, closed terminal) mid-install, it leaves behindengine/cache/install.lockand/or auv-*.lockin the OS temp dir. The engine's embedded installer then waits on these forever ("Waiting for another installation to complete...") since the process that held them is gone.provision.shnow clears these before (re)provisioning.2.
_clean_dbs()can silently leave stale rows between M cells on Windowsconcurrent-processing/run.py's_clean_dbs()usedshutil.rmtree(DB_DIR, ignore_errors=True)with no verification. On Windows, a just-torn-down warm pool's sqlite file handles can still be releasing when the next M cell starts, so the rmtree can silently no-op. This meant runningBENCH_MS=8,16in one invocation showedrows=152/80for M=16 (M=8's 72 leftover rows + M=16's own 80) instead of a clean80/80. Fixed by retrying with a short backoff and raising if the directory truly can't be cleared.How I found these
Reproducing the concurrent-work benchmarks on Windows 11 (RTX 2050, CUDA 13.1). Both issues were 100% reproducible; fault-isolation, concurrent-processing, and data-isolation all match the README's claims once these are fixed.
Testing
fault-isolation(single + 10x reps),concurrent-processing(M=8,16 both together and separately), anddata-isolation(M=32) after these fixes — all produce clean, expected results.