find-gpu-and-train: retry full zone passes on stockout - #56
Open
nima-siboni wants to merge 1 commit into
Open
Conversation
…kout Previously a single pass over the zone list — if every zone was stocked out, the script exited with code 2 and the user had to re-launch. Now wraps the zone loop in an outer retry loop: when a whole pass yields nothing but stockouts, sleep `LORA4VR_RETRY_SLEEP` seconds (default 180) and try again. Non-stockout failures (e.g. quota exhausted) still abort immediately so we don't leave partial VMs behind. Tunables: LORA4VR_RETRY_SLEEP=180 # seconds between passes LORA4VR_MAX_PASSES=0 # 0 = infinite; >0 = cap then exit 2 The pass counter is logged so it's easy to tell how many full sweeps the script has done while waiting for capacity.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
LORA4VR_RETRY_SLEEPseconds (default 180 = 3 min) between passes.LORA4VR_MAX_PASSES(default 0 = infinite) caps the retry count.Quota 'GPUS_ALL_REGIONS' exceeded) still abort immediately — retrying wouldn't help and we don't want to leave partial VMs.Behavior
LORA4VR_RETRY_SLEEPs, run the pass again. The pass counter is included in every zone-attempt log line so it's clear how many full sweeps have happened.Defaults reflect the user's ask: "at the end of the inner loop give it a little bit of pause, 3 mins or so."
Test plan
bash -n scripts/gcp_find_gpu_and_train.sh(syntax OK).LORA4VR_ZONE_LIST=bogus-zone-1 bogus-zone-2 LORA4VR_RETRY_SLEEP=5 LORA4VR_MAX_PASSES=2 bash scripts/gcp_find_gpu_and_train.sh ...— confirm two passes, ~5s between them, exit 2.🤖 Generated with Claude Code