feat(ops): queue autoscaler, staged ramp, stalled analysis remediation, CNI restart recovery#83
Open
anchapin wants to merge 1 commit into
Open
feat(ops): queue autoscaler, staged ramp, stalled analysis remediation, CNI restart recovery#83anchapin wants to merge 1 commit into
anchapin wants to merge 1 commit into
Conversation
…ed analysis remediation, CNI restart recovery
Add new operational scripts for large-scale batch run management:
scripts/queue-autoscale-loop.sh:
- Continuously adjusts worker HPA minReplicas and web-background replicas
based on queue depths and running worker count
- Supports ratio-based simulation queue floor (simulations per worker)
- Optional MANAGE_WEB_BACKGROUND for direct scale control or deference to HPA
scripts/worker-staged-ramp.sh:
- Ramps HPA maxReplicas through configurable stages (2k, 5k, 10k, 15k)
- Validates readiness ratio, non-running pod counts, sandbox failure rate,
and CreateContainerError counts before advancing each stage
- Auto-rolls back to last stable stage on guardrail breach
scripts/remediate-stalled-analyses.sh:
- Detects analyses stuck in na/queued state with no queue progress
- Optionally re-triggers start for stalled analyses via REST API
scripts/clean-data.sh:
- Safely delete projects, analyses, and associated data for a fresh start
- Supports dry-run mode, PV deletion, and namespace targeting
scripts/health-remediation-loop.sh updates:
- Add CNI sandbox timeout remediation: detect FailedCreatePodSandBox +
DeadlineExceeded, restart aws-node on affected node with cooldown
- Update default HPA parameters to match 15k scaling targets
- Add ENFORCE_WORKER_HPA_MAX_REPLICAS flag (0 = disabled)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds four new operational scripts for managing large-scale batch simulation runs, plus significant improvements to the existing health remediation loop.
New Scripts
scripts/queue-autoscale-loop.shContinuously adjusts worker HPA
minReplicasandweb-backgroundreplicas based on live queue depths:web-backgroundreplica management (vs deferring to HPA)scripts/worker-staged-ramp.shRamps worker HPA
maxReplicasthrough stages (default: 2k → 5k → 10k → 15k):CreateContainerError counts, and Terminating pod count before advancing
scripts/remediate-stalled-analyses.shDetects analyses stuck in
na/queuedstate with no queue activity:--executeflag to act--analysis-idfor targeted remediationscripts/clean-data.shSafely deletes projects, analyses, Redis queues, and optionally PVs for a fresh deployment start. Supports dry-run, force mode, and PV deletion.
Updated:
scripts/health-remediation-loop.shFailedCreatePodSandBox + DeadlineExceeded, restartsaws-nodeon theaffected node with a configurable cooldown (
CNI_RESTART_COOLDOWN_SECONDS)MAX_CNI_RESTARTS_PER_CYCLE)ENFORCE_WORKER_HPA_MAX_REPLICASflag (0 = disabled, N = enforce max)