task #751: failover relaunch syncs new pod into pods.conf + recovery commands create-missing#561
Open
superkaiba wants to merge 554 commits into
Open
task #751: failover relaunch syncs new pod into pods.conf + recovery commands create-missing#561superkaiba wants to merge 554 commits into
superkaiba wants to merge 554 commits into
Conversation
## Code-Reviewer Verdict — PASS [task.py]
title: 'A cross-model context-vector patch fails to regi [task.py]
… does not beat the inst [task.py]
…e9 phase=done after 211s. [task.py]
## Code-Reviewer Verdict — PASS [task.py]
# Codex Code Review: SKILL [task.py]
… effort=high write=False po [task.py]
… effort=high write=False po [task.py]
**Task:** #755 — reconciler must persi [task.py]
[task.py]
…gister above i [task.py]
[task.py]
…jg phase=done after 181s. [task.py]
…e (original-body.md snapshott [task.py]
…e per skill spec. The [task.py]
## Reconciler Verdict — FA [task.py]
**Blocker tags:** none **Tier:** trunk **D [task.py]
# Codex Code Review: Step [task.py]
Edited `.claude/agents/analyzer.md` ( [task.py]
[task.py]
…c/fact behavioral-E and re-fold the [task.py]
…ubagent=clean-res [task.py]
…ctive task (statu [task.py]
…ctive task (statu [task.py]
…47 phase=done after 332s. [task.py]
…ctive task (statu [task.py]
…ctive task (statu [task.py]
…pt phase=done after 211s. [task.py]
# Codex Code Review: daily [task.py]
Audited against the original bod [task.py]
…unioned items landed cleanly; [task.py]
…ow (stall detecte [task.py]
[task.py]
Workflow-fix #759 lands cleanly across fi [task.py]
[task.py]
… effort=high write=False po [task.py]
…_warn=1 (c8 kill-cri [task.py]
…issue_697.md commit=d6b8fc3b748218 [task.py]
## Code-Reviewer Verdict — PASS [task.py]
followup_label: input-pca-robustness-cC-to [task.py]
# Codex Code Review: Decod [task.py]
…rom single-variable- [task.py]
…v1 phase=done after 241s. [task.py]
…p/issue-751-plan-v1. [task.py]
…eshold [task.py]
…ra path): PASS.** ** [task.py]
Path used: scratch-worktree cherry-pick ( [task.py]
…itique-codex v2 --> ## Clean-Result [task.py]
[task.py]
…onomous plan-gate (tas [task.py]
Audited against original body.md [task.py]
## Reconciler Verdict — FA [task.py]
…mands create-missing Close a three-link recovery gap that bit pod-664/697/658: a failover-relaunched pod could be RUNNING on the live RunPod API yet absent from pods.conf, the SSH poller then read status=dead on the transport failure, and `pod.py config --update/--refresh-from-api` refused to create the missing row. - pod_config.py: cmd_refresh_from_api + cmd_update CREATE a missing pods.conf row from the live API / user values instead of sys.exit(1); genuine typo/gone still fails loud. (_pod_row_from_live helper.) - backend_poll.py: both failover success paths (_failover_dead_gcp_to_runpod, _relaunch_fresh_runpod) best-effort re-register the new pod after the authoritative sidecar re-point, fail-soft (never derails the running JSON). - poll_pipeline.py: an absent-from-pods.conf pod the live API reports RUNNING is "still registering" -> suppress the dead verdict for up to DEAD_SUPPRESSION_CAP consecutive ticks; a pod IN pods.conf that probes dead stays dead. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes task #751.
Workflow-fix: a GCP→RunPod (and RunPod-wedge) failover relaunched a fresh pod but never registered it in
pods.conf, so SSH/MCP couldn't reach it, the raw-SSH poller false-declaredstatus=dead, and the documented recovery commandspod.py config --update/--refresh-from-apiBOTH errored "pod '' not found in pods.conf". Three additive fixes:scripts/pod_config.py):--updateand--refresh-from-apiCREATE a missingpods.confrow when the pod is RUNNING on the live RunPod API (refresh) or when both--host+--portare given (update), instead ofsys.exit(1). Genuinely-absent/not-RUNNING still fails loud.scripts/backend_poll.py): both failover paths (_failover_dead_gcp_to_runpod,_relaunch_fresh_runpod) call a best-effort registration helper after the authoritative sidecar re-point; the helper NEVER raises (terminal-JSON contract preserved).scripts/poll_pipeline.py): when SSH probe fails for a pod absent frompods.confAND the live API reports RUNNING, suppress thedeadverdict for up toDEAD_SUPPRESSION_CAP(=20 = 2×SSH_FAIL_REFRESH_THRESHOLD, env-overridable) consecutive ticks, then fall through. Pod INpods.confthat probes dead stays dead (scope guard).Reviewed: code-reviewer PASS (no blockers); all 3 ACs realized + tested; 256 tests pass + ruff + workflow_lint clean. Plan auto-approved at 0 GPU-h.
3 sessions reproduced this gap today: pod-664, pod-697, pod-658 (fingerprint
wf-fix-fp:1e207e2050f6).