From 3c9da052737e3c9f69f86742c9255b71959028d5 Mon Sep 17 00:00:00 2001 From: Manas Srivastava Date: Sat, 30 May 2026 16:49:13 +0530 Subject: [PATCH] =?UTF-8?q?ci:=20bump=20kubectl=20rollout-status=20timeout?= =?UTF-8?q?=20180s=20=E2=86=92=20300s?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Today's #198 deploy timed out at 180s even though the new ReplicaSet came up healthy (rollout actually took ~240s on a fresh node with a cold image pull). The job exited red, prod was correct, and the false-failure forced a manual rerun + on-call investigation. Bumping to 300s removes the rerun cycle. Same change ships simultaneously in worker + provisioner deploy.yml. Healthz verify loop extended 6×5s → 12×5s for the same reason: if rollout takes the full timeout, the new pods are only just live when verify starts and the public LB needs a few seconds to route. Co-Authored-By: Claude Opus 4.7 (1M context) --- .github/workflows/deploy.yml | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index d6546220..c1421a15 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -277,10 +277,14 @@ jobs: "deployment/${K8S_DEPLOYMENT}" \ "${K8S_CONTAINER}=${IMAGE}" \ -n "${K8S_NAMESPACE}" + # 2026-05-30: bumped 180s → 300s after a real rollout took ~240s + # (image pull + readiness on a fresh node) and the workflow timed + # out even though the pod came up healthy. The job exited red but + # prod was correct, so the false-failure forced a manual rerun. kubectl rollout status \ "deployment/${K8S_DEPLOYMENT}" \ -n "${K8S_NAMESPACE}" \ - --timeout=180s + --timeout=300s - name: Verify rolled-out image tag matches built version run: | @@ -297,8 +301,10 @@ jobs: - name: Curl live /healthz and confirm new SHA is reported run: | SHORT_SHA="${{ steps.meta.outputs.short_sha }}" - # Allow up to ~30s for the new pod to start serving the public URL. - for i in 1 2 3 4 5 6; do + # 2026-05-30: extended 30s → 60s to match the rollout-timeout bump. + # If rollout took the full 5 min, the new pods are only just live + # when this step starts and the public LB needs a few s to route. + for i in $(seq 1 12); do BODY=$(curl -fsSL --max-time 5 "${HEALTHZ_URL}" || echo "") echo "Attempt ${i}: ${BODY}" if echo "${BODY}" | grep -q "${SHORT_SHA}"; then