Skip to content

fix(mt#4985): Make the timeout classification name the error that actually arrived #12082

fix(mt#4985): Make the timeout classification name the error that actually arrived

fix(mt#4985): Make the timeout classification name the error that actually arrived #12082

Workflow file for this run

name: CI
on:
push:
branches: [main]
pull_request:
branches: [main]
# Manual retrigger for diagnostic verification only — runs the same job
# steps but does NOT satisfy branch protection's required `build` check
# (manual runs aren't `pull_request` events). Use `--ref task/<branch>`
# explicitly; without it, this dispatches against `main`. mt#1469.
workflow_dispatch:
env:
# Secondary hedge for the mt#3623 install flake: disables the streaming
# tarball path implicated upstream (oven-sh/bun#34821; fix PR #34827 is still
# unmerged, so no release carries a fix). Measured at 0 failures in 10
# cold-cache installs with the flag set, against a 3-in-10 unmitigated
# baseline -- mt#3623 records both runs and the method. The per-step retry
# loops remain the load-bearing mitigation.
# Remove both when a bun release carries the upstream fix.
BUN_FEATURE_FLAG_DISABLE_STREAMING_INSTALL: "1"
jobs:
build:
runs-on: ubuntu-latest
# mt#4300: without this, GitHub's 6-hour default applies, and a hung network
# step holds a REQUIRED check `in_progress` — indistinguishable from a slow
# build at the check-list level — for up to 6 hours. Measured healthy run
# 2026-08-19: 9m28s (run 32284790015).
#
# Sizing rule for every job cap in this file (PR #3164 R1): observed healthy
# duration + the LARGEST single step-level cap in the job. A job cap tighter
# than that fires before the step cap does, which loses the whole benefit of
# the step caps — the job-level error says only "job timed out" while the step
# cap names the step that hung. Here: 9.5 + 15 = 24.5, so 30.
timeout-minutes: 30
steps:
- uses: actions/checkout@v4
- name: Setup Bun
uses: oven-sh/setup-bun@v2
with:
bun-version: "1.3.14"
- name: Cache Bun install (shared root + services/reviewer)
uses: actions/cache@v4
with:
path: ~/.bun/install/cache
key: bun-install-${{ runner.os }}-${{ hashFiles('bun.lock', 'services/reviewer/bun.lock') }}
restore-keys: |
bun-install-${{ runner.os }}-
# mt#4300 (PR #3164 R1): the mt#3623 loop retries but never bounded an
# attempt, so a hung registry connection held this step open on the job's
# budget alone. No per-attempt `timeout` here — `bun install` is resumable and
# cache-warm, so killing it mid-fetch is worse than letting it finish; the
# step-level backstop is the right granularity for this one.
- name: Install dependencies
timeout-minutes: 15
run: for i in 1 2 3; do if bun install --frozen-lockfile --no-progress; then break; fi; if [ "$i" = 3 ]; then exit 1; fi; echo "bun install failed (mt#3623 tarball flake) - retry $i"; sleep 5; done
# ripgrep is a RUNTIME dependency of session.grep_search (mt#3163), which
# spawns `rg` directly — it is not a test-only convenience. ubuntu-latest
# does not ship it, which is why that tool had no CI coverage at all until
# mt#3163 added tests that surfaced the gap by failing loudly instead of
# skipping. Installed from the distro repo rather than fetching a release
# binary, so no new download origin enters the CI supply chain.
#
# mt#4300: bounded + retried. The per-attempt bound is `timeout` INSIDE the
# loop, not the step-level `timeout-minutes` — the latter kills the step, it
# does not retry it. 90s is ~9x the measured healthy duration (10s for
# update+install, run 32284790015 2026-08-19T18:00:09→18:00:19Z), so it
# cannot fire on a merely slow mirror. Bounding is what makes the retry
# meaningful: an unbounded retry against a hung mirror just hangs three times.
#
# `sudo timeout`, NOT `timeout sudo`, and the two commands are bounded
# SEPARATELY rather than grouped under one `timeout`. Both choices are about
# signal delivery: `timeout` signals its DIRECT child, so it has to be the
# parent of the apt-get that actually hangs. `timeout sudo apt-get` makes sudo
# the child, and `sudo timeout 180 sh -c '... && ...'` makes sh the child —
# in both cases SIGTERM can stop at the intermediary and the hung apt-get
# keeps running. Separate bounds cost a longer worst case and are the form
# that actually kills the process.
#
# The step-level backstop MUST exceed the loop's worst case or it truncates
# the retries it exists to protect: 3 attempts x (90s + 90s) + 2 x 10s sleep
# = 560s = 9.3 min, so 12. (PR #3164 R1 caught this at 5 — the third attempt
# could never have run. Observed on this PR's own CI: a degraded mirror took
# the step to 2m53s, so the margin is not hypothetical.) It is a true backstop
# — it only fires if a process escapes its inner `timeout`.
- name: Install ripgrep (required by session.grep_search)
timeout-minutes: 12
run: |
for i in 1 2 3; do
if sudo timeout 90 apt-get update && sudo timeout 90 apt-get install -y ripgrep; then break; fi
if [ "$i" = 3 ]; then exit 1; fi
echo "ripgrep install failed or hung (mt#4300) - retry $i"; sleep 10
done
- name: Check formatting
run: bun run format:check
- name: Lint (strict — fails on any warning)
run: bun run lint:strict
# bun run typecheck covers the root tsconfig AND src/cockpit/web/tsconfig.json
# (mt#2424) — the cockpit frontend has its own dedicated typecheck project since
# the root tsconfig excludes it (browser-only lib/DOM settings) and `vite build`
# transpiles it without type-checking.
# mt#3476: the injected skill listing is charged against a shared cumulative
# budget, and going over it silently drops descriptions — a skill listed
# name-only cannot be routed to. Fails on an OWNED description over the cap or
# on the listing total; vendored descriptions are reported, not gated (their
# text is upstream-owned and reverts on refetch, per ADR-015).
- name: Skill listing description budget
run: bun run skills:budget:check
- name: Type check
run: bun run typecheck
- name: Type check (hooks)
run: bun run typecheck:hooks
- name: Type check (scripts)
run: bun run typecheck:scripts
# mt#3102: `packages/**` test files were in NO tsconfig program — the root
# `include` omits `packages`, and package SOURCE only entered a program via
# import reachability from `src/` (which never reaches a test file, nor a
# source file exported only through a barrel). A compile-time assertion
# written there was therefore inert. `if: always()` mirrors the reviewer
# steps so the gate holds regardless of earlier-step failures.
- name: Type check (packages)
if: ${{ always() }}
run: bun run typecheck:packages
# mt#3780: the steps above each run ONE project; nothing asserted the set
# of projects is COMPLETE. Four trees had been found checked-by-nothing
# (mt#3082/mt#3088, mt#2900, mt#3498, mt#3102), three of them only because
# someone hand-ran a negative control. This step fails when a tracked
# .ts/.tsx file is claimed by no project CI runs and is not explicitly
# allowlisted with a reason.
- name: Typecheck coverage invariant
if: ${{ always() }}
run: bun run check:typecheck-coverage
# mt#3935: the sibling invariant for EXECUTION. A test file can be
# committed, reviewed and merged while running in no suite at all — four
# such holes were found before this step, every one of them by a person
# who happened to look rather than by any mechanism. This step fails when
# a tracked *.test.ts/.tsx file is executed by no PR-gating suite and is
# not allowlisted with a reason and an owning task.
- name: Test-reachability invariant
if: ${{ always() }}
run: bun run check:test-reachability
# Run reviewer typecheck even if earlier steps fail so AT-1/AT-2
# ("step always runs regardless of which paths changed or which
# earlier checks fail") hold structurally, not just by step ordering.
# mt#4300 (PR #3164 R1): step-level backstop, same reasoning as the other
# `bun install` steps — retry without a bound left it on the job budget alone.
- name: Type check (services/reviewer)
timeout-minutes: 15
if: ${{ always() }}
working-directory: services/reviewer
run: |
for i in 1 2 3; do if bun install --frozen-lockfile --no-progress; then break; fi; if [ "$i" = 3 ]; then exit 1; fi; echo "bun install failed (mt#3623 tarball flake) - retry $i"; sleep 5; done
bun run typecheck
# mt#3817: services/site shipped a tsconfig but no `typecheck` script and
# no CI step, so five .ts files (astro.config, the Bun static server and
# its logger, the og-cover tool) were checked by nothing. `astro check`
# rather than `tsgo -p` because Astro's docs are explicit that `tsc`
# "ignores .astro files entirely" while `astro check` checks both .astro
# and .ts (https://docs.astro.build/en/guides/typescript/); it also runs
# its own content sync, so no separate `astro sync` step is needed.
# No install step: services/site is a root workspace, so the root
# `bun install` above already hoisted its deps.
- name: Type check (services/site)
if: ${{ always() }}
working-directory: services/site
run: bun run typecheck
# Run the reviewer unit suite (mt#2367). It lives in
# services/reviewer/src/*.test.ts and ran nowhere else in CI before this
# step existed — only the reviewer typecheck ran, so reviewer test
# regressions merged undetected (mt#2346 / mt#2350 each shipped 4 red
# /retrigger tests). if: always() + a self-contained install mirror the
# reviewer typecheck step so the gate holds regardless of step ordering
# or earlier-step failures.
# mt#4300 (PR #3164 R1): step-level backstop, same reasoning as the other
# `bun install` steps.
- name: Test (services/reviewer)
timeout-minutes: 15
if: ${{ always() }}
run: |
for i in 1 2 3; do if bun install --cwd services/reviewer --frozen-lockfile --no-progress; then break; fi; if [ "$i" = 3 ]; then exit 1; fi; echo "bun install failed (mt#3623 tarball flake) - retry $i"; sleep 5; done
bun run test:reviewer
# Reviewer-image module-resolution smoke (mt#2453): replicates the
# reviewer Dockerfile's COPY set in a temp dir and fails if any relative
# import reachable from the boot entry points escapes into root src/
# (the class that crashed bootDomainContainer in deploy 8128911c).
# Needs only bun + the checkout (~2s); no install, env vars, or DB.
- name: Smoke (reviewer image module resolution)
if: ${{ always() }}
run: bun services/reviewer/scripts/smoke-image-module-resolution.ts
# mt#2665: `bun test` 1.2.21 can silently truncate a large
# multi-directory/many-file invocation — exiting 0 with NO completion
# summary line — when several MCP-lifecycle test files (files under
# src/mcp that spawn/track real MCP server subprocesses) run together.
# See docs/testing-patterns.md "Root cause, pinned" for the
# investigation and a minimal 4-file repro. Confirmed (mt#2665) that
# even ALL 11 files under src/mcp together, with nothing else, trigger
# it, and that directory-arg-based exclusion (bunfig.toml
# pathIgnorePatterns) does not reliably prevent it once a parent
# directory is explicitly targeted -- so the "test" script
# (scripts/run-tests-main.ts) now excludes src/mcp/** via explicit file
# enumeration instead, and src/mcp is tested separately below, one file
# per `bun test` process (scripts/run-tests-mcp-isolated.ts). This step
# additionally hardens the main run against that class (and any other
# silent-truncation class, e.g. an older/pinned Bun version) by
# treating the run as failed whenever bun's own completion summary
# ("Ran N tests across M file(s)") is missing, or whenever its own
# "<N> fail" summary line reports N > 0 — both independent of the
# process exit code.
- name: Test
run: |
set -o pipefail
bun run test 2>&1 | tee test-output.log
TEST_EXIT=${PIPESTATUS[0]}
# Note: bun prints singular "1 file" (no trailing "s") when exactly
# one file ran -- "files?" is load-bearing, not decorative.
if ! grep -qE "Ran [0-9]+ tests across [0-9]+ files?" test-output.log; then
echo "::error::bun test did not print a completion summary line (\"Ran N tests across M files\") — the run may have silently truncated (see docs/testing-patterns.md). Treating as a failure regardless of exit code ($TEST_EXIT)."
exit 1
fi
# Verified against real captured `bun test` 1.2.21 output (mt#2665 R1
# review: both local macOS runs and a live GitHub Actions
# ubuntu-latest run's raw log bytes, via `gh api .../logs`,
# inspected with `od -c` for hidden CR/ANSI). The summary block is
# exactly:
# N pass
# M fail
# K expect() calls
# Ran <total> tests across <files> file(s). [<duration>]
# -- one leading space before the digit, no ANSI escapes, no
# trailing CR, even though OTHER lines in the same GitHub Actions
# log (the echoed step-script source) DO carry ANSI color codes;
# bun's own reporter output is unaffected because it correctly
# detects the non-TTY pipe and disables color.
#
# Fail CLOSED, not open: if the completion summary line was found
# (checked above) but the "<N> fail" line can't be located or its
# count can't be parsed, do NOT default to 0 -- that would silently
# pass on an unparseable/malformed summary, recreating exactly the
# "failures reported success" class this task exists to close.
FAIL_LINE=$(grep -E "^ *[0-9]+ fail$" test-output.log | tail -1)
if [ -z "$FAIL_LINE" ]; then
echo "::error::bun test printed a completion summary but its \"<N> fail\" line could not be found — refusing to assume 0 failures. Treating as a failure (fail-closed) regardless of exit code ($TEST_EXIT)."
exit 1
fi
FAIL_COUNT=$(echo "$FAIL_LINE" | grep -oE "[0-9]+")
if [ -z "$FAIL_COUNT" ]; then
echo "::error::bun test's \"<N> fail\" line was found (\"$FAIL_LINE\") but its count could not be parsed — refusing to assume 0 failures. Treating as a failure (fail-closed) regardless of exit code ($TEST_EXIT)."
exit 1
fi
if [ "$FAIL_COUNT" -gt 0 ]; then
echo "::error::bun test reported $FAIL_COUNT failing test(s) in its summary. Treating as a failure regardless of exit code ($TEST_EXIT)."
exit 1
fi
exit "$TEST_EXIT"
# mt#2665: src/mcp/**.test.ts is excluded from the "Test" step above
# (see the comment there) because bun test 1.2.21 silently truncates
# when several of these real-MCP-subprocess files run together --
# confirmed true even for all 11 files alone, with nothing else in the
# invocation. scripts/run-tests-mcp-isolated.ts runs each file in its
# own separate `bun test` process (every combination tried during the
# mt#2665 investigation was healthy at n=1) and applies the same
# completion-summary hardening per file internally.
- name: Test (src/mcp, isolated per file)
if: ${{ always() }}
run: bun run test:mcp-isolated
# mt#2608: the guard system (.claude/hooks/*.test.ts, ~39 files) had no
# CI wiring at all before this step existed — hook regressions merged
# undetected. if: always() mirrors the other independent test gates
# above (services/reviewer, eslint-rules) so this always runs
# regardless of earlier-step outcome.
- name: Test (.claude/hooks)
if: ${{ always() }}
run: bun run test:hooks
# mt#2608: src/cockpit/web (33 files) is excluded from the default
# `bun run test` via bunfig.toml's pathIgnorePatterns and had no CI
# step invoking test:components. It needs its own step (not just
# folding into `test`) because it requires an EXTRA, separate
# `--preload ./tests/dom-setup.ts` (happy-dom globals for
# @testing-library/react) that must load before ./tests/setup.ts and
# is deliberately NOT part of the global bunfig.toml preload chain —
# see the `test:components` script (package.json) and the docblock
# atop tests/dom-setup.ts for why it stays isolated to this one path.
- name: Test (cockpit components)
if: ${{ always() }}
run: bun run test:components
- name: Test (ESLint rules)
if: ${{ always() }}
run: bun run test:eslint-rules
# mt#4129: a completeness check nothing invokes reproduces the outcome it
# exists to prevent — mt#4071 is the demonstration, where the coordinate
# audit reported `UNAUTHORED: interceptor-catalog-regen` from the moment
# mt#4010 shipped that step and it persisted until a human ran the script
# by hand.
#
# ONLY the settings-coverage audit runs here, deliberately. Its inputs are
# `.claude/settings.json` and the committed catalog, both in the repo, so it
# is deterministic. The two older audits
# (`bun run audit:interceptors:corpus`) read the LIVE fire log, whose
# contents change whenever any guard anywhere starts firing — wiring those
# into CI makes a red build a fact about runtime history rather than about
# the commit. That is very likely why they were never wired, and shipping
# them into CI would have turned this PR red on `claim-provenance-scan`,
# a guard name from unrelated in-flight work.
#
# if: always() matches the other independent gates above.
- name: Audit (interceptor settings coverage)
if: ${{ always() }}
run: bun run audit:interceptors
- name: Build
run: bun build src/cli.ts --outdir dist --target bun
# Infra typecheck leg (mt#3817). `infra/index.ts` is the Pulumi program and was
# checked by nothing: infra ships a tsconfig but declares no `typecheck` script
# and had no CI step, and it cannot fold into a root project because it is NOT
# an npm workspace (root `workspaces` is packages/* + services/*), its
# node_modules/ and generated sdks/ are gitignored, and index.ts imports
# @pulumi/pulumi plus the local @pulumi/railway SDK that `pulumi install`
# regenerates from Pulumi.yaml's `packages:` block.
#
# Separate job (not a step in `build`) so the Pulumi CLI install + `pulumi
# install` run in parallel rather than serializing onto the critical path.
# The install sequence mirrors infra-preview.yml, which has been running it
# since mt#2738; PULUMI_ACCESS_TOKEN is the same already-provisioned secret.
#
# It must live in ci.yml specifically: scripts/typecheck-coverage.ts resolves
# the set of projects CI runs by reading THIS file only, so the same step in
# the path-filtered infra-preview.yml would be real coverage the invariant
# could not see.
typecheck-infra:
runs-on: ubuntu-latest
# mt#4300: see the `build` job's note, including the sizing rule. Measured
# healthy run 2026-08-19: 49s (run 32284790015). This job carries THREE bounded
# network steps (bun install, Pulumi CLI, provider SDK), the largest capped at
# 15, so 1 + 15 = 16 → 25. Raised from 15 in PR #3164 R1: at 15 the job cap
# equalled the largest step cap, so the job would have died first and reported
# only "job timed out".
timeout-minutes: 25
steps:
- uses: actions/checkout@v4
- name: Setup Bun
uses: oven-sh/setup-bun@v2
with:
bun-version: "1.3.14"
- name: Cache Bun install (shared root + services/reviewer)
uses: actions/cache@v4
with:
path: ~/.bun/install/cache
key: bun-install-${{ runner.os }}-${{ hashFiles('bun.lock', 'services/reviewer/bun.lock') }}
restore-keys: |
bun-install-${{ runner.os }}-
# mt#4300 (PR #3164 R1): the mt#3623 loop retries but never bounded an
# attempt, so a hung registry connection held this step open on the job's
# budget alone. No per-attempt `timeout` here — `bun install` is resumable and
# cache-warm, so killing it mid-fetch is worse than letting it finish; the
# step-level backstop is the right granularity for this one.
- name: Install dependencies
timeout-minutes: 15
run: for i in 1 2 3; do if bun install --frozen-lockfile --no-progress; then break; fi; if [ "$i" = 3 ]; then exit 1; fi; echo "bun install failed (mt#3623 tarball flake) - retry $i"; sleep 5; done
- uses: actions/setup-node@v4
with:
node-version: "20"
# mt#4950: cache the Pulumi plugin store. This is the vendor-canonical
# pattern — Pulumi's GitHub Actions guide and pulumi/actions#804 both say a
# clean runner re-downloads provider plugins every run, and to cache
# `~/.pulumi/plugins` — and until now this workflow cached nothing on the
# Pulumi path at all (the only cache here is `~/.bun/install/cache`).
#
# Keyed on Pulumi.yaml because that is where the provider and its version
# are declared (`packages.railway.version`), so a version bump misses the
# cache and re-downloads, which is the behaviour you want.
#
# Sizing note, so this is not mistaken for the fix to the timeout below:
# the plugin download is the SMALL half. Measured on run 33851833179
# (2026-09-04), "Downloading provider terraform-provider" through
# "Unpacking provider" took ~3s of a ~90s step. This cache is worth having
# and is close to free; the step's actual cost is the npm install cached
# below it.
# PR #3617 R1/R3, class check — the rule this job now follows: cache only
# DOWNLOADED artifacts, never GENERATED ones. Plugins are downloaded
# binaries in a version-namespaced directory
# (`~/.pulumi/plugins/<name>-v<version>/`), so a CLI upgrade cannot make a
# restored plugin wrong — at worst the new CLI wants a version that is not
# there, misses, and downloads it. The same reasoning covers `Cache Bun
# install` above and the npm cache below, and it is why `infra/sdks` (which
# `pulumi install` GENERATES) is deliberately not cached at all.
- name: Cache Pulumi plugins
uses: actions/cache@v4
with:
path: ~/.pulumi/plugins
key: pulumi-plugins-${{ runner.os }}-${{ hashFiles('infra/Pulumi.yaml') }}
restore-keys: |
pulumi-plugins-${{ runner.os }}-
# mt#4300: the third unprotected raw-network step in this file, bounded and
# retried like the two ripgrep installs above.
#
# `set -o pipefail` is load-bearing and NOT incidental hygiene. GitHub's
# default shell is `bash -e` WITHOUT pipefail, so a pipeline's status is the
# LAST command's: if curl failed, `sh` read empty stdin and exited 0, the step
# went green, and Pulumi was simply absent — surfacing later as a confusing
# failure in a step that looks unrelated. `--max-time` bounds the fetch the
# way `timeout` bounds apt above.
#
# Backstop arithmetic, same rule as the ripgrep steps: 3 x 120s + 2 x 10s
# sleep = 380s = 6.3 min, so 8.
- name: Install Pulumi CLI
timeout-minutes: 8
run: |
set -o pipefail
for i in 1 2 3; do
if curl -fsSL --max-time 120 https://get.pulumi.com | sh; then break; fi
if [ "$i" = 3 ]; then exit 1; fi
echo "pulumi install failed or hung (mt#4300) - retry $i"; sleep 10
done
echo "$HOME/.pulumi/bin" >> "$GITHUB_PATH"
# Regenerates the gitignored Railway terraform-bridge provider SDK and
# installs infra's npm deps — without them, @pulumi/* does not resolve.
#
# mt#4300 (PR #3164 R1): this is a FOURTH network-fetching step, and the
# first pass missed it. It downloads the provider SDK and npm deps, so it can
# hang exactly like the others — but it is spelled `pulumi install`, which
# matches none of the patterns (`apt-get`, `curl`, `npm install`,
# `bun install`) the enumeration searched for. Named-tool-plus-bare-`install`
# is the shape that slipped through; the reviewer caught it.
# mt#4950: cache what `pulumi install` actually spends its time on. The
# step below is TWO operations and only the second is slow: (1) download +
# unpack the provider plugin, ~3s, cached above; (2) `npm install` in
# infra/, which regenerates nothing but installs the generated local SDK
# and its deps — measured at ~87s on run 33851833179 ("added 3 packages in
# 1m"). `infra/sdks` is cached alongside `infra/node_modules` because
# package.json declares `"@pulumi/railway": "file:sdks/railway"`, so
# restoring node_modules without sdks/ would leave a dangling file: link.
#
# PR #3617 R1/R3, both BLOCKING on the same artifact — cache npm's DOWNLOAD
# cache, never `infra/sdks` or `infra/node_modules`.
#
# The first version of this PR cached `infra/node_modules` + `infra/sdks`.
# Two independent review rounds flagged the same hazard from different
# angles: R1 that the key omitted the floating Pulumi CLI version, R3 that
# no key can be trusted to cover every input to a GENERATED artifact. R1
# was answered by adding the CLI version to the key; R3 is the deeper
# objection and it is right — `infra/sdks` is generated by `pulumi
# install`, so caching it means reasoning about what regeneration depends
# on, forever, and a stale restore surfaces as a confusing type error in
# `typecheck:infra` rather than as a cache problem.
#
# `~/.npm` has none of that: it holds content-addressed tarballs keyed by
# integrity hash, so a stale entry is not a thing that exists. `pulumi
# install` then ALWAYS regenerates the SDK and ALWAYS runs npm install —
# the cold path is never skipped, only its network fetches are served
# locally. That is most of the ~109s (228 packages) at none of the risk.
#
# This also dissolves R1's non-blocking sdks<->node_modules coupling note
# and R4's missing-restore-keys note: nothing is coupled any more, and the
# prefix below allows partial restores.
- name: Cache npm download cache (for pulumi install's npm step)
uses: actions/cache@v4
with:
path: ~/.npm
key: infra-npm-${{ runner.os }}-${{ hashFiles('infra/package-lock.json') }}
restore-keys: |
infra-npm-${{ runner.os }}-
# mt#4950: the 120s bound is re-derived, and the comment it replaced was
# the reason it looked safe. That comment read "120s is ~12x the measured
# healthy duration (10s, run 32287929281 2026-08-19T18:33:34→18:33:44Z)."
# The 10s figure was accurate when written and is now 16 days stale: on run
# 33851833179 (2026-09-04) a SUCCEEDING attempt took ~87s, so 120s was 1.4x
# the healthy duration, not 12x — and the attempt before it on that same
# run hit the timeout at exactly 120.0s and was rescued only by the retry
# below. That is the whole mechanism of mt#4950: nothing broke, the install
# grew until ordinary variance started crossing a bound sized for a much
# smaller operation.
#
# 240s is ~2.2x a measured COLD install. Two figures, and the difference
# between them is the point:
#
# run 33851833179, attempt 2: 87s, "added 3 packages in 1m"
# run 33854882366, attempt 1: 109s, "added 228 packages ... in 2m"
#
# The 87s is NOT a cold install and must not be used to size this. That
# attempt ran after attempt 1 had been killed at 120s having already
# installed 225 of the 228 packages — it is a RESUME, and its own log line
# says so ("3 packages"). Sizing a cold-path bound on it would repeat this
# comment's own history: a correct measurement of the wrong population.
# The 109s figure is the first attempt on a run where BOTH caches above
# missed and were written, which is the condition this bound has to
# survive. Backstop: 3 x 240s + 2 x 10s = 740s = 12.3 min, so 13; the
# job's 25-min cap still leaves room for the ~4 min the other steps take.
#
# Note how little headroom the OLD bound had against the real figure: 120s
# against a 109s cold install is 1.1x, so roughly half of cold runs would
# cross it. That is the failure mt#4950 was filed on, quantified.
#
# What would falsify this bound: a cache-MISS run whose attempt still
# exceeds 240s. The caches make a miss rare, which also makes a miss the
# run that matters — read the duration on one before trusting this number.
#
# PR #3617 R4 raised the job's own 25-min cap as blocking, on the
# arithmetic 15 + 8 + 13 = 36 > 25. That is not this file's sizing rule.
# The rule is stated in the job header and is about the LARGEST SINGLE step
# cap, not their sum: "the largest capped at 15, so 1 + 15 = 16 -> 25". It
# exists so that ONE pathological step hits its own bound first and reports
# a cause-specific diagnosis instead of a bare "job timed out". 13 < 15, so
# the largest step cap is unchanged by this PR and 25 still satisfies the
# rule. Three steps simultaneously pegged at their caps is a different
# scenario the rule deliberately does not cover (mt#4300 / PR #3164 R1);
# raising the job cap for it would trade a real property for a speculative
# one. Left at 25 knowingly.
- name: Install infra dependencies + provider SDK
working-directory: infra
timeout-minutes: 13
run: |
for i in 1 2 3; do
set +e
timeout 240 pulumi install
rc=$?
set -e
if [ "$rc" -eq 0 ]; then break; fi
# mt#4950 criterion 2: a retry-exhaustion exit 1 told the reader
# nothing about WHY. `timeout` exits 124 specifically when it killed
# the command, so the two causes are separable at no cost — and they
# call for opposite responses (a hang wants a bigger bound or a
# cache; a non-124 exit wants the install error read).
if [ "$rc" -eq 124 ]; then
echo "pulumi install HUNG - killed by the 240s per-attempt bound (attempt $i/3, mt#4950)"
else
echo "pulumi install FAILED with exit $rc - this is an install error, not a hang (attempt $i/3, mt#4950)"
fi
if [ "$i" = 3 ]; then
if [ "$rc" -eq 124 ]; then
echo "::error::pulumi install hung on all 3 attempts (240s each). The bound may be too tight for a cold install — see mt#4950 before raising it blindly."
else
echo "::error::pulumi install failed with exit $rc on all 3 attempts. This is NOT a timeout; read the install output above for the real cause."
fi
exit "$rc"
fi
sleep 10
done
env:
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
- name: Type check (infra)
run: bun run typecheck:infra
# Forced-TZ leg (mt#3299, gate 5 of the mt#3295 corpus-derived gate wave).
# Runs the main unit suite + hooks suite under a non-UTC, non-US timezone to
# catch tests whose assertions implicitly assume UTC or the author's local
# offset (e.g. hardcoded Date-formatting expectations). Pacific/Kiritimati
# (UTC+14) is deliberately the most extreme available offset — a maximal
# offset surfaces TZ-dependent assertions most reliably. Originating example:
# inject-current-time.test.ts (.claude/hooks) was TZ-dependent and only
# passed in the author's local zone. Separate job (not folded into `build`)
# so a TZ-only failure is visually distinct from a real regression, and so
# this leg's cost is additive/parallel rather than serializing onto the
# critical path.
test-forced-tz:
runs-on: ubuntu-latest
# mt#4300: see the `build` job's note, including the sizing rule. This job
# carries the SECOND copy of the ripgrep install, so it can hang exactly the
# same way. Measured healthy run 2026-08-19: 3m58s (run 32284790015). Largest
# step cap here is 15 (bun install), so 4 + 15 = 19 → 25. Raised from 20 in
# PR #3164 R1, which left only 1 minute of headroom over the rule.
timeout-minutes: 25
env:
TZ: Pacific/Kiritimati
steps:
- uses: actions/checkout@v4
- name: Setup Bun
uses: oven-sh/setup-bun@v2
with:
bun-version: "1.3.14"
- name: Cache Bun install (shared root + services/reviewer)
uses: actions/cache@v4
with:
path: ~/.bun/install/cache
key: bun-install-${{ runner.os }}-${{ hashFiles('bun.lock', 'services/reviewer/bun.lock') }}
restore-keys: |
bun-install-${{ runner.os }}-
# mt#4300 (PR #3164 R1): the mt#3623 loop retries but never bounded an
# attempt, so a hung registry connection held this step open on the job's
# budget alone. No per-attempt `timeout` here — `bun install` is resumable and
# cache-warm, so killing it mid-fetch is worse than letting it finish; the
# step-level backstop is the right granularity for this one.
- name: Install dependencies
timeout-minutes: 15
run: for i in 1 2 3; do if bun install --frozen-lockfile --no-progress; then break; fi; if [ "$i" = 3 ]; then exit 1; fi; echo "bun install failed (mt#3623 tarball flake) - retry $i"; sleep 5; done
# ripgrep is a runtime dependency of session.grep_search (mt#3163); see
# the identical step in the `build` job above for rationale, including the
# mt#4300 bound-and-retry note and the 12-minute backstop arithmetic. Kept
# identical to that step on purpose.
- name: Install ripgrep (required by session.grep_search)
timeout-minutes: 12
run: |
for i in 1 2 3; do
if sudo timeout 90 apt-get update && sudo timeout 90 apt-get install -y ripgrep; then break; fi
if [ "$i" = 3 ]; then exit 1; fi
echo "ripgrep install failed or hung (mt#4300) - retry $i"; sleep 10
done
- name: Test (forced TZ=Pacific/Kiritimati)
run: bun run test
- name: Test hooks (forced TZ=Pacific/Kiritimati)
run: bun run test:hooks