fix(mt#4985): Make the timeout classification name the error that actually arrived #12082
Workflow file for this run
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: CI | |
| on: | |
| push: | |
| branches: [main] | |
| pull_request: | |
| branches: [main] | |
| # Manual retrigger for diagnostic verification only — runs the same job | |
| # steps but does NOT satisfy branch protection's required `build` check | |
| # (manual runs aren't `pull_request` events). Use `--ref task/<branch>` | |
| # explicitly; without it, this dispatches against `main`. mt#1469. | |
| workflow_dispatch: | |
| env: | |
| # Secondary hedge for the mt#3623 install flake: disables the streaming | |
| # tarball path implicated upstream (oven-sh/bun#34821; fix PR #34827 is still | |
| # unmerged, so no release carries a fix). Measured at 0 failures in 10 | |
| # cold-cache installs with the flag set, against a 3-in-10 unmitigated | |
| # baseline -- mt#3623 records both runs and the method. The per-step retry | |
| # loops remain the load-bearing mitigation. | |
| # Remove both when a bun release carries the upstream fix. | |
| BUN_FEATURE_FLAG_DISABLE_STREAMING_INSTALL: "1" | |
| jobs: | |
| build: | |
| runs-on: ubuntu-latest | |
| # mt#4300: without this, GitHub's 6-hour default applies, and a hung network | |
| # step holds a REQUIRED check `in_progress` — indistinguishable from a slow | |
| # build at the check-list level — for up to 6 hours. Measured healthy run | |
| # 2026-08-19: 9m28s (run 32284790015). | |
| # | |
| # Sizing rule for every job cap in this file (PR #3164 R1): observed healthy | |
| # duration + the LARGEST single step-level cap in the job. A job cap tighter | |
| # than that fires before the step cap does, which loses the whole benefit of | |
| # the step caps — the job-level error says only "job timed out" while the step | |
| # cap names the step that hung. Here: 9.5 + 15 = 24.5, so 30. | |
| timeout-minutes: 30 | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Setup Bun | |
| uses: oven-sh/setup-bun@v2 | |
| with: | |
| bun-version: "1.3.14" | |
| - name: Cache Bun install (shared root + services/reviewer) | |
| uses: actions/cache@v4 | |
| with: | |
| path: ~/.bun/install/cache | |
| key: bun-install-${{ runner.os }}-${{ hashFiles('bun.lock', 'services/reviewer/bun.lock') }} | |
| restore-keys: | | |
| bun-install-${{ runner.os }}- | |
| # mt#4300 (PR #3164 R1): the mt#3623 loop retries but never bounded an | |
| # attempt, so a hung registry connection held this step open on the job's | |
| # budget alone. No per-attempt `timeout` here — `bun install` is resumable and | |
| # cache-warm, so killing it mid-fetch is worse than letting it finish; the | |
| # step-level backstop is the right granularity for this one. | |
| - name: Install dependencies | |
| timeout-minutes: 15 | |
| run: for i in 1 2 3; do if bun install --frozen-lockfile --no-progress; then break; fi; if [ "$i" = 3 ]; then exit 1; fi; echo "bun install failed (mt#3623 tarball flake) - retry $i"; sleep 5; done | |
| # ripgrep is a RUNTIME dependency of session.grep_search (mt#3163), which | |
| # spawns `rg` directly — it is not a test-only convenience. ubuntu-latest | |
| # does not ship it, which is why that tool had no CI coverage at all until | |
| # mt#3163 added tests that surfaced the gap by failing loudly instead of | |
| # skipping. Installed from the distro repo rather than fetching a release | |
| # binary, so no new download origin enters the CI supply chain. | |
| # | |
| # mt#4300: bounded + retried. The per-attempt bound is `timeout` INSIDE the | |
| # loop, not the step-level `timeout-minutes` — the latter kills the step, it | |
| # does not retry it. 90s is ~9x the measured healthy duration (10s for | |
| # update+install, run 32284790015 2026-08-19T18:00:09→18:00:19Z), so it | |
| # cannot fire on a merely slow mirror. Bounding is what makes the retry | |
| # meaningful: an unbounded retry against a hung mirror just hangs three times. | |
| # | |
| # `sudo timeout`, NOT `timeout sudo`, and the two commands are bounded | |
| # SEPARATELY rather than grouped under one `timeout`. Both choices are about | |
| # signal delivery: `timeout` signals its DIRECT child, so it has to be the | |
| # parent of the apt-get that actually hangs. `timeout sudo apt-get` makes sudo | |
| # the child, and `sudo timeout 180 sh -c '... && ...'` makes sh the child — | |
| # in both cases SIGTERM can stop at the intermediary and the hung apt-get | |
| # keeps running. Separate bounds cost a longer worst case and are the form | |
| # that actually kills the process. | |
| # | |
| # The step-level backstop MUST exceed the loop's worst case or it truncates | |
| # the retries it exists to protect: 3 attempts x (90s + 90s) + 2 x 10s sleep | |
| # = 560s = 9.3 min, so 12. (PR #3164 R1 caught this at 5 — the third attempt | |
| # could never have run. Observed on this PR's own CI: a degraded mirror took | |
| # the step to 2m53s, so the margin is not hypothetical.) It is a true backstop | |
| # — it only fires if a process escapes its inner `timeout`. | |
| - name: Install ripgrep (required by session.grep_search) | |
| timeout-minutes: 12 | |
| run: | | |
| for i in 1 2 3; do | |
| if sudo timeout 90 apt-get update && sudo timeout 90 apt-get install -y ripgrep; then break; fi | |
| if [ "$i" = 3 ]; then exit 1; fi | |
| echo "ripgrep install failed or hung (mt#4300) - retry $i"; sleep 10 | |
| done | |
| - name: Check formatting | |
| run: bun run format:check | |
| - name: Lint (strict — fails on any warning) | |
| run: bun run lint:strict | |
| # bun run typecheck covers the root tsconfig AND src/cockpit/web/tsconfig.json | |
| # (mt#2424) — the cockpit frontend has its own dedicated typecheck project since | |
| # the root tsconfig excludes it (browser-only lib/DOM settings) and `vite build` | |
| # transpiles it without type-checking. | |
| # mt#3476: the injected skill listing is charged against a shared cumulative | |
| # budget, and going over it silently drops descriptions — a skill listed | |
| # name-only cannot be routed to. Fails on an OWNED description over the cap or | |
| # on the listing total; vendored descriptions are reported, not gated (their | |
| # text is upstream-owned and reverts on refetch, per ADR-015). | |
| - name: Skill listing description budget | |
| run: bun run skills:budget:check | |
| - name: Type check | |
| run: bun run typecheck | |
| - name: Type check (hooks) | |
| run: bun run typecheck:hooks | |
| - name: Type check (scripts) | |
| run: bun run typecheck:scripts | |
| # mt#3102: `packages/**` test files were in NO tsconfig program — the root | |
| # `include` omits `packages`, and package SOURCE only entered a program via | |
| # import reachability from `src/` (which never reaches a test file, nor a | |
| # source file exported only through a barrel). A compile-time assertion | |
| # written there was therefore inert. `if: always()` mirrors the reviewer | |
| # steps so the gate holds regardless of earlier-step failures. | |
| - name: Type check (packages) | |
| if: ${{ always() }} | |
| run: bun run typecheck:packages | |
| # mt#3780: the steps above each run ONE project; nothing asserted the set | |
| # of projects is COMPLETE. Four trees had been found checked-by-nothing | |
| # (mt#3082/mt#3088, mt#2900, mt#3498, mt#3102), three of them only because | |
| # someone hand-ran a negative control. This step fails when a tracked | |
| # .ts/.tsx file is claimed by no project CI runs and is not explicitly | |
| # allowlisted with a reason. | |
| - name: Typecheck coverage invariant | |
| if: ${{ always() }} | |
| run: bun run check:typecheck-coverage | |
| # mt#3935: the sibling invariant for EXECUTION. A test file can be | |
| # committed, reviewed and merged while running in no suite at all — four | |
| # such holes were found before this step, every one of them by a person | |
| # who happened to look rather than by any mechanism. This step fails when | |
| # a tracked *.test.ts/.tsx file is executed by no PR-gating suite and is | |
| # not allowlisted with a reason and an owning task. | |
| - name: Test-reachability invariant | |
| if: ${{ always() }} | |
| run: bun run check:test-reachability | |
| # Run reviewer typecheck even if earlier steps fail so AT-1/AT-2 | |
| # ("step always runs regardless of which paths changed or which | |
| # earlier checks fail") hold structurally, not just by step ordering. | |
| # mt#4300 (PR #3164 R1): step-level backstop, same reasoning as the other | |
| # `bun install` steps — retry without a bound left it on the job budget alone. | |
| - name: Type check (services/reviewer) | |
| timeout-minutes: 15 | |
| if: ${{ always() }} | |
| working-directory: services/reviewer | |
| run: | | |
| for i in 1 2 3; do if bun install --frozen-lockfile --no-progress; then break; fi; if [ "$i" = 3 ]; then exit 1; fi; echo "bun install failed (mt#3623 tarball flake) - retry $i"; sleep 5; done | |
| bun run typecheck | |
| # mt#3817: services/site shipped a tsconfig but no `typecheck` script and | |
| # no CI step, so five .ts files (astro.config, the Bun static server and | |
| # its logger, the og-cover tool) were checked by nothing. `astro check` | |
| # rather than `tsgo -p` because Astro's docs are explicit that `tsc` | |
| # "ignores .astro files entirely" while `astro check` checks both .astro | |
| # and .ts (https://docs.astro.build/en/guides/typescript/); it also runs | |
| # its own content sync, so no separate `astro sync` step is needed. | |
| # No install step: services/site is a root workspace, so the root | |
| # `bun install` above already hoisted its deps. | |
| - name: Type check (services/site) | |
| if: ${{ always() }} | |
| working-directory: services/site | |
| run: bun run typecheck | |
| # Run the reviewer unit suite (mt#2367). It lives in | |
| # services/reviewer/src/*.test.ts and ran nowhere else in CI before this | |
| # step existed — only the reviewer typecheck ran, so reviewer test | |
| # regressions merged undetected (mt#2346 / mt#2350 each shipped 4 red | |
| # /retrigger tests). if: always() + a self-contained install mirror the | |
| # reviewer typecheck step so the gate holds regardless of step ordering | |
| # or earlier-step failures. | |
| # mt#4300 (PR #3164 R1): step-level backstop, same reasoning as the other | |
| # `bun install` steps. | |
| - name: Test (services/reviewer) | |
| timeout-minutes: 15 | |
| if: ${{ always() }} | |
| run: | | |
| for i in 1 2 3; do if bun install --cwd services/reviewer --frozen-lockfile --no-progress; then break; fi; if [ "$i" = 3 ]; then exit 1; fi; echo "bun install failed (mt#3623 tarball flake) - retry $i"; sleep 5; done | |
| bun run test:reviewer | |
| # Reviewer-image module-resolution smoke (mt#2453): replicates the | |
| # reviewer Dockerfile's COPY set in a temp dir and fails if any relative | |
| # import reachable from the boot entry points escapes into root src/ | |
| # (the class that crashed bootDomainContainer in deploy 8128911c). | |
| # Needs only bun + the checkout (~2s); no install, env vars, or DB. | |
| - name: Smoke (reviewer image module resolution) | |
| if: ${{ always() }} | |
| run: bun services/reviewer/scripts/smoke-image-module-resolution.ts | |
| # mt#2665: `bun test` 1.2.21 can silently truncate a large | |
| # multi-directory/many-file invocation — exiting 0 with NO completion | |
| # summary line — when several MCP-lifecycle test files (files under | |
| # src/mcp that spawn/track real MCP server subprocesses) run together. | |
| # See docs/testing-patterns.md "Root cause, pinned" for the | |
| # investigation and a minimal 4-file repro. Confirmed (mt#2665) that | |
| # even ALL 11 files under src/mcp together, with nothing else, trigger | |
| # it, and that directory-arg-based exclusion (bunfig.toml | |
| # pathIgnorePatterns) does not reliably prevent it once a parent | |
| # directory is explicitly targeted -- so the "test" script | |
| # (scripts/run-tests-main.ts) now excludes src/mcp/** via explicit file | |
| # enumeration instead, and src/mcp is tested separately below, one file | |
| # per `bun test` process (scripts/run-tests-mcp-isolated.ts). This step | |
| # additionally hardens the main run against that class (and any other | |
| # silent-truncation class, e.g. an older/pinned Bun version) by | |
| # treating the run as failed whenever bun's own completion summary | |
| # ("Ran N tests across M file(s)") is missing, or whenever its own | |
| # "<N> fail" summary line reports N > 0 — both independent of the | |
| # process exit code. | |
| - name: Test | |
| run: | | |
| set -o pipefail | |
| bun run test 2>&1 | tee test-output.log | |
| TEST_EXIT=${PIPESTATUS[0]} | |
| # Note: bun prints singular "1 file" (no trailing "s") when exactly | |
| # one file ran -- "files?" is load-bearing, not decorative. | |
| if ! grep -qE "Ran [0-9]+ tests across [0-9]+ files?" test-output.log; then | |
| echo "::error::bun test did not print a completion summary line (\"Ran N tests across M files\") — the run may have silently truncated (see docs/testing-patterns.md). Treating as a failure regardless of exit code ($TEST_EXIT)." | |
| exit 1 | |
| fi | |
| # Verified against real captured `bun test` 1.2.21 output (mt#2665 R1 | |
| # review: both local macOS runs and a live GitHub Actions | |
| # ubuntu-latest run's raw log bytes, via `gh api .../logs`, | |
| # inspected with `od -c` for hidden CR/ANSI). The summary block is | |
| # exactly: | |
| # N pass | |
| # M fail | |
| # K expect() calls | |
| # Ran <total> tests across <files> file(s). [<duration>] | |
| # -- one leading space before the digit, no ANSI escapes, no | |
| # trailing CR, even though OTHER lines in the same GitHub Actions | |
| # log (the echoed step-script source) DO carry ANSI color codes; | |
| # bun's own reporter output is unaffected because it correctly | |
| # detects the non-TTY pipe and disables color. | |
| # | |
| # Fail CLOSED, not open: if the completion summary line was found | |
| # (checked above) but the "<N> fail" line can't be located or its | |
| # count can't be parsed, do NOT default to 0 -- that would silently | |
| # pass on an unparseable/malformed summary, recreating exactly the | |
| # "failures reported success" class this task exists to close. | |
| FAIL_LINE=$(grep -E "^ *[0-9]+ fail$" test-output.log | tail -1) | |
| if [ -z "$FAIL_LINE" ]; then | |
| echo "::error::bun test printed a completion summary but its \"<N> fail\" line could not be found — refusing to assume 0 failures. Treating as a failure (fail-closed) regardless of exit code ($TEST_EXIT)." | |
| exit 1 | |
| fi | |
| FAIL_COUNT=$(echo "$FAIL_LINE" | grep -oE "[0-9]+") | |
| if [ -z "$FAIL_COUNT" ]; then | |
| echo "::error::bun test's \"<N> fail\" line was found (\"$FAIL_LINE\") but its count could not be parsed — refusing to assume 0 failures. Treating as a failure (fail-closed) regardless of exit code ($TEST_EXIT)." | |
| exit 1 | |
| fi | |
| if [ "$FAIL_COUNT" -gt 0 ]; then | |
| echo "::error::bun test reported $FAIL_COUNT failing test(s) in its summary. Treating as a failure regardless of exit code ($TEST_EXIT)." | |
| exit 1 | |
| fi | |
| exit "$TEST_EXIT" | |
| # mt#2665: src/mcp/**.test.ts is excluded from the "Test" step above | |
| # (see the comment there) because bun test 1.2.21 silently truncates | |
| # when several of these real-MCP-subprocess files run together -- | |
| # confirmed true even for all 11 files alone, with nothing else in the | |
| # invocation. scripts/run-tests-mcp-isolated.ts runs each file in its | |
| # own separate `bun test` process (every combination tried during the | |
| # mt#2665 investigation was healthy at n=1) and applies the same | |
| # completion-summary hardening per file internally. | |
| - name: Test (src/mcp, isolated per file) | |
| if: ${{ always() }} | |
| run: bun run test:mcp-isolated | |
| # mt#2608: the guard system (.claude/hooks/*.test.ts, ~39 files) had no | |
| # CI wiring at all before this step existed — hook regressions merged | |
| # undetected. if: always() mirrors the other independent test gates | |
| # above (services/reviewer, eslint-rules) so this always runs | |
| # regardless of earlier-step outcome. | |
| - name: Test (.claude/hooks) | |
| if: ${{ always() }} | |
| run: bun run test:hooks | |
| # mt#2608: src/cockpit/web (33 files) is excluded from the default | |
| # `bun run test` via bunfig.toml's pathIgnorePatterns and had no CI | |
| # step invoking test:components. It needs its own step (not just | |
| # folding into `test`) because it requires an EXTRA, separate | |
| # `--preload ./tests/dom-setup.ts` (happy-dom globals for | |
| # @testing-library/react) that must load before ./tests/setup.ts and | |
| # is deliberately NOT part of the global bunfig.toml preload chain — | |
| # see the `test:components` script (package.json) and the docblock | |
| # atop tests/dom-setup.ts for why it stays isolated to this one path. | |
| - name: Test (cockpit components) | |
| if: ${{ always() }} | |
| run: bun run test:components | |
| - name: Test (ESLint rules) | |
| if: ${{ always() }} | |
| run: bun run test:eslint-rules | |
| # mt#4129: a completeness check nothing invokes reproduces the outcome it | |
| # exists to prevent — mt#4071 is the demonstration, where the coordinate | |
| # audit reported `UNAUTHORED: interceptor-catalog-regen` from the moment | |
| # mt#4010 shipped that step and it persisted until a human ran the script | |
| # by hand. | |
| # | |
| # ONLY the settings-coverage audit runs here, deliberately. Its inputs are | |
| # `.claude/settings.json` and the committed catalog, both in the repo, so it | |
| # is deterministic. The two older audits | |
| # (`bun run audit:interceptors:corpus`) read the LIVE fire log, whose | |
| # contents change whenever any guard anywhere starts firing — wiring those | |
| # into CI makes a red build a fact about runtime history rather than about | |
| # the commit. That is very likely why they were never wired, and shipping | |
| # them into CI would have turned this PR red on `claim-provenance-scan`, | |
| # a guard name from unrelated in-flight work. | |
| # | |
| # if: always() matches the other independent gates above. | |
| - name: Audit (interceptor settings coverage) | |
| if: ${{ always() }} | |
| run: bun run audit:interceptors | |
| - name: Build | |
| run: bun build src/cli.ts --outdir dist --target bun | |
| # Infra typecheck leg (mt#3817). `infra/index.ts` is the Pulumi program and was | |
| # checked by nothing: infra ships a tsconfig but declares no `typecheck` script | |
| # and had no CI step, and it cannot fold into a root project because it is NOT | |
| # an npm workspace (root `workspaces` is packages/* + services/*), its | |
| # node_modules/ and generated sdks/ are gitignored, and index.ts imports | |
| # @pulumi/pulumi plus the local @pulumi/railway SDK that `pulumi install` | |
| # regenerates from Pulumi.yaml's `packages:` block. | |
| # | |
| # Separate job (not a step in `build`) so the Pulumi CLI install + `pulumi | |
| # install` run in parallel rather than serializing onto the critical path. | |
| # The install sequence mirrors infra-preview.yml, which has been running it | |
| # since mt#2738; PULUMI_ACCESS_TOKEN is the same already-provisioned secret. | |
| # | |
| # It must live in ci.yml specifically: scripts/typecheck-coverage.ts resolves | |
| # the set of projects CI runs by reading THIS file only, so the same step in | |
| # the path-filtered infra-preview.yml would be real coverage the invariant | |
| # could not see. | |
| typecheck-infra: | |
| runs-on: ubuntu-latest | |
| # mt#4300: see the `build` job's note, including the sizing rule. Measured | |
| # healthy run 2026-08-19: 49s (run 32284790015). This job carries THREE bounded | |
| # network steps (bun install, Pulumi CLI, provider SDK), the largest capped at | |
| # 15, so 1 + 15 = 16 → 25. Raised from 15 in PR #3164 R1: at 15 the job cap | |
| # equalled the largest step cap, so the job would have died first and reported | |
| # only "job timed out". | |
| timeout-minutes: 25 | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Setup Bun | |
| uses: oven-sh/setup-bun@v2 | |
| with: | |
| bun-version: "1.3.14" | |
| - name: Cache Bun install (shared root + services/reviewer) | |
| uses: actions/cache@v4 | |
| with: | |
| path: ~/.bun/install/cache | |
| key: bun-install-${{ runner.os }}-${{ hashFiles('bun.lock', 'services/reviewer/bun.lock') }} | |
| restore-keys: | | |
| bun-install-${{ runner.os }}- | |
| # mt#4300 (PR #3164 R1): the mt#3623 loop retries but never bounded an | |
| # attempt, so a hung registry connection held this step open on the job's | |
| # budget alone. No per-attempt `timeout` here — `bun install` is resumable and | |
| # cache-warm, so killing it mid-fetch is worse than letting it finish; the | |
| # step-level backstop is the right granularity for this one. | |
| - name: Install dependencies | |
| timeout-minutes: 15 | |
| run: for i in 1 2 3; do if bun install --frozen-lockfile --no-progress; then break; fi; if [ "$i" = 3 ]; then exit 1; fi; echo "bun install failed (mt#3623 tarball flake) - retry $i"; sleep 5; done | |
| - uses: actions/setup-node@v4 | |
| with: | |
| node-version: "20" | |
| # mt#4950: cache the Pulumi plugin store. This is the vendor-canonical | |
| # pattern — Pulumi's GitHub Actions guide and pulumi/actions#804 both say a | |
| # clean runner re-downloads provider plugins every run, and to cache | |
| # `~/.pulumi/plugins` — and until now this workflow cached nothing on the | |
| # Pulumi path at all (the only cache here is `~/.bun/install/cache`). | |
| # | |
| # Keyed on Pulumi.yaml because that is where the provider and its version | |
| # are declared (`packages.railway.version`), so a version bump misses the | |
| # cache and re-downloads, which is the behaviour you want. | |
| # | |
| # Sizing note, so this is not mistaken for the fix to the timeout below: | |
| # the plugin download is the SMALL half. Measured on run 33851833179 | |
| # (2026-09-04), "Downloading provider terraform-provider" through | |
| # "Unpacking provider" took ~3s of a ~90s step. This cache is worth having | |
| # and is close to free; the step's actual cost is the npm install cached | |
| # below it. | |
| # PR #3617 R1/R3, class check — the rule this job now follows: cache only | |
| # DOWNLOADED artifacts, never GENERATED ones. Plugins are downloaded | |
| # binaries in a version-namespaced directory | |
| # (`~/.pulumi/plugins/<name>-v<version>/`), so a CLI upgrade cannot make a | |
| # restored plugin wrong — at worst the new CLI wants a version that is not | |
| # there, misses, and downloads it. The same reasoning covers `Cache Bun | |
| # install` above and the npm cache below, and it is why `infra/sdks` (which | |
| # `pulumi install` GENERATES) is deliberately not cached at all. | |
| - name: Cache Pulumi plugins | |
| uses: actions/cache@v4 | |
| with: | |
| path: ~/.pulumi/plugins | |
| key: pulumi-plugins-${{ runner.os }}-${{ hashFiles('infra/Pulumi.yaml') }} | |
| restore-keys: | | |
| pulumi-plugins-${{ runner.os }}- | |
| # mt#4300: the third unprotected raw-network step in this file, bounded and | |
| # retried like the two ripgrep installs above. | |
| # | |
| # `set -o pipefail` is load-bearing and NOT incidental hygiene. GitHub's | |
| # default shell is `bash -e` WITHOUT pipefail, so a pipeline's status is the | |
| # LAST command's: if curl failed, `sh` read empty stdin and exited 0, the step | |
| # went green, and Pulumi was simply absent — surfacing later as a confusing | |
| # failure in a step that looks unrelated. `--max-time` bounds the fetch the | |
| # way `timeout` bounds apt above. | |
| # | |
| # Backstop arithmetic, same rule as the ripgrep steps: 3 x 120s + 2 x 10s | |
| # sleep = 380s = 6.3 min, so 8. | |
| - name: Install Pulumi CLI | |
| timeout-minutes: 8 | |
| run: | | |
| set -o pipefail | |
| for i in 1 2 3; do | |
| if curl -fsSL --max-time 120 https://get.pulumi.com | sh; then break; fi | |
| if [ "$i" = 3 ]; then exit 1; fi | |
| echo "pulumi install failed or hung (mt#4300) - retry $i"; sleep 10 | |
| done | |
| echo "$HOME/.pulumi/bin" >> "$GITHUB_PATH" | |
| # Regenerates the gitignored Railway terraform-bridge provider SDK and | |
| # installs infra's npm deps — without them, @pulumi/* does not resolve. | |
| # | |
| # mt#4300 (PR #3164 R1): this is a FOURTH network-fetching step, and the | |
| # first pass missed it. It downloads the provider SDK and npm deps, so it can | |
| # hang exactly like the others — but it is spelled `pulumi install`, which | |
| # matches none of the patterns (`apt-get`, `curl`, `npm install`, | |
| # `bun install`) the enumeration searched for. Named-tool-plus-bare-`install` | |
| # is the shape that slipped through; the reviewer caught it. | |
| # mt#4950: cache what `pulumi install` actually spends its time on. The | |
| # step below is TWO operations and only the second is slow: (1) download + | |
| # unpack the provider plugin, ~3s, cached above; (2) `npm install` in | |
| # infra/, which regenerates nothing but installs the generated local SDK | |
| # and its deps — measured at ~87s on run 33851833179 ("added 3 packages in | |
| # 1m"). `infra/sdks` is cached alongside `infra/node_modules` because | |
| # package.json declares `"@pulumi/railway": "file:sdks/railway"`, so | |
| # restoring node_modules without sdks/ would leave a dangling file: link. | |
| # | |
| # PR #3617 R1/R3, both BLOCKING on the same artifact — cache npm's DOWNLOAD | |
| # cache, never `infra/sdks` or `infra/node_modules`. | |
| # | |
| # The first version of this PR cached `infra/node_modules` + `infra/sdks`. | |
| # Two independent review rounds flagged the same hazard from different | |
| # angles: R1 that the key omitted the floating Pulumi CLI version, R3 that | |
| # no key can be trusted to cover every input to a GENERATED artifact. R1 | |
| # was answered by adding the CLI version to the key; R3 is the deeper | |
| # objection and it is right — `infra/sdks` is generated by `pulumi | |
| # install`, so caching it means reasoning about what regeneration depends | |
| # on, forever, and a stale restore surfaces as a confusing type error in | |
| # `typecheck:infra` rather than as a cache problem. | |
| # | |
| # `~/.npm` has none of that: it holds content-addressed tarballs keyed by | |
| # integrity hash, so a stale entry is not a thing that exists. `pulumi | |
| # install` then ALWAYS regenerates the SDK and ALWAYS runs npm install — | |
| # the cold path is never skipped, only its network fetches are served | |
| # locally. That is most of the ~109s (228 packages) at none of the risk. | |
| # | |
| # This also dissolves R1's non-blocking sdks<->node_modules coupling note | |
| # and R4's missing-restore-keys note: nothing is coupled any more, and the | |
| # prefix below allows partial restores. | |
| - name: Cache npm download cache (for pulumi install's npm step) | |
| uses: actions/cache@v4 | |
| with: | |
| path: ~/.npm | |
| key: infra-npm-${{ runner.os }}-${{ hashFiles('infra/package-lock.json') }} | |
| restore-keys: | | |
| infra-npm-${{ runner.os }}- | |
| # mt#4950: the 120s bound is re-derived, and the comment it replaced was | |
| # the reason it looked safe. That comment read "120s is ~12x the measured | |
| # healthy duration (10s, run 32287929281 2026-08-19T18:33:34→18:33:44Z)." | |
| # The 10s figure was accurate when written and is now 16 days stale: on run | |
| # 33851833179 (2026-09-04) a SUCCEEDING attempt took ~87s, so 120s was 1.4x | |
| # the healthy duration, not 12x — and the attempt before it on that same | |
| # run hit the timeout at exactly 120.0s and was rescued only by the retry | |
| # below. That is the whole mechanism of mt#4950: nothing broke, the install | |
| # grew until ordinary variance started crossing a bound sized for a much | |
| # smaller operation. | |
| # | |
| # 240s is ~2.2x a measured COLD install. Two figures, and the difference | |
| # between them is the point: | |
| # | |
| # run 33851833179, attempt 2: 87s, "added 3 packages in 1m" | |
| # run 33854882366, attempt 1: 109s, "added 228 packages ... in 2m" | |
| # | |
| # The 87s is NOT a cold install and must not be used to size this. That | |
| # attempt ran after attempt 1 had been killed at 120s having already | |
| # installed 225 of the 228 packages — it is a RESUME, and its own log line | |
| # says so ("3 packages"). Sizing a cold-path bound on it would repeat this | |
| # comment's own history: a correct measurement of the wrong population. | |
| # The 109s figure is the first attempt on a run where BOTH caches above | |
| # missed and were written, which is the condition this bound has to | |
| # survive. Backstop: 3 x 240s + 2 x 10s = 740s = 12.3 min, so 13; the | |
| # job's 25-min cap still leaves room for the ~4 min the other steps take. | |
| # | |
| # Note how little headroom the OLD bound had against the real figure: 120s | |
| # against a 109s cold install is 1.1x, so roughly half of cold runs would | |
| # cross it. That is the failure mt#4950 was filed on, quantified. | |
| # | |
| # What would falsify this bound: a cache-MISS run whose attempt still | |
| # exceeds 240s. The caches make a miss rare, which also makes a miss the | |
| # run that matters — read the duration on one before trusting this number. | |
| # | |
| # PR #3617 R4 raised the job's own 25-min cap as blocking, on the | |
| # arithmetic 15 + 8 + 13 = 36 > 25. That is not this file's sizing rule. | |
| # The rule is stated in the job header and is about the LARGEST SINGLE step | |
| # cap, not their sum: "the largest capped at 15, so 1 + 15 = 16 -> 25". It | |
| # exists so that ONE pathological step hits its own bound first and reports | |
| # a cause-specific diagnosis instead of a bare "job timed out". 13 < 15, so | |
| # the largest step cap is unchanged by this PR and 25 still satisfies the | |
| # rule. Three steps simultaneously pegged at their caps is a different | |
| # scenario the rule deliberately does not cover (mt#4300 / PR #3164 R1); | |
| # raising the job cap for it would trade a real property for a speculative | |
| # one. Left at 25 knowingly. | |
| - name: Install infra dependencies + provider SDK | |
| working-directory: infra | |
| timeout-minutes: 13 | |
| run: | | |
| for i in 1 2 3; do | |
| set +e | |
| timeout 240 pulumi install | |
| rc=$? | |
| set -e | |
| if [ "$rc" -eq 0 ]; then break; fi | |
| # mt#4950 criterion 2: a retry-exhaustion exit 1 told the reader | |
| # nothing about WHY. `timeout` exits 124 specifically when it killed | |
| # the command, so the two causes are separable at no cost — and they | |
| # call for opposite responses (a hang wants a bigger bound or a | |
| # cache; a non-124 exit wants the install error read). | |
| if [ "$rc" -eq 124 ]; then | |
| echo "pulumi install HUNG - killed by the 240s per-attempt bound (attempt $i/3, mt#4950)" | |
| else | |
| echo "pulumi install FAILED with exit $rc - this is an install error, not a hang (attempt $i/3, mt#4950)" | |
| fi | |
| if [ "$i" = 3 ]; then | |
| if [ "$rc" -eq 124 ]; then | |
| echo "::error::pulumi install hung on all 3 attempts (240s each). The bound may be too tight for a cold install — see mt#4950 before raising it blindly." | |
| else | |
| echo "::error::pulumi install failed with exit $rc on all 3 attempts. This is NOT a timeout; read the install output above for the real cause." | |
| fi | |
| exit "$rc" | |
| fi | |
| sleep 10 | |
| done | |
| env: | |
| PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }} | |
| - name: Type check (infra) | |
| run: bun run typecheck:infra | |
| # Forced-TZ leg (mt#3299, gate 5 of the mt#3295 corpus-derived gate wave). | |
| # Runs the main unit suite + hooks suite under a non-UTC, non-US timezone to | |
| # catch tests whose assertions implicitly assume UTC or the author's local | |
| # offset (e.g. hardcoded Date-formatting expectations). Pacific/Kiritimati | |
| # (UTC+14) is deliberately the most extreme available offset — a maximal | |
| # offset surfaces TZ-dependent assertions most reliably. Originating example: | |
| # inject-current-time.test.ts (.claude/hooks) was TZ-dependent and only | |
| # passed in the author's local zone. Separate job (not folded into `build`) | |
| # so a TZ-only failure is visually distinct from a real regression, and so | |
| # this leg's cost is additive/parallel rather than serializing onto the | |
| # critical path. | |
| test-forced-tz: | |
| runs-on: ubuntu-latest | |
| # mt#4300: see the `build` job's note, including the sizing rule. This job | |
| # carries the SECOND copy of the ripgrep install, so it can hang exactly the | |
| # same way. Measured healthy run 2026-08-19: 3m58s (run 32284790015). Largest | |
| # step cap here is 15 (bun install), so 4 + 15 = 19 → 25. Raised from 20 in | |
| # PR #3164 R1, which left only 1 minute of headroom over the rule. | |
| timeout-minutes: 25 | |
| env: | |
| TZ: Pacific/Kiritimati | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Setup Bun | |
| uses: oven-sh/setup-bun@v2 | |
| with: | |
| bun-version: "1.3.14" | |
| - name: Cache Bun install (shared root + services/reviewer) | |
| uses: actions/cache@v4 | |
| with: | |
| path: ~/.bun/install/cache | |
| key: bun-install-${{ runner.os }}-${{ hashFiles('bun.lock', 'services/reviewer/bun.lock') }} | |
| restore-keys: | | |
| bun-install-${{ runner.os }}- | |
| # mt#4300 (PR #3164 R1): the mt#3623 loop retries but never bounded an | |
| # attempt, so a hung registry connection held this step open on the job's | |
| # budget alone. No per-attempt `timeout` here — `bun install` is resumable and | |
| # cache-warm, so killing it mid-fetch is worse than letting it finish; the | |
| # step-level backstop is the right granularity for this one. | |
| - name: Install dependencies | |
| timeout-minutes: 15 | |
| run: for i in 1 2 3; do if bun install --frozen-lockfile --no-progress; then break; fi; if [ "$i" = 3 ]; then exit 1; fi; echo "bun install failed (mt#3623 tarball flake) - retry $i"; sleep 5; done | |
| # ripgrep is a runtime dependency of session.grep_search (mt#3163); see | |
| # the identical step in the `build` job above for rationale, including the | |
| # mt#4300 bound-and-retry note and the 12-minute backstop arithmetic. Kept | |
| # identical to that step on purpose. | |
| - name: Install ripgrep (required by session.grep_search) | |
| timeout-minutes: 12 | |
| run: | | |
| for i in 1 2 3; do | |
| if sudo timeout 90 apt-get update && sudo timeout 90 apt-get install -y ripgrep; then break; fi | |
| if [ "$i" = 3 ]; then exit 1; fi | |
| echo "ripgrep install failed or hung (mt#4300) - retry $i"; sleep 10 | |
| done | |
| - name: Test (forced TZ=Pacific/Kiritimati) | |
| run: bun run test | |
| - name: Test hooks (forced TZ=Pacific/Kiritimati) | |
| run: bun run test:hooks |