Post-Deploy Health Monitor #4014
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Post-Deploy Health Monitor | |
| # mt#1302 — Automated external post-deploy outcome+health verification across | |
| # all deployed services. Runs on a cron cadence so outages are detected within | |
| # minutes rather than discovered by accident. | |
| # | |
| # Architecture notes (per spec § Scope expansion 2026-06-10): | |
| # | |
| # HOST: This workflow runs on GitHub Actions — EXTERNAL to all Railway services. | |
| # A monitor that detects "is service X up?" cannot run inside X, nor can it | |
| # alert through infra that may itself be the thing that's down (dependency | |
| # cycle). GitHub stays up when Railway/Minsky is down. | |
| # | |
| # PRIMARY ALERT: open/update a GitHub P0 issue per service+failure-class. | |
| # De-duped so a sustained outage updates ONE issue, not N. GitHub is the | |
| # always-available substrate regardless of Minsky infra health. | |
| # | |
| # SECONDARY ALERT (best-effort): when minsky-mcp is up, also POST an | |
| # asks_create coordination.notify over hosted MCP so it surfaces on the | |
| # cockpit AsksPage. Wrapped in try/catch — its failure MUST NOT suppress | |
| # the primary GitHub-issue path. | |
| # | |
| # PER-SERVICE CHECKS: | |
| # (a) Latest Railway deploy terminal status — alert on FAILED/CRASHED. | |
| # Catches the mt#1991 build-failure class (bun install --frozen-lockfile). | |
| # (b) GET <service>/health returns 200 — alert on non-200/timeout. | |
| # Catches the mt#2345 runtime-crash-after-green-build class. | |
| # (c) Deployed image digest lags the newest registry digest (mt#3251) — | |
| # catches a service that stays HEALTHY on an OLD image because its | |
| # deploy pipeline silently stopped shipping new ones. Image-source | |
| # services only (reviewer, minsky-mcp, minsky-ops); needs no additional | |
| # secret — uses GHCR's anonymous registry-token flow. See the module | |
| # doc-comment in scripts/post-deploy-health-monitor.ts for the full | |
| # Covers / Does NOT cover breakdown. | |
| # All three checks are independent: deploy-status alone misses (b) and (c); | |
| # /health alone misses build failures and a frozen-but-healthy deploy. | |
| # | |
| # A CHECK THAT COULD NOT RUN IS NOT A CHECK THAT PASSED (mt#3921). Any of the | |
| # three failing to complete raises a `check-failed` alert and makes the | |
| # service's verdict DEGRADED, because the monitor has observed nothing about | |
| # it. That is separate from "not applicable" — a repo-source service has no | |
| # configured image, so (c) genuinely does not apply to it and is silent. | |
| # From 2026-08-05 to 2026-08-10 this distinction did not exist: (a) errored | |
| # `Not Authorized` on every run for all five services, (c) was consequently | |
| # skipped, and every service still reported HEALTHY with 0 alerts. See | |
| # packages/domain/src/deployment/monitor-verdict.ts for the scoring rule. | |
| # | |
| # All five services are checked; none has an empty serviceId. (An empty | |
| # serviceId in deploy.config.ts would skip a service by data, not by name.) | |
| # | |
| # SECRETS REQUIRED (set as repository secrets): | |
| # RAILWAY_MCP_TOKEN — Railway ACCOUNT/WORKSPACE token, passed to the | |
| # script as RAILWAY_TOKEN. It must be account- or | |
| # workspace-scoped for two reasons: the five | |
| # services span FOUR distinct Railway projects, and | |
| # the script authenticates with `Authorization: | |
| # Bearer`, which per Railway's public-API reference | |
| # is not the header a project token uses | |
| # (`Project-Access-Token`). The repo's generic | |
| # RAILWAY_TOKEN secret is a PROJECT token and was | |
| # the cause of the 4.5-day blind spot above; it is | |
| # still used by cockpit-preview.yml and must not be | |
| # re-scoped (mt#3890). | |
| # MINSKY_MCP_AUTH_TOKEN — Bearer token for the hosted minsky-mcp (secondary path only) | |
| # GITHUB_TOKEN — auto-provided by GitHub Actions | |
| # | |
| # CROSS-REFERENCES: | |
| # mt#1991 — workspace-COPY Dockerfile guard (upstream prevention) | |
| # mt#2345 — reviewer Railway deploy watchPatterns (companion detection) | |
| # mt#1280 — original outage that motivates this task | |
| # scripts/smoke-post-deploy-health-monitor.ts — verification artifact | |
| on: | |
| schedule: | |
| # Every 10 minutes. Low-frequency is correct: the mt#2365 data shows | |
| # <1% steady-state failure rate across services. A high-frequency | |
| # dedicated poller is over-engineered for this failure class. | |
| - cron: "*/10 * * * *" | |
| workflow_dispatch: | |
| inputs: | |
| dry_run: | |
| description: "Dry run (log findings but do not open/update issues)" | |
| required: false | |
| default: "false" | |
| type: choice | |
| options: | |
| - "false" | |
| - "true" | |
| env: | |
| # Secondary hedge for the mt#3623 install flake: disables the streaming | |
| # tarball path implicated upstream (oven-sh/bun#34821; fix PR #34827 is still | |
| # unmerged, so no release carries a fix). Measured at 0 failures in 10 | |
| # cold-cache installs with the flag set, against a 3-in-10 unmitigated | |
| # baseline -- mt#3623 records both runs and the method. The per-step retry | |
| # loops remain the load-bearing mitigation. | |
| # Remove both when a bun release carries the upstream fix. | |
| BUN_FEATURE_FLAG_DISABLE_STREAMING_INSTALL: "1" | |
| jobs: | |
| monitor: | |
| name: monitor | |
| runs-on: ubuntu-latest | |
| # Give the script enough time to probe all services and write issues. | |
| timeout-minutes: 5 | |
| permissions: | |
| issues: write | |
| contents: read | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Setup Bun | |
| uses: oven-sh/setup-bun@v2 | |
| with: | |
| bun-version: "1.3.14" | |
| # bun install is required because the monitor script dynamically imports | |
| # services/*/deploy.config.ts files, which in turn import | |
| # @minsky/shared/deployment-config. Bun needs the workspace symlinks | |
| # established by bun install to resolve that import. | |
| - name: Install dependencies | |
| run: for i in 1 2 3; do if bun install --frozen-lockfile; then break; fi; if [ "$i" = 3 ]; then exit 1; fi; echo "bun install failed (mt#3623 tarball flake) - retry $i"; sleep 5; done | |
| - name: Run post-deploy health monitor | |
| env: | |
| # mt#3921 — the account/workspace-scoped secret, not the generic | |
| # project-scoped RAILWAY_TOKEN. See SECRETS REQUIRED above. No | |
| # fallback is wired on purpose: falling back to the wrong-scoped | |
| # secret is the state this task exists to end, and an absent | |
| # credential now raises check-failed alerts rather than passing | |
| # silently. | |
| RAILWAY_TOKEN: ${{ secrets.RAILWAY_MCP_TOKEN }} | |
| MINSKY_MCP_AUTH_TOKEN: ${{ secrets.MINSKY_MCP_AUTH_TOKEN }} | |
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | |
| GITHUB_REPO: ${{ github.repository }} | |
| # Use event-name guard so 'inputs' is never accessed on schedule events | |
| # (where github.event.inputs is undefined). On workflow_dispatch the user | |
| # value is used; on schedule it falls back to 'false'. | |
| DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run || 'false' }} | |
| run: bun scripts/post-deploy-health-monitor.ts |