Corpus nightly #6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Corpus nightly | |
| on: | |
| schedule: | |
| # 03:17 UTC — off the hour, which is when scheduled runs queue worst. | |
| # A scheduled workflow in a public repository is disabled automatically | |
| # after 60 days without repository activity, so a job that goes quiet is | |
| # worth checking here before it is debugged. | |
| - cron: "17 3 * * *" | |
| workflow_dispatch: | |
| # One at a time. Two concurrent runs would double the request rate every | |
| # scanned origin sees from GitHub's address range. | |
| concurrency: corpus-nightly | |
| permissions: | |
| contents: read | |
| jobs: | |
| scan: | |
| runs-on: ubuntu-latest | |
| # The list is about 400 curated domains, held under 500 by its test, and | |
| # a night scans all of it: at concurrency 2 and about 63 s per site per | |
| # worker that is about 220 minutes, inside the script's 240-minute | |
| # deadline. status.json keeps dead and robots-blocked domains out, which | |
| # is most of what made the old 1913-entry list slow, and the scanner's | |
| # own 180 s budget per scan keeps one slow origin from eating the margin. | |
| # This cap is the backstop behind the deadline. | |
| timeout-minutes: 300 | |
| steps: | |
| - uses: actions/checkout@v7 | |
| - name: Install pnpm | |
| uses: pnpm/action-setup@v6 | |
| - name: Setup Node.js | |
| uses: actions/setup-node@v7 | |
| with: | |
| node-version: 22 | |
| cache: "pnpm" | |
| - name: Install dependencies | |
| run: pnpm install --frozen-lockfile | |
| # No build step: the script runs through `tsx` against `packages/core/src` | |
| # and never reads `dist/`. Building would spend minutes against the cap and | |
| # give the nightly a second way to go red for reasons unrelated to scanning. | |
| # `node --import tsx` rather than `npx tsx`: the script writes its partial | |
| # summary from a SIGTERM handler, and each wrapper process between the | |
| # runner and node is one more place that signal can stop. | |
| # Exit codes are distinct on purpose: 1 is a scan that broke an | |
| # invariant — the finding this job exists for — while 3 (cancelled), | |
| # 4 (deadline), 5 (scanned nothing) and 6 (the runner threw) are the job | |
| # failing rather than a site. `scan-site-list.ts` documents the set. | |
| - name: Scan the site list | |
| run: node --import tsx scripts/scan-site-list.ts --limit=500 --allow-partial=1 | |
| # `if: always()` on purpose: the summary is most wanted on the run that | |
| # failed, and the scan step exits non-zero exactly when it found something. | |
| - name: Upload the summary | |
| if: always() | |
| uses: actions/upload-artifact@v7 | |
| with: | |
| name: corpus-nightly | |
| path: reports/corpus-nightly.json | |
| retention-days: 14 | |
| # Default is `warn`, which turns a run that produced no summary into a | |
| # green upload step under a red job — the one case where the operator | |
| # most needs to be told there is nothing to read. | |
| if-no-files-found: error |