ci: refresh published bot IP ranges weekly - #19
Merged
Conversation
The bundled snapshot in src/bot-ranges.ts is the one part of identity verification that rots on its own. Vendors rotate their published prefixes, and a stale list doesn't fail loudly — it quietly starts returning 'spoofed' for legitimate crawlers. That is the verdict people act on, so silent drift is the expensive failure mode. Runs Mondays 06:00 UTC (plus manual dispatch) and opens a PR rather than committing to main, because a range change alters what downstream sites report about real traffic and deserves eyes on the diff. Three guards, so the automation can't make things worse than no automation: - The refresh script already aborts on a failed feed or a list that shrinks by more than half, rather than writing a truncated snapshot. - BOT_RANGES_CAPTURED_AT moves on every run, so a timestamp-only diff is discarded instead of opening a weekly no-op PR. - typecheck and the test suite run against the new snapshot before the PR is opened; the suite asserts every bundled prefix still compiles. Verified by running the script live: all four feeds fetched (372/20/8/12 prefixes, matching the committed snapshot exactly) and the timestamp-only diff was correctly identified as a skip.
This was referenced Aug 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bundled range snapshot in
src/bot-ranges.tsis the one part of identity verification that rots on its own. Vendors rotate their published prefixes, and a stale list doesn't fail loudly — it quietly starts returningspoofedfor legitimate crawlers. Sincespoofedis the verdict anyone would act on, silent drift is the expensive failure mode.Runs Mondays 06:00 UTC, plus
workflow_dispatch.Opens a PR, doesn't commit to main
A range change alters what downstream sites report about real traffic, so the diff deserves eyes. A large removal especially — that flips previously-
verifiedtraffic tospoofed, and is worth checking against the vendor before merging.Three guards
So the automation can't make things worse than having none:
refresh-bot-ranges.mjsalready aborts on a failed feed or a list that shrinks by >50%, rather than writing a truncated snapshot.BOT_RANGES_CAPTURED_ATchanges every run, so a timestamp-only diff is discarded — no weekly no-op PR to train people into rubber-stamping.typecheck+ the test suite run against the new snapshot before the PR opens. The suite asserts every bundled prefix still compiles, so a malformed feed can't ship.Verified live
Ran the script against the real feeds before committing:
All four match the committed snapshot exactly, and the timestamp-only diff was correctly identified as a skip.
Note
Merging a refresh PR doesn't publish. Bump
package.jsonseparately when you want new ranges to reach consumers — which also means a rotation takes two steps to land downstream, worth knowing if a vendor ever moves ranges abruptly.🤖 Generated with Claude Code