fix: publish the merged bot-coverage expansion as 0.10.0 - #17
Merged
Conversation
The AI_BOT_PATTERN expansion (bb8dbbe, merged 2026-04-28) never shipped. release.yml only fires on a package.json version change, and the last bump was 749d275 on 2026-04-26 — so main has carried 43 bot patterns while npm 0.9.0 has 23. Same version number, different content. 22 crawlers are missing from the published build, so production traffic from all of them lands as `is_ai_bot: false` / `ua_category: other` and drops out of every AI-filtered chart: Applebot, Claude-SearchBot, Claude-Web, Google-CloudVertexBot, Google-Agent, GoogleAgent-Mariner, Gemini-Deep-Research, cohere, Amzn-SearchBot, NovaAct, AzureAI-SearchBot, meta-externalfetcher, meta-webindexer, DeepSeek, PanguBot, Webzio-Extended, omgili, Timpibot, Grok, Manus-User, quillbot, MyCentralAIScraperBot Confirmed against ~120 days of production data: bare `Applebot/0.1` UAs (11.3k events) are labelled bot_name 'Apple' but carry is_ai_bot false, while `Applebot-Extended` UAs (2.5k) classify correctly — because only the latter token is in the published pattern. Adds a regression test pinning the invariant that let this through: any UA parseBotName maps to an AI vendor must also match AI_BOT_PATTERN. The two lists were maintained independently, so a bot could get a friendly bot_name while silently staying out of is_ai_bot. Also pins the inverse — SEO, monitoring, and classic search crawlers must NOT be declared-crawler.
Contributor
Author
|
Follow-up completed, and superseded. This PR noted that consumers on
The root cause this PR fixed — a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The published package is missing 22 of 43 bot patterns
AI_BOT_PATTERNwas expanded in bb8dbbe (merged 2026-04-28), but the version was last bumped in 749d275 on 2026-04-26.release.ymlonly fires on apackage.jsonversion change, so the expansion never published.mainand npm0.9.0share a version number and differ in content.Missing from the published build:
Every one of these currently lands in production as
is_ai_bot: false/ua_category: other, so the traffic drops out of any AI-filtered insight.Confirmed against production data
~120 days of
agent_visitevents across apideck.com:bot_nameis_ai_bot…Safari/605.1.15 (Applebot/0.1; …)Apple…(compatible; Applebot/0.1; …)Apple…(compatible; Applebot-Extended/0.1; …)AppleOnly
Applebot-Extendedis in the published pattern, so bareApplebot— which is the majority of Apple's crawl — is invisible. The AI numbers on every dashboard are undercounts, and the undercount is uneven per vendor.The class of bug, not just this instance
parseBotNameandAI_BOT_PATTERNare two independently-maintained lists of the same bots. When they drift, a crawler gets a friendlybot_namewhile silently staying out ofis_ai_bot— the failure is invisible, because the event still looks well-formed.This PR pins the invariant both ways:
parseBotNamemaps to an AI vendor must matchAI_BOT_PATTERN— 26 real UA strings, including the full-Safari-UA shape Apple actually sends (token appended at the end), which is what slipped through.declared-crawler— Ahrefs, Semrush, Majestic, DataForSEO, bingbot, PetalBot, Yandex, Baiduspider. These get friendly labels too and must stay out of AI charts.Tests: 161 → 187, all passing. The new suite fails against published
0.9.0.Merging this
release.ymlpicks up the0.10.0bump on merge, cuts the tag and release, and dispatchespublish.yml. No manual step.Consumers on
^0.9.0(website, developer-docs) need a follow-up bump to^0.10.0— caret on0.xwon't cross the minor.🤖 Generated with Claude Code