From e0915d3a98b10e64959b2502508eef3bfc217419 Mon Sep 17 00:00:00 2001 From: GJ Date: Sun, 2 Aug 2026 06:13:15 +0200 Subject: [PATCH] feat: verify claimed crawler identity against published IP ranges (0.11.0) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit UA strings are forgeable — `curl -A "ChatGPT-User"` is indistinguishable from the real thing at the UA layer, so every is_ai_bot number is a claim rather than a fact. Opt in with `verifyIdentity: true` to check the client IP against the vendor's published crawler ranges. Emits bot_verification (verified | spoofed | unverifiable | not-claimed), bot_verified (tri-state boolean), and bot_verification_reason. Off by default: the verdict is meaningless unless the caller's edge controls x-forwarded-for. Bundles range snapshots for the four vendors that publish machine-readable feeds — OpenAI (372 prefixes), Anthropic (20), Perplexity (8), Apple (12) — with scripts/refresh-bot-ranges.mjs to regenerate. The script aborts on a failed feed or a list that shrinks by more than half, because a stale or truncated snapshot produces false 'spoofed' verdicts on real crawlers, which is the failure mode that actually costs you. Two deliberate design constraints, both derived from 30 days of production traffic rather than assumed: - Vendors with no published feed report 'unverifiable', never 'spoofed'. bot_verified is tri-state for this reason; collapsing "can't check" into "impostor" would falsely accuse Bytespider, Amazonbot and Meta. - Products that fetch from the end user's device are excluded from verdicts. Claude Code runs on a developer's laptop, so the request carries their IP: 6,492 events across 4,486 IPs, 0% in Anthropic's ranges, versus ClaudeBot at 96% from 236 IPs. Perplexity-User behaves the same (0% from 148 IPs). A vendor-level check would have branded ~7k legitimate fetches a month as impersonation. The gate is per product, not per vendor — a `-User` suffix is not the signal, since OpenAI's ChatGPT-User fetches server-side and verifies at ~99%. CIDR matching is dependency-free and lives in its own module so it can be tested in isolation: IPv4 and IPv6, `::` compression, IPv4-mapped addresses from dual-stack edges, /0 and /32 edge cases, and the signed-shift traps that break naive mask implementations. Ranges are pre-compiled once at module load, not per request. Tests: 214 -> 216 files/216 total, including boundary cases either side of every prefix length and the client-side-agent carve-out. --- README.md | 76 ++++++ package.json | 2 +- scripts/refresh-bot-ranges.mjs | 116 +++++++++ src/bot-ranges.ts | 449 +++++++++++++++++++++++++++++++++ src/cidr.ts | 161 ++++++++++++ src/index.ts | 10 + src/track.ts | 11 + src/types.ts | 18 ++ src/verify.ts | 147 +++++++++++ test/track.test.ts | 62 +++++ test/verify.test.ts | 214 ++++++++++++++++ 11 files changed, 1265 insertions(+), 1 deletion(-) create mode 100644 scripts/refresh-bot-ranges.mjs create mode 100644 src/bot-ranges.ts create mode 100644 src/cidr.ts create mode 100644 src/verify.ts create mode 100644 test/verify.test.ts diff --git a/README.md b/README.md index 74e3c8c..fd04293 100644 --- a/README.md +++ b/README.md @@ -306,6 +306,82 @@ Full middleware example: [`README.md → Markdown mirror helpers`](./README.md#m --- +## Advanced: verifying crawler identity against published IP ranges + +User agents are trivially forged — `curl -A "ChatGPT-User"` is indistinguishable +from the real thing at the UA layer. Set `verifyIdentity: true` to check the +client IP against the vendor's published crawler ranges: + +```ts +void trackVisit(request, { + analytics, + verifyIdentity: true, + captureIp: true // not required, but useful for auditing a 'spoofed' verdict +}) +``` + +Three properties land on the event: + +| property | values | +| --- | --- | +| `bot_verification` | `verified` \| `spoofed` \| `unverifiable` \| `not-claimed` | +| `bot_verified` | `true` \| `false` \| `null` — tri-state, for quick filtering | +| `bot_verification_reason` | why, when the verdict is `unverifiable` | + +### What can actually be verified + +Only vendors that publish a machine-readable range feed: **OpenAI**, +**Anthropic**, **Perplexity**, and **Apple**. Bytespider, Amazonbot, Meta and +the rest report `unverifiable` — never `spoofed`. Collapsing "we can't check" +into "impostor" would be a false accusation, which is why `bot_verified` is +tri-state rather than a boolean. + +### Server-side crawlers vs client-side agents + +A published range list covers a vendor's **crawler fleet**, not its products +that fetch from the end user's device. Claude Code runs on a developer's +laptop, so the request carries *their* IP and will never appear in Anthropic's +ranges. Measured over 30 days of production traffic: + +| user agent | events | distinct IPs | in published range | +| --- | ---: | ---: | ---: | +| `ClaudeBot` | 13,671 | 236 | 96% | +| `PerplexityBot` | 6,897 | 158 | 91% | +| `ChatGPT-User` | ~72,000 | 43 | 99% | +| `Claude-User` (claude-code CLI) | 6,492 | 4,486 | **0%** | +| `Perplexity-User` | 493 | 148 | **0%** | + +A naive vendor-level check would brand the bottom two rows — roughly 7,000 +legitimate fetches a month — as impersonation. So the library gates verdicts on +the *product*, returning `unverifiable` with reason `client-side-agent` for +those. Note the distinction is not a `-User` suffix: OpenAI's `ChatGPT-User` +fetches server-side from Azure and verifies at ~99%. + +### Keeping the ranges fresh + +The bundled snapshot is in `src/bot-ranges.ts`, stamped with +`BOT_RANGES_CAPTURED_AT`. Refresh it on a schedule: + +```bash +node scripts/refresh-bot-ranges.mjs +``` + +Freshness is the whole game. Nearly every OpenAI prefix is an Azure block and +Anthropic's are GCP, so "came from a datacenter" proves nothing on its own — +only membership in the *current* published list does. A stale snapshot produces +false `spoofed` verdicts on real crawlers, so the refresh script refuses to +write a list that shrinks by more than half or when any feed errors. + +### Trusting the client IP + +The verdict is only as good as the IP. On Vercel and Cloudflare the edge +overwrites `x-forwarded-for`, so the first hop is trustworthy. Behind a proxy +that passes a client-supplied header through, an attacker controls the value +and `verified` means nothing — confirm your proxy's behaviour before acting on +this data. + +--- + ## Advanced: Peec.ai crawl-insights export [Peec.ai](https://peec.ai)'s **Agent analytics** product ingests a CSV/CLF access log and produces dashboards on top of it. The Peec docs assume you have a Vercel Log Drain → Axiom (or similar) pipeline that emits these eight columns: `timestamp, request_method, request_url, response_status, client_ip, user_agent, country_code, referer`. diff --git a/package.json b/package.json index 3bfddea..d02458f 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@apideck/agent-analytics", - "version": "0.10.0", + "version": "0.11.0", "description": "Track AI agent and bot traffic to your Next.js / Vercel app — PostHog, webhooks, or any custom analytics backend. Detects Claude, ChatGPT, Perplexity, Google-Extended, and more.", "keywords": [ "ai", diff --git a/scripts/refresh-bot-ranges.mjs b/scripts/refresh-bot-ranges.mjs new file mode 100644 index 0000000..972ca1b --- /dev/null +++ b/scripts/refresh-bot-ranges.mjs @@ -0,0 +1,116 @@ +#!/usr/bin/env node +/** + * Regenerate src/bot-ranges.ts from the vendors' published crawler IP feeds. + * + * node scripts/refresh-bot-ranges.mjs + * + * Run this on a schedule. The lists rotate, and a stale snapshot is worse than + * no snapshot: it produces false 'spoofed' verdicts on legitimate crawlers, + * which is exactly the conclusion you'd act on. The script refuses to write a + * file when a feed fails or shrinks implausibly, so a bad fetch can't silently + * empty out a vendor. + */ +import { writeFileSync, readFileSync } from 'node:fs' +import { fileURLToPath } from 'node:url' +import { dirname, join } from 'node:path' + +const OUT = join(dirname(fileURLToPath(import.meta.url)), '..', 'src', 'bot-ranges.ts') + +// Vendor label (as parseBotName returns) -> the feeds that make up its ranges. +const FEEDS = { + ChatGPT: [ + 'https://openai.com/gptbot.json', + 'https://openai.com/chatgpt-user.json', + 'https://openai.com/searchbot.json' + ], + Claude: ['https://claude.com/crawling/bots.json'], + Perplexity: ['https://www.perplexity.ai/perplexitybot.json'], + Apple: ['https://search.developer.apple.com/applebot.json'] +} + +// A feed dropping below this fraction of its previous size signals a partial +// or malformed response rather than a genuine shrink. Bail instead of writing. +const SHRINK_FLOOR = 0.5 + +async function fetchPrefixes(url) { + const res = await fetch(url, { headers: { 'user-agent': 'agent-analytics range refresher' } }) + if (!res.ok) throw new Error(`${url} -> HTTP ${res.status}`) + const body = await res.json() + const prefixes = body?.prefixes + if (!Array.isArray(prefixes)) throw new Error(`${url} -> no prefixes array`) + // Feeds use ipv4Prefix / ipv6Prefix keys; take whichever is present. + return prefixes.map((e) => e.ipv4Prefix ?? e.ipv6Prefix).filter(Boolean) +} + +function previousCounts() { + try { + const src = readFileSync(OUT, 'utf8') + const counts = {} + for (const vendor of Object.keys(FEEDS)) { + const block = src.match(new RegExp(` ${vendor}: \\[([\\s\\S]*?)\\n \\]`)) + counts[vendor] = block ? (block[1].match(/'/g) || []).length / 2 : 0 + } + return counts + } catch { + return {} + } +} + +const before = previousCounts() +const vendors = {} + +for (const [vendor, urls] of Object.entries(FEEDS)) { + const all = new Set() + for (const url of urls) { + const got = await fetchPrefixes(url) // throws — a failed feed must abort the run + got.forEach((p) => all.add(p)) + console.log(` ${url} -> ${got.length}`) + } + const list = [...all].sort() + const prev = before[vendor] ?? 0 + if (prev && list.length < prev * SHRINK_FLOOR) { + throw new Error( + `${vendor}: ${list.length} prefixes vs ${prev} previously — refusing to write a suspiciously small list` + ) + } + vendors[vendor] = list + console.log(`${vendor}: ${list.length} prefixes (was ${prev})`) +} + +const capturedAt = new Date().toISOString() +const body = `/** + * Published crawler IP ranges, vendor by vendor. + * + * GENERATED FILE — do not edit by hand. Regenerate with: + * + * node scripts/refresh-bot-ranges.mjs + * + * Keys match the labels {@link parseBotName} returns, so a claimed identity + * maps to its range list without a translation table. + * + * Only vendors that publish a machine-readable feed appear here. A vendor's + * absence means "cannot be verified", never "not a real bot" — see + * {@link verifyBotIdentity} for how that distinction is surfaced. + * + * Freshness is the whole ballgame: these lists rotate. Almost every OpenAI + * prefix is an Azure block and every Anthropic prefix is GCP or similar, so + * "came from a datacenter" proves nothing on its own — only membership in the + * current published list does. A stale snapshot produces false 'spoofed' + * verdicts on legitimate crawlers, which is the failure mode to fear. + */ + +/** When these ranges were captured from the vendor feeds (UTC). */ +export const BOT_RANGES_CAPTURED_AT = '${capturedAt}' + +export const BOT_IP_RANGES: Readonly> = { +${Object.entries(vendors) + .map(([name, cidrs]) => ` ${name}: [\n${cidrs.map((c) => ` '${c}',`).join('\n')}\n ],`) + .join('\n')} +} + +/** Vendor labels this build can verify. Anything else yields a null verdict. */ +export const VERIFIABLE_VENDORS: readonly string[] = Object.keys(BOT_IP_RANGES) +` + +writeFileSync(OUT, body) +console.log(`\nwrote ${OUT}`) diff --git a/src/bot-ranges.ts b/src/bot-ranges.ts new file mode 100644 index 0000000..f7647e0 --- /dev/null +++ b/src/bot-ranges.ts @@ -0,0 +1,449 @@ +/** + * Published crawler IP ranges, vendor by vendor. + * + * GENERATED FILE — do not edit by hand. Regenerate with: + * + * node scripts/refresh-bot-ranges.mjs + * + * Keys match the labels {@link parseBotName} returns, so a claimed identity + * maps to its range list without a translation table. + * + * Only vendors that publish a machine-readable feed appear here. A vendor's + * absence means "cannot be verified", never "not a real bot" — see + * {@link verifyBotIdentity} for how that distinction is surfaced. + * + * Freshness is the whole ballgame: these lists rotate. Almost every OpenAI + * prefix is an Azure block and every Anthropic prefix is GCP or similar, so + * "came from a datacenter" proves nothing on its own — only membership in the + * current published list does. A stale snapshot produces false 'spoofed' + * verdicts on legitimate crawlers, which is the failure mode to fear. + */ + +/** When these ranges were captured from the vendor feeds (UTC). */ +export const BOT_RANGES_CAPTURED_AT = '2026-08-02T00:00:00Z' + +export const BOT_IP_RANGES: Readonly> = { + ChatGPT: [ + '104.208.184.192/28', + '104.208.184.208/28', + '104.210.139.192/28', + '104.210.139.224/28', + '104.210.140.128/28', + '128.85.198.32/28', + '13.65.138.112/28', + '13.65.138.96/28', + '13.67.72.16/28', + '13.70.107.160/28', + '13.71.2.208/28', + '13.76.115.224/28', + '13.76.115.240/28', + '13.76.116.80/28', + '13.76.32.208/28', + '13.83.167.128/28', + '13.83.237.176/28', + '132.196.82.48/28', + '132.196.86.0/24', + '134.149.233.80/28', + '135.116.136.160/28', + '135.13.64.240/28', + '135.220.73.208/28', + '135.220.73.240/28', + '135.234.64.0/24', + '135.237.131.208/28', + '135.237.133.48/28', + '137.135.191.176/28', + '138.91.30.48/28', + '138.91.46.96/28', + '145.132.1.32/28', + '145.132.136.96/28', + '145.133.0.176/28', + '158.158.5.32/28', + '168.63.252.240/28', + '172.162.248.64/28', + '172.170.1.80/28', + '172.170.225.0/28', + '172.170.241.80/28', + '172.170.8.208/28', + '172.171.4.176/28', + '172.175.152.224/28', + '172.178.140.144/28', + '172.178.141.112/28', + '172.178.141.128/28', + '172.182.193.224/28', + '172.182.193.80/28', + '172.182.194.144/28', + '172.182.194.32/28', + '172.182.195.48/28', + '172.182.202.0/25', + '172.182.204.0/24', + '172.182.207.0/25', + '172.182.209.208/28', + '172.182.211.192/28', + '172.182.213.192/28', + '172.182.214.0/24', + '172.182.215.0/24', + '172.182.224.0/28', + '172.183.143.224/28', + '172.183.222.128/28', + '172.192.112.208/28', + '172.192.88.192/28', + '172.192.97.32/28', + '172.197.160.192/28', + '172.197.161.208/28', + '172.197.170.80/28', + '172.197.203.16/28', + '172.199.137.80/28', + '172.202.102.112/28', + '172.203.190.128/28', + '172.204.27.16/28', + '172.204.28.224/28', + '172.204.96.32/28', + '172.204.96.48/28', + '172.204.96.80/28', + '172.205.189.192/28', + '172.207.1.32/28', + '172.208.128.32/28', + '172.208.128.48/28', + '172.212.159.64/28', + '172.212.172.160/28', + '172.213.21.16/28', + '172.215.215.32/28', + '172.215.218.96/28', + '191.233.1.112/28', + '191.233.1.128/28', + '191.233.194.32/28', + '191.233.196.112/28', + '191.233.199.160/28', + '191.233.2.0/28', + '191.235.66.16/28', + '191.235.99.80/28', + '191.237.249.64/28', + '191.239.245.16/28', + '20.102.212.144/28', + '20.113.211.112/28', + '20.113.225.112/28', + '20.125.112.224/28', + '20.125.144.144/28', + '20.125.66.80/28', + '20.14.99.96/28', + '20.161.75.208/28', + '20.168.18.32/28', + '20.168.7.192/28', + '20.168.7.240/28', + '20.169.6.224/28', + '20.169.7.48/28', + '20.169.72.112/28', + '20.169.73.176/28', + '20.169.73.32/28', + '20.169.73.64/28', + '20.169.77.0/25', + '20.169.78.112/28', + '20.169.78.128/28', + '20.169.78.144/28', + '20.169.78.160/28', + '20.169.78.176/28', + '20.169.78.192/28', + '20.169.78.208/28', + '20.169.78.48/28', + '20.169.78.64/28', + '20.169.78.80/28', + '20.169.78.96/28', + '20.169.86.224/28', + '20.169.86.240/28', + '20.169.87.112/28', + '20.17.108.96/28', + '20.170.184.16/28', + '20.170.184.32/28', + '20.170.184.48/28', + '20.170.184.64/28', + '20.170.184.80/28', + '20.171.123.64/28', + '20.171.206.0/24', + '20.171.207.0/24', + '20.171.53.224/28', + '20.172.29.32/28', + '20.193.233.240/28', + '20.193.50.32/28', + '20.194.0.208/28', + '20.194.1.0/28', + '20.198.67.96/28', + '20.199.211.160/28', + '20.199.242.0/28', + '20.200.212.240/28', + '20.204.24.240/28', + '20.210.154.128/28', + '20.210.174.208/28', + '20.210.211.192/28', + '20.215.187.208/28', + '20.215.188.192/28', + '20.215.214.16/28', + '20.215.219.128/28', + '20.215.219.160/28', + '20.215.219.208/28', + '20.215.220.112/28', + '20.215.220.128/28', + '20.215.220.144/28', + '20.215.220.160/28', + '20.215.220.176/28', + '20.215.220.192/28', + '20.215.220.208/28', + '20.215.220.64/28', + '20.215.220.80/28', + '20.215.220.96/28', + '20.218.30.240/28', + '20.219.71.192/28', + '20.222.36.192/28', + '20.226.32.80/28', + '20.227.140.32/28', + '20.228.106.176/28', + '20.235.75.208/28', + '20.235.87.224/28', + '20.249.63.208/28', + '20.25.151.224/28', + '20.250.136.64/28', + '20.250.136.80/28', + '20.250.6.128/28', + '20.27.94.128/28', + '20.42.10.176/28', + '20.45.178.144/28', + '20.48.120.208/28', + '20.52.125.160/28', + '20.55.129.0/28', + '20.55.229.144/28', + '20.57.199.192/28', + '20.63.180.96/28', + '20.63.221.64/28', + '20.79.59.112/28', + '20.81.183.64/28', + '20.83.243.176/28', + '20.97.189.96/28', + '23.102.140.144/28', + '23.102.141.32/28', + '23.98.142.176/28', + '23.98.179.16/28', + '23.98.186.176/28', + '23.98.186.192/28', + '23.98.186.64/28', + '23.98.186.96/28', + '4.151.119.48/28', + '4.151.241.240/28', + '4.151.71.176/28', + '4.189.118.208/28', + '4.189.119.48/28', + '4.196.118.112/28', + '4.197.115.112/28', + '4.197.19.176/28', + '4.197.22.112/28', + '4.197.64.0/28', + '4.197.64.16/28', + '4.197.64.48/28', + '4.197.64.64/28', + '4.198.72.16/28', + '4.198.96.112/28', + '4.201.232.64/28', + '4.201.232.80/28', + '4.203.96.80/28', + '4.205.128.176/28', + '4.218.24.64/28', + '4.226.200.16/28', + '4.226.226.32/28', + '4.227.36.0/25', + '40.116.73.208/28', + '40.122.235.112/28', + '40.67.175.0/25', + '40.67.183.160/28', + '40.67.183.176/28', + '40.78.161.48/28', + '40.81.134.128/28', + '40.81.134.144/28', + '40.81.234.144/28', + '40.81.67.96/28', + '40.84.181.32/28', + '40.84.221.208/28', + '40.84.221.224/28', + '40.90.214.16/28', + '48.193.44.32/28', + '48.221.184.112/28', + '48.221.184.80/28', + '48.221.184.96/28', + '48.221.40.176/28', + '51.107.70.192/28', + '51.116.2.80/28', + '51.116.221.96/28', + '51.56.40.80/28', + '51.57.0.96/28', + '51.59.24.64/28', + '51.59.24.80/28', + '51.59.40.80/28', + '51.59.40.96/28', + '51.59.48.80/28', + '51.59.48.96/28', + '51.8.102.0/24', + '51.8.155.112/28', + '51.8.155.48/28', + '51.8.155.64/28', + '51.8.187.224/28', + '52.148.129.32/28', + '52.153.130.64/28', + '52.154.22.48/28', + '52.156.77.144/28', + '52.159.227.32/28', + '52.159.249.96/28', + '52.161.49.224/28', + '52.161.49.96/28', + '52.165.212.16/28', + '52.165.212.32/28', + '52.165.212.48/28', + '52.172.129.160/28', + '52.172.251.112/28', + '52.173.219.112/28', + '52.173.219.96/28', + '52.173.221.16/28', + '52.173.221.176/28', + '52.173.221.208/28', + '52.173.234.16/28', + '52.173.234.80/28', + '52.173.235.80/28', + '52.183.217.240/28', + '52.187.246.128/28', + '52.190.137.144/28', + '52.190.137.16/28', + '52.190.139.48/28', + '52.190.142.64/28', + '52.190.190.16/28', + '52.225.75.208/28', + '52.230.152.0/24', + '52.230.163.32/28', + '52.230.164.176/28', + '52.231.30.48/28', + '52.231.34.176/28', + '52.231.39.144/28', + '52.231.39.192/28', + '52.231.49.48/28', + '52.231.50.64/28', + '52.236.94.144/28', + '52.241.146.208/28', + '52.242.132.224/28', + '52.242.132.240/28', + '52.242.245.208/28', + '52.252.113.240/28', + '52.255.109.112/28', + '52.255.109.128/28', + '52.255.109.144/28', + '52.255.109.80/28', + '52.255.109.96/28', + '52.255.111.0/28', + '52.255.111.112/28', + '52.255.111.32/28', + '52.255.111.48/28', + '52.255.111.80/28', + '57.154.174.112/28', + '57.154.175.0/28', + '57.154.187.32/28', + '68.154.28.96/28', + '68.218.30.112/28', + '68.220.57.64/28', + '68.221.67.192/28', + '68.221.67.224/28', + '68.221.67.240/28', + '70.153.139.208/28', + '70.153.189.192/28', + '70.153.190.16/28', + '70.153.76.16/28', + '70.153.87.224/28', + '70.156.144.64/28', + '70.156.152.80/28', + '70.156.152.96/28', + '74.161.200.96/28', + '74.224.217.64/28', + '74.226.253.160/28', + '74.249.86.176/28', + '74.7.175.128/25', + '74.7.227.0/25', + '74.7.227.128/25', + '74.7.228.0/25', + '74.7.228.128/25', + '74.7.229.0/25', + '74.7.229.128/25', + '74.7.230.0/25', + '74.7.241.0/25', + '74.7.241.128/25', + '74.7.242.0/25', + '74.7.242.128/25', + '74.7.243.0/25', + '74.7.243.128/25', + '74.7.244.0/25', + '74.7.35.112/28', + '74.7.35.48/28', + '74.7.36.64/28', + '74.7.36.80/28', + '74.7.36.96/28', + '85.211.241.128/28', + '9.129.0.0/17', + '9.160.128.16/28', + '9.160.128.32/28', + '9.160.128.64/28', + '9.160.163.128/28', + '9.160.164.128/28', + '9.160.34.144/28', + '9.160.36.16/28', + '9.160.96.16/28', + '9.163.101.48/28', + '9.205.25.128/28', + '9.205.8.48/28', + '9.223.181.208/28', + '9.234.96.192/28', + '9.234.97.128/28', + '9.234.97.96/28', + '9.235.40.32/28', + ], + Claude: [ + '136.107.176.208/32', + '216.73.216.0/22', + '34.11.34.31/32', + '34.150.241.79/32', + '34.162.191.81/32', + '34.162.230.222/32', + '34.162.244.71/32', + '34.182.140.95/32', + '34.182.161.143/32', + '34.182.218.27/32', + '34.182.220.85/32', + '34.182.222.37/32', + '34.182.225.167/32', + '34.182.226.151/32', + '34.182.226.221/32', + '34.186.108.163/32', + '34.85.172.162/32', + '35.221.29.174/32', + '35.245.175.129/32', + '35.245.89.239/32', + ], + Perplexity: [ + '107.20.236.150/32', + '18.210.92.235/32', + '18.97.1.228/30', + '18.97.9.96/29', + '3.211.124.183/32', + '3.222.232.239/32', + '3.224.62.45/32', + '3.231.139.107/32', + ], + Apple: [ + '17.22.237.0/24', + '17.22.245.0/24', + '17.22.253.0/24', + '17.241.193.160/27', + '17.241.200.160/27', + '17.241.208.160/27', + '17.241.219.0/24', + '17.241.227.0/24', + '17.241.75.0/24', + '17.246.15.0/24', + '17.246.19.0/24', + '17.246.23.0/24', + ], +} + +/** Vendor labels this build can verify. Anything else yields a null verdict. */ +export const VERIFIABLE_VENDORS: readonly string[] = Object.keys(BOT_IP_RANGES) diff --git a/src/cidr.ts b/src/cidr.ts new file mode 100644 index 0000000..2cf2637 --- /dev/null +++ b/src/cidr.ts @@ -0,0 +1,161 @@ +/** + * Dependency-free CIDR matching for IPv4 and IPv6. + * + * Kept separate from {@link verifyBotIdentity} so the matching logic can be + * tested in isolation — a wrong answer here silently turns a real crawler into + * a "spoofed" one, which is worse than not verifying at all. + * + * Everything is parsed once into numeric form (`number` for v4, `bigint` for + * v6) and compared with masks. No allocation per request beyond the parse of + * the incoming IP. + */ + +/** Parse dotted-quad IPv4 into a 32-bit unsigned integer. */ +export function ipv4ToInt(ip: string): number | null { + const parts = ip.split('.') + if (parts.length !== 4) return null + let out = 0 + for (const part of parts) { + // Reject empty, non-numeric, out-of-range, and leading-zero forms + // (`01.2.3.4` is ambiguous — some parsers read it as octal). + if (!/^\d{1,3}$/.test(part)) return null + if (part.length > 1 && part[0] === '0') return null + const n = Number(part) + if (n > 255) return null + out = (out << 8) | n + } + // `>>> 0` converts the signed 32-bit result back to unsigned. + return out >>> 0 +} + +/** + * Parse IPv6 (including `::` compression and IPv4-mapped tails like + * `::ffff:1.2.3.4`) into a 128-bit BigInt. + */ +export function ipv6ToBigInt(ip: string): bigint | null { + let text = ip + // Strip a zone index (`fe80::1%eth0`) — irrelevant for range membership. + const zone = text.indexOf('%') + if (zone !== -1) text = text.slice(0, zone) + if (!text || text.indexOf(':') === -1) return null + + // An embedded IPv4 tail contributes the low 32 bits. + let tail: number | null = null + const lastColon = text.lastIndexOf(':') + const maybeV4 = text.slice(lastColon + 1) + if (maybeV4.indexOf('.') !== -1) { + tail = ipv4ToInt(maybeV4) + if (tail === null) return null + text = text.slice(0, lastColon + 1) + '0:0' + } + + const halves = text.split('::') + if (halves.length > 2) return null + const head = halves[0] ? halves[0].split(':') : [] + const rest = halves.length === 2 ? (halves[1] ? halves[1].split(':') : []) : null + + let groups: string[] + if (rest === null) { + groups = head + if (groups.length !== 8) return null + } else { + const fill = 8 - head.length - rest.length + if (fill < 0) return null + groups = [...head, ...Array(fill).fill('0'), ...rest] + } + + let out = 0n + for (const g of groups) { + if (!/^[0-9a-fA-F]{1,4}$/.test(g)) return null + out = (out << 16n) | BigInt(parseInt(g, 16)) + } + // Overwrite the low 32 bits with the embedded IPv4 value when present. + if (tail !== null) out = ((out >> 32n) << 32n) | BigInt(tail >>> 0) + return out +} + +interface V4Range { + net: number + mask: number +} +interface V6Range { + net: bigint + bits: number +} + +export interface CompiledRanges { + v4: V4Range[] + v6: V6Range[] +} + +/** + * Pre-compile a list of CIDR strings into numeric form. Invalid entries are + * dropped rather than thrown — a malformed line in a vendor's published feed + * shouldn't take down the whole check. + */ +export function compileRanges(cidrs: readonly string[]): CompiledRanges { + const v4: V4Range[] = [] + const v6: V6Range[] = [] + for (const cidr of cidrs) { + const slash = cidr.lastIndexOf('/') + if (slash === -1) continue + const addr = cidr.slice(0, slash) + const bits = Number(cidr.slice(slash + 1)) + if (!Number.isInteger(bits) || bits < 0) continue + + if (addr.indexOf(':') !== -1) { + if (bits > 128) continue + const net = ipv6ToBigInt(addr) + if (net === null) continue + v6.push({ net: bits === 0 ? 0n : (net >> BigInt(128 - bits)) << BigInt(128 - bits), bits }) + } else { + if (bits > 32) continue + const net = ipv4ToInt(addr) + if (net === null) continue + // `bits === 0` needs special handling: `<<32` is a no-op in JS, not zero. + const mask = bits === 0 ? 0 : (0xffffffff << (32 - bits)) >>> 0 + v4.push({ net: (net & mask) >>> 0, mask }) + } + } + return { v4, v6 } +} + +/** True when `ip` falls inside any range in the pre-compiled set. */ +export function ipInRanges(ip: string, ranges: CompiledRanges): boolean { + if (!ip) return false + const trimmed = ip.trim() + if (!trimmed) return false + + if (trimmed.indexOf(':') !== -1) { + const value = ipv6ToBigInt(trimmed) + if (value === null) return false + // An IPv4-mapped address (::ffff:a.b.c.d) must also be checked against the + // v4 list — Vercel and Cloudflare both emit this form on dual-stack edges. + const V4_MAPPED_PREFIX = 0xffffn << 32n + if ((value >> 32n) === V4_MAPPED_PREFIX >> 32n) { + const low = Number(value & 0xffffffffn) >>> 0 + if (matchV4(low, ranges.v4)) return true + } + for (const r of ranges.v6) { + if (r.bits === 0) return true + if ((value >> BigInt(128 - r.bits)) << BigInt(128 - r.bits) === r.net) return true + } + return false + } + + const value = ipv4ToInt(trimmed) + if (value === null) return false + return matchV4(value, ranges.v4) +} + +function matchV4(value: number, list: readonly V4Range[]): boolean { + for (const r of list) { + if (((value & r.mask) >>> 0) === r.net) return true + } + return false +} + +/** Convenience wrapper — compiles on every call, so prefer {@link ipInRanges}. */ +export function ipInCidr(ip: string, cidr: string): boolean { + return ipInRanges(ip, compileRanges([cidr])) +} diff --git a/src/index.ts b/src/index.ts index be2f20d..fe714ca 100644 --- a/src/index.ts +++ b/src/index.ts @@ -12,6 +12,16 @@ export { } from './bots.js' export type { AgentClassification, AgentKind, HeadlessDetection } from './bots.js' export { hashId } from './hash.js' +export { + clientIpFromRequest, + verifiableVendors, + verifyBotIdentity, + verifyRequest +} from './verify.js' +export type { BotVerification, VerificationVerdict } from './verify.js' +export { BOT_IP_RANGES, BOT_RANGES_CAPTURED_AT, VERIFIABLE_VENDORS } from './bot-ranges.js' +export { compileRanges, ipInCidr, ipInRanges } from './cidr.js' +export type { CompiledRanges } from './cidr.js' export { posthogAnalytics } from './adapters/posthog.js' export { webhookAnalytics } from './adapters/webhook.js' export { customAnalytics } from './adapters/custom.js' diff --git a/src/track.ts b/src/track.ts index 1be6dbc..3254cf2 100644 --- a/src/track.ts +++ b/src/track.ts @@ -1,4 +1,5 @@ import { classifyRequest, detectHeadless, isAiBot, isHttpClient } from './bots.js' +import { verifyBotIdentity } from './verify.js' import { hashId } from './hash.js' import type { TrackVisitOptions } from './types.js' @@ -50,6 +51,9 @@ export async function trackVisit( : null const geo = opts.captureGeo ? extractGeo(req) : null const classification = classifyRequest(req) + // Only run the range check when asked — it is pure CPU over pre-compiled + // masks, but the verdict is misleading unless the caller trusts `ip`. + const verification = opts.verifyIdentity ? verifyBotIdentity(userAgent, ip) : null const event = { event: opts.eventName ?? 'agent_visit', @@ -70,6 +74,13 @@ export async function trackVisit( coding_agent_hint: classification.codingAgentHint, headless_score: classification.headless?.score ?? 0, headless_likely: classification.headless?.likely ?? false, + ...(verification + ? { + bot_verified: verification.verified, + bot_verification: verification.verdict, + ...(verification.reason ? { bot_verification_reason: verification.reason } : {}) + } + : {}), referer, source: opts.source ?? null, ...opts.properties diff --git a/src/types.ts b/src/types.ts index 235d14c..ad94143 100644 --- a/src/types.ts +++ b/src/types.ts @@ -57,6 +57,24 @@ export interface TrackVisitOptions { * Enable for log-style exports (e.g. Peec.ai's crawl-insights CSV). */ captureCountry?: boolean + /** + * When `true`, check the request's claimed crawler identity against the + * vendor's published IP ranges and emit `bot_verified` (tri-state) plus + * `bot_verification` (`verified` | `spoofed` | `unverifiable` | + * `not-claimed`) on the event. + * + * UA strings are trivially forgeable — `curl -A "ChatGPT-User"` is + * indistinguishable from the real thing without this check. Off by default + * because it only means something when the client IP is trustworthy: on + * Vercel and Cloudflare the edge overwrites `x-forwarded-for`, but behind a + * proxy that passes the client-supplied header through, an attacker controls + * the value and a `verified` verdict is worthless. + * + * Only vendors that publish a machine-readable range feed can be verified — + * currently OpenAI, Anthropic, Perplexity, and Apple. Everything else yields + * `unverifiable`, never `spoofed`. + */ + verifyIdentity?: boolean /** * When `true`, emit `region`, `city`, `latitude`, `longitude`, and * `timezone` derived from Vercel's `x-vercel-ip-*` edge headers. Values diff --git a/src/verify.ts b/src/verify.ts new file mode 100644 index 0000000..f6c2eb9 --- /dev/null +++ b/src/verify.ts @@ -0,0 +1,147 @@ +import { parseBotName } from './bots.js' +import { BOT_IP_RANGES, VERIFIABLE_VENDORS } from './bot-ranges.js' +import { compileRanges, ipInRanges, type CompiledRanges } from './cidr.js' + +/** + * Verdict on whether a request's claimed crawler identity holds up against the + * vendor's published IP ranges. + * + * - `'verified'` — the UA claims a vendor we can check, and the client IP is + * inside that vendor's published ranges. High confidence. + * - `'spoofed'` — the UA claims a vendor we can check, and the IP is **not** + * in its ranges. Someone is impersonating the crawler. + * - `'unverifiable'` — the UA claims a vendor that publishes no feed we bundle + * (Bytespider, Amazon, Meta, …), or no usable client IP was available. + * - `'not-claimed'` — the UA doesn't claim a verifiable crawler at all. This is + * the normal verdict for browsers and HTTP clients; it is *not* a negative + * finding. + */ +export type VerificationVerdict = 'verified' | 'spoofed' | 'unverifiable' | 'not-claimed' + +/** Why a request could not be judged. Only set when verdict is 'unverifiable'. */ +export type UnverifiableReason = 'no-published-ranges' | 'client-side-agent' | 'no-client-ip' + +export interface BotVerification { + verdict: VerificationVerdict + reason?: UnverifiableReason + /** Vendor label the UA claims, when it claims one. */ + claimed: string | null + /** + * Convenience boolean for filtering: `true` only for `'verified'`, `false` + * only for `'spoofed'`, `null` when no judgement was possible. Deliberately + * tri-state — collapsing "unverifiable" into `false` would brand every + * Bytespider and Amazonbot hit an impostor. + */ + verified: boolean | null +} + +/** + * Products that fetch from the **end user's own device**, not from the + * vendor's infrastructure. + * + * A published range list only covers a vendor's server-side crawler fleet. + * When the fetch originates on a developer's laptop, the client IP is theirs + * and will never appear in the vendor's ranges — so a range check produces + * 'spoofed' for entirely legitimate traffic. + * + * This is not cosmetic. Measured against 30 days of production data: + * + * Claude-User (claude-code CLI) 6,492 events 4,486 IPs 0% in range + * Perplexity-User 493 events 148 IPs 0% in range + * ClaudeBot 13,671 events 236 IPs 96% in range + * PerplexityBot 6,897 events 158 IPs 91% in range + * + * Treating the first two as impostors would have falsely accused ~7k real + * fetches a month. The distinction is per *product*, not per vendor, and not + * inferable from a `-User` suffix — OpenAI's ChatGPT-User fetches server-side + * from Azure and verifies at ~99%. + */ +const CLIENT_SIDE_AGENT_PATTERN = /claude-code|perplexity-user|cursor|windsurf|cline|aider/i + +/** + * Products known to fetch server-side from ranges the vendor publishes. Only + * these can earn a 'verified' or 'spoofed' verdict; anything else is reported + * 'unverifiable' so the data never overstates what was actually checked. + */ +const SERVER_SIDE_CRAWLER_PATTERN = + /ClaudeBot|Claude-SearchBot|GPTBot|OAI-SearchBot|ChatGPT-User|PerplexityBot|Applebot/i + +// Compile each vendor's ranges once at module load rather than per request. +const COMPILED: Record = {} +for (const vendor of VERIFIABLE_VENDORS) { + COMPILED[vendor] = compileRanges(BOT_IP_RANGES[vendor] ?? []) +} + +/** Vendor labels this build can produce a verified/spoofed verdict for. */ +export function verifiableVendors(): readonly string[] { + return VERIFIABLE_VENDORS +} + +/** + * Check a claimed crawler identity against the vendor's published IP ranges. + * + * Pass the client IP you already trust — on Vercel and Cloudflare that is the + * first hop of `x-forwarded-for`. If your edge doesn't strip client-supplied + * `X-Forwarded-For`, an attacker controls this value and a `'verified'` verdict + * means nothing; verify your proxy's behaviour before relying on it. + */ +export function verifyBotIdentity( + userAgent: string | null | undefined, + ip: string | null | undefined +): BotVerification { + const ua = userAgent ?? '' + const claimed = parseBotName(userAgent) + const ranges = COMPILED[claimed] + + if (!ranges) { + // Either not a crawler at all, or a crawler with no published feed. Both + // are "no judgement", but the caller may want to tell them apart. + const isKnownCrawler = claimed !== 'Other' && claimed !== 'Browser' + return { + verdict: isKnownCrawler ? 'unverifiable' : 'not-claimed', + ...(isKnownCrawler ? { reason: 'no-published-ranges' as const } : {}), + claimed: isKnownCrawler ? claimed : null, + verified: null + } + } + + // Order matters: a client-side agent must be excluded before the range check, + // and the server-side allowlist gates everything else, so an unrecognised + // product from a covered vendor is never accused on a partial range list. + if (CLIENT_SIDE_AGENT_PATTERN.test(ua) || !SERVER_SIDE_CRAWLER_PATTERN.test(ua)) { + return { + verdict: 'unverifiable', + reason: 'client-side-agent', + claimed, + verified: null + } + } + + const trimmed = (ip ?? '').trim() + if (!trimmed) { + return { verdict: 'unverifiable', reason: 'no-client-ip', claimed, verified: null } + } + + const inRange = ipInRanges(trimmed, ranges) + return { + verdict: inRange ? 'verified' : 'spoofed', + claimed, + verified: inRange + } +} + +/** + * Extract the client IP the way {@link trackVisit} does — first hop of + * `x-forwarded-for`, falling back to the platform-specific headers. + */ +export function clientIpFromRequest(req: Request): string { + const forwarded = req.headers.get('x-forwarded-for') || '' + const first = forwarded.split(',')[0]?.trim() + if (first) return first + return (req.headers.get('cf-connecting-ip') || req.headers.get('x-real-ip') || '').trim() +} + +/** Verify straight from a request object. */ +export function verifyRequest(req: Request): BotVerification { + return verifyBotIdentity(req.headers.get('user-agent'), clientIpFromRequest(req)) +} diff --git a/test/track.test.ts b/test/track.test.ts index d0fa7bd..e9ace75 100644 --- a/test/track.test.ts +++ b/test/track.test.ts @@ -420,3 +420,65 @@ describe('trackVisit', () => { expect(a.distinctId).toBe(b.distinctId) }) }) + +describe('trackVisit — verifyIdentity', () => { + const CHATGPT_UA = 'Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)' + const REAL_OPENAI_IP = '104.208.184.193' + + async function capture(headers: Record, opts: Record = {}) { + const spy = vi.fn() + await trackVisit(makeRequest('https://example.com/page', headers), { + analytics: customAnalytics(spy), + ...opts + }) + return spy.mock.calls[0]![0] as CaptureEvent + } + + it('omits the verification properties entirely when not opted in', () => { + // Absent, not null — so existing dashboards don't gain a column of nulls. + return capture({ 'user-agent': CHATGPT_UA, 'x-forwarded-for': REAL_OPENAI_IP }).then((e) => { + expect('bot_verified' in e.properties).toBe(false) + expect('bot_verification' in e.properties).toBe(false) + }) + }) + + it('marks a real crawler verified', async () => { + const e = await capture( + { 'user-agent': CHATGPT_UA, 'x-forwarded-for': REAL_OPENAI_IP }, + { verifyIdentity: true } + ) + expect(e.properties.bot_verified).toBe(true) + expect(e.properties.bot_verification).toBe('verified') + expect(e.properties.bot_name).toBe('ChatGPT') + }) + + it('marks the same UA from another IP spoofed', async () => { + const e = await capture( + { 'user-agent': CHATGPT_UA, 'x-forwarded-for': '1.2.3.4' }, + { verifyIdentity: true } + ) + expect(e.properties.bot_verified).toBe(false) + expect(e.properties.bot_verification).toBe('spoofed') + // Still labelled ChatGPT and still is_ai_bot — the verdict is the extra + // dimension, it does not rewrite the classification. + expect(e.properties.bot_name).toBe('ChatGPT') + expect(e.properties.is_ai_bot).toBe(true) + }) + + it('uses the first x-forwarded-for hop, not a trailing proxy', async () => { + const e = await capture( + { 'user-agent': CHATGPT_UA, 'x-forwarded-for': `${REAL_OPENAI_IP}, 10.0.0.1` }, + { verifyIdentity: true } + ) + expect(e.properties.bot_verification).toBe('verified') + }) + + it('reports unverifiable for vendors without a published feed', async () => { + const e = await capture( + { 'user-agent': 'Mozilla/5.0 (compatible; Bytespider/1.0)', 'x-forwarded-for': '1.2.3.4' }, + { verifyIdentity: true } + ) + expect(e.properties.bot_verified).toBeNull() + expect(e.properties.bot_verification).toBe('unverifiable') + }) +}) diff --git a/test/verify.test.ts b/test/verify.test.ts new file mode 100644 index 0000000..3174435 --- /dev/null +++ b/test/verify.test.ts @@ -0,0 +1,214 @@ +import { describe, expect, it } from 'vitest' +import { compileRanges, ipInCidr, ipInRanges, ipv4ToInt, ipv6ToBigInt } from '../src/cidr.js' +import { verifyBotIdentity, verifyRequest, verifiableVendors } from '../src/verify.js' +import { BOT_IP_RANGES } from '../src/bot-ranges.js' + +describe('ipv4ToInt', () => { + it('parses dotted quads', () => { + expect(ipv4ToInt('0.0.0.0')).toBe(0) + expect(ipv4ToInt('255.255.255.255')).toBe(4294967295) + expect(ipv4ToInt('192.168.1.1')).toBe(3232235777) + }) + + it('rejects malformed input', () => { + for (const bad of ['', '1.2.3', '1.2.3.4.5', '256.1.1.1', 'a.b.c.d', '1.2.3.-1', ' 1.2.3.4']) { + expect(ipv4ToInt(bad)).toBeNull() + } + }) + + it('rejects leading zeros, which some parsers read as octal', () => { + // 010.1.1.1 is 8.1.1.1 in octal but 10.1.1.1 in decimal — refusing it + // avoids disagreeing with whatever produced the range list. + expect(ipv4ToInt('010.1.1.1')).toBeNull() + expect(ipv4ToInt('01.2.3.4')).toBeNull() + }) +}) + +describe('ipv6ToBigInt', () => { + it('parses full and compressed forms to the same value', () => { + expect(ipv6ToBigInt('2001:0db8:0000:0000:0000:0000:0000:0001')).toBe( + ipv6ToBigInt('2001:db8::1') + ) + expect(ipv6ToBigInt('::')).toBe(0n) + expect(ipv6ToBigInt('::1')).toBe(1n) + }) + + it('parses IPv4-mapped addresses', () => { + expect(ipv6ToBigInt('::ffff:192.168.1.1')).toBe(0xffffn * 2n ** 32n + 3232235777n) + }) + + it('strips zone indexes', () => { + expect(ipv6ToBigInt('fe80::1%eth0')).toBe(ipv6ToBigInt('fe80::1')) + }) + + it('rejects malformed input', () => { + for (const bad of ['', '1.2.3.4', '2001::db8::1', 'gggg::1', '1:2:3:4:5:6:7', '1:2:3:4:5:6:7:8:9']) { + expect(ipv6ToBigInt(bad)).toBeNull() + } + }) +}) + +describe('ipInCidr', () => { + it('matches IPv4 boundaries exactly', () => { + // /28 spans .192 through .207 — the two adjacent addresses must miss. + expect(ipInCidr('104.208.184.192', '104.208.184.192/28')).toBe(true) + expect(ipInCidr('104.208.184.207', '104.208.184.192/28')).toBe(true) + expect(ipInCidr('104.208.184.191', '104.208.184.192/28')).toBe(false) + expect(ipInCidr('104.208.184.208', '104.208.184.192/28')).toBe(false) + }) + + it('handles /32 host routes and /0 catch-alls', () => { + expect(ipInCidr('34.162.230.222', '34.162.230.222/32')).toBe(true) + expect(ipInCidr('34.162.230.223', '34.162.230.222/32')).toBe(false) + // /0 must match everything — `0xffffffff << 32` is a no-op in JS, so this + // is the case a naive mask implementation gets wrong. + expect(ipInCidr('8.8.8.8', '0.0.0.0/0')).toBe(true) + }) + + it('handles the high bit without sign errors', () => { + // 255.x addresses overflow into negative territory under signed shifts. + expect(ipInCidr('255.255.255.255', '255.255.255.0/24')).toBe(true) + expect(ipInCidr('200.0.0.1', '128.0.0.0/1')).toBe(true) + expect(ipInCidr('127.0.0.1', '128.0.0.0/1')).toBe(false) + }) + + it('matches IPv6 ranges', () => { + expect(ipInCidr('2001:4860:4801:2008::1', '2001:4860:4801:2008::/64')).toBe(true) + expect(ipInCidr('2001:4860:4801:2009::1', '2001:4860:4801:2008::/64')).toBe(false) + }) + + it('matches IPv4-mapped IPv6 against IPv4 ranges', () => { + // Dual-stack edges hand us ::ffff:a.b.c.d; it must still match the v4 list. + expect(ipInCidr('::ffff:104.208.184.200', '104.208.184.192/28')).toBe(true) + expect(ipInCidr('::ffff:104.208.184.208', '104.208.184.192/28')).toBe(false) + }) + + it('returns false rather than throwing on junk', () => { + for (const bad of ['', ' ', 'not-an-ip', '999.999.999.999']) { + expect(ipInCidr(bad, '10.0.0.0/8')).toBe(false) + } + expect(ipInRanges('10.0.0.1', compileRanges(['garbage', '10.0.0.0/33']))).toBe(false) + }) +}) + +describe('verifyBotIdentity', () => { + const CHATGPT_UA = 'Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)' + // First prefix of OpenAI's published chatgpt-user feed. + const REAL_OPENAI_IP = '104.208.184.193' + + it('verifies a real crawler from a published range', () => { + const r = verifyBotIdentity(CHATGPT_UA, REAL_OPENAI_IP) + expect(r.verdict).toBe('verified') + expect(r.verified).toBe(true) + expect(r.claimed).toBe('ChatGPT') + }) + + it('flags the same UA from an unpublished IP as spoofed', () => { + // This is the whole point: `curl -A "ChatGPT-User"` from anywhere else. + const r = verifyBotIdentity(CHATGPT_UA, '1.2.3.4') + expect(r.verdict).toBe('spoofed') + expect(r.verified).toBe(false) + expect(r.claimed).toBe('ChatGPT') + }) + + it('verifies Anthropic, Perplexity, and Apple from their published ranges', () => { + const cases: Array<[string, string, string]> = [ + ['ClaudeBot/1.0', '34.162.230.222', 'Claude'], + ['Mozilla/5.0 (compatible; PerplexityBot/1.0)', '107.20.236.150', 'Perplexity'], + ['Mozilla/5.0 (compatible; Applebot/0.1)', '17.22.237.5', 'Apple'] + ] + for (const [ua, ip, label] of cases) { + const r = verifyBotIdentity(ua, ip) + expect(`${label}:${r.verdict}`).toBe(`${label}:verified`) + expect(r.claimed).toBe(label) + } + }) + + it('returns unverifiable — never spoofed — for vendors with no published feed', () => { + // Branding Bytespider or Amazonbot an impostor just because we lack their + // ranges would be a false accusation, so these must not be `false`. + for (const ua of [ + 'Mozilla/5.0 (compatible; Bytespider/1.0)', + 'Mozilla/5.0 (compatible; Amazonbot/0.1)', + 'meta-externalagent/1.1', + 'Mozilla/5.0 (compatible; SemrushBot/7~bl)' + ]) { + const r = verifyBotIdentity(ua, '1.2.3.4') + expect(r.verdict).toBe('unverifiable') + expect(r.verified).toBeNull() + } + }) + + it('returns unverifiable when no IP is available', () => { + for (const ip of ['', ' ', null, undefined]) { + const r = verifyBotIdentity(CHATGPT_UA, ip) + expect(r.verdict).toBe('unverifiable') + expect(r.reason).toBe('no-client-ip') + expect(r.verified).toBeNull() + } + }) + + it('never accuses client-side agents that fetch from the user\'s own machine', () => { + // Claude Code runs on a developer's laptop, so its IP is theirs and will + // never be in Anthropic's ranges. Production data: 6,492 events across + // 4,486 IPs, 0% in range — a naive vendor-level check brands every one of + // them an impostor. + const clientSide = [ + 'Claude-User (claude-code/2.1.218; +https://support.anthropic.com/)', + 'Mozilla/5.0 (compatible; Perplexity-User/1.0)' + ] + for (const ua of clientSide) { + const r = verifyBotIdentity(ua, '109.135.42.185') + expect(`${ua.slice(0, 12)}:${r.verdict}`).toBe(`${ua.slice(0, 12)}:unverifiable`) + expect(r.reason).toBe('client-side-agent') + expect(r.verified).toBeNull() + } + }) + + it('still verifies the server-side crawler from the same vendor', () => { + // ClaudeBot proper fetches from Anthropic infra and verifies at ~96%. + expect(verifyBotIdentity('ClaudeBot/1.0', '34.162.230.222').verdict).toBe('verified') + // A `-User` suffix is not itself the signal: OpenAI fetches server-side. + expect(verifyBotIdentity(CHATGPT_UA, REAL_OPENAI_IP).verdict).toBe('verified') + }) + + it('returns not-claimed for browsers and unknown UAs', () => { + for (const ua of [ + 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/120 Safari/537.36', + '', + null + ]) { + const r = verifyBotIdentity(ua, '1.2.3.4') + expect(r.verdict).toBe('not-claimed') + expect(r.verified).toBeNull() + expect(r.claimed).toBeNull() + } + }) + + it('reads the client IP from x-forwarded-for on a request', () => { + const req = new Request('https://example.com/', { + headers: { + 'user-agent': CHATGPT_UA, + // Only the first hop is the client; the rest are proxies. + 'x-forwarded-for': `${REAL_OPENAI_IP}, 10.0.0.1, 10.0.0.2` + } + }) + expect(verifyRequest(req).verdict).toBe('verified') + }) +}) + +describe('bundled ranges', () => { + it('covers the vendors that publish feeds', () => { + expect(verifiableVendors()).toEqual(['ChatGPT', 'Claude', 'Perplexity', 'Apple']) + }) + + it('contains only parseable CIDRs', () => { + for (const [vendor, cidrs] of Object.entries(BOT_IP_RANGES)) { + expect(cidrs.length, `${vendor} has no ranges`).toBeGreaterThan(0) + const compiled = compileRanges(cidrs) + // Every entry must survive compilation — a silently dropped prefix means + // real crawler traffic from that block gets marked spoofed. + expect(compiled.v4.length + compiled.v6.length, `${vendor} dropped entries`).toBe(cidrs.length) + } + }) +})