diff --git a/README.md b/README.md index 74e3c8c..fd04293 100644 --- a/README.md +++ b/README.md @@ -306,6 +306,82 @@ Full middleware example: [`README.md → Markdown mirror helpers`](./README.md#m --- +## Advanced: verifying crawler identity against published IP ranges + +User agents are trivially forged — `curl -A "ChatGPT-User"` is indistinguishable +from the real thing at the UA layer. Set `verifyIdentity: true` to check the +client IP against the vendor's published crawler ranges: + +```ts +void trackVisit(request, { + analytics, + verifyIdentity: true, + captureIp: true // not required, but useful for auditing a 'spoofed' verdict +}) +``` + +Three properties land on the event: + +| property | values | +| --- | --- | +| `bot_verification` | `verified` \| `spoofed` \| `unverifiable` \| `not-claimed` | +| `bot_verified` | `true` \| `false` \| `null` — tri-state, for quick filtering | +| `bot_verification_reason` | why, when the verdict is `unverifiable` | + +### What can actually be verified + +Only vendors that publish a machine-readable range feed: **OpenAI**, +**Anthropic**, **Perplexity**, and **Apple**. Bytespider, Amazonbot, Meta and +the rest report `unverifiable` — never `spoofed`. Collapsing "we can't check" +into "impostor" would be a false accusation, which is why `bot_verified` is +tri-state rather than a boolean. + +### Server-side crawlers vs client-side agents + +A published range list covers a vendor's **crawler fleet**, not its products +that fetch from the end user's device. Claude Code runs on a developer's +laptop, so the request carries *their* IP and will never appear in Anthropic's +ranges. Measured over 30 days of production traffic: + +| user agent | events | distinct IPs | in published range | +| --- | ---: | ---: | ---: | +| `ClaudeBot` | 13,671 | 236 | 96% | +| `PerplexityBot` | 6,897 | 158 | 91% | +| `ChatGPT-User` | ~72,000 | 43 | 99% | +| `Claude-User` (claude-code CLI) | 6,492 | 4,486 | **0%** | +| `Perplexity-User` | 493 | 148 | **0%** | + +A naive vendor-level check would brand the bottom two rows — roughly 7,000 +legitimate fetches a month — as impersonation. So the library gates verdicts on +the *product*, returning `unverifiable` with reason `client-side-agent` for +those. Note the distinction is not a `-User` suffix: OpenAI's `ChatGPT-User` +fetches server-side from Azure and verifies at ~99%. + +### Keeping the ranges fresh + +The bundled snapshot is in `src/bot-ranges.ts`, stamped with +`BOT_RANGES_CAPTURED_AT`. Refresh it on a schedule: + +```bash +node scripts/refresh-bot-ranges.mjs +``` + +Freshness is the whole game. Nearly every OpenAI prefix is an Azure block and +Anthropic's are GCP, so "came from a datacenter" proves nothing on its own — +only membership in the *current* published list does. A stale snapshot produces +false `spoofed` verdicts on real crawlers, so the refresh script refuses to +write a list that shrinks by more than half or when any feed errors. + +### Trusting the client IP + +The verdict is only as good as the IP. On Vercel and Cloudflare the edge +overwrites `x-forwarded-for`, so the first hop is trustworthy. Behind a proxy +that passes a client-supplied header through, an attacker controls the value +and `verified` means nothing — confirm your proxy's behaviour before acting on +this data. + +--- + ## Advanced: Peec.ai crawl-insights export [Peec.ai](https://peec.ai)'s **Agent analytics** product ingests a CSV/CLF access log and produces dashboards on top of it. The Peec docs assume you have a Vercel Log Drain → Axiom (or similar) pipeline that emits these eight columns: `timestamp, request_method, request_url, response_status, client_ip, user_agent, country_code, referer`. diff --git a/package.json b/package.json index 3bfddea..d02458f 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@apideck/agent-analytics", - "version": "0.10.0", + "version": "0.11.0", "description": "Track AI agent and bot traffic to your Next.js / Vercel app — PostHog, webhooks, or any custom analytics backend. Detects Claude, ChatGPT, Perplexity, Google-Extended, and more.", "keywords": [ "ai", diff --git a/scripts/refresh-bot-ranges.mjs b/scripts/refresh-bot-ranges.mjs new file mode 100644 index 0000000..972ca1b --- /dev/null +++ b/scripts/refresh-bot-ranges.mjs @@ -0,0 +1,116 @@ +#!/usr/bin/env node +/** + * Regenerate src/bot-ranges.ts from the vendors' published crawler IP feeds. + * + * node scripts/refresh-bot-ranges.mjs + * + * Run this on a schedule. The lists rotate, and a stale snapshot is worse than + * no snapshot: it produces false 'spoofed' verdicts on legitimate crawlers, + * which is exactly the conclusion you'd act on. The script refuses to write a + * file when a feed fails or shrinks implausibly, so a bad fetch can't silently + * empty out a vendor. + */ +import { writeFileSync, readFileSync } from 'node:fs' +import { fileURLToPath } from 'node:url' +import { dirname, join } from 'node:path' + +const OUT = join(dirname(fileURLToPath(import.meta.url)), '..', 'src', 'bot-ranges.ts') + +// Vendor label (as parseBotName returns) -> the feeds that make up its ranges. +const FEEDS = { + ChatGPT: [ + 'https://openai.com/gptbot.json', + 'https://openai.com/chatgpt-user.json', + 'https://openai.com/searchbot.json' + ], + Claude: ['https://claude.com/crawling/bots.json'], + Perplexity: ['https://www.perplexity.ai/perplexitybot.json'], + Apple: ['https://search.developer.apple.com/applebot.json'] +} + +// A feed dropping below this fraction of its previous size signals a partial +// or malformed response rather than a genuine shrink. Bail instead of writing. +const SHRINK_FLOOR = 0.5 + +async function fetchPrefixes(url) { + const res = await fetch(url, { headers: { 'user-agent': 'agent-analytics range refresher' } }) + if (!res.ok) throw new Error(`${url} -> HTTP ${res.status}`) + const body = await res.json() + const prefixes = body?.prefixes + if (!Array.isArray(prefixes)) throw new Error(`${url} -> no prefixes array`) + // Feeds use ipv4Prefix / ipv6Prefix keys; take whichever is present. + return prefixes.map((e) => e.ipv4Prefix ?? e.ipv6Prefix).filter(Boolean) +} + +function previousCounts() { + try { + const src = readFileSync(OUT, 'utf8') + const counts = {} + for (const vendor of Object.keys(FEEDS)) { + const block = src.match(new RegExp(` ${vendor}: \\[([\\s\\S]*?)\\n \\]`)) + counts[vendor] = block ? (block[1].match(/'/g) || []).length / 2 : 0 + } + return counts + } catch { + return {} + } +} + +const before = previousCounts() +const vendors = {} + +for (const [vendor, urls] of Object.entries(FEEDS)) { + const all = new Set() + for (const url of urls) { + const got = await fetchPrefixes(url) // throws — a failed feed must abort the run + got.forEach((p) => all.add(p)) + console.log(` ${url} -> ${got.length}`) + } + const list = [...all].sort() + const prev = before[vendor] ?? 0 + if (prev && list.length < prev * SHRINK_FLOOR) { + throw new Error( + `${vendor}: ${list.length} prefixes vs ${prev} previously — refusing to write a suspiciously small list` + ) + } + vendors[vendor] = list + console.log(`${vendor}: ${list.length} prefixes (was ${prev})`) +} + +const capturedAt = new Date().toISOString() +const body = `/** + * Published crawler IP ranges, vendor by vendor. + * + * GENERATED FILE — do not edit by hand. Regenerate with: + * + * node scripts/refresh-bot-ranges.mjs + * + * Keys match the labels {@link parseBotName} returns, so a claimed identity + * maps to its range list without a translation table. + * + * Only vendors that publish a machine-readable feed appear here. A vendor's + * absence means "cannot be verified", never "not a real bot" — see + * {@link verifyBotIdentity} for how that distinction is surfaced. + * + * Freshness is the whole ballgame: these lists rotate. Almost every OpenAI + * prefix is an Azure block and every Anthropic prefix is GCP or similar, so + * "came from a datacenter" proves nothing on its own — only membership in the + * current published list does. A stale snapshot produces false 'spoofed' + * verdicts on legitimate crawlers, which is the failure mode to fear. + */ + +/** When these ranges were captured from the vendor feeds (UTC). */ +export const BOT_RANGES_CAPTURED_AT = '${capturedAt}' + +export const BOT_IP_RANGES: Readonly> = { +${Object.entries(vendors) + .map(([name, cidrs]) => ` ${name}: [\n${cidrs.map((c) => ` '${c}',`).join('\n')}\n ],`) + .join('\n')} +} + +/** Vendor labels this build can verify. Anything else yields a null verdict. */ +export const VERIFIABLE_VENDORS: readonly string[] = Object.keys(BOT_IP_RANGES) +` + +writeFileSync(OUT, body) +console.log(`\nwrote ${OUT}`) diff --git a/src/bot-ranges.ts b/src/bot-ranges.ts new file mode 100644 index 0000000..f7647e0 --- /dev/null +++ b/src/bot-ranges.ts @@ -0,0 +1,449 @@ +/** + * Published crawler IP ranges, vendor by vendor. + * + * GENERATED FILE — do not edit by hand. Regenerate with: + * + * node scripts/refresh-bot-ranges.mjs + * + * Keys match the labels {@link parseBotName} returns, so a claimed identity + * maps to its range list without a translation table. + * + * Only vendors that publish a machine-readable feed appear here. A vendor's + * absence means "cannot be verified", never "not a real bot" — see + * {@link verifyBotIdentity} for how that distinction is surfaced. + * + * Freshness is the whole ballgame: these lists rotate. Almost every OpenAI + * prefix is an Azure block and every Anthropic prefix is GCP or similar, so + * "came from a datacenter" proves nothing on its own — only membership in the + * current published list does. A stale snapshot produces false 'spoofed' + * verdicts on legitimate crawlers, which is the failure mode to fear. + */ + +/** When these ranges were captured from the vendor feeds (UTC). */ +export const BOT_RANGES_CAPTURED_AT = '2026-08-02T00:00:00Z' + +export const BOT_IP_RANGES: Readonly> = { + ChatGPT: [ + '104.208.184.192/28', + '104.208.184.208/28', + '104.210.139.192/28', + '104.210.139.224/28', + '104.210.140.128/28', + '128.85.198.32/28', + '13.65.138.112/28', + '13.65.138.96/28', + '13.67.72.16/28', + '13.70.107.160/28', + '13.71.2.208/28', + '13.76.115.224/28', + '13.76.115.240/28', + '13.76.116.80/28', + '13.76.32.208/28', + '13.83.167.128/28', + '13.83.237.176/28', + '132.196.82.48/28', + '132.196.86.0/24', + '134.149.233.80/28', + '135.116.136.160/28', + '135.13.64.240/28', + '135.220.73.208/28', + '135.220.73.240/28', + '135.234.64.0/24', + '135.237.131.208/28', + '135.237.133.48/28', + '137.135.191.176/28', + '138.91.30.48/28', + '138.91.46.96/28', + '145.132.1.32/28', + '145.132.136.96/28', + '145.133.0.176/28', + '158.158.5.32/28', + '168.63.252.240/28', + '172.162.248.64/28', + '172.170.1.80/28', + '172.170.225.0/28', + '172.170.241.80/28', + '172.170.8.208/28', + '172.171.4.176/28', + '172.175.152.224/28', + '172.178.140.144/28', + '172.178.141.112/28', + '172.178.141.128/28', + '172.182.193.224/28', + '172.182.193.80/28', + '172.182.194.144/28', + '172.182.194.32/28', + '172.182.195.48/28', + '172.182.202.0/25', + '172.182.204.0/24', + '172.182.207.0/25', + '172.182.209.208/28', + '172.182.211.192/28', + '172.182.213.192/28', + '172.182.214.0/24', + '172.182.215.0/24', + '172.182.224.0/28', + '172.183.143.224/28', + '172.183.222.128/28', + '172.192.112.208/28', + '172.192.88.192/28', + '172.192.97.32/28', + '172.197.160.192/28', + '172.197.161.208/28', + '172.197.170.80/28', + '172.197.203.16/28', + '172.199.137.80/28', + '172.202.102.112/28', + '172.203.190.128/28', + '172.204.27.16/28', + '172.204.28.224/28', + '172.204.96.32/28', + '172.204.96.48/28', + '172.204.96.80/28', + '172.205.189.192/28', + '172.207.1.32/28', + '172.208.128.32/28', + '172.208.128.48/28', + '172.212.159.64/28', + '172.212.172.160/28', + '172.213.21.16/28', + '172.215.215.32/28', + '172.215.218.96/28', + '191.233.1.112/28', + '191.233.1.128/28', + '191.233.194.32/28', + '191.233.196.112/28', + '191.233.199.160/28', + '191.233.2.0/28', + '191.235.66.16/28', + '191.235.99.80/28', + '191.237.249.64/28', + '191.239.245.16/28', + '20.102.212.144/28', + '20.113.211.112/28', + '20.113.225.112/28', + '20.125.112.224/28', + '20.125.144.144/28', + '20.125.66.80/28', + '20.14.99.96/28', + '20.161.75.208/28', + '20.168.18.32/28', + '20.168.7.192/28', + '20.168.7.240/28', + '20.169.6.224/28', + '20.169.7.48/28', + '20.169.72.112/28', + '20.169.73.176/28', + '20.169.73.32/28', + '20.169.73.64/28', + '20.169.77.0/25', + '20.169.78.112/28', + '20.169.78.128/28', + '20.169.78.144/28', + '20.169.78.160/28', + '20.169.78.176/28', + '20.169.78.192/28', + '20.169.78.208/28', + '20.169.78.48/28', + '20.169.78.64/28', + '20.169.78.80/28', + '20.169.78.96/28', + '20.169.86.224/28', + '20.169.86.240/28', + '20.169.87.112/28', + '20.17.108.96/28', + '20.170.184.16/28', + '20.170.184.32/28', + '20.170.184.48/28', + '20.170.184.64/28', + '20.170.184.80/28', + '20.171.123.64/28', + '20.171.206.0/24', + '20.171.207.0/24', + '20.171.53.224/28', + '20.172.29.32/28', + '20.193.233.240/28', + '20.193.50.32/28', + '20.194.0.208/28', + '20.194.1.0/28', + '20.198.67.96/28', + '20.199.211.160/28', + '20.199.242.0/28', + '20.200.212.240/28', + '20.204.24.240/28', + '20.210.154.128/28', + '20.210.174.208/28', + '20.210.211.192/28', + '20.215.187.208/28', + '20.215.188.192/28', + '20.215.214.16/28', + '20.215.219.128/28', + '20.215.219.160/28', + '20.215.219.208/28', + '20.215.220.112/28', + '20.215.220.128/28', + '20.215.220.144/28', + '20.215.220.160/28', + '20.215.220.176/28', + '20.215.220.192/28', + '20.215.220.208/28', + '20.215.220.64/28', + '20.215.220.80/28', + '20.215.220.96/28', + '20.218.30.240/28', + '20.219.71.192/28', + '20.222.36.192/28', + '20.226.32.80/28', + '20.227.140.32/28', + '20.228.106.176/28', + '20.235.75.208/28', + '20.235.87.224/28', + '20.249.63.208/28', + '20.25.151.224/28', + '20.250.136.64/28', + '20.250.136.80/28', + '20.250.6.128/28', + '20.27.94.128/28', + '20.42.10.176/28', + '20.45.178.144/28', + '20.48.120.208/28', + '20.52.125.160/28', + '20.55.129.0/28', + '20.55.229.144/28', + '20.57.199.192/28', + '20.63.180.96/28', + '20.63.221.64/28', + '20.79.59.112/28', + '20.81.183.64/28', + '20.83.243.176/28', + '20.97.189.96/28', + '23.102.140.144/28', + '23.102.141.32/28', + '23.98.142.176/28', + '23.98.179.16/28', + '23.98.186.176/28', + '23.98.186.192/28', + '23.98.186.64/28', + '23.98.186.96/28', + '4.151.119.48/28', + '4.151.241.240/28', + '4.151.71.176/28', + '4.189.118.208/28', + '4.189.119.48/28', + '4.196.118.112/28', + '4.197.115.112/28', + '4.197.19.176/28', + '4.197.22.112/28', + '4.197.64.0/28', + '4.197.64.16/28', + '4.197.64.48/28', + '4.197.64.64/28', + '4.198.72.16/28', + '4.198.96.112/28', + '4.201.232.64/28', + '4.201.232.80/28', + '4.203.96.80/28', + '4.205.128.176/28', + '4.218.24.64/28', + '4.226.200.16/28', + '4.226.226.32/28', + '4.227.36.0/25', + '40.116.73.208/28', + '40.122.235.112/28', + '40.67.175.0/25', + '40.67.183.160/28', + '40.67.183.176/28', + '40.78.161.48/28', + '40.81.134.128/28', + '40.81.134.144/28', + '40.81.234.144/28', + '40.81.67.96/28', + '40.84.181.32/28', + '40.84.221.208/28', + '40.84.221.224/28', + '40.90.214.16/28', + '48.193.44.32/28', + '48.221.184.112/28', + '48.221.184.80/28', + '48.221.184.96/28', + '48.221.40.176/28', + '51.107.70.192/28', + '51.116.2.80/28', + '51.116.221.96/28', + '51.56.40.80/28', + '51.57.0.96/28', + '51.59.24.64/28', + '51.59.24.80/28', + '51.59.40.80/28', + '51.59.40.96/28', + '51.59.48.80/28', + '51.59.48.96/28', + '51.8.102.0/24', + '51.8.155.112/28', + '51.8.155.48/28', + '51.8.155.64/28', + '51.8.187.224/28', + '52.148.129.32/28', + '52.153.130.64/28', + '52.154.22.48/28', + '52.156.77.144/28', + '52.159.227.32/28', + '52.159.249.96/28', + '52.161.49.224/28', + '52.161.49.96/28', + '52.165.212.16/28', + '52.165.212.32/28', + '52.165.212.48/28', + '52.172.129.160/28', + '52.172.251.112/28', + '52.173.219.112/28', + '52.173.219.96/28', + '52.173.221.16/28', + '52.173.221.176/28', + '52.173.221.208/28', + '52.173.234.16/28', + '52.173.234.80/28', + '52.173.235.80/28', + '52.183.217.240/28', + '52.187.246.128/28', + '52.190.137.144/28', + '52.190.137.16/28', + '52.190.139.48/28', + '52.190.142.64/28', + '52.190.190.16/28', + '52.225.75.208/28', + '52.230.152.0/24', + '52.230.163.32/28', + '52.230.164.176/28', + '52.231.30.48/28', + '52.231.34.176/28', + '52.231.39.144/28', + '52.231.39.192/28', + '52.231.49.48/28', + '52.231.50.64/28', + '52.236.94.144/28', + '52.241.146.208/28', + '52.242.132.224/28', + '52.242.132.240/28', + '52.242.245.208/28', + '52.252.113.240/28', + '52.255.109.112/28', + '52.255.109.128/28', + '52.255.109.144/28', + '52.255.109.80/28', + '52.255.109.96/28', + '52.255.111.0/28', + '52.255.111.112/28', + '52.255.111.32/28', + '52.255.111.48/28', + '52.255.111.80/28', + '57.154.174.112/28', + '57.154.175.0/28', + '57.154.187.32/28', + '68.154.28.96/28', + '68.218.30.112/28', + '68.220.57.64/28', + '68.221.67.192/28', + '68.221.67.224/28', + '68.221.67.240/28', + '70.153.139.208/28', + '70.153.189.192/28', + '70.153.190.16/28', + '70.153.76.16/28', + '70.153.87.224/28', + '70.156.144.64/28', + '70.156.152.80/28', + '70.156.152.96/28', + '74.161.200.96/28', + '74.224.217.64/28', + '74.226.253.160/28', + '74.249.86.176/28', + '74.7.175.128/25', + '74.7.227.0/25', + '74.7.227.128/25', + '74.7.228.0/25', + '74.7.228.128/25', + '74.7.229.0/25', + '74.7.229.128/25', + '74.7.230.0/25', + '74.7.241.0/25', + '74.7.241.128/25', + '74.7.242.0/25', + '74.7.242.128/25', + '74.7.243.0/25', + '74.7.243.128/25', + '74.7.244.0/25', + '74.7.35.112/28', + '74.7.35.48/28', + '74.7.36.64/28', + '74.7.36.80/28', + '74.7.36.96/28', + '85.211.241.128/28', + '9.129.0.0/17', + '9.160.128.16/28', + '9.160.128.32/28', + '9.160.128.64/28', + '9.160.163.128/28', + '9.160.164.128/28', + '9.160.34.144/28', + '9.160.36.16/28', + '9.160.96.16/28', + '9.163.101.48/28', + '9.205.25.128/28', + '9.205.8.48/28', + '9.223.181.208/28', + '9.234.96.192/28', + '9.234.97.128/28', + '9.234.97.96/28', + '9.235.40.32/28', + ], + Claude: [ + '136.107.176.208/32', + '216.73.216.0/22', + '34.11.34.31/32', + '34.150.241.79/32', + '34.162.191.81/32', + '34.162.230.222/32', + '34.162.244.71/32', + '34.182.140.95/32', + '34.182.161.143/32', + '34.182.218.27/32', + '34.182.220.85/32', + '34.182.222.37/32', + '34.182.225.167/32', + '34.182.226.151/32', + '34.182.226.221/32', + '34.186.108.163/32', + '34.85.172.162/32', + '35.221.29.174/32', + '35.245.175.129/32', + '35.245.89.239/32', + ], + Perplexity: [ + '107.20.236.150/32', + '18.210.92.235/32', + '18.97.1.228/30', + '18.97.9.96/29', + '3.211.124.183/32', + '3.222.232.239/32', + '3.224.62.45/32', + '3.231.139.107/32', + ], + Apple: [ + '17.22.237.0/24', + '17.22.245.0/24', + '17.22.253.0/24', + '17.241.193.160/27', + '17.241.200.160/27', + '17.241.208.160/27', + '17.241.219.0/24', + '17.241.227.0/24', + '17.241.75.0/24', + '17.246.15.0/24', + '17.246.19.0/24', + '17.246.23.0/24', + ], +} + +/** Vendor labels this build can verify. Anything else yields a null verdict. */ +export const VERIFIABLE_VENDORS: readonly string[] = Object.keys(BOT_IP_RANGES) diff --git a/src/cidr.ts b/src/cidr.ts new file mode 100644 index 0000000..2cf2637 --- /dev/null +++ b/src/cidr.ts @@ -0,0 +1,161 @@ +/** + * Dependency-free CIDR matching for IPv4 and IPv6. + * + * Kept separate from {@link verifyBotIdentity} so the matching logic can be + * tested in isolation — a wrong answer here silently turns a real crawler into + * a "spoofed" one, which is worse than not verifying at all. + * + * Everything is parsed once into numeric form (`number` for v4, `bigint` for + * v6) and compared with masks. No allocation per request beyond the parse of + * the incoming IP. + */ + +/** Parse dotted-quad IPv4 into a 32-bit unsigned integer. */ +export function ipv4ToInt(ip: string): number | null { + const parts = ip.split('.') + if (parts.length !== 4) return null + let out = 0 + for (const part of parts) { + // Reject empty, non-numeric, out-of-range, and leading-zero forms + // (`01.2.3.4` is ambiguous — some parsers read it as octal). + if (!/^\d{1,3}$/.test(part)) return null + if (part.length > 1 && part[0] === '0') return null + const n = Number(part) + if (n > 255) return null + out = (out << 8) | n + } + // `>>> 0` converts the signed 32-bit result back to unsigned. + return out >>> 0 +} + +/** + * Parse IPv6 (including `::` compression and IPv4-mapped tails like + * `::ffff:1.2.3.4`) into a 128-bit BigInt. + */ +export function ipv6ToBigInt(ip: string): bigint | null { + let text = ip + // Strip a zone index (`fe80::1%eth0`) — irrelevant for range membership. + const zone = text.indexOf('%') + if (zone !== -1) text = text.slice(0, zone) + if (!text || text.indexOf(':') === -1) return null + + // An embedded IPv4 tail contributes the low 32 bits. + let tail: number | null = null + const lastColon = text.lastIndexOf(':') + const maybeV4 = text.slice(lastColon + 1) + if (maybeV4.indexOf('.') !== -1) { + tail = ipv4ToInt(maybeV4) + if (tail === null) return null + text = text.slice(0, lastColon + 1) + '0:0' + } + + const halves = text.split('::') + if (halves.length > 2) return null + const head = halves[0] ? halves[0].split(':') : [] + const rest = halves.length === 2 ? (halves[1] ? halves[1].split(':') : []) : null + + let groups: string[] + if (rest === null) { + groups = head + if (groups.length !== 8) return null + } else { + const fill = 8 - head.length - rest.length + if (fill < 0) return null + groups = [...head, ...Array(fill).fill('0'), ...rest] + } + + let out = 0n + for (const g of groups) { + if (!/^[0-9a-fA-F]{1,4}$/.test(g)) return null + out = (out << 16n) | BigInt(parseInt(g, 16)) + } + // Overwrite the low 32 bits with the embedded IPv4 value when present. + if (tail !== null) out = ((out >> 32n) << 32n) | BigInt(tail >>> 0) + return out +} + +interface V4Range { + net: number + mask: number +} +interface V6Range { + net: bigint + bits: number +} + +export interface CompiledRanges { + v4: V4Range[] + v6: V6Range[] +} + +/** + * Pre-compile a list of CIDR strings into numeric form. Invalid entries are + * dropped rather than thrown — a malformed line in a vendor's published feed + * shouldn't take down the whole check. + */ +export function compileRanges(cidrs: readonly string[]): CompiledRanges { + const v4: V4Range[] = [] + const v6: V6Range[] = [] + for (const cidr of cidrs) { + const slash = cidr.lastIndexOf('/') + if (slash === -1) continue + const addr = cidr.slice(0, slash) + const bits = Number(cidr.slice(slash + 1)) + if (!Number.isInteger(bits) || bits < 0) continue + + if (addr.indexOf(':') !== -1) { + if (bits > 128) continue + const net = ipv6ToBigInt(addr) + if (net === null) continue + v6.push({ net: bits === 0 ? 0n : (net >> BigInt(128 - bits)) << BigInt(128 - bits), bits }) + } else { + if (bits > 32) continue + const net = ipv4ToInt(addr) + if (net === null) continue + // `bits === 0` needs special handling: `<<32` is a no-op in JS, not zero. + const mask = bits === 0 ? 0 : (0xffffffff << (32 - bits)) >>> 0 + v4.push({ net: (net & mask) >>> 0, mask }) + } + } + return { v4, v6 } +} + +/** True when `ip` falls inside any range in the pre-compiled set. */ +export function ipInRanges(ip: string, ranges: CompiledRanges): boolean { + if (!ip) return false + const trimmed = ip.trim() + if (!trimmed) return false + + if (trimmed.indexOf(':') !== -1) { + const value = ipv6ToBigInt(trimmed) + if (value === null) return false + // An IPv4-mapped address (::ffff:a.b.c.d) must also be checked against the + // v4 list — Vercel and Cloudflare both emit this form on dual-stack edges. + const V4_MAPPED_PREFIX = 0xffffn << 32n + if ((value >> 32n) === V4_MAPPED_PREFIX >> 32n) { + const low = Number(value & 0xffffffffn) >>> 0 + if (matchV4(low, ranges.v4)) return true + } + for (const r of ranges.v6) { + if (r.bits === 0) return true + if ((value >> BigInt(128 - r.bits)) << BigInt(128 - r.bits) === r.net) return true + } + return false + } + + const value = ipv4ToInt(trimmed) + if (value === null) return false + return matchV4(value, ranges.v4) +} + +function matchV4(value: number, list: readonly V4Range[]): boolean { + for (const r of list) { + if (((value & r.mask) >>> 0) === r.net) return true + } + return false +} + +/** Convenience wrapper — compiles on every call, so prefer {@link ipInRanges}. */ +export function ipInCidr(ip: string, cidr: string): boolean { + return ipInRanges(ip, compileRanges([cidr])) +} diff --git a/src/index.ts b/src/index.ts index be2f20d..fe714ca 100644 --- a/src/index.ts +++ b/src/index.ts @@ -12,6 +12,16 @@ export { } from './bots.js' export type { AgentClassification, AgentKind, HeadlessDetection } from './bots.js' export { hashId } from './hash.js' +export { + clientIpFromRequest, + verifiableVendors, + verifyBotIdentity, + verifyRequest +} from './verify.js' +export type { BotVerification, VerificationVerdict } from './verify.js' +export { BOT_IP_RANGES, BOT_RANGES_CAPTURED_AT, VERIFIABLE_VENDORS } from './bot-ranges.js' +export { compileRanges, ipInCidr, ipInRanges } from './cidr.js' +export type { CompiledRanges } from './cidr.js' export { posthogAnalytics } from './adapters/posthog.js' export { webhookAnalytics } from './adapters/webhook.js' export { customAnalytics } from './adapters/custom.js' diff --git a/src/track.ts b/src/track.ts index 1be6dbc..3254cf2 100644 --- a/src/track.ts +++ b/src/track.ts @@ -1,4 +1,5 @@ import { classifyRequest, detectHeadless, isAiBot, isHttpClient } from './bots.js' +import { verifyBotIdentity } from './verify.js' import { hashId } from './hash.js' import type { TrackVisitOptions } from './types.js' @@ -50,6 +51,9 @@ export async function trackVisit( : null const geo = opts.captureGeo ? extractGeo(req) : null const classification = classifyRequest(req) + // Only run the range check when asked — it is pure CPU over pre-compiled + // masks, but the verdict is misleading unless the caller trusts `ip`. + const verification = opts.verifyIdentity ? verifyBotIdentity(userAgent, ip) : null const event = { event: opts.eventName ?? 'agent_visit', @@ -70,6 +74,13 @@ export async function trackVisit( coding_agent_hint: classification.codingAgentHint, headless_score: classification.headless?.score ?? 0, headless_likely: classification.headless?.likely ?? false, + ...(verification + ? { + bot_verified: verification.verified, + bot_verification: verification.verdict, + ...(verification.reason ? { bot_verification_reason: verification.reason } : {}) + } + : {}), referer, source: opts.source ?? null, ...opts.properties diff --git a/src/types.ts b/src/types.ts index 235d14c..ad94143 100644 --- a/src/types.ts +++ b/src/types.ts @@ -57,6 +57,24 @@ export interface TrackVisitOptions { * Enable for log-style exports (e.g. Peec.ai's crawl-insights CSV). */ captureCountry?: boolean + /** + * When `true`, check the request's claimed crawler identity against the + * vendor's published IP ranges and emit `bot_verified` (tri-state) plus + * `bot_verification` (`verified` | `spoofed` | `unverifiable` | + * `not-claimed`) on the event. + * + * UA strings are trivially forgeable — `curl -A "ChatGPT-User"` is + * indistinguishable from the real thing without this check. Off by default + * because it only means something when the client IP is trustworthy: on + * Vercel and Cloudflare the edge overwrites `x-forwarded-for`, but behind a + * proxy that passes the client-supplied header through, an attacker controls + * the value and a `verified` verdict is worthless. + * + * Only vendors that publish a machine-readable range feed can be verified — + * currently OpenAI, Anthropic, Perplexity, and Apple. Everything else yields + * `unverifiable`, never `spoofed`. + */ + verifyIdentity?: boolean /** * When `true`, emit `region`, `city`, `latitude`, `longitude`, and * `timezone` derived from Vercel's `x-vercel-ip-*` edge headers. Values diff --git a/src/verify.ts b/src/verify.ts new file mode 100644 index 0000000..f6c2eb9 --- /dev/null +++ b/src/verify.ts @@ -0,0 +1,147 @@ +import { parseBotName } from './bots.js' +import { BOT_IP_RANGES, VERIFIABLE_VENDORS } from './bot-ranges.js' +import { compileRanges, ipInRanges, type CompiledRanges } from './cidr.js' + +/** + * Verdict on whether a request's claimed crawler identity holds up against the + * vendor's published IP ranges. + * + * - `'verified'` — the UA claims a vendor we can check, and the client IP is + * inside that vendor's published ranges. High confidence. + * - `'spoofed'` — the UA claims a vendor we can check, and the IP is **not** + * in its ranges. Someone is impersonating the crawler. + * - `'unverifiable'` — the UA claims a vendor that publishes no feed we bundle + * (Bytespider, Amazon, Meta, …), or no usable client IP was available. + * - `'not-claimed'` — the UA doesn't claim a verifiable crawler at all. This is + * the normal verdict for browsers and HTTP clients; it is *not* a negative + * finding. + */ +export type VerificationVerdict = 'verified' | 'spoofed' | 'unverifiable' | 'not-claimed' + +/** Why a request could not be judged. Only set when verdict is 'unverifiable'. */ +export type UnverifiableReason = 'no-published-ranges' | 'client-side-agent' | 'no-client-ip' + +export interface BotVerification { + verdict: VerificationVerdict + reason?: UnverifiableReason + /** Vendor label the UA claims, when it claims one. */ + claimed: string | null + /** + * Convenience boolean for filtering: `true` only for `'verified'`, `false` + * only for `'spoofed'`, `null` when no judgement was possible. Deliberately + * tri-state — collapsing "unverifiable" into `false` would brand every + * Bytespider and Amazonbot hit an impostor. + */ + verified: boolean | null +} + +/** + * Products that fetch from the **end user's own device**, not from the + * vendor's infrastructure. + * + * A published range list only covers a vendor's server-side crawler fleet. + * When the fetch originates on a developer's laptop, the client IP is theirs + * and will never appear in the vendor's ranges — so a range check produces + * 'spoofed' for entirely legitimate traffic. + * + * This is not cosmetic. Measured against 30 days of production data: + * + * Claude-User (claude-code CLI) 6,492 events 4,486 IPs 0% in range + * Perplexity-User 493 events 148 IPs 0% in range + * ClaudeBot 13,671 events 236 IPs 96% in range + * PerplexityBot 6,897 events 158 IPs 91% in range + * + * Treating the first two as impostors would have falsely accused ~7k real + * fetches a month. The distinction is per *product*, not per vendor, and not + * inferable from a `-User` suffix — OpenAI's ChatGPT-User fetches server-side + * from Azure and verifies at ~99%. + */ +const CLIENT_SIDE_AGENT_PATTERN = /claude-code|perplexity-user|cursor|windsurf|cline|aider/i + +/** + * Products known to fetch server-side from ranges the vendor publishes. Only + * these can earn a 'verified' or 'spoofed' verdict; anything else is reported + * 'unverifiable' so the data never overstates what was actually checked. + */ +const SERVER_SIDE_CRAWLER_PATTERN = + /ClaudeBot|Claude-SearchBot|GPTBot|OAI-SearchBot|ChatGPT-User|PerplexityBot|Applebot/i + +// Compile each vendor's ranges once at module load rather than per request. +const COMPILED: Record = {} +for (const vendor of VERIFIABLE_VENDORS) { + COMPILED[vendor] = compileRanges(BOT_IP_RANGES[vendor] ?? []) +} + +/** Vendor labels this build can produce a verified/spoofed verdict for. */ +export function verifiableVendors(): readonly string[] { + return VERIFIABLE_VENDORS +} + +/** + * Check a claimed crawler identity against the vendor's published IP ranges. + * + * Pass the client IP you already trust — on Vercel and Cloudflare that is the + * first hop of `x-forwarded-for`. If your edge doesn't strip client-supplied + * `X-Forwarded-For`, an attacker controls this value and a `'verified'` verdict + * means nothing; verify your proxy's behaviour before relying on it. + */ +export function verifyBotIdentity( + userAgent: string | null | undefined, + ip: string | null | undefined +): BotVerification { + const ua = userAgent ?? '' + const claimed = parseBotName(userAgent) + const ranges = COMPILED[claimed] + + if (!ranges) { + // Either not a crawler at all, or a crawler with no published feed. Both + // are "no judgement", but the caller may want to tell them apart. + const isKnownCrawler = claimed !== 'Other' && claimed !== 'Browser' + return { + verdict: isKnownCrawler ? 'unverifiable' : 'not-claimed', + ...(isKnownCrawler ? { reason: 'no-published-ranges' as const } : {}), + claimed: isKnownCrawler ? claimed : null, + verified: null + } + } + + // Order matters: a client-side agent must be excluded before the range check, + // and the server-side allowlist gates everything else, so an unrecognised + // product from a covered vendor is never accused on a partial range list. + if (CLIENT_SIDE_AGENT_PATTERN.test(ua) || !SERVER_SIDE_CRAWLER_PATTERN.test(ua)) { + return { + verdict: 'unverifiable', + reason: 'client-side-agent', + claimed, + verified: null + } + } + + const trimmed = (ip ?? '').trim() + if (!trimmed) { + return { verdict: 'unverifiable', reason: 'no-client-ip', claimed, verified: null } + } + + const inRange = ipInRanges(trimmed, ranges) + return { + verdict: inRange ? 'verified' : 'spoofed', + claimed, + verified: inRange + } +} + +/** + * Extract the client IP the way {@link trackVisit} does — first hop of + * `x-forwarded-for`, falling back to the platform-specific headers. + */ +export function clientIpFromRequest(req: Request): string { + const forwarded = req.headers.get('x-forwarded-for') || '' + const first = forwarded.split(',')[0]?.trim() + if (first) return first + return (req.headers.get('cf-connecting-ip') || req.headers.get('x-real-ip') || '').trim() +} + +/** Verify straight from a request object. */ +export function verifyRequest(req: Request): BotVerification { + return verifyBotIdentity(req.headers.get('user-agent'), clientIpFromRequest(req)) +} diff --git a/test/track.test.ts b/test/track.test.ts index d0fa7bd..e9ace75 100644 --- a/test/track.test.ts +++ b/test/track.test.ts @@ -420,3 +420,65 @@ describe('trackVisit', () => { expect(a.distinctId).toBe(b.distinctId) }) }) + +describe('trackVisit — verifyIdentity', () => { + const CHATGPT_UA = 'Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)' + const REAL_OPENAI_IP = '104.208.184.193' + + async function capture(headers: Record, opts: Record = {}) { + const spy = vi.fn() + await trackVisit(makeRequest('https://example.com/page', headers), { + analytics: customAnalytics(spy), + ...opts + }) + return spy.mock.calls[0]![0] as CaptureEvent + } + + it('omits the verification properties entirely when not opted in', () => { + // Absent, not null — so existing dashboards don't gain a column of nulls. + return capture({ 'user-agent': CHATGPT_UA, 'x-forwarded-for': REAL_OPENAI_IP }).then((e) => { + expect('bot_verified' in e.properties).toBe(false) + expect('bot_verification' in e.properties).toBe(false) + }) + }) + + it('marks a real crawler verified', async () => { + const e = await capture( + { 'user-agent': CHATGPT_UA, 'x-forwarded-for': REAL_OPENAI_IP }, + { verifyIdentity: true } + ) + expect(e.properties.bot_verified).toBe(true) + expect(e.properties.bot_verification).toBe('verified') + expect(e.properties.bot_name).toBe('ChatGPT') + }) + + it('marks the same UA from another IP spoofed', async () => { + const e = await capture( + { 'user-agent': CHATGPT_UA, 'x-forwarded-for': '1.2.3.4' }, + { verifyIdentity: true } + ) + expect(e.properties.bot_verified).toBe(false) + expect(e.properties.bot_verification).toBe('spoofed') + // Still labelled ChatGPT and still is_ai_bot — the verdict is the extra + // dimension, it does not rewrite the classification. + expect(e.properties.bot_name).toBe('ChatGPT') + expect(e.properties.is_ai_bot).toBe(true) + }) + + it('uses the first x-forwarded-for hop, not a trailing proxy', async () => { + const e = await capture( + { 'user-agent': CHATGPT_UA, 'x-forwarded-for': `${REAL_OPENAI_IP}, 10.0.0.1` }, + { verifyIdentity: true } + ) + expect(e.properties.bot_verification).toBe('verified') + }) + + it('reports unverifiable for vendors without a published feed', async () => { + const e = await capture( + { 'user-agent': 'Mozilla/5.0 (compatible; Bytespider/1.0)', 'x-forwarded-for': '1.2.3.4' }, + { verifyIdentity: true } + ) + expect(e.properties.bot_verified).toBeNull() + expect(e.properties.bot_verification).toBe('unverifiable') + }) +}) diff --git a/test/verify.test.ts b/test/verify.test.ts new file mode 100644 index 0000000..3174435 --- /dev/null +++ b/test/verify.test.ts @@ -0,0 +1,214 @@ +import { describe, expect, it } from 'vitest' +import { compileRanges, ipInCidr, ipInRanges, ipv4ToInt, ipv6ToBigInt } from '../src/cidr.js' +import { verifyBotIdentity, verifyRequest, verifiableVendors } from '../src/verify.js' +import { BOT_IP_RANGES } from '../src/bot-ranges.js' + +describe('ipv4ToInt', () => { + it('parses dotted quads', () => { + expect(ipv4ToInt('0.0.0.0')).toBe(0) + expect(ipv4ToInt('255.255.255.255')).toBe(4294967295) + expect(ipv4ToInt('192.168.1.1')).toBe(3232235777) + }) + + it('rejects malformed input', () => { + for (const bad of ['', '1.2.3', '1.2.3.4.5', '256.1.1.1', 'a.b.c.d', '1.2.3.-1', ' 1.2.3.4']) { + expect(ipv4ToInt(bad)).toBeNull() + } + }) + + it('rejects leading zeros, which some parsers read as octal', () => { + // 010.1.1.1 is 8.1.1.1 in octal but 10.1.1.1 in decimal — refusing it + // avoids disagreeing with whatever produced the range list. + expect(ipv4ToInt('010.1.1.1')).toBeNull() + expect(ipv4ToInt('01.2.3.4')).toBeNull() + }) +}) + +describe('ipv6ToBigInt', () => { + it('parses full and compressed forms to the same value', () => { + expect(ipv6ToBigInt('2001:0db8:0000:0000:0000:0000:0000:0001')).toBe( + ipv6ToBigInt('2001:db8::1') + ) + expect(ipv6ToBigInt('::')).toBe(0n) + expect(ipv6ToBigInt('::1')).toBe(1n) + }) + + it('parses IPv4-mapped addresses', () => { + expect(ipv6ToBigInt('::ffff:192.168.1.1')).toBe(0xffffn * 2n ** 32n + 3232235777n) + }) + + it('strips zone indexes', () => { + expect(ipv6ToBigInt('fe80::1%eth0')).toBe(ipv6ToBigInt('fe80::1')) + }) + + it('rejects malformed input', () => { + for (const bad of ['', '1.2.3.4', '2001::db8::1', 'gggg::1', '1:2:3:4:5:6:7', '1:2:3:4:5:6:7:8:9']) { + expect(ipv6ToBigInt(bad)).toBeNull() + } + }) +}) + +describe('ipInCidr', () => { + it('matches IPv4 boundaries exactly', () => { + // /28 spans .192 through .207 — the two adjacent addresses must miss. + expect(ipInCidr('104.208.184.192', '104.208.184.192/28')).toBe(true) + expect(ipInCidr('104.208.184.207', '104.208.184.192/28')).toBe(true) + expect(ipInCidr('104.208.184.191', '104.208.184.192/28')).toBe(false) + expect(ipInCidr('104.208.184.208', '104.208.184.192/28')).toBe(false) + }) + + it('handles /32 host routes and /0 catch-alls', () => { + expect(ipInCidr('34.162.230.222', '34.162.230.222/32')).toBe(true) + expect(ipInCidr('34.162.230.223', '34.162.230.222/32')).toBe(false) + // /0 must match everything — `0xffffffff << 32` is a no-op in JS, so this + // is the case a naive mask implementation gets wrong. + expect(ipInCidr('8.8.8.8', '0.0.0.0/0')).toBe(true) + }) + + it('handles the high bit without sign errors', () => { + // 255.x addresses overflow into negative territory under signed shifts. + expect(ipInCidr('255.255.255.255', '255.255.255.0/24')).toBe(true) + expect(ipInCidr('200.0.0.1', '128.0.0.0/1')).toBe(true) + expect(ipInCidr('127.0.0.1', '128.0.0.0/1')).toBe(false) + }) + + it('matches IPv6 ranges', () => { + expect(ipInCidr('2001:4860:4801:2008::1', '2001:4860:4801:2008::/64')).toBe(true) + expect(ipInCidr('2001:4860:4801:2009::1', '2001:4860:4801:2008::/64')).toBe(false) + }) + + it('matches IPv4-mapped IPv6 against IPv4 ranges', () => { + // Dual-stack edges hand us ::ffff:a.b.c.d; it must still match the v4 list. + expect(ipInCidr('::ffff:104.208.184.200', '104.208.184.192/28')).toBe(true) + expect(ipInCidr('::ffff:104.208.184.208', '104.208.184.192/28')).toBe(false) + }) + + it('returns false rather than throwing on junk', () => { + for (const bad of ['', ' ', 'not-an-ip', '999.999.999.999']) { + expect(ipInCidr(bad, '10.0.0.0/8')).toBe(false) + } + expect(ipInRanges('10.0.0.1', compileRanges(['garbage', '10.0.0.0/33']))).toBe(false) + }) +}) + +describe('verifyBotIdentity', () => { + const CHATGPT_UA = 'Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)' + // First prefix of OpenAI's published chatgpt-user feed. + const REAL_OPENAI_IP = '104.208.184.193' + + it('verifies a real crawler from a published range', () => { + const r = verifyBotIdentity(CHATGPT_UA, REAL_OPENAI_IP) + expect(r.verdict).toBe('verified') + expect(r.verified).toBe(true) + expect(r.claimed).toBe('ChatGPT') + }) + + it('flags the same UA from an unpublished IP as spoofed', () => { + // This is the whole point: `curl -A "ChatGPT-User"` from anywhere else. + const r = verifyBotIdentity(CHATGPT_UA, '1.2.3.4') + expect(r.verdict).toBe('spoofed') + expect(r.verified).toBe(false) + expect(r.claimed).toBe('ChatGPT') + }) + + it('verifies Anthropic, Perplexity, and Apple from their published ranges', () => { + const cases: Array<[string, string, string]> = [ + ['ClaudeBot/1.0', '34.162.230.222', 'Claude'], + ['Mozilla/5.0 (compatible; PerplexityBot/1.0)', '107.20.236.150', 'Perplexity'], + ['Mozilla/5.0 (compatible; Applebot/0.1)', '17.22.237.5', 'Apple'] + ] + for (const [ua, ip, label] of cases) { + const r = verifyBotIdentity(ua, ip) + expect(`${label}:${r.verdict}`).toBe(`${label}:verified`) + expect(r.claimed).toBe(label) + } + }) + + it('returns unverifiable — never spoofed — for vendors with no published feed', () => { + // Branding Bytespider or Amazonbot an impostor just because we lack their + // ranges would be a false accusation, so these must not be `false`. + for (const ua of [ + 'Mozilla/5.0 (compatible; Bytespider/1.0)', + 'Mozilla/5.0 (compatible; Amazonbot/0.1)', + 'meta-externalagent/1.1', + 'Mozilla/5.0 (compatible; SemrushBot/7~bl)' + ]) { + const r = verifyBotIdentity(ua, '1.2.3.4') + expect(r.verdict).toBe('unverifiable') + expect(r.verified).toBeNull() + } + }) + + it('returns unverifiable when no IP is available', () => { + for (const ip of ['', ' ', null, undefined]) { + const r = verifyBotIdentity(CHATGPT_UA, ip) + expect(r.verdict).toBe('unverifiable') + expect(r.reason).toBe('no-client-ip') + expect(r.verified).toBeNull() + } + }) + + it('never accuses client-side agents that fetch from the user\'s own machine', () => { + // Claude Code runs on a developer's laptop, so its IP is theirs and will + // never be in Anthropic's ranges. Production data: 6,492 events across + // 4,486 IPs, 0% in range — a naive vendor-level check brands every one of + // them an impostor. + const clientSide = [ + 'Claude-User (claude-code/2.1.218; +https://support.anthropic.com/)', + 'Mozilla/5.0 (compatible; Perplexity-User/1.0)' + ] + for (const ua of clientSide) { + const r = verifyBotIdentity(ua, '109.135.42.185') + expect(`${ua.slice(0, 12)}:${r.verdict}`).toBe(`${ua.slice(0, 12)}:unverifiable`) + expect(r.reason).toBe('client-side-agent') + expect(r.verified).toBeNull() + } + }) + + it('still verifies the server-side crawler from the same vendor', () => { + // ClaudeBot proper fetches from Anthropic infra and verifies at ~96%. + expect(verifyBotIdentity('ClaudeBot/1.0', '34.162.230.222').verdict).toBe('verified') + // A `-User` suffix is not itself the signal: OpenAI fetches server-side. + expect(verifyBotIdentity(CHATGPT_UA, REAL_OPENAI_IP).verdict).toBe('verified') + }) + + it('returns not-claimed for browsers and unknown UAs', () => { + for (const ua of [ + 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/120 Safari/537.36', + '', + null + ]) { + const r = verifyBotIdentity(ua, '1.2.3.4') + expect(r.verdict).toBe('not-claimed') + expect(r.verified).toBeNull() + expect(r.claimed).toBeNull() + } + }) + + it('reads the client IP from x-forwarded-for on a request', () => { + const req = new Request('https://example.com/', { + headers: { + 'user-agent': CHATGPT_UA, + // Only the first hop is the client; the rest are proxies. + 'x-forwarded-for': `${REAL_OPENAI_IP}, 10.0.0.1, 10.0.0.2` + } + }) + expect(verifyRequest(req).verdict).toBe('verified') + }) +}) + +describe('bundled ranges', () => { + it('covers the vendors that publish feeds', () => { + expect(verifiableVendors()).toEqual(['ChatGPT', 'Claude', 'Perplexity', 'Apple']) + }) + + it('contains only parseable CIDRs', () => { + for (const [vendor, cidrs] of Object.entries(BOT_IP_RANGES)) { + expect(cidrs.length, `${vendor} has no ranges`).toBeGreaterThan(0) + const compiled = compileRanges(cidrs) + // Every entry must survive compilation — a silently dropped prefix means + // real crawler traffic from that block gets marked spoofed. + expect(compiled.v4.length + compiled.v6.length, `${vendor} dropped entries`).toBe(cidrs.length) + } + }) +})