feat: verify claimed crawler identity against published IP ranges

README.md

@@ -306,6 +306,82 @@ Full middleware example: [`README.md → Markdown mirror helpers`](./README.md#m
 
 ---
 
+## Advanced: verifying crawler identity against published IP ranges
+
+User agents are trivially forged — `curl -A "ChatGPT-User"` is indistinguishable
+from the real thing at the UA layer. Set `verifyIdentity: true` to check the
+client IP against the vendor's published crawler ranges:
+
+```ts
+void trackVisit(request, {
+  analytics,
+  verifyIdentity: true,
+  captureIp: true // not required, but useful for auditing a 'spoofed' verdict
+})
+```
+
+Three properties land on the event:
+
+| property | values |
+| --- | --- |
+| `bot_verification` | `verified` \| `spoofed` \| `unverifiable` \| `not-claimed` |
+| `bot_verified` | `true` \| `false` \| `null` — tri-state, for quick filtering |
+| `bot_verification_reason` | why, when the verdict is `unverifiable` |
+
+### What can actually be verified
+
+Only vendors that publish a machine-readable range feed: **OpenAI**,
+**Anthropic**, **Perplexity**, and **Apple**. Bytespider, Amazonbot, Meta and
+the rest report `unverifiable` — never `spoofed`. Collapsing "we can't check"
+into "impostor" would be a false accusation, which is why `bot_verified` is
+tri-state rather than a boolean.
+
+### Server-side crawlers vs client-side agents
+
+A published range list covers a vendor's **crawler fleet**, not its products
+that fetch from the end user's device. Claude Code runs on a developer's
+laptop, so the request carries *their* IP and will never appear in Anthropic's
+ranges. Measured over 30 days of production traffic:
+
+| user agent | events | distinct IPs | in published range |
+| --- | ---: | ---: | ---: |
+| `ClaudeBot` | 13,671 | 236 | 96% |
+| `PerplexityBot` | 6,897 | 158 | 91% |
+| `ChatGPT-User` | ~72,000 | 43 | 99% |
+| `Claude-User` (claude-code CLI) | 6,492 | 4,486 | **0%** |
+| `Perplexity-User` | 493 | 148 | **0%** |
+
+A naive vendor-level check would brand the bottom two rows — roughly 7,000
+legitimate fetches a month — as impersonation. So the library gates verdicts on
+the *product*, returning `unverifiable` with reason `client-side-agent` for
+those. Note the distinction is not a `-User` suffix: OpenAI's `ChatGPT-User`
+fetches server-side from Azure and verifies at ~99%.
+
+### Keeping the ranges fresh
+
+The bundled snapshot is in `src/bot-ranges.ts`, stamped with
+`BOT_RANGES_CAPTURED_AT`. Refresh it on a schedule:
+
+```bash
+node scripts/refresh-bot-ranges.mjs
+```
+
+Freshness is the whole game. Nearly every OpenAI prefix is an Azure block and
+Anthropic's are GCP, so "came from a datacenter" proves nothing on its own —
+only membership in the *current* published list does. A stale snapshot produces
+false `spoofed` verdicts on real crawlers, so the refresh script refuses to
+write a list that shrinks by more than half or when any feed errors.
+
+### Trusting the client IP
+
+The verdict is only as good as the IP. On Vercel and Cloudflare the edge
+overwrites `x-forwarded-for`, so the first hop is trustworthy. Behind a proxy
+that passes a client-supplied header through, an attacker controls the value
+and `verified` means nothing — confirm your proxy's behaviour before acting on
+this data.
+
+---
+
 ## Advanced: Peec.ai crawl-insights export
 
 [Peec.ai](https://peec.ai)'s **Agent analytics** product ingests a CSV/CLF access log and produces dashboards on top of it. The Peec docs assume you have a Vercel Log Drain → Axiom (or similar) pipeline that emits these eight columns: `timestamp, request_method, request_url, response_status, client_ip, user_agent, country_code, referer`.

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@apideck/agent-analytics",
-  "version": "0.10.0",
+  "version": "0.11.0",
   "description": "Track AI agent and bot traffic to your Next.js / Vercel app — PostHog, webhooks, or any custom analytics backend. Detects Claude, ChatGPT, Perplexity, Google-Extended, and more.",
   "keywords": [
     "ai",

scripts/refresh-bot-ranges.mjs

@@ -0,0 +1,116 @@
+#!/usr/bin/env node
+/**
+ * Regenerate src/bot-ranges.ts from the vendors' published crawler IP feeds.
+ *
+ *     node scripts/refresh-bot-ranges.mjs
+ *
+ * Run this on a schedule. The lists rotate, and a stale snapshot is worse than
+ * no snapshot: it produces false 'spoofed' verdicts on legitimate crawlers,
+ * which is exactly the conclusion you'd act on. The script refuses to write a
+ * file when a feed fails or shrinks implausibly, so a bad fetch can't silently
+ * empty out a vendor.
+ */
+import { writeFileSync, readFileSync } from 'node:fs'
+import { fileURLToPath } from 'node:url'
+import { dirname, join } from 'node:path'
+
+const OUT = join(dirname(fileURLToPath(import.meta.url)), '..', 'src', 'bot-ranges.ts')
+
+// Vendor label (as parseBotName returns) -> the feeds that make up its ranges.
+const FEEDS = {
+  ChatGPT: [
+    'https://openai.com/gptbot.json',
+    'https://openai.com/chatgpt-user.json',
+    'https://openai.com/searchbot.json'
+  ],
+  Claude: ['https://claude.com/crawling/bots.json'],
+  Perplexity: ['https://www.perplexity.ai/perplexitybot.json'],
+  Apple: ['https://search.developer.apple.com/applebot.json']
+}
+
+// A feed dropping below this fraction of its previous size signals a partial
+// or malformed response rather than a genuine shrink. Bail instead of writing.
+const SHRINK_FLOOR = 0.5
+
+async function fetchPrefixes(url) {
+  const res = await fetch(url, { headers: { 'user-agent': 'agent-analytics range refresher' } })
+  if (!res.ok) throw new Error(`${url} -> HTTP ${res.status}`)
+  const body = await res.json()
+  const prefixes = body?.prefixes
+  if (!Array.isArray(prefixes)) throw new Error(`${url} -> no prefixes array`)
+  // Feeds use ipv4Prefix / ipv6Prefix keys; take whichever is present.
+  return prefixes.map((e) => e.ipv4Prefix ?? e.ipv6Prefix).filter(Boolean)
+}
+
+function previousCounts() {
+  try {
+    const src = readFileSync(OUT, 'utf8')
+    const counts = {}
+    for (const vendor of Object.keys(FEEDS)) {
+      const block = src.match(new RegExp(`  ${vendor}: \\[([\\s\\S]*?)\\n  \\]`))
+      counts[vendor] = block ? (block[1].match(/'/g) || []).length / 2 : 0
+    }
+    return counts
+  } catch {
+    return {}
+  }
+}
+
+const before = previousCounts()
+const vendors = {}
+
+for (const [vendor, urls] of Object.entries(FEEDS)) {
+  const all = new Set()
+  for (const url of urls) {
+    const got = await fetchPrefixes(url) // throws — a failed feed must abort the run
+    got.forEach((p) => all.add(p))
+    console.log(`  ${url} -> ${got.length}`)
+  }
+  const list = [...all].sort()
+  const prev = before[vendor] ?? 0
+  if (prev && list.length < prev * SHRINK_FLOOR) {
+    throw new Error(
+      `${vendor}: ${list.length} prefixes vs ${prev} previously — refusing to write a suspiciously small list`
+    )
+  }
+  vendors[vendor] = list
+  console.log(`${vendor}: ${list.length} prefixes (was ${prev})`)
+}
+
+const capturedAt = new Date().toISOString()
+const body = `/**
+ * Published crawler IP ranges, vendor by vendor.
+ *
+ * GENERATED FILE — do not edit by hand. Regenerate with:
+ *
+ *     node scripts/refresh-bot-ranges.mjs
+ *
+ * Keys match the labels {@link parseBotName} returns, so a claimed identity
+ * maps to its range list without a translation table.
+ *
+ * Only vendors that publish a machine-readable feed appear here. A vendor's
+ * absence means "cannot be verified", never "not a real bot" — see
+ * {@link verifyBotIdentity} for how that distinction is surfaced.
+ *
+ * Freshness is the whole ballgame: these lists rotate. Almost every OpenAI
+ * prefix is an Azure block and every Anthropic prefix is GCP or similar, so
+ * "came from a datacenter" proves nothing on its own — only membership in the
+ * current published list does. A stale snapshot produces false 'spoofed'
+ * verdicts on legitimate crawlers, which is the failure mode to fear.
+ */
+
+/** When these ranges were captured from the vendor feeds (UTC). */
+export const BOT_RANGES_CAPTURED_AT = '${capturedAt}'
+
+export const BOT_IP_RANGES: Readonly<Record<string, readonly string[]>> = {
+${Object.entries(vendors)
+  .map(([name, cidrs]) => `  ${name}: [\n${cidrs.map((c) => `    '${c}',`).join('\n')}\n  ],`)
+  .join('\n')}
+}
+
+/** Vendor labels this build can verify. Anything else yields a null verdict. */
+export const VERIFIABLE_VENDORS: readonly string[] = Object.keys(BOT_IP_RANGES)
+`
+
+writeFileSync(OUT, body)
+console.log(`\nwrote ${OUT}`)