feat: agentPolicy() — decide what to do with an agent, by intent

[Gdewilde]

Classification tells you which agent knocked. It doesn't tell you what to do about it — and the obvious answer, "block the AI bots", is wrong in a way that costs real money.

The distinction this is built on

A vendor's bulk crawler and its user-facing fetcher have opposite economics while wearing similar names:

• GPTBot — corpus collection. You get nothing back per fetch.
• ChatGPT-User — a person just asked about you, and an assistant is reading the page for them. That's a distribution channel. Charging for it means charging for your own marketing.

Same for ClaudeBot vs Claude-User, PerplexityBot vs Perplexity-User.

It isn't a marginal split

Measured on 30 days of AI traffic to apideck.com:

intent	events	share	default action
retrieval	310,882	60.1%	allow — this is demand
training	199,013	38.5%	meter
unknown	7,579	1.5%	allow
tooling	4	0.0%	allow

A blanket bot block would have discarded 60% of that traffic — all of it people asking questions about the product.

API

const decision = agentPolicy(req, { verify: true, onTraining: 'charge' })

if (decision.action === 'block')  return new Response(null, { status: 403 })
if (decision.action === 'charge') return paymentRequired(decision)

Maps a request to allow / meter / charge / block via an intent of retrieval / training / search / tooling / unknown. One knob per intent, plus an allowList that overrides everything. Defaults are conservative: meter training, allow the rest.

Verified against the real UA shapes:

action   intent     label        what
charge   training   ChatGPT      OpenAI training crawl
allow    retrieval  ChatGPT      person asked ChatGPT
charge   training   Claude       Anthropic training crawl
allow    retrieval  Claude       Claude Code on a laptop
allow    search     Google       search index
allow    tooling    curl         script or coding agent

Design notes

• Intent is an explicit list, not a pattern guess. A -User suffix is not the marker — OpenAI's ChatGPT-User fetches server-side from Azure, while Claude Code's Claude-User runs on a developer's laptop. Guessing here is exactly how you end up billing your own demand.
• Applebot is search; Applebot-Extended is training. Same vendor, and that distinction is the whole reason Apple publishes two tokens.
• Only spoofed blocks when verify is on. unverifiable must never block, or you refuse every vendor without a published feed plus every coding agent running locally — the same trap feat: verify claimed crawler identity against published IP ranges #18 documented.
• No payment rails, no network calls, no dependencies. This is the policy layer. Settlement belongs to Stripe's MPP or an x402 SDK; the library should not hold money or inherit PCI scope.

Why now

Stripe and Tempo's Machine Payments Protocol (18 March 2026) and x402 both settle at the HTTP layer — the same layer this middleware already occupies. Metering is the prerequisite for either, and the retrieval/training split is the policy that makes metering sane rather than self-harming.

This PR is the classification and policy half only. A paymentRequired() helper emitting the 402 challenge would be the natural follow-up, and should stay an adapter over someone else's rail.

Tests: 216 → 229.

🤖 Generated with Claude Code