beacon

Make your site legible to AI crawlers, and find out which ones actually came.

Beacon does two things. It serves a clean Markdown twin of every page to clients that ask for one, and it reports what visited - which crawler, from where, and whether the User-Agent was telling the truth.

MIT licensed. It ships with a server you can run yourself, so none of this has to go through anyone else's platform.

Packages

Package	What it is
@snowseo/beacon	The client. Generates Markdown twins, serves them by content negotiation, reports hits. Node, Next.js, Workers, Deno, Bun, or any fetch-based framework.
@snowseo/beacon-server	The receiving half. Crawler verification (published IP ranges, datacenter detection, reverse DNS, Web Bot Auth) plus a reference ingest server.

Quick start

npm install @snowseo/beacon
npx beacon build dist

That writes a .md twin next to every HTML page. Then serve them:

import { createBeacon } from "@snowseo/beacon";

const beacon = createBeacon({ siteUrl: "https://example.com", dir: "dist" });

Full instructions for Next.js, Express, Cloudflare Workers, Astro, Deno and Bun are in packages/beacon.

PHP and WordPress are served by a separate client that speaks the same protocol: one auto_prepend_file line, no application code touched. It lives with the WordPress plugin, which is what it is built into.

Reporting only

Measuring AI crawlers and serving them Markdown are separate jobs. If you only want the measurement, that is the whole config:

import { createBeacon } from "@snowseo/beacon";

export const beacon = createBeacon({
  analytics: { key: process.env.SNOW_API_KEY, collector: "snow-analytics" },
});

No siteUrl — a key belongs to one site, so the collector resolves it. No resolve, because there are no twins to serve. And strictNegotiation defaults to false here, so a beacon that answers nothing never turns your own JSON endpoints into 406s.

collector takes a name from COLLECTORS: "snow-analytics" or "snowseo". Naming it beats spelling out a URL, which is easy to get subtly wrong — a bare origin, or the wrong mount path, fails as an opaque 401 rather than a 404.

Running your own collector

The analytics half is not tied to any hosted service. Point it wherever you like:

BEACON_KEYS=$(openssl rand -hex 24) npx @snowseo/beacon-server
# [beacon] listening on http://0.0.0.0:8787/beacon/hits

createBeacon({
  siteUrl: "https://example.com",
  dir: "dist",
  analytics: { key: "...", endpoint: "https://beacon.example.com" },
});

Two routes, no more: POST /beacon/hits and GET /health. It classifies and verifies exactly as the hosted service does - the same ingestBatch function runs in both - and stores to SQLite or Postgres.

The wire contract is frozen and documented in PROTOCOL.md, so writing your own collector in another language is a supported path, not a hack.

Why the server half exists at all

A User-Agent is a claim, and it costs nothing to forge. Beacon's client deliberately draws no conclusions: it reports raw facts and lets the server decide, because a client that graded its own traffic would just be repeating what the crawler told it.

Verification uses three signals, weakest to strongest:

• Published IP ranges. Providers publish the addresses their crawlers use.
• Reverse DNS. A PTR under an authenticating suffix, forward-confirmed.
• Web Bot Auth. An RFC 9421 signature from a signer the registry lists. The only one that is proof rather than inference.

None of that is a service you call. The agents it matches, by User-Agent or by the key directory they sign with, are plain lists in crawlers.ts, and the range feeds are URLs in crawler-ranges.ts that your own server fetches straight from the provider. Read them, and send a PR to add one - CONTRIBUTING.md says what an entry needs.

Results land in four states - signed, verified, unverified, spoofed_suspected. unverified means there was nothing to check against; spoofed_suspected means something checkable contradicts the claim. Keeping those apart matters more than it sounds: collapsed together, "we could not tell" becomes an accusation.

Contributing

This is where beacon is developed. Issues and pull requests are read and merged here, and the npm packages are published from this repository's tags.

The most common contribution by far is a crawler we do not recognise yet. See CONTRIBUTING.md for what a registry entry needs, and for running the suites - including the real Apache end-to-end run.

Hosted SnowSEO

SnowSEO runs this same code and adds what is not in this repository: a dashboard, attribution from crawl to citation, and history beyond what your own database holds. Using it is optional - beacon works completely without it.

License

MIT.