telegram-graph-crawler

A crawler that builds a channel-to-channel graph of public Telegram channels, following "Forwarded from" provenance links, bio/description links, and in-message links between channels.

Implemented as a Cloudflare Worker (TypeScript, native HTMLRewriter streaming parser, no HTML-parsing dependency) backed by D1, running on a Cron Trigger.

How it works

• Source: t.me/s/<channel> — Telegram's public, unauthenticated channel-preview page. No API credentials, no login, no private groups or invite-link channels.
• Signals extracted per channel:
  1. Bio/description links (curated, self-declared affiliation)
  2. In-message links (incidental mention)
  3. "Forwarded from" (directed, timestamped provenance)
• Storage: channels (status: pending/done/failed) and edges (weighted, incremented on repeat sightings) tables in D1. A newly discovered channel is inserted as pending before it's ever fetched, and a channel's full outcome (new pending rows + edges + its own status) commits as one atomic batch() transaction — crash-safe by construction, verified against real mid-batch restart tests.
• Redirect handling: dead-end redirects (personal Telegram accounts mis-extracted as channel links) are classified and failed immediately rather than sitting in the retry queue for weeks; genuine transient failures (network errors, rate limits) are left pending and retried automatically.
• No full message-text archival — graph structure and light per-channel metadata only.

Scraping approach

This only touches Telegram's public, unauthenticated t.me/s/<channel> preview pages — the same HTML anyone gets by visiting that URL in a browser, logged out. No API keys, no login, no private data.

• Bounded batches (BATCH_SIZE) on a Cron Trigger, not back-to-back requests.
• A self-identifying User-Agent: Mozilla/5.0 (compatible; MapOfTelegramBot/0.1).
• No proxy rotation, no User-Agent spoofing, no CAPTCHA bypassing, no detection evasion of any kind. If Telegram wants to block this, the User-Agent makes that trivial.

Running it

cd worker
npm install
npx wrangler d1 execute map-of-telegram --file=schema.sql   # create your own D1 database first and update wrangler.toml's database_id
npx wrangler deploy

The Cron Trigger in wrangler.toml runs the crawl on a schedule once deployed. See src/index.ts for the batch size and per-channel page limits.