OpenCrawl

Distributed browser rendering as a service. Workers contribute real Chrome browsers, users pay credits to crawl any JavaScript-rendered page.

Built for tools like OpenClaw that run on headless VPS environments without real browser access.

中文文档

Why OpenCrawl?

AI agents and scraping tools on VPS/cloud servers often can't render JavaScript-heavy pages — they lack a real browser. Solutions like Puppeteer or Playwright are resource-heavy (4GB+ RAM) and hard to maintain.

OpenCrawl solves this by crowdsourcing real Chrome browsers:

Anyone can install the Chrome extension and become a Worker
Workers earn credits for each page they render
Users spend credits to crawl any URL via a simple API
Results are stored on Cloudflare R2 (zero egress fees)
Workers' cookies and sessions are isolated via incognito mode

User (API)                Platform (FastAPI)           Worker (Chrome Extension)
    │                          │                             │
    │  POST /api/crawl         │        WebSocket /ws        │
    │  {url, selector?}        │◄────────────────────────────┤
    ├─────────────────────────►│                             │
    │                          │  dispatch task + upload URL  │
    │                          ├────────────────────────────►│
    │                          │                             │ open tab (incognito)
    │                          │                             │ render JavaScript
    │                          │                             │ extract DOM content
    │                          │          Cloudflare R2      │
    │                          │         (zero egress)       │
    │                          │               ▲ PUT         │
    │                          │               ├─────────────┤
    │                          │  taskComplete  │             │
    │  credits: user -1        │◄──────────────┤             │
    │  credits: worker +1      │               │             │
    │                          │               │             │
    │  {downloadUrl}           │               │             │
    │◄─────────────────────────┤               │             │
    │                          │               │             │
    │  GET downloadUrl ────────┼──────► R2 download          │

OpenClaw Integration

OpenCrawl is designed as a browser rendering backend for OpenClaw agents running on headless VPS environments.

Instead of installing Chromium + Playwright on your server (4GB+ RAM, complex setup), point your OpenClaw agent to OpenCrawl's API:

import requests

# Fetch any JS-rendered page through real Chrome browsers
res = requests.post("https://your-opencrawl-server/api/crawl",
    headers={"Authorization": "Bearer ak_your_key"},
    json={"url": "https://example.com", "selector": ".main-content"})

data = res.json()
# data["downloadUrl"] → download the rendered page content from R2

This gives your VPS-hosted agent access to a pool of real browsers without any local browser installation.

Features

Dual Crawl Modes — lite (no images/CSS, 0.1 credit, ~3s) and full (complete render, 1 credit, ~8s)
Search API — Multi-engine search (DuckDuckGo + Google + Bing + Baidu), Brave Search API compatible
Credit System — Users spend credits, Workers earn credits
API Key Auth — Each user gets a unique API key
Admin Panel — Create users, recharge credits, view stats
User Panel — Check balance, view API key, usage examples
Dashboard — Real-time monitoring of Workers, tasks, history
Worker Concurrency — Single Worker handles multiple tasks in parallel (multi-tab)
Privacy Protection — Crawling in incognito windows, isolating Worker cookies
URL Blacklist — Blocks localhost, internal IPs, cloud metadata, dangerous ports
Version Control — Outdated Workers are rejected and prompted to update
Auto Cleanup — R2 objects expire after 1 day, zero storage cost
One-Click Registration — Sign up on the homepage, get 1,000 free credits

Quick Start

1. Set Up Cloudflare R2

Create a Cloudflare account
Go to R2 Object Storage → Create bucket (name: opencrawl)
Go to R2 → Manage R2 API Tokens → Create API Token
- Permissions: Object Read & Write

You'll need these values:

Variable	Description	Where to find
`R2_ACCOUNT_ID`	Cloudflare Account ID	Dashboard sidebar, 32-char string
`R2_ACCESS_KEY_ID`	R2 API Access Key	Shown after creating API Token
`R2_SECRET_ACCESS_KEY`	R2 API Secret Key	Shown once after creating API Token

2. Deploy the Server

git clone https://github.com/hlyylly/OpenCrawl.git
cd OpenCrawl
pip install -r requirements.txt

cp .env.example .env
# Edit .env with your R2 credentials and admin key

uvicorn server:app --host 0.0.0.0 --port 9877

3. Create Your First User

curl -X POST http://localhost:9877/api/admin/create-key \
  -H "Authorization: Bearer your_admin_key" \
  -H "Content-Type: application/json" \
  -d '{"name": "my-account", "credits": 100}'

Or just visit http://localhost:9877 and click Register.

4. Install the Chrome Extension (Worker)

Open chrome://extensions/, enable Developer Mode
Click "Load unpacked", select the extension/ directory
Click the extension icon, configure:
- Server URL: ws://your-server-ip:9877/ws
- API Key: your key (optional, for earning credits)
Click "Save & Reconnect"
Recommended: Go to extension details → Enable "Allow in incognito" for cookie isolation

API Reference

Crawl a Page

# Lite mode (no images/CSS, faster, 0.1 credit)
curl -X POST http://your-server:9877/api/crawl \
  -H "Authorization: Bearer ak_xxx" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "mode": "lite"}'

# Full mode (complete render, 1 credit)
curl -X POST http://your-server:9877/api/crawl \
  -H "Authorization: Bearer ak_xxx" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "mode": "full", "selector": ".article"}'

# linkFilter — customize which links to extract (defaults to file links only, e.g. pdf/xlsx)
#   "all"  → all links; custom → comma-separated substrings/regex matched against URL or anchor text
curl -X POST http://your-server:9877/api/crawl \
  -H "Authorization: Bearer ak_xxx" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://post.smzdm.com/p/xxx", "linkFilter": "go.smzdm.com,jd.com"}'
# response data.links → [{text, url}, ...] of matched links

Params: selector scopes extraction, mode controls rendering (lite/full), linkFilter controls which links are returned (omit/file = file links only, all = all, or custom comma-separated substrings/regex).

Response:

{
  "success": true,
  "url": "https://example.com",
  "mode": "lite",
  "r2Key": "tasks/xxx.json",
  "downloadUrl": "https://...signed-r2-url..."
}

Search (Brave Search API Compatible)

# Lite search — DuckDuckGo only (0.1 credit)
curl -X POST http://your-server:9877/api/search \
  -H "Authorization: Bearer ak_xxx" \
  -H "Content-Type: application/json" \
  -d '{"q": "python tutorial", "mode": "lite"}'

# Full search — DuckDuckGo + Google + Bing + Baidu parallel (3 credits, ~20-30 results)
curl -X POST http://your-server:9877/api/search \
  -H "Authorization: Bearer ak_xxx" \
  -H "Content-Type: application/json" \
  -d '{"q": "python tutorial", "mode": "full"}'

Response (Brave Search compatible):

{
  "success": true,
  "query": "python tutorial",
  "type": "search",
  "mode": "full",
  "engines": ["duckduckgo", "bing", "google", "baidu"],
  "web": {
    "results": [
      {"title": "...", "url": "https://...", "description": "...", "source": "duckduckgo"},
      {"title": "...", "url": "https://...", "description": "...", "source": "google"}
    ]
  }
}

Check Balance

curl http://your-server:9877/api/balance \
  -H "Authorization: Bearer ak_xxx"

Platform Status (Public)

curl http://your-server:9877/api/status

Admin — Create User

curl -X POST http://your-server:9877/api/admin/create-key \
  -H "Authorization: Bearer admin_key" \
  -H "Content-Type: application/json" \
  -d '{"name": "username", "credits": 100}'

Admin — Recharge Credits

curl -X POST http://your-server:9877/api/admin/recharge \
  -H "Authorization: Bearer admin_key" \
  -H "Content-Type: application/json" \
  -d '{"apiKey": "ak_xxx", "credits": 50}'

Pricing

API	Mode	Credits	Speed	Description
`/api/crawl`	`lite`	0.1	~3s	No images/CSS, fast extraction
`/api/crawl`	`full`	1.0	~8s	Full JS render
`/api/search`	`lite`	0.1	~5s	DuckDuckGo single engine
`/api/search`	`full`	3.0	~10s	4 engines parallel, 20-30 results

Web Pages

Path	Description
`/`	Dashboard + one-click registration
`/admin`	Admin panel (requires Admin Key)
`/user`	User panel (requires API Key)

Tech Stack

Component	Technology
Server	Python / FastAPI / uvicorn
Storage	Cloudflare R2 (S3-compatible, zero egress)
Credits	JSON file (data/users.json)
Worker	Chrome Extension (Manifest V3)
Communication	HTTP REST + WebSocket

R2 Free Tier

Resource	Free/month
Storage	10 GB
Write (PUT)	1M requests
Read (GET)	10M requests
Egress	Unlimited free

With 30KB average per task and 1-day auto-expiry, up to 30,000 tasks/day for free.

Security

URL Blacklist — Blocks localhost, private IPs (10.x, 192.168.x, 172.16-31.x), cloud metadata (169.254.169.254), dangerous ports (22, 3306, 6379...)
Incognito Isolation — Crawling tabs open in incognito windows, completely isolating Worker's personal cookies and sessions
Signed URLs — R2 upload/download URLs are time-limited (10min upload, 1hr download)
Upload Verification — Server verifies R2 upload via HEAD before confirming task completion
Heartbeat Detection — Stale Worker connections are automatically cleaned up after 30s
Worker Version Control — Outdated Workers are rejected with update prompt, no tasks assigned

License

MIT

Name		Name	Last commit message	Last commit date
extension		extension
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
README_CN.md		README_CN.md
admin.html		admin.html
architecture.md		architecture.md
dashboard.html		dashboard.html
local_server.py		local_server.py
requirements.txt		requirements.txt
server.py		server.py
user.html		user.html

Navigation Menu

Repository files navigation

OpenCrawl

Why OpenCrawl?

OpenClaw Integration

Features

Quick Start

1. Set Up Cloudflare R2

2. Deploy the Server

3. Create Your First User

4. Install the Chrome Extension (Worker)

API Reference

Crawl a Page

Search (Brave Search API Compatible)

Check Balance

Platform Status (Public)

Admin — Create User

Admin — Recharge Credits

Pricing

Web Pages

Tech Stack

R2 Free Tier

Security

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

OpenCrawl

Why OpenCrawl?

OpenClaw Integration

Features

Quick Start

1. Set Up Cloudflare R2

2. Deploy the Server

3. Create Your First User

4. Install the Chrome Extension (Worker)

API Reference

Crawl a Page

Search (Brave Search API Compatible)

Check Balance

Platform Status (Public)

Admin — Create User

Admin — Recharge Credits

Pricing

Web Pages

Tech Stack

R2 Free Tier

Security

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages