distributed-crawling

Here are 3 public repositories matching this topic...

hlyylly / OpenCrawl

Distributed browser rendering as a service. Built for OpenClaw and AI agents on headless VPS without real browsers. Workers earn credits, users spend credits.

chrome-extension web-scraping fastapi ai-agent distributed-crawling cloudflare-r2 browser-rendering openclaw

Updated Jul 22, 2026
Python

DenimEvert / crawlab-capsolver

Star

Crawlab & CapSolver integration. Solve reCAPTCHA & Turnstile for distributed web crawling systems.

selenium web-scraping scrapy turnstile puppeteer crawlab distributed-crawling capsolver

Updated Jan 12, 2026
Python

mory-dev / hexdigest-crawl-sharding

Star

Deterministic crawler sharding, lease-aware SQLite state, and safe multi-worker launches.

python open-source crawler sqlite web-scraping scrapy sharding hexdigest distributed-crawling parallel-crawling worker-orchestration crawl-queue hexdigest-crawl-sharding

Updated Aug 28, 2026
Python

Add this topic to your repo

To associate your repository with the distributed-crawling topic, visit your repo's landing page and select "manage topics."

Learn more