Deterministic crawler sharding, lease-aware SQLite state, and safe multi-worker launches.
python open-source crawler sqlite web-scraping scrapy sharding hexdigest distributed-crawling parallel-crawling worker-orchestration crawl-queue hexdigest-crawl-sharding
-
Updated
Aug 28, 2026 - Python