Awesome Scrapers
A curated list of scrapers, crawlers, browser automation tools, extraction libraries, and managed services.
⚠️ = known maintenance concern; verify repository status before adoption.
LLMs understand page structure, extract via natural language, and output LLM-ready formats.
Tool
Stars
Language
Description
Firecrawl 175k
TypeScript
Websites → LLM-ready markdown or structured data via API.
browser-use 112k
Python
AI agents that control a browser to complete tasks autonomously.
Crawl4AI 81k
Python
LLM-friendly web crawler with structured extraction.
Docling 66k
Python
IBM — parse PDFs, DOCX into AI-ready output.
ScrapeGraphAI 30k
Python
Graph pipelines + LLMs to extract data via plain English.
Stagehand 24k
TypeScript
Browser automation combining natural language with code precision.
Skyvern 23k
Python
Browser workflows with computer vision + LLMs, no selectors needed.
Jina Reader 12k
TypeScript
Any URL → LLM-friendly markdown with vision model support.
llm-scraper 7k
TypeScript
Structured data from any webpage using LLMs with Zod schemas.
Spider 3k
Rust
Async, configurable web crawler with streaming and browser support.
(⬆ back to top )
🥷 Stealth & Anti-Detection The cat-and-mouse game of modern scraping.
Tool
Stars
Language
Description
Scrapling 77k
Python
Adaptive scraping with built-in anti-detection and auto-matching.
SeleniumBase 13k
Python
Browser automation with UC (Undetected Chrome) mode.
Camoufox 12k
Python
Firefox-based browser automation project with fingerprint-management features.
curl_cffi 6k
Python
HTTP client with browser TLS/JA3/HTTP2 fingerprint impersonation.
Nodriver 5k
Python
Successor to undetected-chromedriver — direct CDP, no WebDriver.
Botasaurus 6k
Python
Scraping framework with anti-detection, parallelism, and caching.
Patchright 4k
JavaScript
Undetected Playwright fork that passes bot detection.
(⬆ back to top )
🔌 MCP Servers (Model Context Protocol) Connect LLM agents (Claude, GPT, etc.) directly to scraping tools.
Server
Stars
Description
Playwright MCP 37k
Browser automation via accessibility snapshots (by Microsoft).
Firecrawl MCP 7k
Web scraping and search in Claude/Cursor via Firecrawl API.
Browserbase MCP 3k
Cloud browser control with Stagehand AI.
Bright Data MCP 3k
Web access with geo-unblocking and bot evasion.
(⬆ back to top )
The foundation for dynamic/JS-heavy scraping.
Tool
Stars
Language
Description
Puppeteer 96k
JavaScript
Google's Chrome/Firefox control via DevTools Protocol.
Playwright 95k
Multi
Cross-browser automation (Chromium, Firefox, WebKit) by Microsoft.
Selenium 34k
Multi
The OG browser automation (W3C WebDriver standard).
Crawlee 26k
TypeScript
Scraping/automation library with proxy rotation by Apify.
(⬆ back to top )
🕷️ Web Scraping Frameworks
Tool
Stars
Description
Scrapy 64k
Python scraping framework — middleware, pipelines, extensions.
MechanicalSoup 5k
Stateful browser-like interaction for simple scraping.
scrapy-playwright 1k
Playwright integration for Scrapy — JS rendering with full pipeline.
Tool
Stars
Description
Colly 25k
Fast scraping framework for Go.
Katana 17k
Crawling and spidering framework by ProjectDiscovery.
Ferret 6k
Declarative scraping with FQL query language.
Tool
Stars
Description
Nokogiri 6k
Standard HTML/XML parser for Ruby.
(⬆ back to top )
The network layer — making requests that look human.
Tool
Stars
Language
Description
aiohttp 17k
Python
Async HTTP client/server for high-concurrency scraping.
httpx 15k
Python
Async/sync HTTP client with HTTP/2 support.
curl_cffi 6k
Python
HTTP client impersonating browser TLS fingerprints (also in Stealth).
got-scraping 769
Node.js
HTTP client with header/TLS mimicry by Apify.
(⬆ back to top )
Tool
Stars
Language
Description
Cheerio 30k
JavaScript
jQuery-like HTML manipulation for Node.js.
goquery 15k
Go
jQuery-like HTML selector for Go.
jsoup 11k
Java
HTML parser with CSS selectors and XSS sanitization.
AngleSharp 6k
C#
W3C-compliant HTML5 parser for .NET.
Beautiful Soup -
Python
Most popular Python HTML/XML parser.
lxml 3k
Python
Fast XML/HTML parser with XPath and XSLT.
html5ever 3k
Rust
Browser-grade HTML5 parser from Mozilla Servo.
selectolax 2k
Python
5-30x faster than Beautiful Soup using Lexbor engine.
parsel 1k
Python
CSS/XPath selectors for HTML+JSON (powers Scrapy).
(⬆ back to top )
📝 Content & Text Extraction
Pull clean text out of messy HTML — essential for LLM/RAG pipelines.
Tool
Stars
Language
Description
Readability.js 11k
JavaScript
Mozilla's article extractor (powers Firefox Reader View).
Trafilatura 7k
Python
Web text extraction with metadata and language detection.
html2text 2k
Python
HTML → clean Markdown.
Markdownify 2k
Python
Flexible HTML-to-Markdown with customizable options.
newspaper4k 1k
Python
News article extraction with NLP and multilingual support.
(⬆ back to top )
Platforms frequently change APIs and block scrapers. Check issue trackers for current status.
Tool
Stars
Platform
Description
Instaloader 13k
Instagram
Posts, stories, reels, highlights with metadata.
TikTok-Api 7k
TikTok
Unofficial API wrapper for Python.
PRAW 4k
Reddit
Official Python Reddit API Wrapper.
(⬆ back to top )
Tool
Stars
Description
yt-dlp 188k
YouTube and 1000+ sites (fork of youtube-dl).
lux 32k
Go video downloader — 40+ sites (formerly annie).
spotdl 26k
Spotify tracks/playlists with metadata and album art.
gallery-dl 19k
Image galleries from 100+ sites (Pixiv, Twitter, Reddit).
(⬆ back to top )
📄 Document & PDF Extraction
Tool
Stars
Language
Description
Docling 66k
Python
IBM — PDFs, DOCX, PPTX into AI-ready output.
Unstructured 15k
Python
ETL pipeline for documents → structured data for LLMs.
pdfplumber 11k
Python
Text, tables, and layout from PDFs with precision.
PyMuPDF 11k
Python
Fast PDF/XPS/EPUB extraction and rendering.
Tabula 7k
Java
Data tables from PDFs. ⚠️
pdfminer.six 7k
Python
PDF text extraction with layout analysis.
Camelot 4k
Python
PDF table extraction — lattice and stream modes.
tabula-py 2k
Python
Python wrapper for Tabula. ⚠️
(⬆ back to top )
Tool
Stars
Language
Description
Tesseract 76k
C++
Google's OCR engine — 100+ languages.
PaddleOCR 89k
Python
Lightweight OCR — 100+ languages with LLM integration.
EasyOCR 30k
Python
Ready-to-use OCR — 80+ languages, PyTorch.
pytesseract 6k
Python
Python wrapper for Tesseract.
(⬆ back to top )
Tool
Type
Description
The Graph Open Source (3k)
Blockchain indexing via GraphQL subgraphs.
Subsquid Open Source
Blockchain indexer — 50k+ blocks/sec.
Dune Analytics SaaS
SQL-based blockchain analytics.
Etherscan APIs Freemium API
REST APIs for Ethereum data.
(⬆ back to top )
Pay-per-request services that handle proxies, browsers, and anti-bot for you.
Service
Best For
Key Feature
Apify Full-stack platform
Marketplace of pre-built Actors plus Crawlee-based development and managed execution.
ScrapingBee Simple API access
JavaScript rendering, screenshots, and search APIs.
ZenRows Anti-bot bypass
Managed scraping API with browser rendering and proxy rotation.
ScrapFly Multi-API
Scraping, screenshots, extraction, and crawler APIs.
Browserless Headless browsers
Headless Chrome in Docker, BrowserQL, self-hostable.
Browserbase AI browser agents
Cloud browsers for AI, session persistence, Stagehand integration.
Oxylabs Enterprise
ML-driven proxy rotation, e-commerce specialized.
Bright Data Scale
Web Unlocker with CAPTCHA solving, geo-routing, mobile UA.
SerpApi SERP data
Structured results from Google, Bing, Yahoo.
ScraperAPI Getting started
Proxy rotation, browser rendering, and structured scraping endpoints.
(⬆ back to top )
Service
Method
Pricing
Supports
2Captcha Human workers
Usage-based; see current pricing
reCAPTCHA, Turnstile, FunCaptcha, GeeTest, image.
Anti-Captcha Human workers
Usage-based; see current pricing
reCAPTCHA, hCaptcha, FunCaptcha, Turnstile.
CapSolver Automated solving
Usage-based; see current pricing
reCAPTCHA, AWS WAF, Cloudflare, GeeTest.
CapMonster Cloud Automated solving
Usage-based; see current pricing
reCAPTCHA, hCaptcha, and Turnstile.
(⬆ back to top )
Provider
Coverage
Pricing
Highlights
Bright Data Residential, ISP, mobile, and datacenter proxies
Pay-as-you-go
Broad proxy and web-data product portfolio with geographic targeting.
Oxylabs Residential, ISP, mobile, and datacenter proxies
Contact sales
Enterprise proxy services with country, city, and ZIP targeting options.
Decodo Residential, ISP, mobile, and datacenter proxies
Published plans
Geographic targeting and rotating or sticky sessions.
IPRoyal Residential, ISP, mobile, and datacenter proxies
Published plans
Proxy services with geographic targeting and pay-as-you-go options.
NetNut Residential, ISP, mobile, and datacenter proxies
Contact sales
ISP-sourced proxy connectivity and geographic targeting.
Webshare Residential and datacenter proxies
Published plans
Self-service proxy plans and a limited free tier.
(⬆ back to top )
(⬆ back to top )
🪦 Deprecated Tools Graveyard
Dead Tool
Why
Use Instead
PhantomJS
Archived 2018
Playwright, Puppeteer.
CasperJS
Depended on PhantomJS
Playwright, Puppeteer.
Nightmare
Unmaintained since 2020
Playwright.
Zombie.js
Unmaintained
Playwright, Puppeteer.
SlimerJS
Unmaintained, Gecko-based
Playwright (Firefox).
Splash
Scrapinghub, deprecated
Scrapy-Playwright.
twint
Archived Mar 2023, blocked by Twitter
Official API.
Goutte (PHP)
Deprecated by Symfony
Symfony BrowserKit + DomCrawler.
snscrape
Unmaintained since Nov 2023
Official APIs.
undetected-chromedriver
Aging, last push Jul 2025
Nodriver, Camoufox.
puppeteer-extra-stealth
Unmaintained since Jul 2024
Patchright, Camoufox.
playwright-stealth
Unmaintained since Nov 2023
Patchright, Camoufox.
curl-impersonate
Unmaintained since Jul 2024
curl_cffi.
GoogleScraper
Unmaintained since Jul 2021
SerpApi.
pyautogui
Unmaintained since Aug 2024
pytesseract + Playwright.
SikuliX
Stale, niche
Playwright, pytesseract.
(⬆ back to top )
Some links in the Managed Scraping APIs, CAPTCHA Solving, Proxy Providers, and Try Free sections are affiliate/referral links. These help support the maintenance of this list. All tools are included based on merit — affiliate status does not influence placement or rankings.
(⬆ back to top )
List
Description
awesome-ai AI APIs, tools, frameworks, platforms, and learning resources.
awesome-robotics Robotics frameworks, simulators, and platforms.
awesome-web3-ai Web3 x AI tools, agent frameworks, and protocols.
Contributions welcome! Please read the contribution guidelines first.
Add tools you've actually used or evaluated
Include star count and language where applicable
Note if a tool is unmaintained (last commit >1 year ago)
Commercial tools/services are fine but must be clearly labeled
To the extent possible under law, Edward Tay has waived all copyright and related or neighboring rights to this work.
