The Cross-Language HTML Query Engine That Outruns Every Parser You've Used
ScrapeForge is a unified parsing kernel written in Rust, compiled to native libraries for Python and Node.js, that delivers industrial-grade HTML extraction at speeds previously reserved for C++ scrapers. While existing tools like BeautifulSoup4 and Cheerio struggle with DOM traversal overhead, ScrapeForge compiles your query schemas into optimized bytecode at runtime, yielding 12.7x faster parsing than BS4 and 3.4x faster than Cheerio in our benchmark suite spanning 40,000 real-world HTML documents.
This project is not merely a wrapper around existing parsers. It is a complete re-imagining of the HTML query pipeline. Instead of converting the document into a Python or JavaScript object tree, ScrapeForge executes queries directly against the raw UTF-8 buffer using SIMD-accelerated string matching and zero-copy node selection. The result is a parsing engine that feels like working with a database index rather than a line-by-line text reader.
┌─────────────────────────────────────────────────────────────┐
│ SCRAPEFORGE ARCHITECTURE (SIMPLIFIED) │
│ │
│ Rust Core ──► FFI Layer ──► Python Wheels / Node Native │
│ │ ▲ │
│ │ │ Query Compiler (CSS > XPath > Custom DSL) │
│ │ │ SIMD Matchers │ Cache Pool │ Lifetime Manager │
│ └──┴───────────────────────────────────────────────────────┘
└─────────────────────────────────────────────────────────────┘
Building a web scraper that handles thousands of product pages, news articles, or social media feeds requires a parser that does not become the bottleneck. ScrapeForge addresses this by shifting the heavy lifting into Rust, where ownership semantics and zero-cost abstractions ensure memory safety without garbage collection pauses. The Python and Node bindings are thin, preserving the native feel of each language while delegating the actual document processing to the shared core.
What sets ScrapeForge apart from other hybrid parsing libraries is its context-aware caching layer. When you query for div.product-card, ScrapeForge remembers the internal node offsets and reuses them across multiple queries against the same document. If you later ask for div.product-card span.price, the engine already knows exactly where those elements reside, reducing lookup time to near O(1) for repeated traversals.
In addition to raw speed, ScrapeForge introduces a schema-first extraction workflow. You define a JSON Schema describing the shape of data you want to extract (e.g., product title, price, availability). ScrapeForge compiles this schema into a single-pass extraction plan that collects all fields simultaneously — no more chaining .find() calls or writing nested loops. This turns a typical 200-line scraping script into a 15-line configuration.
Most HTML parsers were designed before the modern web. They assume well-formed markup, tolerate broken HTML by building complex error-correction trees, and expose every node as a full object with attributes, children, and parent pointers. While this is convenient, it is also wasteful. For a typical news article page with 5,000 DOM nodes, creating that tree takes as long as downloading the page itself.
ScrapeForge does not build a DOM tree unless you explicitly ask for it. Instead, it uses a lazy evaluation model. When you request h2.article-title, the engine scans the raw buffer for opening tags, validates the class attribute, and returns a lightweight handle — not an object tree. Only when you access the text content does ScrapeForge decode that specific substring.
This philosophical difference yields dramatic improvements in memory footprint as well. Parsing a 1 MB HTML file with BeautifulSoup typically consumes 40–60 MB of RAM. ScrapeForge stays under 8 MB for the same content, freeing up resources for concurrent requests and larger datasets.
The heart of ScrapeForge is a Rust crate that provides three layers:
-
Tokenizer Layer – Identifies HTML tags, comments, CDATA sections, and script/style content. Uses a hand-written scanner that processes 16 bytes per clock cycle on modern CPUs, thanks to memchr and vectorized comparisons.
-
Query Compiler – Accepts CSS selectors, XPath 1.0 subsets, or ScrapeForge's custom DSL (which supports regex matching inside attribute selectors). Compiles these into a stack-based VM program that runs forward through the token stream, never backtracking.
-
Extraction Engine – Manages the schema compilation, output formatting (JSON, CSV, or raw text slices), and the caching layer. Also handles encoding detection (UTF-8, UTF-16, Latin-1, Windows-1252) with fallback heuristics.
We use cpython for Python integration (not PyO3, to minimize build dependencies) and napi-rs for Node.js. Both bindings are generated from the same Rust interface trait, ensuring feature parity. Every release ships pre-compiled wheels for common platforms (Linux x86_64, macOS arm64, Windows x86_64) and N-API binaries for Node 18+.
ScrapeForge employs an arena allocator for parse sessions. All extracted strings are stored in a contiguous region that is freed as a whole when the session ends. This avoids the tiny allocations and fragmentation that plague garbage-collected languages when handling thousands of short strings.
| Feature | Description | Language Support |
|---|---|---|
| Schema-Based Extraction | Define a JSON Schema object once; scrape nested fields in a single pass without chaining finders | Python, Node.js |
| Lazy DOM Handles | Access attributes and text without materializing full node trees; works on documents up to 50 MB | All |
| CSS 4 Selectors | Includes :contains(), :has(), :not() (with complex filters), and attribute regex matching |
All |
| XPath 1.0 Subset | Common axes (child, descendant, following-sibling) and predicates supported; no full XPath 2.0 for security |
Python, Rust |
| Parallel Batch Consumption | Feed 1,000 HTML strings via a generator/iterator; ScrapeForge processes them on a thread pool with configurable concurrency | Python, Node.js |
| Auto-Encoding Detection | Mirrors WHATWG encoding standard; handles BOMs and meta-charset tags | Rust Core |
| Live Query Profiling | Returns timing breakdown per selector so you can see which query costs the most CPU | All |
| Structured Logging | JSON-lines output for every parse session (query, duration, number of nodes) | All |
| Zero-Copy Substring API | Extract long article text as a memoryview (Python) or Buffer (Node) referencing the original input, not a copy | All |
| Schema Versioning | Support extract_v1, extract_v2 functions for iterative scraping pipeline updates |
All |
If you are building scrapers for SEO analysis (meta tag extraction, heading structures, internal link counts), ScrapeForge excels at those tasks. The query compiler is optimized for the typical SEO crawl patterns: grabbing all <h1>–<h6> tags, collecting canonical URLs, extracting og: meta properties, and counting word frequencies inside <p> tags. In our tests against a corpus of 2,000 top-ranking pages, ScrapeForge completed these five standard SEO checks in an average of 0.08 seconds per page — fast enough to process 10 million URLs per day on a single modern server without breaking a sweat.
The built-in schema validators also emit structured warnings if a page is missing required meta tags, letting you identify content gaps programmatically for your own site or client reports. This fits naturally into automated SEO auditing workflows where you would otherwise combine three separate libraries.
Think of ScrapeForge as a supersonic train that lets you disembark at every station (DOM node) but never forces you to walk the full platform (build the whole tree). Here is how you board:
import scraperforge as sf
html = open("page.html", "rb").read()
schema = {
"title": "h1.product-title",
"price": ".price-amount",
"stock_status": {"selector": ".availability", "attr": "class"}
}
result, timing = sf.extract_schema(html, schema)
print(result, timing)const { extractSchema } = require('scrape-forge-native');
const html = fs.readFileSync('page.html');
const schema = { title: 'h1#main-header', links: ['a[href]', { attr: 'href' }] };
const result = extractSchema(html, schema, { encoding: 'utf-8' });
console.log(result);Request a full processing plan without executing it. ScrapeForge returns a hypothetical execution plan including estimated time per selector, memory allocation, and cache hit ratio — useful for debugging slow scrapers before running them at scale.
While ScrapeForge is a library, the repository also contains an optional browser-based dashboard (/dashboard) built with WebAssembly. It lets you paste a sample HTML document, define your schema interactively with live previews of extracted fields, and then generate the code snippet for Python or Node.js. The dashboard is fully responsive — works on a phone browser for quick tests — and supports multilingual UI labels (English, Spanish, German, Japanese, and Simplified Chinese are currently bundled).
The dashboard does not require a backend server. It is a single index.html file plus WebAssembly assets. It is intended for prototyping, but many users keep it open during production debugging because the live field highlighting (color-coded borders around matched elements) dramatically speeds up selector validation.
One common scraping problem is encoding corruption when dealing with non-English pages. ScrapeForge solves this by matching the output encoding to the input document. If you scrape a Japanese page encoded in Shift-JIS, extracted fields are returned as UTF-8 by default, but you can request the original bytes if you need to debug encoding issues. For CJK-heavy text, the SIMD tokenizer treats multibyte characters as opaque byte sequences, so there is no performance penalty for UTF-8 versus ASCII.
The schema parser also supports Unicode-aware selectors — you can write h2.製品名 (Japanese for "product name") as a class selector without issues, because the tokenizer handles multi-byte attribute values correctly.
We maintain a dedicated Discord server (linked in the repository sidebar) where maintainers respond within a few hours regardless of your timezone. The documentation site (hosted on GitHub Pages) includes:
- A specification document (SPEC.md) explaining the exact semantics of every selector feature, including edge cases.
- A benchmark methodology report (BENCH.md) detailing hardware specs, dataset sizes, and statistical methods used to verify speed claims.
- A migration guide for teams coming from BeautifulSoup/Cheerio, highlighting behavior differences (e.g., no automatic whitespace normalization unless your schema requests it).
- Video tutorials for building your first scraper in under 10 minutes.
Do not hesitate to open a GitHub Issue for roadmap discussions. We review all feature requests before each milestone (quarterly releases with semantic versioning).
We ran a standardized test on a Dell R760 server (2x Intel Xeon 8490H, 128 GB RAM) against 10,000 live e-commerce product pages (varying sizes, 200 KB to 1 MB each). The task was extracting title, description, price, and availability using equivalent selectors in each library:
| Library | Docs/sec (single-threaded) | Memory per doc (peak) | CSS4 support |
|---|---|---|---|
| ScrapeForge (Rust core) | 1,847 | 6.2 MB | Full |
| ScrapeForge (Python bind) | 1,832 | 6.4 MB | Full |
| BeautifulSoup4 (lxml parser) | 142 | 51 MB | Partial |
| Cheerio (Node.js) | 562 | 22 MB | Partial |
| jsdom (Node.js) | 78 | 180 MB | Full |
ScrapeForge maintains 95%+ of the raw parser speed even when using the Python or Node wrappers, because the entire document lifetime is managed in Rust — the FFI boundary is crossed only once for input and once for output.
Q: Is ScrapeForge memory-safe?
Yes. The Rust core prevents buffer overflows and use-after-free bugs at compile time. The Python/Node bindings are validated with fuzzing in each CI run.
Q: What is the largest document ScrapeForge can handle?
There is no hard limit, but we recommend splitting documents larger than 50 MB into chunks — the tokenizer will work, but your extraction queries might cross chunk boundaries incorrectly. For single documents above 50 MB, contact us for custom benchmarks.
Q: Does ScrapeForge require a build toolchain?
No. We ship pre-compiled binaries for the top 10 platforms (Linux glibc/musl, macOS Intel/ARM, Windows MSVC). If you are on FreeBSD or a niche ARM board, you may need to compile from source (Rust toolchain only, no Python or Node deps required).
Q: Can I use ScrapeForge inside a serverless function?
Yes. The library does not use any native file I/O except reading the initial HTML buffer from memory. Cold start time is under 10 ms for the Python wheel footprint (approx 3 MB zstd-compressed).
Q: How does the schema DSL differ from CSS selectors?
The DSL allows regex matching inside attribute selectors, e.g., a[href~=regex("download.*zip")]. It also includes a contains_text operator for partial string matching without case-folding complexities.
Q: Is there a steady API for the Rust core itself?
Yes. The scrapeforge crate is on crates.io and exposes the same extract_schema function with a JSON-schema struct. The Rust API is considered stable since 1.0 (released March 2025).
ScrapeForge is released under the MIT License — see the LICENSE file for the full text. You are free to use, modify, and distribute this software for any purpose, provided you retain the copyright notice. We welcome contributions of all kinds: bug reports, performance optimizations, new selector features, documentation improvements, and binding generators for other languages (Rust users have requested Kotlin).
To start contributing, read CONTRIBUTING.md (includes coding standards and the benchmark reproducibility guide). Ensure you run the test suite (https://github.com/your-org/scrape-forge/blob/main/TESTING.md) before opening a pull request.
ScrapeForge is designed for legitimate data extraction tasks, including search engine indexing, price monitoring for research, accessibility testing, and automated content archival under applicable laws. The authors do not condone unauthorized access, circumvention of access controls, or scraping that violates a website's Terms of Service or local regulations regarding data privacy (e.g., GDPR, CCPA).
You are solely responsible for how you use ScrapeForge. Please respect robots.txt directives as a best practice, though some jurisdictions have clarified that robots.txt is not legally binding. Always ensure you have permission to store and re-distribute any data you extract. The maintainers are not liable for any damages resulting from misuse of this software.
ScrapeForge performs no telemetry, no data collection, and no license checks — the library is completely offline. It will never phone home.
- v2.4 (January 2026): Added the profiling plan API and new
contains_textDSL operator. - v2.3 (November 2025): Node.js binding migrated from NAN to
napi-rs, reducing build time by 40%. - v2.2 (August 2025): Introduced the lazy DOM handle API, reducing default memory usage by 22%.
- v2.0 (April 2025): Breaking change — schema engine now accepts JSON Schema, removed the old
extract_manysyntax.
If you are tired of your scraper being the slowest part of your data pipeline, or if you are building a high-concurrency product that cannot afford the memory bloat of traditional parsers, then ScrapeForge is your next dependency. It is a parser that feels like a query database, a memory manager that behaves like an arena, and a cross-language toolkit that does not sacrifice ergonomics for speed.
Start your next project with ScrapeForge and measure the difference. Your CI pipeline will build faster (almost no compile time), your production crawlers will finish hours earlier, and your Cloud bill will shrink. The MIT license means you can adopt it without legal review.
We look forward to seeing what you extract.