perf: use cheerio/slim to reduce module load time - #4114
Draft
metalwarrior665 wants to merge 2 commits into
Draft
Conversation
`cheerio`'s main entrypoint statically imports `undici` (for
`cheerio.fromURL()`), `parse5`, `encoding-sniffer` and `whatwg-encoding`.
`CheerioCrawler` parses the body with `htmlparser2`'s `parseDocument()` and
only hands the resulting document to `cheerio.load()`, so none of that code
is ever reachable - but it is compiled on every process start. `undici`
alone pulls in ~1.5 MB of CommonJS plus a chunk of Node's internal modules.
Switching to the `cheerio/slim` entrypoint (same `load()`, htmlparser2-only)
cuts `import { CheerioCrawler } from '@crawlee/cheerio'` from ~385 ms to
~306 ms locally, which closes almost the whole import-time gap against v3.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5TMh6KxDLgUrjMHtwK8ui
`htmlToText()`, `extractMicrodata()` and `parseOpenGraph()` all accept either a
raw HTML string or an existing `CheerioAPI`, but awaited `import('cheerio')`
before checking which one they got. Callers passing a `$` they already have -
the common path inside a `CheerioCrawler` handler - paid ~85 ms of module
compilation for a `load()` that was never called. Moving the import into the
string branch drops that to ~2 ms.
`parseHandlesFromHtml()` always parses a string, but does so with
`{ xml: { decodeEntities: true } }`. A truthy `xml` option sets
`_useHtmlParser2`, so cheerio already parses and serializes it with htmlparser2
and dom-serializer instead of parse5 - the slim entrypoint is byte-identical
here and skips the parse5 + undici imports.
The remaining `await import('cheerio')` sites (`parseWithCheerio` /
`waitForSelector` on HttpCrawler, JSDOMCrawler, Playwright and Puppeteer) call
`load()` on an HTML string with no options, so they genuinely use parse5.
Switching those to slim would change the parse tree - htmlparser2 does not
imply `<html>/<head>/<body>`, does no `<table>` foster parenting, and serializes
void SVG elements differently - so they are left alone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5TMh6KxDLgUrjMHtwK8ui
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This was pure regression. v3 Actor loads modules in
422 msvs605 msof v4. I will do more testing once this is merged to identify other potential issues but that this was the biggest.AI summary
Switch from the full cheerio entrypoint to the slim variant to improve startup performance.
The slim entrypoint excludes cheerio's parse5 and undici dependencies, which are only needed for
cheerio.load(string)andcheerio.fromURL(). Since cheerio-crawler always passes a pre-parsed htmlparser2 document toload(), this code path is never reached. Importing the unused dependencies adds ~100ms of overhead on every module load.Changes:
cheerio/sliminstead ofcheerioin cheerio-crawlerhttps://claude.ai/code/session_01J5TMh6KxDLgUrjMHtwK8ui