perf!: import only cheerio/slim in cheerio/http crawlers and utils - #4114
Open
metalwarrior665 wants to merge 5 commits into
Open
perf!: import only cheerio/slim in cheerio/http crawlers and utils#4114metalwarrior665 wants to merge 5 commits into
metalwarrior665 wants to merge 5 commits into
Conversation
`cheerio`'s main entrypoint statically imports `undici` (for
`cheerio.fromURL()`), `parse5`, `encoding-sniffer` and `whatwg-encoding`.
`CheerioCrawler` parses the body with `htmlparser2`'s `parseDocument()` and
only hands the resulting document to `cheerio.load()`, so none of that code
is ever reachable - but it is compiled on every process start. `undici`
alone pulls in ~1.5 MB of CommonJS plus a chunk of Node's internal modules.
Switching to the `cheerio/slim` entrypoint (same `load()`, htmlparser2-only)
cuts `import { CheerioCrawler } from '@crawlee/cheerio'` from ~385 ms to
~306 ms locally, which closes almost the whole import-time gap against v3.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5TMh6KxDLgUrjMHtwK8ui
`htmlToText()`, `extractMicrodata()` and `parseOpenGraph()` all accept either a
raw HTML string or an existing `CheerioAPI`, but awaited `import('cheerio')`
before checking which one they got. Callers passing a `$` they already have -
the common path inside a `CheerioCrawler` handler - paid ~85 ms of module
compilation for a `load()` that was never called. Moving the import into the
string branch drops that to ~2 ms.
`parseHandlesFromHtml()` always parses a string, but does so with
`{ xml: { decodeEntities: true } }`. A truthy `xml` option sets
`_useHtmlParser2`, so cheerio already parses and serializes it with htmlparser2
and dom-serializer instead of parse5 - the slim entrypoint is byte-identical
here and skips the parse5 + undici imports.
The remaining `await import('cheerio')` sites (`parseWithCheerio` /
`waitForSelector` on HttpCrawler, JSDOMCrawler, Playwright and Puppeteer) call
`load()` on an HTML string with no options, so they genuinely use parse5.
Switching those to slim would change the parse tree - htmlparser2 does not
imply `<html>/<head>/<body>`, does no `<table>` foster parenting, and serializes
void SVG elements differently - so they are left alone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5TMh6KxDLgUrjMHtwK8ui
This reverts commit 243e86f. The deferral only paid off because `@crawlee/utils` loaded a different cheerio entrypoint than `CheerioCrawler` did. The next commit points the utils at `cheerio/slim` as well, so the module is already resident by the time these helpers run and the branch buys nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J5TMh6KxDLgUrjMHtwK8ui
`cheerio`'s main entrypoint statically imports `undici` (1 MB, for the
`fromURL()` helper crawlee never calls) and `parse5`. Measured cold, that entry
costs ~194 ms to load against a 23 ms bare-node baseline, of which ~113 ms is
undici and friends and ~15 ms is parse5. `cheerio/slim` is the htmlparser2-only
build and costs ~65 ms.
Since a scraper is either HTTP-based or browser-based, route the two families at
different entrypoints so that the HTTP ones never pay for the browser ones:
- `@crawlee/cheerio`, `@crawlee/http` and `@crawlee/utils` use `cheerio/slim`.
- `@crawlee/jsdom`, `@crawlee/linkedom`, `@crawlee/playwright` and
`@crawlee/puppeteer` keep the full entrypoint, and now say why in a comment.
`@crawlee/linkedom` imported it statically, so merely importing the package
loaded undici; it is dynamic now, matching the other three.
htmlparser2 is not a drop-in for parse5, so this is a deliberate behaviour
change on the HTTP side, in two places:
- `HttpCrawler`'s `parseWithCheerio()` / `waitForSelector()` parsed with parse5
and never passed `xmlMode`, so the XML feeds this crawler serves came out
mangled - `<link>` is a void element in HTML, so every `<link>` in an RSS feed
read back empty. They now parse with htmlparser2 in xml mode when the response
content type says XML, which is what `CheerioCrawler` has always done. Covered
by a new test against a small RSS fixture.
- `htmlToText()` and friends parse strings with htmlparser2 now, so they agree
with the `$` a `CheerioCrawler` handler passes them. Previously the same
document gave different text depending on which of the two you handed over.
One spec difference had to be handled by hand: an HTML parser drops a single
newline right after `<pre>`, htmlparser2 keeps it, so `htmlToText()` strips it
itself and both inputs now produce the spec-correct output.
Browser crawlers are untouched: `page.content()` is the browser's own
serialization of its DOM, and only a spec-compliant HTML5 parser reproduces the
tree the browser had, so selectors copied out of devtools keep working.
Measured: `import { CheerioCrawler }` 385 ms -> 306 ms; first
`HttpCrawler.parseWithCheerio()` 117 ms -> 37 ms. Verified with a loader hook
that neither a `CheerioCrawler` nor an `HttpCrawler` process loads
`cheerio/dist/esm/index.js`, `undici` or `parse5` any more, including when every
`@crawlee/utils` HTML helper is called.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J5TMh6KxDLgUrjMHtwK8ui
metalwarrior665
changed the title
metalwarrior665
changed the title
`feat(http): add DomCrawler with a pluggable DOM parser` (#4104) moved the code this branch had edited, so all three conflicts were "these lines live elsewhere now" rather than disagreements. Master's version of each conflicting file was taken as-is, and the branch's intent re-applied where the code went: - `cheerio-crawler.ts` -> `cheerio-parser.ts`. Master wrote the new file from the pre-branch source, so taking it plain would have silently put `undici` back in `CheerioCrawler`'s import graph. Re-applied `cheerio/slim` there, with the parser-choice comment. - `jsdom-crawler.ts`: the two annotated `import('cheerio')` sites are gone, replaced by one shared fallback in `DomCrawler`. The comment moved there. - `linkedom-crawler.ts`: dropped entirely. The static full-cheerio import this branch made dynamic no longer exists - `linkedom-parser.ts` does not import cheerio at all. `DomCrawler`'s fallback now passes `xmlMode` from the response content type, so jsdom and linkedom get the same XML fix `HttpCrawler` got on this branch. It stays on the full entrypoint on purpose: both are spec-compliant DOM implementations, so parse5 is the parser that agrees with them - and a top-level `xmlMode` keeps parse5 for HTML while switching to htmlparser2 only for XML, which parse5 cannot parse at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J5TMh6KxDLgUrjMHtwK8ui
metalwarrior665
marked this pull request as ready for review
September 9, 2026 16:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR does multiple things and might be better to split but I want to get your feedback first. I will also cleanup the AI comments later.
We currently use htmlparser2 for CheerioCrawler but parse5 for everything else. That doesn't feel logical. I went with a split
Import cheerio/slim where we only need
htmlparser2. This also accidentally gets rid of the undici import that we don't use anywhere.Small fix of using
xmlModeexplicitly in http crawler for xml, that part was broken because it used parse5 beforeEDIT: Got a bit surprised by the recent merge of DomCrawler but it shouldn't change anything substantial about this
AI summary
Switch from the full cheerio entrypoint to the slim variant to improve startup performance.
The slim entrypoint excludes cheerio's parse5 and undici dependencies, which are only needed for
cheerio.load(string)andcheerio.fromURL(). Since cheerio-crawler always passes a pre-parsed htmlparser2 document toload(), this code path is never reached. Importing the unused dependencies adds ~100ms of overhead on every module load.Changes:
cheerio/sliminstead ofcheerioin cheerio-crawlerhttps://claude.ai/code/session_01J5TMh6KxDLgUrjMHtwK8ui