Skip to content

Commit 8a42d57

Browse files
chore: release @alauda/doom (#363)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
1 parent bff6f07 commit 8a42d57

3 files changed

Lines changed: 47 additions & 47 deletions

File tree

‎.changeset/search-index-oversized-tokens.md‎

Lines changed: 0 additions & 46 deletions
This file was deleted.

‎packages/doom/CHANGELOG.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,51 @@
11
# @alauda/doom
22

3+
## 2.8.2
4+
5+
### Patch Changes
6+
7+
- [#362](https://github.com/alauda/doom/pull/362) [`bff6f07`](https://github.com/alauda/doom/commit/bff6f07056d7fb005ae6e5f3376d964a8e279a54) Thanks [@youyongsong](https://github.com/youyongsong)! - Keep pathologically long tokens out of the browser-side search index.
8+
9+
Opening the search box on the connectors documentation site crashed the tab with
10+
`Out of Memory`. The index file it downloads is 1.1 MB; building the index from
11+
it took **1970 MB of heap and 18.2 seconds** of blocked main thread. Measured on
12+
the live site's index, in Node, with rspress's own `LocalProvider` parameters —
13+
a browser costs more, because the tab also holds the DOM, React and the rest of
14+
the page.
15+
16+
The cause was three pages whose YAML samples embed a base64-encoded SVG icon in
17+
a `tekton.dev/icon` annotation — 11,819, 4,243 and 2,995 characters. rspress
18+
configures FlexSearch with `tokenize: 'full'`, which indexes _every substring_
19+
of every token, so a token's cost grows with the square of its length. Those
20+
three blobs are **2% of the indexed text and were 91% of the memory**: the
21+
twenty longest tokens alone accounted for 50.7% of the 6.1 million substrings
22+
the index expanded into.
23+
24+
It is worth being precise that this is not about site size, because the obvious
25+
reading is wrong. acp-docs has **3.7× more indexed text** across 914 pages and
26+
builds its index in **230 MB and 4.3 seconds**. Its longest token is 184
27+
characters. Length is the whole story.
28+
29+
A new `doom-search-index` plugin now strips tokens longer than 64 characters in
30+
the `modifySearchIndexData` hook, just before the index file is written — an
31+
official rspress hook, so nothing internal is patched or aliased. The same index
32+
then builds in **72 MB and 1.1 seconds**. 64 leaves real prose alone: the
33+
longest English words run 20-30 characters and long code identifiers 40-50.
34+
35+
Two things this deliberately does not do:
36+
- **CJK runs are never stripped.** A long unpunctuated Chinese sentence matches
37+
`\p{L}{65,}` exactly the way a base64 blob does, and removing it would
38+
silently make that text unsearchable. Those runs are already handled by the
39+
dedicated CJK index, which splits them per character.
40+
- **`toc[].charIndex` is shifted by the same amount the content lost.** The
41+
search UI uses those offsets to decide which heading a hit belongs to;
42+
shortening the content without moving them would attribute hits to the wrong
43+
section and link to the wrong anchor.
44+
45+
When something is stripped the build prints a warning naming the page and the
46+
longest token, because a base64 payload or a credential sitting in a code sample
47+
is worth telling the author about. Silence is the normal case.
48+
349
## 2.8.1
450

551
### Patch Changes

‎packages/doom/package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "@alauda/doom",
3-
"version": "2.8.1",
3+
"version": "2.8.2",
44
"type": "module",
55
"description": "Doctor Doom making docs.",
66
"repository": {

0 commit comments

Comments
 (0)