|
1 | 1 | # @alauda/doom |
2 | 2 |
|
| 3 | +## 2.8.2 |
| 4 | + |
| 5 | +### Patch Changes |
| 6 | + |
| 7 | +- [#362](https://github.com/alauda/doom/pull/362) [`bff6f07`](https://github.com/alauda/doom/commit/bff6f07056d7fb005ae6e5f3376d964a8e279a54) Thanks [@youyongsong](https://github.com/youyongsong)! - Keep pathologically long tokens out of the browser-side search index. |
| 8 | + |
| 9 | + Opening the search box on the connectors documentation site crashed the tab with |
| 10 | + `Out of Memory`. The index file it downloads is 1.1 MB; building the index from |
| 11 | + it took **1970 MB of heap and 18.2 seconds** of blocked main thread. Measured on |
| 12 | + the live site's index, in Node, with rspress's own `LocalProvider` parameters — |
| 13 | + a browser costs more, because the tab also holds the DOM, React and the rest of |
| 14 | + the page. |
| 15 | + |
| 16 | + The cause was three pages whose YAML samples embed a base64-encoded SVG icon in |
| 17 | + a `tekton.dev/icon` annotation — 11,819, 4,243 and 2,995 characters. rspress |
| 18 | + configures FlexSearch with `tokenize: 'full'`, which indexes _every substring_ |
| 19 | + of every token, so a token's cost grows with the square of its length. Those |
| 20 | + three blobs are **2% of the indexed text and were 91% of the memory**: the |
| 21 | + twenty longest tokens alone accounted for 50.7% of the 6.1 million substrings |
| 22 | + the index expanded into. |
| 23 | + |
| 24 | + It is worth being precise that this is not about site size, because the obvious |
| 25 | + reading is wrong. acp-docs has **3.7× more indexed text** across 914 pages and |
| 26 | + builds its index in **230 MB and 4.3 seconds**. Its longest token is 184 |
| 27 | + characters. Length is the whole story. |
| 28 | + |
| 29 | + A new `doom-search-index` plugin now strips tokens longer than 64 characters in |
| 30 | + the `modifySearchIndexData` hook, just before the index file is written — an |
| 31 | + official rspress hook, so nothing internal is patched or aliased. The same index |
| 32 | + then builds in **72 MB and 1.1 seconds**. 64 leaves real prose alone: the |
| 33 | + longest English words run 20-30 characters and long code identifiers 40-50. |
| 34 | + |
| 35 | + Two things this deliberately does not do: |
| 36 | + - **CJK runs are never stripped.** A long unpunctuated Chinese sentence matches |
| 37 | + `\p{L}{65,}` exactly the way a base64 blob does, and removing it would |
| 38 | + silently make that text unsearchable. Those runs are already handled by the |
| 39 | + dedicated CJK index, which splits them per character. |
| 40 | + - **`toc[].charIndex` is shifted by the same amount the content lost.** The |
| 41 | + search UI uses those offsets to decide which heading a hit belongs to; |
| 42 | + shortening the content without moving them would attribute hits to the wrong |
| 43 | + section and link to the wrong anchor. |
| 44 | + |
| 45 | + When something is stripped the build prints a warning naming the page and the |
| 46 | + longest token, because a base64 payload or a credential sitting in a code sample |
| 47 | + is worth telling the author about. Silence is the normal case. |
| 48 | + |
3 | 49 | ## 2.8.1 |
4 | 50 |
|
5 | 51 | ### Patch Changes |
|
0 commit comments