OOM due to FlexSearch tokenization `full` and synchrous export

[lyleaigbedion]

Hi,

I only have 20 documents and Flexsearch could not process them all when tokenization was set to full.

Took a deeper look and the use of index.export() should be async

nextapps-de/flexsearch#270

The combination of Patching the tokenization to forward and patching index.export fixed my problem

Would it be possible for the tokenization setting to be controled via the config?

[dimitry-cb]

@lyleaigbedion can you please share you patch? Dealing with the same issue

[scalvert]

Appreciate the report! Taking a look at #12 now. Thanks for contributing!

[scalvert]

Fixed and released in https://github.com/scalvert/docusaurus-plugin-mcp-server/releases/tag/v0.11.0. Thanks again!