[Feature request] Specialized language segmentation for mixed-language pages

[atomarna]

I would like to make use of the newly introduced specialized language segmentation on a mixed-language site. Specifically, I have a text in English interspaced with quotes in Japanese. Since the site has an English lang attribute, the Japanese quotes are indexed as single entries (as shown in the playground) and searching for words within the quotes does not return any results, only searching for the entire quote or its begining does.

I am not sure how common the use case is, but it would be extremely useful if I could turn on the segmentation for a site with an English lang attribute, provided it is applied only to the non-Latin script present on the pages.

[atomarna]

Looking at the codebase, implementing a switch allowing one to choose the behavior preceding commit #1002 would likely solve my issue. Sacrificing the compound word handling would be worth it to improve the mixed-language index.

Indeed, playing with the playground a bit, I find the results satisfactory when using --force-language zh (for some reason, which I do not understand, zh works much better than jp in this case, even though the quotes are in Japanese; using Firefox, specifically).

The only thing preventing me from using the second approach is that the site itself is multilingual (i.e. not only an English-Japanese mix) and using the --force-language option collapses the separate indexes into a single one. I would need it to preserve the separate language indexes and only change the way these are created (e.g. according to the zh rules instead of the en rules).

[atomarna]

I inadvertently closed the issue, I am sorry for the mistake.