Improve search (make fuzzy again and more robust)

[nikku]

Proposed Changes

We found various issues with our existing search, both in terms of robustness and behavior:

• Under certain circumstances searching loops infinitely (https://github.com/camunda/web-modeler/pull/10940)
• Search is no longer fuzzy, i.e. matches terms across all fields, and hence is overly strict (Improve search (make fuzzy again and more robust) #932 (comment))
• Search does not return all tokens for matched entries, making it hard to integrate it properly (Use search bpmn-js#2235)
• Lack of test coverage around contract and other crucial aspects

This PR solves these issues, while keeping the generally desirable properties:

• We sort search results semantically, preferring
  • start of word
  • start of term (in word)
  • we sort alphabetically (where items are equal in semantics)

The overall goal is to offer a snappy, robust, and intuitive search behavior, and I'd like to put up to a test if we accomplished that:

Try it out via

npx @bpmn-io/sr bpmn-io/bpmn-js#use-search -l bpmn-io/diagram-js#fix-search

Checklist

To ensure you provided everything we need to look at your PR:

• Brief textual description of the changes present
• Visual demo attached
• Steps to try out present, i.e. using the @bpmn-io/sr tool
• Related issue linked via Closes {LINK_TO_ISSUE} or Related to {LINK_TO_ISSUE}

[nikku]

Additional finding, debugging the implementation:

We do no longer tokenize between spaces, this makes it impossible to match two works, such as this:

[lmbateman]

Is there a playground where I can try this out? Alternatively, if you need my review, could you please do a quick screen capture to demonstrate the changes? Thanks!

[nikku]

Cf. bpmn-io/bpmn-js#2235 (comment). I'll give you an E2E demo once I'm done with implementing the search, unless you want a demo earlier, to discuss what we do (did), and why.

[nikku]

I updated this PR to be fuzzy again, and solve all other outstanding issues.

Just now tests revealed that we're not fully fuzzy, actually match too many things, cf. original comment in the popup menu that I subscribe to:

// every word of `search term` should be included in one of the searchable fields

Our current behavior, leading to more search results popping up is:

// at least one word of `search term` should be included in one of the searchable fields

Need to do another round tomorrow.

[misiekhardcore]

One general question that crossed my mind when I was playing with it:
Do we want to match partial searches? What I mean
There is an element named User task and user types in the search Ustk so few letters taken from the name in the order of occurance "User task".

[nikku]

Do we want to match partial searches?

This initiative aims to unify search across all places where we use it.

Given the new infrastructure in place (global search utility) integrators can choose what kind of search experience they want.

At the core we want to offer a search that is simple enough, downstream you may plug-in your fuzzy search provider, cf. camunda/camunda-bpmn-js#367.

[nikku]

Squashed the changes, @misiekhardcore ready for another look.