Webmail header search (from:/to:/subject:) uses $regex causing full collection scan on large mailboxes

[MartinCerny-awin]

When searching via the webmail UI with field-specific queries like from:user@example.com, the search takes 30+ seconds for users with 100K+ messages. The root cause is that prepare-search-filter.js and search-query.js both use $regex with $elemMatch on the headers array, which results in a full collection scan regardless of indexes.

The webmail sends GET /users/{userId}/search?from=user@example.com, which generates:

db.messages.aggregate([
    { $match: {
        user: ObjectId("..."),
        $and: [{
          headers: { $elemMatch: {
            key: "from",
            value: { $regex: "user@example\\.com", $options: "i" }
          }}
        }]
    }},
    { $group: { _id: 1, n: { $sum: 1 } } }
  ])

on 180k messages this takes 30s

I think this could be solved by using $text search instead.

[NickOvt]

Hello!
Considering that usually these searches are basic indeed I think that trying out a $text search instead of regex is a good idea. I'll work on it and test whether there would be any improvements using $text.

[martian-str33t]

PR #1005 addresses the timeout/crash aspect of this issue — it prevents $text searches from failing with MaxTimeMSExpired on large mailboxes by using the correct 2-minute timeout instead of the 3-second one.

I am working on another PR which will resolve the issue with slow header search, by adding indexed search.* fields

[martian-str33t]

This should be resolved by #1008 , which adds indexed search.* fields at message ingestion and routes all search queries to these fields instead of scanning the headers array with $elemMatch regex. On a 213K+ message mailbox, subject search went from 3.17s to 0.39s (8x), from-email search from 36.85s to 0.19s (196x), and to-email search from timeout (180s+) to 0.35s (>500x).