feat: search backend behind feature flag

[jordanh]

Description

Implements search backend behind a feature flag. This PR is largely for discussion and testing (esp. on staging); but ultimately should be safe to merge.

Demo

Loom: https://www.loom.com/share/6ea18954e7314cc8a2fa8fd2c6b9d6f6

Testing scenarios

[Please list all the testing scenarios a reviewer has to check before approving the PR]

• Scenario A

  • Step 1
  • Step 2...

• Scenario B

  • Step 1
  • Step 2....

Final checklist

• I checked the code review guidelines
• I have added Metrics Representative as reviewer(s) if my PR invovles metrics/data/analytics related changes
• I have performed a self-review of my code, the same way I'd do it for any other team member
• I have tested all cases I listed in the testing scenarios and I haven't found any issues or regressions
• Whenever I took a non-obvious choice I added a comment explaining why I did it this way
• I added the label Skip Maintainer Review Indicating the PR only requires reviewer review and can be merged right after it's approved if the PR introduces only minor changes, does not contain any architectural changes or does not introduce any new patterns and I think one review is sufficient'
• PR title is human readable and could be used in changelog

[jordanh]

After a bunch of research into how to chunk text for search, I've moved this from a simple chunker to a "sliding window" algorithm that attempts to retain sentence boundaries, where able.

[mattkrick]

-1 a Recursive Character Level Chunking algo (this one) also retains sentence boundaries & where appropriate it merges multiple sentences, paragraphs, sections, etc. which means fewer chunks and less repeated data. Why not both? Wouldn't it make more sense to add an overlap parameter?

If you really don't like the recursive approach, we should build a separate chunker & let the data dictate which one to use, because from what i've seen, a recursive char chunker typically outperforms a naive sliding window, but a recursive chunk with a sliding window isn't a bad option!

[jordanh]

Ideally, we want to chunk based on what we know of the data. If it's markdown, we want to use a chunker that performs well on markdown. I'd prefer to merge this as is and make that a separate, targeted enhancement

[mattkrick]

Do we have some benchmark data we can test this out with? Getting rid of our recursive chunker seems like a step backwards, especially if the new one is going to create more chunks due to overlaps. The recursive chunker handles section headings and paragraph breaks! The new one does not.

That said... why squabble over chunking when we can just do away with most of it??

Qwen3 uses less memory, outputs the same 1024 dimension size, and has 32768 max tokens. That's 64x more tokens
https://huggingface.co/Qwen/Qwen3-Embedding-0.6B

[jordanh]

pg-native full-text search

[jordanh]

I had to create a new loader because the default loader doesn't return plaintext content. I think it might be better to have renamed the existing loader to pagesNoPlaintext or something to make it clearer what that loader does...

[jordanh]

After doing some reading, adding some structured text to be embedded seems to be a best practice...

[jordanh]

Since Pages are going to be embedded over and over again, we had to put a hook in there someplace to remove old chunks. This seemed to be the right place

[mattkrick]

-1 It seems very expensive to start over every time. I think we can do better!
Let's say a page has 1000 chunks.
We don't want to re-embed every single chunk on every keystroke.
After we re-tokenize, let's compare the embedText and only embed if it's changed.

[jordanh]

It's not clear that we should keep this mutation after releasing to GA, but it sure is handy for development.

[mattkrick]

private schema?

[jordanh]

This is a key change, rather than embed all of these items in parallel, I decided to prioritize latency over throughput now that users will be hitting the embedder to do semantic search. I believe this is the right thing to do.

Now, the embedder will prioritize executing search even over embedding new/updated objects, by using this simple redis priority lock

[jordanh]

I've introduced the concept of "high" and "low" priority embeddings. "high" priority are user searches. "low" are catching up on embedding new/updated objects

[mattkrick]

-1 doing it this way means we cannot run multiple replicas in the future since we're keeping a count of high priority embeddings on 1 replica instead of in a distributed state.

-1 The need to have a priority on top of the priority that is assigned to the item in the job queue tells me the system isn't working. What do we need to change about getEmbedderPriority(0) in order to make it fit your use case?

[jordanh]

getEmbedderPriority(0) was to pick the model, not the priority in embedding queue.

I'll have to think about this just a bit more. The embedder replicas were picking from a queue (the job table) ... but they were processing chunks in blocks. Those blocks sometimes take a long time to complete, blocking an embedder for a long while (timing out user requests). We might need to move to queuing chunks instead of objects of multiple chunks.

I'll need to think on this some more...

[mattkrick]

I think the problem is that your search function is cutting the line. It needs that query embedded NOW, which I totally get! ...but the webserver should never call a model directly. If it does, that cuts the line, which seems great at first, but what happens when 2 search queries come in, both high priority? We're back to where we were without any queue at all.

My .02:

• create a new workflow called something like embedAndReply which, after embedding, it replies via pubsub & that resolves a promise
• modify getEmbedderPriority to handle priorities that are higher than 0 (or use 0 for search and make everything that was 0 a 1)

So, a new search comes in, the query gets put at the front of the line & when the job completes, it resolves the promise.

[mattkrick]

hitting pg with 2 getJob queries (true, false) every 250ms is ambitious. This is already the most expensive query we have & this change would run it 40x more.
Perhaps instead we subscribe to a channel runNow in redis.
And instead of sleep, we have Promise.race(sleep, wakeUp)
And on redis message for that channel, we resolve wakeUp, if it exists.
And we publish to runNow whenever we add an item to the queue.
i'm sure there's something more eloquent, but the goal here is to not tank our DB performance for the whole app when the queue runs dry.

[mattkrick]

0 curious why this loop was needed vs. just calling this.next()?

[mattkrick]

-1 doing it this way means we cannot run multiple replicas in the future since we're keeping a count of high priority embeddings on 1 replica instead of in a distributed state.

-1 The need to have a priority on top of the priority that is assigned to the item in the job queue tells me the system isn't working. What do we need to change about getEmbedderPriority(0) in order to make it fit your use case?

[mattkrick]

-1 a Recursive Character Level Chunking algo (this one) also retains sentence boundaries & where appropriate it merges multiple sentences, paragraphs, sections, etc. which means fewer chunks and less repeated data. Why not both? Wouldn't it make more sense to add an overlap parameter?

If you really don't like the recursive approach, we should build a separate chunker & let the data dictate which one to use, because from what i've seen, a recursive char chunker typically outperforms a naive sliding window, but a recursive chunk with a sliding window isn't a bad option!

[mattkrick]

-1 don't take away our type safety!

[mattkrick]

can you move this to a helper function & possibly reuse for lexical & semantic?

[mattkrick]

-1 search needs some validation and/or a rateLimiter on it

[jordanh]

Let's do that as a fast follow on. I'll create an issue for it

[mattkrick]

-1 break this into a helper function. each objectType should have its own function

[mattkrick]

-1 waaaay too much casting as any used throughout. this is totally fine for exploratory coding & creating proof of concepts, but when code has to be shared & maintained, it's a bunch of landmines. I see that you went through the trouble of turning this function into a generic, but then that's not used?

[mattkrick]

-1 no as any

[mattkrick]

+1 websearch_to_tsquery may be more appropriate? it'll allow you to quote phrases when 1 word should follow another, etc.

[mattkrick]

Just some notes to myself:

• Putting userId on EmbeddingsMetadata doesn't help us because Page.userId is only the person who created the page, not the people who can access it.
• EmbeddingsMetadata for pages isn't necessary. We should be able to grab all that data from the Page table itself
• If we can first get all the Pages a user has access to, then we can perform a cosine similarity from there. We just need a pageId column on the Embeddings_n table.
• Part of me wonders if we should have 1 embeddings table per object type
• discussion topics could generate their text and language and store them on ReflectionGroup
• meeting templates could do the same thing
• pages already have their full text persisted at all times
• I think an EmbeddingsPages_modelId table makes the most sense. This is because when you perform a search, you're probably searching for a document. Sure, if you want to search across documents and discussion topics, we can perform 2 searches simultaneously, but then we can use different models, the indexed vectors will take up less RAM, and we don't need to have a polymorphic objectType='page'/pageId column. It also simplifies development because each objectType will be a different query & we just patch them together without the need for really complex queries that use RBAC for some objectTypes and teamId for others.

Gameplan:

• remove plaintextContent from Page
• Add pageId to the JobQueue
• put a unique constraint on pageId + model on the JobQueue. that way we attempt to enqueue updates all we want, but only the first will get added. that first will then fetch the latest value
• always write to embedText on the embedding table. This means we don't have a fullText, plaintext version of the page anywhere. I think that's OK because we don't use it anywhere except debugging
• This also means we only need to join PageAccess and Embeddings_qwen3! This is a huge win for performance
• we can also put a updatedAt on the Embeddings_ table. This could be a good sanity check where we don't want to embed something that was updatedAt recently, but that would require hitting the DB & we don't really know what to compare it to. Page.updatedAt is always going to be >= Embeddings_.updatedAt.

[jordanh]

@mattkrick closing this now!