Combining Image-Retrieval with Algolia/Typesense?

[alexisdrakopoulos]

I am looking to build a nice multi-modal searching tool. Queries may include subsets of images, text and filters.

Tools like algolia/typesense make the text/filter search quite easy to setup. How would you combine it to work elegantly with CLIP Retrieval approaches?

Would you just search both separately and then combine the rankings somehow? Can you use one to augment the other?

Any suggestions are greatly appreciated.

[rom1504]

There are many different ways to implement subset search, you can read some in https://rom1504.medium.com/semantic-search-with-embeddings-index-anything-8fb18556443c
Or https://blog.vespa.ai/redefining-hybrid-search-possibilities-with-vespa/

2 simple methods are

• Compute one index per subset
• Search with a large K then post filter

Regarding text + image search, you can do an average of both text and image embedding, see the notebook about this in this repo

[alexisdrakopoulos]

Thanks!

I wonder if instead of large k -> filter there's a way to just keep searching for the next nearest neighbor until k are found with filters applied.

Kind of like

results = []
while len(results) < k:
candidate = search_next()
if not_filter(candidate):
results.append(candidate)

Style approach?

[rom1504]

You can but it's much faster to search with a big k first

[alexisdrakopoulos]

Yea to be clear it was pseudocode I'm just thinking if filters are super specific the right object may be only in 15 results out of millions.

So could do the above loop 50k or 100k at a time

[alexisdrakopoulos]

Yea to be clear it was pseudocode I'm just thinking if filters are super specific the right object may be only in 15 results out of millions.

So could do the above loop 50k or 100k at a time