Question about FT.SEARCH indices

[yatriks]

Hello,

I know this is a bit strange of a question, so please bear with me.

Right now, to experiment with the KQIR, I'm using an index on 3 fields, d TAG, s NUMERIC, e NUMERIC. All my queries are range queries on s and e for a particular d (There are thousands of keys with d, all sharing a prefix). This is really fast generally, but as I'm at around ~100,000 d, the search command is starting to get slower.

This database will likely need to be shared anyway is it grows from 100k -> 1M unique values of d, and I think a relatively simple ZSET structure would work well in that case anyway.

But for the sake of learning about the internal architecture, I'm curious -- if I create an index per d with a PREFIX, I don't think there are any hard limitations, but architecturally what are the issues with this? What, if any, is the issue with 100k indexes that are distinct on PREFIX? 1M? Where are the technical bottlenecks?

Thanks a lot for your time and help 🙏

[PragmaTwice]

Sorry for the late reply.

Could you provide some real data and queries? From the description alone, it's hard to fully understand what queries you're running and what your data looks like.

As things stand, for the three fields d TAG, s NUMERIC, e NUMERIC -- if you haven't declared NOINDEX, Kvrocks Search will create three separate indexes, one for each field. The index selection algorithm then tries to pick the most suitable one.

There are two shortcomings in the current implementation:

• First, the cost model for index selection is static -- it doesn't account for the actual distribution of your data, which can lead to suboptimal index choices. I've considered letting users provide hints for better selection, but haven't had time to implement that yet.
• Second, composite indexes are not yet supported. If your query is something like WHERE d == .. AND s == .., a composite index would make it extremely fast, instead of the current approach of filtering through one index and then scanning.

You can use FT.EXPLAINSQL to see what the current query plan looks like.

[yatriks]

Thank you for the response 🙏

Tha queries are always of the form select d where d==VAL and s<=t and e>t (sometimes query for multiple t chained with or).

Explain is

project *: (filter (and e >= T, s < T): tag-scan d, VAL)

s, e, and t are floats. Is that doing the tag scan first?

The fixed d in the query, as well as the fact that I can use PREFIX to select all keys for d is why I was curious about multiple indexes

Thanks again for your time

[PragmaTwice]

project *: (filter (and e >= T, s < T): tag-scan d, VAL)

means it will use the index of d.

it wil jump to the first key that d == VAL, and scan over it until the last key that d == VAL, and filter out via the condition (and e >= T, s < T).

[yatriks]

This makes sense, thank you 🙏 and thanks for walking me through the query plan.