Use skip index for labels

[R-omk]

In this case, the index type 'tokenbf_v1' is suitable.
Before extracting the field from a json object, it need to check the presence of the key or the value using the hasToken function.
It also need to pay attention that word breaks are not allowed in tokens for indexing.

[adubovikov]

Hi @R-omk ,

sorry for late reply, yes this is a good idea, the problem is that size of this index will be huge and insert will be a bit slower..
let us play around it and make it optional.

[R-omk]

let us play around it and make it optional.

Yes, it's a good decision.

I also noticed that you can use an min/max index for the timestamp field, you can create a separate issue or include it as part of this one.

[R-omk]

the problem is that size of this index will be huge and insert will be a bit slower

By the way, this is quite consistent with the general logging recommendation that the number of unique combination should be small. Therefore, this is not a big deal, since a new entry of labels will actually be added only once per partition (one per day for example).

[adubovikov]

yes, but lets think big and expect a lot of uniq tags from different sources :-) I.e. voice / IoT that includes IP/hostname.

[adubovikov]

but you are right - the update/insert should(MUST) not be offten.