Indexing language function to get size of an array

[dainiusjocas]

Is your feature request related to a problem? Please describe.
Having a field with the length of text in terms of tokens helps with calculating the "cost" of sending the text to an LLM. Vespa already has an embedder which is type="hugging-face-tokenizer" that spits out token ids. Currently, a custom docproc is needed to get the token count.

Describe the solution you'd like
Indexing language should have an operator that given an array returns the lenght of the array.

Describe alternatives you've considered
A custom document processor to get the job done.

Additional context
https://docs.vespa.ai/en/reference/writing/indexing-language.html#get_language

[hmusum]

Do you want this as a metric? In that case https://docs.vespa.ai/en/reference/operations/metrics/container.html#embedder_sequence_length might give you what you. Otherwise, can you give some more details about what you want to achieve?

[dainiusjocas]

Metric doesn't work for our use case, as it is average over some period of time. Per document is needed.

Overall, we need to return a number of tokens with the search response. Therefore, some tokenizer is needed. To get that away from the query path we want to put the number of tokens into a Vespa document, so that it is returned with a summary class and then processed in the application layer.

Vespa has a huggingface tokenizer support. It can be run from the indexing language. But storing raw token ids would use way more storage than storing just a single number. Currently there is no good way to get the size of array in the indexing language. This issue asks for an indexing language command to get the array length of currently processed value when it is array.

[hmusum]

Thanks for clarifying the use case, we'll discuss this. A custom docproc is as you say a workaround.

[bratseth]

Do you only need this for the returned result (i.e. after filtering, ranking, grouping, summary fetching), but you want to do it on the indexing side to avoid re-tokenizing the query response?

[dainiusjocas]

Yes, only need this for the returned result. Pre-calculating during indexing seems to be logical approach.

[bratseth]

I think you can just take the length of the returned text and divide by the average token length (about 4)?