Open
Description
Please describe the module you would like to add to bricks
Similar to #347 but returning the amount of tokens instead of classes. See here: https://platform.openai.com/tokenizer
Do you already have an implementation?
Tiktoken has a basic implementation in their docs, e.g. for gpt-3.5.-turbo. We just need to calculate the number of tokens of a text input.
Additional context
Also use this to calculate the complexity of a paragraph for RAG; for instance, > 1000 tokens means a high complexity.