add custom tokenizer

[drahnreb]

Description

Regardless of selected model, tiktoken was always used to do the tokenization. This PR removes this dependency and adds the ability to replace tiktoken with a custom tokenizer.

Related Issues

None

Changes Made

• Add tokenizer parameter to replace default tiktoken encoding/decoding.
• Added Tokenizer and a default TiktokenTokenizer in utils.
• Kept tiktoken_model_name parameter for backwards compatibility.
• Updated Readme.
• Added example for google/gemma based GemmaTokenizer.

Checklist

• Changes tested locally
• Code reviewed
• Documentation updated (if necessary)

[drahnreb]

@danielaskdd Fixed linting and rebased. Please review. Kindly let me know if changes are needed.

[danielaskdd]

Thanks for sharing. There some issues in the PR which I already fix it after merge:

• tokenizer does not add to global_config for default setting: cd01ec6
• We use num_fragment instead of summary_max_tokens to determine where LLM summarization is need now: 4ae5246