All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
- TiktokenTokenizer for accurate OpenAI token counting using the tiktoken library
- Support for precise token counting with GPT-4, GPT-3.5-turbo, and other OpenAI models
- Comprehensive test suite for TiktokenTokenizer with 16 test cases
- Usage example demonstrating TiktokenTokenizer integration
- Documentation updates in README.md covering token counting options
- Updated mix.exs to include tiktoken dependency (~> 0.4.1)
- Enhanced README.md with token counting section and installation instructions
- Initial PromptVault implementation with context management
- PretendTokenizer for token estimation using fnord library
- Support for text, tool call, and media messages
- EEx and Liquid template engines
- Context compaction with SummarizeHistory strategy
- Comprehensive test coverage
- Full documentation and examples