Since parallel running Sessions and Chats is very common and Costs have risen, managing Tokens and therefore Cache hits is very important.
It would be very useful to know for every response how long this is cached.
- When is the cache lost
- Show this on Session info in the sessions list
- show this in the Session
This would allow taking care which session to prioritize.
Since parallel running Sessions and Chats is very common and Costs have risen, managing Tokens and therefore Cache hits is very important.
It would be very useful to know for every response how long this is cached.
This would allow taking care which session to prioritize.