Optimize Input Tokens

[azmy60]

Problem

• Long or too much messages can decrease the quality of AI response.

• All messages get sent per request, which means more messages = more cost.

A quote from LangGraph docs:

Long conversations pose a challenge to today's LLMs. The full history may not even fit inside an LLM's context window, resulting in an irrecoverable error. Even if your LLM technically supports the full context length, most LLMs still perform poorly over long contexts. They get "distracted" by stale or off-topic content, all while suffering from slower response times and higher costs.