[Core] Roadmap for handling context overflow

[yiranwu0]

Any help is appreciated!

Thit task demands a considerable amount of effort. If you have insights, suggestions, or can contribute in any way, your help would be immensely valued.

Problem Description

(Continue from #9) Current LLM have limited context size / token limit (gpt3.5turbo: 4096, gpt4 8192, etc). Although the current max_token limit from OpenAI is sufficient for many tasks, the token limit will be always exceeded with the conversation running. autogen.Completion will raise this InvalidRequestError that indicates the context size is exceeded since autogen doesn’t have a way to handle long context sizes.

Potential Methods

1. Compression: we can utilize LLMs to compress previous messages to reduce context size.
2. Retrieve related history messages: we can retrieve the most related messages based on the latest message.
3. Truncation: a simple way is keep the recent k messages and truncate all previous messages. We can also implement some truncation mechanisms, such as remove failed code executions.
4. A mixture of methods above.

Some References

• https://community.openai.com/t/when-conversation-grows-larger-did-chatgpt-include-all-of-the-conversation-context-for-every-chat/121876/7
• https://ai.stackexchange.com/questions/38150/how-does-chatgpt-retain-the-context-of-previous-questions
• From @juanmacuevas: MemWalker processed long context intro a tree of summaries and link to the paper
• LongLLMLingua: local model for compression.
• From @MrXandbadas: MemGPT supporting Autogen Agent discussion

### Compression & Truncation
- [ ] https://github.com/microsoft/autogen/pull/131
- [ ] https://github.com/microsoft/autogen/pull/421
- [ ] https://github.com/microsoft/autogen/pull/443
- [ ] Allow async compression
- [ ] https://github.com/microsoft/autogen/pull/497
- [ ] https://github.com/microsoft/autogen/issues/685

### Retrieval
- [ ] Explore memGPT agent

[OrderAndCh4oS]

Probably a slight tangent, but I'm finding error stack traces to a major culprit of context overflow

[kazunator]

Would using something like Llama Index help?

[kazunator]

could this work? https://memgpt.ai/

[juanmacuevas]

MemWalker processed long context intro a tree of summaries
Could this approach be applied for autogen? link to the paper

[sonichi]

@Hacker0912 Are you interested in this topic?