Some recent PRs have explored very flexible/dynamic tools that frequently generate runtime errors even for calls that typecheck, e.g. #693 #687
Currently, RetryLLMHandler has nice and intuitive behavior for statically detectable failures within call_assistant, but simply appends runtime tool execution errors to the history and continues, silently and indefinitely. However, in the presence of these new tools and/or a requirement to use tools for a final answer (#549), it's not uncommon in practice for this to lead to failed tool calls ad infinitum, something I found when working on #694 #695
Perhaps we should add some heuristics to RetryLLMHandler to recognize and deal with extended sequences of tool execution errors, capping their length, cleaning the conversation history and terminating the assistant loop according to some possibly user-defined criterion.
Some recent PRs have explored very flexible/dynamic tools that frequently generate runtime errors even for calls that typecheck, e.g. #693 #687
Currently,
RetryLLMHandlerhas nice and intuitive behavior for statically detectable failures withincall_assistant, but simply appends runtime tool execution errors to the history and continues, silently and indefinitely. However, in the presence of these new tools and/or a requirement to use tools for a final answer (#549), it's not uncommon in practice for this to lead to failed tool calls ad infinitum, something I found when working on #694 #695Perhaps we should add some heuristics to
RetryLLMHandlerto recognize and deal with extended sequences of tool execution errors, capping their length, cleaning the conversation history and terminating the assistant loop according to some possibly user-defined criterion.