Improve error messaging for short output token lengths

[suryabdev]

When the output token length is smaller than required for a particular use case, the user sees parsing errors. It is unclear to them that the issue might be with the shorter token length. They might see errors like

Error while parsing tool call from model output: The model output does not contain any JSON blob.

for ToolCallingAgents and

The code blob is invalid, because the regex pattern ```(?:py|python)?\n(.*?)\n``` was not found in code_blob

for CodeAgents

Issues: #1783, #1732, etc. There are older threads were the issue is mentioned. #201 (comment), Workaround was to increase the output tokens

That happens with some models. Sometimes it can be tweaked by increasing max_tokens or num_ctx

It is also mentioned that longer contexts are better in an example in the docs

smolagents/docs/source/en/guided_tour.md

Line 189 in f76dee1

num_ctx=8192, # ollama default is 2048 which will fail horribly. 8192 works for easy tasks, more is better. Check https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator to calculate how much VRAM this will need for the selected model.

This PR tries to

1. Add a warning message
2. Add a note in the documentation

Testing

After the change it should look something like this. (For the screenshot I simulated the situation by modifying the code flow on my local setup)

[suryabdev]

For the tool calling agent, I've added some details to the error message. This gets fed back to the ReAct loop but only the user can edit the model parameters. Maybe we should print it at the end of the trace?

For the CodeAgent, Shall I add a similar error in parse_code_blobs?

smolagents/src/smolagents/utils.py

Line 224 in f76dee1

Your code snippet is invalid, because the regex pattern {code_block_tags[0]}(.*?){code_block_tags[1]} was not found in it.

[suryabdev]

Maybe we should print it at the end of the trace?

Actually thinking again, we shouldn't pass this part to the ReAct loop. It could confuse the model
Maybe print a warning message that doesn't get added to the agent's memory

[suryabdev]

we shouldn't pass this part to the ReAct loop. It could confuse the model

Modified the code to print a warning. The warning wont get added to the error and hence shouldn't get added to the agent's memory.
It won't be visible to the LLM in the next steps

smolagents/src/smolagents/agents.py

Line 596 in f042c0c

action_step.error = e

[suryabdev]

Perhaps in the "How to debug your agent" section, We could add another point
https://huggingface.co/docs/smolagents/en/tutorials/building_good_agents#how-to-debug-your-agent
An expanded/more clear version of the following 👇

5. Increase token length

If you experience issues with parsing code (for CodeAgents) or JSON objects (for ToolCallingAgents) like the following, it could be due to insufficient output token length. Please adjust your model's parameters accordingly.

Error while parsing tool call from model output: The model output does not contain any JSON blob.

for ToolCallingAgents and

The code blob is invalid, because the regex pattern ```(?:py|python)?\n(.*?)\n``` was not found in code_blob

for CodeAgents

[suryabdev]

cc: @albertvillanova (For review)
I have some clarifications, Please take a look when you are free

[aymeric-roucher]

Perhaps in the "How to debug your agent" section, We could add another point https://huggingface.co/docs/smolagents/en/tutorials/building_good_agents#how-to-debug-your-agent An expanded/more clear version of the following 👇

5. Increase token length

If you experience issues with parsing code (for CodeAgents) or JSON objects (for ToolCallingAgents) like the following, it could be due to insufficient output token length. Please adjust your model's parameters accordingly.

Error while parsing tool call from model output: The model output does not contain any JSON blob.

for ToolCallingAgents and

The code blob is invalid, because the regex pattern ```(?:py|python)?\n(.*?)\n``` was not found in code_blob

for CodeAgents

I think this is indeed the best way to go! It's more future-proof

[suryabdev]

Added some simple tests to check the exact string. If the error message is changed in parse_code_blobs or parse_json_blob the note in the failing test will serve as a reminder to change the logic in agents.py

[suryabdev]

Warnings will look like the following

[suryabdev]

@aymeric-roucher thanks for the comments on the PR
I changed the message to a warning, added a note in the documentation and tests.
Please take another look when you are free