Currently if the model provider returns an error due to overfilling the context window, the error is just displayed to the user.
So far this is most likely to come up with very large commits and the /commit command. In that case, we could workaround in various ways:
- Just truncate the diffs by chopping off the end and proceed as usual (asking the user first, unless yolo mode is enabled)
- Send multiple requests to the model, asking it to summarize the diffs piecewise and then submitting the summaries for creation of the final commit messages
We could implement solution 1 as the quick/easy route for now.
One question would be how to find the right place to truncate the input, without running a tokenizer locally. Probably it's good enough to just use the returned error's info about the number of tokens submitted versus the limit and chop proportionally, repeating if needed. Making a few calls to the provider to find the right amount of text to submit is not ideal but probably acceptable in terms of latency.
Currently if the model provider returns an error due to overfilling the context window, the error is just displayed to the user.
So far this is most likely to come up with very large commits and the
/commitcommand. In that case, we could workaround in various ways:We could implement solution 1 as the quick/easy route for now.
One question would be how to find the right place to truncate the input, without running a tokenizer locally. Probably it's good enough to just use the returned error's info about the number of tokens submitted versus the limit and chop proportionally, repeating if needed. Making a few calls to the provider to find the right amount of text to submit is not ideal but probably acceptable in terms of latency.