Handling rate-limits

[fabiopires]

Hey there,

I'm having some issues with rate-limit with Anthropic, and while I know I can potentially reach out to them and negotiate a custom deal, I was wondering if it could be a way to handle these differently (without raising an exception).

I'm thinking in either adding a delay when these are triggered, or even allowing a state of the analysis to be saved and reused a few days later?

Thank you in advance.

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/vulnhuntr/LLMs.py", line 96, in send_message
    return self.client.messages.create(
  File "/usr/local/lib/python3.10/site-packages/anthropic/_utils/_utils.py", line 275, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/anthropic/resources/messages/messages.py", line 901, in create
    return self._post(
  File "/usr/local/lib/python3.10/site-packages/anthropic/_base_client.py", line 1279, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
  File "/usr/local/lib/python3.10/site-packages/anthropic/_base_client.py", line 956, in request
    return self._request(
  File "/usr/local/lib/python3.10/site-packages/anthropic/_base_client.py", line 1045, in _request
    return self._retry_request(
  File "/usr/local/lib/python3.10/site-packages/anthropic/_base_client.py", line 1094, in _retry_request
    return self._request(
  File "/usr/local/lib/python3.10/site-packages/anthropic/_base_client.py", line 1045, in _request
    return self._retry_request(
  File "/usr/local/lib/python3.10/site-packages/anthropic/_base_client.py", line 1094, in _retry_request
    return self._request(
  File "/usr/local/lib/python3.10/site-packages/anthropic/_base_client.py", line 1045, in _request
    return self._retry_request(
  File "/usr/local/lib/python3.10/site-packages/anthropic/_base_client.py", line 1094, in _retry_request
    return self._request(
  File "/usr/local/lib/python3.10/site-packages/anthropic/_base_client.py", line 1060, in _request
    raise self._make_status_error_from_response(err.response) from None
anthropic.RateLimitError: Error code: 429 - {'type': 'error', 'error': {'type': 'rate_limit_error', 'message': 'This request would exceed your organization’s rate limit of 40,000 input tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits; see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase.'}}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/vulnhuntr", line 8, in <module>
    sys.exit(run())
  File "/usr/local/lib/python3.10/site-packages/vulnhuntr/__main__.py", line 464, in run
    secondary_analysis_report: Response = llm.chat(vuln_specific_user_prompt, response_model=Response)
  File "/usr/local/lib/python3.10/site-packages/vulnhuntr/LLMs.py", line 68, in chat
    response = self.send_message(messages, max_tokens, response_model)
  File "/usr/local/lib/python3.10/site-packages/vulnhuntr/LLMs.py", line 105, in send_message
    raise RateLimitError("Request was rate-limited") from e
vulnhuntr.LLMs.RateLimitError: Request was rate-limited

[rationalpsyche]

Hi, I created a fork of the project where I handle rate limits by retrying for a chosen number of iterations, each time with an increasing delay. I also considered maintaining a state of the analysis but at this stage I would rather not introduce more complexity.

[robomotic]

Hi, I created a fork of the project where I handle rate limits by retrying for a chosen number of iterations, each time with an increasing delay. I also considered maintaining a state of the analysis but at this stage I would rather not introduce more complexity.

Love your fork, are you adding more languages ?

[robomotic]

I also added rate limiters and state checkpoints:
https://github.com/robomotic/vulnhuntr
Need to perform some serious testing!