Skip to content

Commit 4bfa8e7

Browse files
mhumzaarainclaudesamuelvkwong
authored
LLM client rate limiting (#242)
* docs: design for shared LLM client with built-in rate limiting Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: extract_data uses user message and extra_body Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: implementation plan for shared LLM client with rate limiting Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(core): add LLM rate-limit settings * test: add LLM rate-limit error builders * feat(core): add rate-limit gate with sync and async helpers * fix(core): use clamped pause for gate budget decision run_through_gate(_async) armed the gate with the clamped pause but made the defer-vs-wait decision against the raw, unclamped Retry-After. When a server sent Retry-After above header_ceiling, this caused a spurious RateLimited even though the clamped window fit the caller's budget. Drop the divergent early-raise and let the loop's wait_until_open(_async) decide from the armed (clamped) _blocked_until, as it already did correctly. * feat(core): add throttled LLMClient and AsyncChatClient * refactor: use core LLMClient with built-in rate limiting across apps * fix(core): give RateLimited a default message; clarify test * docs: design for proactive LLM_MAX_RPM request cap Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: implementation plan for LLM_MAX_RPM request cap Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(core): add LLM_MAX_RPM setting * feat(core): add RpmLimiter and wire it into run_through_gate * feat(core): cap LLM requests per minute via shared RpmLimiter * Clean up: Added few comments and cleaned up specs files * Fix: few minor issue that were suggested by Gemini and Coderabbitai * fix(core): make RpmLimiter a sliding-window to match provider quota * refactor(core): remove proactive RPM limiter Different LLM providers use different RPM quotas, so a single proactive per-process cap is awkward to configure meaningfully. Reactive rate limiting (the shared 429 backoff gate plus transient retries) already handles provider limits regardless of their configuration, so drop the proactive RpmLimiter entirely. Removes the RpmLimiter class, the rpm parameter on run_through_gate/ run_through_gate_async (and the redundant post-acquire gate re-check), the _LLM_RPM_LIMITER global, the LLM_MAX_RPM setting/env var, and all related tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(core): route absurd Retry-After to exponential backoff Rename LLM_RATE_LIMIT_FALLBACK_MAX_SECONDS to LLM_RATE_LIMIT_BACKOFF_MAX_SECONDS (and the gate's fallback_max param/attr to backoff_max) to better describe what it caps. Change the header ceiling from a clamp into a sanity threshold: a Retry-After >= LLM_RATE_LIMIT_HEADER_CEILING_SECONDS is now treated as absurd and ignored, falling back to the shared exponential backoff ladder instead of blocking the gate for the full ceiling. * test: adapt merged-in LLM mocks to new client signature Tests from main (#241) mocked parse() without extra_body and asserted a system-role message, matching the old ChatClient. The new core LLMClient always passes extra_body and sends the prompt as a user message. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Samuel Kwong <samvictor232@gmail.com>
1 parent 63b1a5c commit 4bfa8e7

15 files changed

Lines changed: 1017 additions & 104 deletions

File tree

example.env

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -117,6 +117,11 @@ EXTERNAL_LLM_PROVIDER_API_KEY=your_external_llm_provider_api_key_here
117117
# Indicates if a local LLaMA.cpp container in development should use the GPU for inference.
118118
# During production SGLang always requires a GPU.
119119
LLAMACPP_USE_GPU=false
120+
# extract_data passes LLM_EXTRA_BODY (JSON) to the provider; the default disables provider
121+
# "thinking" for cleaner structured output. The LLM request timeout, rate-limit gate, and
122+
# transient-retry knobs have sensible defaults in settings (LLM_REQUEST_TIMEOUT_SECONDS,
123+
# LLM_RATE_LIMIT_*, LLM_TRANSIENT_RETRY_*); override them here only if needed.
124+
LLM_EXTRA_BODY='{"chat_template_kwargs": {"enable_thinking": false}}'
120125

121126
# The language of the example reports that will be seeded to the development database.
122127
# Possible values are 'en' or 'de'.

radis/chats/templates/chats/chat.html

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,9 @@
2525
{% endblock heading %}
2626
{% block content %}
2727
<div id="chat-with-form">
28+
{% if error %}
29+
<div class="alert alert-warning" role="alert">{{ error }}</div>
30+
{% endif %}
2831
{% if report %}
2932
<div class="card mb-3" x-data="{full: false}">
3033
<div class="card-body">

radis/chats/utils/chat_client.py

Lines changed: 0 additions & 68 deletions
This file was deleted.

radis/chats/utils/testing_helpers.py

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import asyncio
22
from unittest.mock import MagicMock
33

4+
import httpx
45
import openai
56
from faker import Faker
67
from pydantic import BaseModel
@@ -16,7 +17,7 @@ def create_question_body() -> str:
1617
return " ".join(question_body)
1718

1819

19-
def create_async_openai_client_mock(content: str) -> openai.AsyncOpenAI:
20+
def create_async_openai_client_mock(content: str | None) -> openai.AsyncOpenAI:
2021
openai_mock = MagicMock()
2122
mock_response = MagicMock(choices=[MagicMock(message=MagicMock(content=content))])
2223
future = asyncio.Future()
@@ -25,8 +26,21 @@ def create_async_openai_client_mock(content: str) -> openai.AsyncOpenAI:
2526
return openai_mock
2627

2728

28-
def create_openai_client_mock(content: BaseModel) -> openai.OpenAI:
29+
def create_openai_client_mock(content: BaseModel | None) -> openai.OpenAI:
2930
openai_mock = MagicMock()
3031
mock_response = MagicMock(choices=[MagicMock(message=MagicMock(parsed=content))])
3132
openai_mock.beta.chat.completions.parse.return_value = mock_response
3233
return openai_mock
34+
35+
36+
def make_rate_limit_error(headers: dict[str, str] | None = None) -> openai.RateLimitError:
37+
"""Build a real openai.RateLimitError carrying chosen response headers (e.g. retry-after)."""
38+
request = httpx.Request("POST", "http://testserver/v1/chat/completions")
39+
response = httpx.Response(429, headers=headers or {}, request=request)
40+
return openai.RateLimitError("rate limited", response=response, body=None)
41+
42+
43+
def make_connection_error() -> openai.APIConnectionError:
44+
"""Build a real openai.APIConnectionError (a transient, non-429 error)."""
45+
request = httpx.Request("POST", "http://testserver/v1/chat/completions")
46+
return openai.APIConnectionError(message="connection failed", request=request)

radis/chats/views.py

Lines changed: 54 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -16,10 +16,11 @@
1616

1717
from radis.chats.forms import CreateChatForm, PromptForm
1818
from radis.chats.tables import ChatTable
19+
from radis.core.utils.llm_client import AsyncChatClient
20+
from radis.core.utils.rate_limit import RateLimited
1921
from radis.reports.models import Report
2022

2123
from .models import Chat, ChatMessage, ChatRole
22-
from .utils.chat_client import AsyncChatClient
2324

2425

2526
@require_GET
@@ -63,26 +64,42 @@ async def chat_create_view(request: AuthenticatedHttpRequest) -> HttpResponse:
6364

6465
client = AsyncChatClient()
6566

66-
# Generate an answer for the user prompt
67-
answer = await client.chat(
68-
[
69-
{"role": "system", "content": instructions_system_prompt},
70-
{"role": "user", "content": user_prompt},
71-
],
72-
)
73-
74-
# Generate a title for the chat
67+
try:
68+
# Generate an answer for the user prompt
69+
answer = await client.chat(
70+
[
71+
{"role": "system", "content": instructions_system_prompt},
72+
{"role": "user", "content": user_prompt},
73+
],
74+
)
75+
except RateLimited:
76+
return render(
77+
request,
78+
"chats/_chat.html",
79+
{
80+
"chat": None,
81+
"report": report,
82+
"chat_messages": [],
83+
"form": form,
84+
"error": "The LLM service is busy. Please try again in a moment.",
85+
},
86+
)
87+
88+
# Generate a title for the chat. The title is secondary, so if it is rate-limited
89+
# we keep the answer and fall back to the user prompt instead of failing the request.
7590
title_system_prompt = Template(settings.CHAT_GENERATE_TITLE_SYSTEM_PROMPT).substitute(
7691
{"num_words": 6, "user_prompt": user_prompt, "assistant_response": answer}
7792
)
78-
79-
title = await client.chat(
80-
[
81-
{"role": "system", "content": title_system_prompt},
82-
{"role": "user", "content": user_prompt},
83-
],
84-
max_completion_tokens=20,
85-
)
93+
try:
94+
title = await client.chat(
95+
[
96+
{"role": "system", "content": title_system_prompt},
97+
{"role": "user", "content": user_prompt},
98+
],
99+
max_completion_tokens=20,
100+
)
101+
except RateLimited:
102+
title = user_prompt
86103
title = title.strip().rstrip(string.punctuation)[:100]
87104

88105
chat = await Chat.objects.acreate(owner=request.user, title=title, report=report)
@@ -172,7 +189,25 @@ async def chat_update_view(request: AuthenticatedHttpRequest, pk: int) -> HttpRe
172189
messages.append({"role": "user", "content": prompt})
173190

174191
client = AsyncChatClient()
175-
response = await client.chat(messages)
192+
try:
193+
response = await client.chat(messages)
194+
except RateLimited:
195+
return render(
196+
request,
197+
"chats/_chat.html",
198+
{
199+
"chat": chat,
200+
"report": chat.report,
201+
"chat_messages": [
202+
message
203+
async for message in chat.messages.exclude(role=ChatRole.SYSTEM).order_by(
204+
"id"
205+
)
206+
],
207+
"form": form,
208+
"error": "The LLM service is busy. Please try again in a moment.",
209+
},
210+
)
176211

177212
await ChatMessage.objects.acreate(chat=chat, role=ChatRole.USER, content=prompt)
178213
await ChatMessage.objects.acreate(chat=chat, role=ChatRole.ASSISTANT, content=response)
Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
from typing import cast
2+
from unittest.mock import MagicMock, patch
3+
4+
import pytest
5+
from pydantic import BaseModel
6+
7+
from radis.chats.utils.testing_helpers import (
8+
create_async_openai_client_mock,
9+
create_openai_client_mock,
10+
make_rate_limit_error,
11+
)
12+
from radis.core.utils.llm_client import (
13+
_LLM_GATE,
14+
AsyncChatClient,
15+
LLMClient,
16+
LLMResponseError,
17+
)
18+
from radis.core.utils.rate_limit import RateLimited
19+
20+
21+
class _Schema(BaseModel):
22+
value: str
23+
24+
25+
@pytest.fixture(autouse=True)
26+
def reset_gate():
27+
_LLM_GATE.reset()
28+
yield
29+
_LLM_GATE.reset()
30+
31+
32+
def test_llm_client_sets_max_retries_and_timeout(settings):
33+
settings.LLM_REQUEST_TIMEOUT_SECONDS = 42.0
34+
with patch("openai.OpenAI") as openai_cls:
35+
LLMClient()
36+
kwargs = openai_cls.call_args.kwargs
37+
assert kwargs["max_retries"] == 0
38+
assert kwargs["timeout"] == 42.0
39+
40+
41+
def test_extract_data_uses_user_message_and_extra_body(settings):
42+
settings.LLM_EXTRA_BODY = {"foo": "bar"}
43+
mock = cast(MagicMock, create_openai_client_mock(_Schema(value="hi")))
44+
with patch("openai.OpenAI", return_value=mock):
45+
result = LLMClient().extract_data("the prompt", _Schema)
46+
assert isinstance(result, _Schema)
47+
call = mock.beta.chat.completions.parse.call_args.kwargs
48+
assert call["messages"] == [{"role": "user", "content": "the prompt"}]
49+
assert call["extra_body"] == {"foo": "bar"}
50+
51+
52+
def test_extract_data_recovers_after_one_rate_limit():
53+
# No real wait: the injected 429 carries retry-after: 0. (`_LLM_GATE` reads its backoff
54+
# settings once at import time, so overriding them via `settings` here would be a no-op.)
55+
mock = cast(MagicMock, create_openai_client_mock(_Schema(value="ok")))
56+
calls = {"n": 0}
57+
success_response = mock.beta.chat.completions.parse.return_value
58+
59+
def flaky(**kwargs):
60+
calls["n"] += 1
61+
if calls["n"] == 1:
62+
raise make_rate_limit_error({"retry-after": "0"})
63+
return success_response
64+
65+
mock.beta.chat.completions.parse.side_effect = flaky
66+
with patch("openai.OpenAI", return_value=mock):
67+
result = LLMClient().extract_data("p", _Schema)
68+
assert isinstance(result, _Schema)
69+
assert result.value == "ok"
70+
assert calls["n"] == 2
71+
72+
73+
def test_extract_data_defers_when_rate_limit_exceeds_budget():
74+
mock = cast(MagicMock, create_openai_client_mock(_Schema(value="never")))
75+
76+
def always_429(**kwargs):
77+
raise make_rate_limit_error({"retry-after": "600"})
78+
79+
mock.beta.chat.completions.parse.side_effect = always_429
80+
with patch("openai.OpenAI", return_value=mock):
81+
with pytest.raises(RateLimited):
82+
LLMClient().extract_data("p", _Schema, max_wait=300.0)
83+
84+
85+
@pytest.mark.asyncio
86+
async def test_chat_returns_content():
87+
mock = create_async_openai_client_mock("the answer")
88+
with patch("openai.AsyncOpenAI", return_value=mock):
89+
answer = await AsyncChatClient().chat([{"role": "user", "content": "hi"}])
90+
assert answer == "the answer"
91+
92+
93+
@pytest.mark.asyncio
94+
async def test_chat_defers_when_rate_limit_exceeds_budget():
95+
mock = cast(MagicMock, create_async_openai_client_mock("never"))
96+
97+
async def always_429(**kwargs):
98+
raise make_rate_limit_error({"retry-after": "600"})
99+
100+
mock.chat.completions.create.side_effect = always_429
101+
with patch("openai.AsyncOpenAI", return_value=mock):
102+
with pytest.raises(RateLimited):
103+
await AsyncChatClient().chat([{"role": "user", "content": "hi"}], max_wait=20.0)
104+
105+
106+
@pytest.mark.asyncio
107+
async def test_chat_raises_when_content_is_none():
108+
# A refusal/empty completion yields content=None; surface it explicitly instead of
109+
# asserting (which would be stripped under python -O).
110+
mock = create_async_openai_client_mock(None)
111+
with patch("openai.AsyncOpenAI", return_value=mock):
112+
with pytest.raises(LLMResponseError):
113+
await AsyncChatClient().chat([{"role": "user", "content": "hi"}])
114+
115+
116+
def test_extract_data_raises_when_parsed_is_none():
117+
mock = create_openai_client_mock(None)
118+
with patch("openai.OpenAI", return_value=mock):
119+
with pytest.raises(LLMResponseError):
120+
LLMClient().extract_data("p", _Schema)
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
from django.conf import settings as dj_settings
2+
3+
4+
def test_llm_rate_limit_settings_have_expected_defaults():
5+
assert dj_settings.LLM_REQUEST_TIMEOUT_SECONDS == 60.0
6+
assert dj_settings.LLM_EXTRA_BODY == {"chat_template_kwargs": {"enable_thinking": False}}
7+
assert dj_settings.LLM_RATE_LIMIT_BACKOFF_BASE_SECONDS == 2.0
8+
assert dj_settings.LLM_RATE_LIMIT_BACKOFF_MAX_SECONDS == 120.0
9+
assert dj_settings.LLM_RATE_LIMIT_HEADER_CEILING_SECONDS == 1800.0
10+
assert dj_settings.LLM_RATE_LIMIT_MAX_WAIT_SECONDS == 300.0
11+
assert dj_settings.LLM_RATE_LIMIT_INTERACTIVE_MAX_WAIT_SECONDS == 20.0
12+
assert dj_settings.LLM_TRANSIENT_RETRY_ATTEMPTS == 2
13+
assert dj_settings.LLM_TRANSIENT_RETRY_BASE_SECONDS == 1.0

0 commit comments

Comments
 (0)