Skip to content

Commit 32c89a5

Browse files
committed
docs(h2o): record the rename hook's two scope limits
Review correctly pointed out that only the async path is covered. The @client decorator dispatches async_pre_call_deployment_hook from its async wrapper only; the sync wrapper goes straight to the original function. My earlier claim that both completion and acompletion are covered was too broad. It does not affect how this hook is deployed: it is registered only in the proxy config, and the proxy maps /chat/completions to acompletion, so all proxied traffic is async. Recorded anyway so nobody assumes an in-process sync caller is covered. Also records the second limit: the predicate reads merged kwargs and so cannot tell a deployment-configured max_completion_tokens from a caller-supplied one, meaning a request sending BOTH fields to a max_tokens-native deployment is rewritten. Logic left as-is deliberately; the alternative narrows a real Azure case, and the reported traffic sends max_tokens alone.
1 parent cdf049d commit 32c89a5

1 file changed

Lines changed: 20 additions & 0 deletions

File tree

litellm/integrations/h2o/litellm_max_tokens_rename_hook.py

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,26 @@
5454
non-2025 Azure) are left untouched, including when they share a model group
5555
with an Azure deployment.
5656
57+
Two scope limits worth knowing:
58+
59+
* ASYNC PATH ONLY. `async_pre_call_deployment_hook` is dispatched by the
60+
@client decorator's ASYNC wrapper (litellm/utils.py, in wrapper_async).
61+
The sync wrapper does not dispatch it, so a direct sync
62+
`litellm.completion()` or `Router._completion()` bypasses this hook. That
63+
is fine for how the hook is deployed: it is registered only in the proxy
64+
config, and the proxy maps /chat/completions to `acompletion`
65+
(proxy/route_llm_request.py), so all proxied traffic takes the async path.
66+
Code embedding litellm in-process and calling sync `completion()` would
67+
not get the rename.
68+
69+
* The predicate reads the MERGED kwargs, which cannot distinguish a
70+
deployment-configured `max_completion_tokens` from a caller-supplied one.
71+
A request sending BOTH `max_tokens` and `max_completion_tokens` to a
72+
max_tokens-native deployment is therefore rewritten. The reported case,
73+
and everything h2oGPTe core emits, sends `max_tokens` alone. Keying only
74+
on `additional_drop_params` would remove this edge, at the cost of not
75+
firing for an Azure deployment configured with a ceiling but no drop.
76+
5777
INTERACTION WITH THE CAP HOOK
5878
-----------------------------
5979
`litellm_max_tokens_cap_hook` clips both fields down to the deployment

0 commit comments

Comments
 (0)