v1.37.19

🚨 SSO on LiteLLM Proxy will be enforced behind a license from this release

• If you use SSO on the litellm admin UI + Proxy and want a license, meet with us here: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

What's Changed

• [Fix] only run check_request_disconnection logic for maximum 10 mins by @ishaan-jaff in #3741
• Adding decoding of base64 image data for gemini pro 1.5 by @hmcp22 in #3711
• [Feat] Enforce user has a valid license when using SSO on LiteLLM Proxy by @ishaan-jaff in #3742
• [FEAT] Async VertexAI Image Generation by @ishaan-jaff in #3739
• [Feat] Router/ Proxy - set cooldown_time based on Azure exception headers by @ishaan-jaff in #3716
• fix divide by 0 bug on slack alerting by @ishaan-jaff in #3745
• Standardize slack exception msg format by @ishaan-jaff in #3747
• Another dictionary changed size during iteration error by @phact in #3657
• feat(proxy_server.py): allow admin to return rejected response as string to user by @krrishdholakia in #3740
• [Fix] - raise 404 from /team/info when team does not exist by @ishaan-jaff in #3749
• webhook support for budget alerts by @krrishdholakia in #3748
• [Fix] - raise Exception when trying to update/delete a non-existent team by @ishaan-jaff in #3750
• [FEAT] - add litellm.Router - abatch_completion_one_model_multiple_requests by @ishaan-jaff in #3751

New Contributors

• @hmcp22 made their first contribution in #3711

Full Changelog: v1.37.17...v1.37.19

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.37.19

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

v1.37.17

What's Changed

• fix(utils.py): drop response_format if 'drop_params=True' for gpt-4 by @krrishdholakia in #3724
• fix(vertex_ai.py): support passing in result of tool call to vertex by @krrishdholakia in #3729
• feat(proxy_cli.py): support json logs on proxy by @krrishdholakia in #3737

Full Changelog: v1.37.16...v1.37.17

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.37.17

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Failed ❌	19	21.889823638252498	1.60686072057369	1.60686072057369	481	481	17.351264000012634	105.27215500002285
/health/liveliness	Failed ❌	18	22.18578070962989	15.68108985940312	15.68108985940312	4694	4694	16.798868000023504	1270.1498669999864
/health/readiness	Failed ❌	18	22.51649214501728	15.687771192960598	15.687771192960598	4696	4696	16.936380999993617	1206.0035970000058
Aggregated	Failed ❌	18	22.328690804782045	32.975721772937405	32.975721772937405	9871	9871	16.798868000023504	1270.1498669999864

v1.37.16

What's Changed

• fix - allow non master key to access llm_utils_routes by @ishaan-jaff in #3710
• fix(bedrock_httpx.py): move anthropic bedrock calls to httpx by @krrishdholakia in #3708
• [Feat] Admin UI - use base_model for Slack Alerts by @ishaan-jaff in #3713
• [Admin UI] show max input tokens on UI by @ishaan-jaff in #3714
• fix(proxy_server.py): fix setting model id for db models by @krrishdholakia in #3715

Full Changelog: v1.37.14...v1.37.16

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.37.16

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Failed ❌	9	10.2880556709407	1.5629325106711098	1.5629325106711098	468	468	7.436624999968444	83.99098699999286
/health/liveliness	Failed ❌	8	10.80103857402248	15.632664706092875	15.632664706092875	4681	4681	6.298579000031168	1272.475381999982
/health/readiness	Failed ❌	8	10.780497224867714	15.712815091255495	15.712815091255495	4705	4705	6.286180000017794	650.4576310000232
Aggregated	Failed ❌	8	10.766867369799249	32.90841230801948	32.90841230801948	9854	9854	6.286180000017794	1272.475381999982

v1.37.14

What's Changed

• Work with custom LANGSMITH_BASE_URL by @msabramo in #3703

Full Changelog: v1.37.13...v1.37.14

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.37.14

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Failed ❌	9	11.887553362704857	1.6297242673296988	1.6297242673296988	488	488	7.5520679999954154	178.089099999994
/health/liveliness	Failed ❌	8	10.902480935483862	15.529134924350613	15.529134924350613	4650	4650	6.3291929999991225	907.1240070000499
/health/readiness	Failed ❌	8	11.157714575899545	15.68609607304835	15.68609607304835	4697	4697	6.4579570000091735	1189.8105640000267
Aggregated	Failed ❌	8	11.073253457447953	32.84495526472866	32.84495526472866	9835	9835	6.3291929999991225	1189.8105640000267

v1.37.13-stable

Full Changelog: v1.37.13...v1.37.13-stable

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.37.13-stable

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

v1.37.13

What's Changed

• [Fix]- router/proxy show better client side errors when no_healthy deployments available by @ishaan-jaff in #3679
• [Fix] Flush langfuse logs on proxy shutdown by @ishaan-jaff in #3681
• Allow non-admins to use /engines/{model}/chat/completions by @msabramo in #3663
• Fix datetime.datetime.utcnow DeprecationWarning by @msabramo in #3686
• [Fix] - include model name in cool down alerts by @ishaan-jaff in #3690
• feat(lago.py): Enable Usage-based billing with lago by @krrishdholakia in #3685
• [UI] End User Spend - Fix Timezone diff bug by @ishaan-jaff in #3692
• [Feat] token_counter endpoint by @ishaan-jaff in #3682
• Timeout param: custom_llm_provider needs to be set before setting timeout by @edwinjosegeorge in #3645
• [Fix] AI Studio (Gemini API) returns invalid 1 index instead of 0 when "stream": false by @ishaan-jaff in #3693
• fix(proxy_server.py): check + get end-user obj even for master key calls by @krrishdholakia in #3575
• [Feat] Support Anthropic tools-2024-05-16 - Set Custom Anthropic Custom Headers by @ishaan-jaff in #3694
• [Feat] Admin UI - show model prices as Per 1M tokens by @ishaan-jaff in #3696
• Add commented set_verbose line to proxy_config by @msabramo in #3699
• [Fix] Polish Models Page - set max width per column, fix bug with selecting models by @ishaan-jaff in #3698
• [UI] Fix Round Team Spend, and Show Key Alias on Top API Keys by @ishaan-jaff in #3700
• [Fix] allow users to opt into specific alert types + Introduce spend_report alert type by @ishaan-jaff in #3702
• fix(replicate.py): move replicate calls to being async by @krrishdholakia in #3704
• [FEAT] add cost tracking for Fine Tuned OpenAI ft:davinci-002 and ft:babbage-002 by @ishaan-jaff in #3705
• Exclude custom headers from response if the value is None or empty string by @paneru-rajan in #3701
• Fix(router.py): Kill a bug that forced Azure OpenAI to have an API ke… by @Manouchehri in #3706

https://docs.litellm.ai/docs/providers/anthropic#forcing-anthropic-tool-use

New Contributors

• @edwinjosegeorge made their first contribution in #3645

Full Changelog: v1.37.12...v1.37.13

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.37.13

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

v1.37.12-stable

What's Changed

• feat(proxy_server.py): JWT-Auth improvements by @krrishdholakia in #3666
• feat(proxy_server.py): new /end_user/info endpoint by @krrishdholakia in #3652

Full Changelog: v1.37.11...v1.37.12-stable

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.37.12-stable

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

v1.37.12

What's Changed

• feat(proxy_server.py): JWT-Auth improvements by @krrishdholakia in #3666
• feat(proxy_server.py): new /end_user/info endpoint by @krrishdholakia in #3652

Full Changelog: v1.37.11...v1.37.12

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.37.12

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

v1.37.11

What's Changed

• feat(proxy_server.py): Enabling Admin to control general settings on proxy ui by @krrishdholakia in #3660
• [Fix] Mask API Keys from Predibase AuthenticationErrors by @ishaan-jaff in #3662
• [FIX] raise alerts for exceptions on /completions endpoint by @ishaan-jaff in #3661
• Updated Ollama cost models to include LLaMa3 and Mistral/Mixtral Instruct series by @kmheckel in #3543

New Contributors

• @kmheckel made their first contribution in #3543

Full Changelog: v1.37.10...v1.37.11

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.37.11

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.37.11

v1.37.10

What's Changed

• Fix pkg_resources warning by @msabramo in #3602
• Update pydantic code to fix warnings by @msabramo in #3600
• Add ability to customize slack report frequency by @msabramo in #3622
• Duplicate code by @rkataria1000 in #3594
• [Feature] Add cache to disk by @antonioloison in #3266
• Logfire Integration by @elisalimli in #3444
• Ignore 0 failures and 0s latency in daily slack reports by @taralika in #3599
• feat - reset spend per team, api_key [Only Master Key] by @ishaan-jaff in #3626
• docs - use discord alerting by @ishaan-jaff in #3634
• Revert "Logfire Integration" by @krrishdholakia in #3637
• [Feat] Proxy - cancel tasks when fast api request is cancelled by @ishaan-jaff in #3640
• [Feat] Proxy + router - don't cooldown on 4XX error that are not 429, 408, 401 by @ishaan-jaff in #3651
• cloned gpt-4o models into openrouter/openai in costs&context.json by @paul-gauthier in #3647
• [Fix] - Alerting on /completions - don't raise hanging request alert for /completions by @ishaan-jaff in #3653
• Fix Proxy Server - only show API base, Model server log exceptions, not on client side by @ishaan-jaff in #3655
• [Fix] Revert #3600 #3600 by @ishaan-jaff in #3664

New Contributors

• @rkataria1000 made their first contribution in #3594
• @antonioloison made their first contribution in #3266
• @taralika made their first contribution in #3599

Full Changelog: v1.37.9...v1.37.10

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.37.10

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat