Releases: BerriAI/litellm
Releases · BerriAI/litellm
v1.65.0.rc
What's Changed
- Nova Canvas complete image generation tasks (#9177) by @krrishdholakia in #9525
- [Feature]: Support for Fine-Tuned Vertex AI LLMs by @ishaan-jaff in #9542
- feat(prisma-migrations): add baseline db migration file by @krrishdholakia in #9565
- Add Daily User Spend Aggregate view - allows UI Usage tab to work > 1m rows by @krrishdholakia in #9538
- Support Gemini audio token cost tracking + fix openai audio input token cost tracking by @krrishdholakia in #9535
- [Reliability Fixes] - Gracefully handle exceptions when DB is having an outage by @ishaan-jaff in #9533
- [Reliability Fix] - Allow Pods to startup + passing /health/readiness when
allow_requests_on_db_unavailable: Trueand DB is down by @ishaan-jaff in #9569 - Add OpenAI gpt-4o-transcribe support by @krrishdholakia in #9517
- Allow viewing keyinfo on request logs by @krrishdholakia in #9568
- Allow team admins to add/update/delete models on UI + show api base and model id on request logs by @krrishdholakia in #9572
- Litellm fix db testing by @krrishdholakia in #9593
- Litellm new UI build by @krrishdholakia in #9601
- Support max_completion_tokens on Mistral by @Cmancuso in #9589
- Revert "Support max_completion_tokens on Mistral" by @krrishdholakia in #9604
- fix(mistral_chat_transformation.py): add missing comma by @krrishdholakia in #9606
- Support discovering gemini, anthropic, xai models by calling their
/v1/modelendpoint by @krrishdholakia in #9530 - Connect UI to "LiteLLM_DailyUserSpend" spend table - enables usage tab to work at 1m+ spend logs by @krrishdholakia in #9603
- Update README.md by @krrishdholakia in #9616
- fix(proxy_server.py): get master key from environment, if not set in … by @krrishdholakia in #9617
New Contributors
Full Changelog: v1.64.1-nightly...v1.65.0.rc
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.65.0.rc
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 540.0 | 624.3702232376279 | 5.474375767460597 | 0.0 | 1637 | 0 | 487.16235000006236 | 2776.3750889999983 |
| Aggregated | Failed ❌ | 540.0 | 624.3702232376279 | 5.474375767460597 | 0.0 | 1637 | 0 | 487.16235000006236 | 2776.3750889999983 |
Contributors
Cmancuso, krrishdholakia, and ishaan-jaff
Assets 4
v1.65.0-nightly
What's Changed
- Nova Canvas complete image generation tasks (#9177) by @krrishdholakia in #9525
- [Feature]: Support for Fine-Tuned Vertex AI LLMs by @ishaan-jaff in #9542
- feat(prisma-migrations): add baseline db migration file by @krrishdholakia in #9565
- Add Daily User Spend Aggregate view - allows UI Usage tab to work > 1m rows by @krrishdholakia in #9538
- Support Gemini audio token cost tracking + fix openai audio input token cost tracking by @krrishdholakia in #9535
- [Reliability Fixes] - Gracefully handle exceptions when DB is having an outage by @ishaan-jaff in #9533
- [Reliability Fix] - Allow Pods to startup + passing /health/readiness when
allow_requests_on_db_unavailable: Trueand DB is down by @ishaan-jaff in #9569 - Add OpenAI gpt-4o-transcribe support by @krrishdholakia in #9517
- Allow viewing keyinfo on request logs by @krrishdholakia in #9568
- Allow team admins to add/update/delete models on UI + show api base and model id on request logs by @krrishdholakia in #9572
- Litellm fix db testing by @krrishdholakia in #9593
- Litellm new UI build by @krrishdholakia in #9601
- Support max_completion_tokens on Mistral by @Cmancuso in #9589
- Revert "Support max_completion_tokens on Mistral" by @krrishdholakia in #9604
- fix(mistral_chat_transformation.py): add missing comma by @krrishdholakia in #9606
- Support discovering gemini, anthropic, xai models by calling their
/v1/modelendpoint by @krrishdholakia in #9530 - Connect UI to "LiteLLM_DailyUserSpend" spend table - enables usage tab to work at 1m+ spend logs by @krrishdholakia in #9603
- Update README.md by @krrishdholakia in #9616
- fix(proxy_server.py): get master key from environment, if not set in … by @krrishdholakia in #9617
New Contributors
Full Changelog: v1.64.1-nightly...v1.65.0-nightly
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.65.0-nightly
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 530.0 | 578.5163737856723 | 5.6521867286945735 | 0.0 | 1689 | 0 | 483.77581299996564 | 4841.4478800000325 |
| Aggregated | Failed ❌ | 530.0 | 578.5163737856723 | 5.6521867286945735 | 0.0 | 1689 | 0 | 483.77581299996564 | 4841.4478800000325 |
Contributors
Cmancuso, krrishdholakia, and ishaan-jaff
Assets 4
v1.64.1.dev1
What's Changed
- Nova Canvas complete image generation tasks (#9177) by @krrishdholakia in #9525
- [Feature]: Support for Fine-Tuned Vertex AI LLMs by @ishaan-jaff in #9542
- feat(prisma-migrations): add baseline db migration file by @krrishdholakia in #9565
- Add Daily User Spend Aggregate view - allows UI Usage tab to work > 1m rows by @krrishdholakia in #9538
- Support Gemini audio token cost tracking + fix openai audio input token cost tracking by @krrishdholakia in #9535
- [Reliability Fixes] - Gracefully handle exceptions when DB is having an outage by @ishaan-jaff in #9533
- [Reliability Fix] - Allow Pods to startup + passing /health/readiness when
allow_requests_on_db_unavailable: Trueand DB is down by @ishaan-jaff in #9569 - Add OpenAI gpt-4o-transcribe support by @krrishdholakia in #9517
- Allow viewing keyinfo on request logs by @krrishdholakia in #9568
- Allow team admins to add/update/delete models on UI + show api base and model id on request logs by @krrishdholakia in #9572
Full Changelog: v1.64.1-nightly...v1.64.1.dev1
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.64.1.dev1
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.64.1.dev1
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.64.1.dev1
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 530.0 | 588.3618842036032 | 5.564625107213408 | 0.0 | 1665 | 0 | 469.31838699998707 | 4872.995692000018 |
| Aggregated | Failed ❌ | 530.0 | 588.3618842036032 | 5.564625107213408 | 0.0 | 1665 | 0 | 469.31838699998707 | 4872.995692000018 |
Contributors
krrishdholakia and ishaan-jaff
Assets 4
v1.64.1-nightly
What's Changed
- Log 'api_base' on spend logs by @krrishdholakia in #9509
- [Fix] Use StandardLoggingPayload for GCS Pub Sub Logging Integration by @ishaan-jaff in #9508
- [Feat] Support for exposing MCP tools on litellm proxy by @ishaan-jaff in #9426
- fix(invoke_handler.py): remove hard coded final usage chunk on bedrock streaming usage by @krrishdholakia in #9512
- Add vertexai topLogprobs support by @krrishdholakia in #9518
- Update model_prices_and_context_window.json by @superpoussin22 in #9459
- fix vertex ai multimodal embedding translation by @krrishdholakia in #9471
- ci(publish-migrations.yml): add action for publishing prisma db migrations by @krrishdholakia in #9537
- [Feat - New Model] Add VertexAI
gemini-2.0-flash-liteand Google AI Studiogemini-2.0-flash-liteby @ishaan-jaff in #9523 - Support
litellm.api_basefor vertex_ai + gemini/ across completion, embedding, image_generation by @krrishdholakia in #9516
Full Changelog: 1.64.0.dev1...v1.64.1-nightly
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.64.1-nightly
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 530.0 | 583.0955007522234 | 5.641309914749418 | 0.0 | 1687 | 0 | 483.3096179999643 | 5048.277267999993 |
| Aggregated | Failed ❌ | 530.0 | 583.0955007522234 | 5.641309914749418 | 0.0 | 1687 | 0 | 483.3096179999643 | 5048.277267999993 |
Contributors
krrishdholakia, superpoussin22, and ishaan-jaff
Assets 4
1.64.0.dev1
What's Changed
- Fix route check for non-proxy admins on jwt auth by @krrishdholakia in #9454
- docs(predibase): fix typo by @luisegarduno in #9464
- build(deps): bump next from 14.2.21 to 14.2.25 in /ui/litellm-dashboard by @dependabot in #9458
- [Feat] Add OpenAI Web Search Tool Call Support - Initial support by @ishaan-jaff in #9465
- Refactor vertex ai passthrough routes - fixes unpredictable behaviour w/ auto-setting default_vertex_region on router model add by @krrishdholakia in #9467
- [Feat] Add testing for
litellm.supports_web_search()and render supports_web_search on model hub by @ishaan-jaff in #9469 - Litellm dev 03 22 2025 release note by @krrishdholakia in #9475
- build: add new vertex text embedding model by @krrishdholakia in #9476
- enables viewing all wildcard models on /model/info by @krrishdholakia in #9473
- Litellm redis semantic caching by @tylerhutcherson in #9356
New Contributors
- @luisegarduno made their first contribution in #9464
Full Changelog: v1.63.14-nightly...1.64.0.dev1
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-1.64.0.dev1
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 530.0 | 595.2345953515114 | 5.639441403576418 | 0.0 | 1687 | 0 | 488.5956009999859 | 4882.238139000037 |
| Aggregated | Failed ❌ | 530.0 | 595.2345953515114 | 5.639441403576418 | 0.0 | 1687 | 0 | 488.5956009999859 | 4882.238139000037 |
Contributors
krrishdholakia, tylerhutcherson, and 3 other contributors
Assets 4
v1.63.14-stable.patch1
Full Changelog: v1.63.14-stable...v1.63.14-stable.patch1
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:litellm_stable_release_branch-v1.63.14-stable.patch1
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Passed ✅ | 240.0 | 255.95973818329688 | 6.163881077009807 | 0.0 | 1844 | 0 | 213.38151600002675 | 2969.717206000041 |
| Aggregated | Passed ✅ | 240.0 | 255.95973818329688 | 6.163881077009807 | 0.0 | 1844 | 0 | 213.38151600002675 | 2969.717206000041 |
v1.63.14-stable
Full Changelog: v1.63.14-nightly...v1.63.14-stable
🚨 Issue identified on Bedrock streaming usage calculation - fixed in v1.63.14-stable.patch 1 - https://github.com/BerriAI/litellm/releases/tag/v1.63.14-stable.patch1
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:litellm_stable_release_branch-v1.63.14-stable
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Passed ✅ | 190.0 | 207.58892887694674 | 6.219076098228175 | 0.0 | 1861 | 0 | 165.81997500003354 | 2278.4240120000163 |
| Aggregated | Passed ✅ | 190.0 | 207.58892887694674 | 6.219076098228175 | 0.0 | 1861 | 0 | 165.81997500003354 | 2278.4240120000163 |
v1.63.14.rc
What's Changed
- Modify completion handler for SageMaker to use payload from
prepared_requestby @andjsmi in #9326 - Arize integration Fix by @nate-mar in #9338
- Fix get_llm_provider error + add model name to tpm/rpm cache key (enables wildcard models to work w/ usage-based-routing) by @krrishdholakia in #9355
- Litellm dev 03 18 2025 p1 v3 by @krrishdholakia in #9354
- [Bug Fix] Arize AI Logging Integration with LiteLLM by @ishaan-jaff in #9352
- build(model_prices_and_context_window.json): fix azure gpt-4o pricing by @krrishdholakia in #9361
- Contributor PR: Fix sagemaker too little data for content error by @krrishdholakia in #9335
- [Feat] - API - Allow using dynamic Arize AI Spaces on LiteLLM by @ishaan-jaff in #9353
- fix(common_utils.py): handle cris only model by @krrishdholakia in #9363
- docs(litellm_proxy): correct parameter assignment in litellm proxy docs by @colesmcintosh in #9375
- Feature flag checking LiteLLM_CredentialsTable by @krrishdholakia in #9376
- fix(lowest_tpm_rpm_v2.py): support batch writing increments to redis by @krrishdholakia in #9357
- Support 'prisma migrate' for db schema changes by @krrishdholakia in #9382
- Fix latency redis by @emerzon in #9387
- Revert "Fix latency redis" by @krrishdholakia in #9388
- build(model_prices_and_context_window.json): add o1-pro pricing by @krrishdholakia in #9397
- [Bug Fix] - Azure OpenAI - ensure SSL verification runs by @ishaan-jaff in #9341
- [Feat] - Allow building custom prompt management integration by @ishaan-jaff in #9384
- Litellm fix icons by @azdolinski in #9374
- [UI Improvement] Use local icons for model providers instead of downloading them by @ishaan-jaff in #9399
- fix(internal_user_endpoints.py): re-introduce upsert on user not found by @krrishdholakia in #9395
- docs: Include Phoenix Page into sidebar under observability by @SrilakshmiC in #9332
- fix(types/utils.py): support openai 'file' message type by @krrishdholakia in #9402
- feat: Add support for custom OPENROUTER_API_BASE via get_secret in co… by @graysonchen in #9369
- fix: VertexAI outputDimensionality configuration by @JamesGuthrie in #9366
- docs(vertexai): fix typo in required env variables by @Shiti in #9325
- Update perplexity.md by @jollyolliel in #9290
- fix: VoyageAI
prompt_tokenalways empty by @lucasra1 in #9260 - build(deps): bump litellm from 1.55.3 to 1.61.15 in /cookbook/litellm-ollama-docker-image by @dependabot in #9422
- [Feat] OpenAI o1-pro Responses API streaming support by @ishaan-jaff in #9419
- [Feat] Add OpenAI o1-pro support on Responses API by @ishaan-jaff in #9415
- [Docs - Draft] LiteLLM x MCP Interface by @ishaan-jaff in #9435
- support returning api-base on pass-through endpoints + consistently return 404 if team not found in DB by @krrishdholakia in #9439
- fix(handle_error.py): make cooldown error more descriptive by @krrishdholakia in #9438
- Consistent anthropic response_format streaming/non-streaming behaviour by @krrishdholakia in #9437
- New Azure Models (GPT-4.5-Preview, Mistral Small 3.1) by @emerzon in #9453
- Set max size limit to in-memory cache item - prevents OOM errors by @krrishdholakia in #9448
- fix(model_param_helper.py): update
_get_litellm_supported_transcription_kwargs()to use proper annotations fromTranscriptionCreateParamsNonStreaming& ``TranscriptionCreateParamsStreaming` by @hsaeed3 in #9451 - [Feat] LiteLLM x MCP Bridge - Use MCP Tools with LiteLLM by @ishaan-jaff in #9436
- fix(model_param_helper.py): update _get_litellm_supported_transcription_kwargs() to use proper annotations from TranscriptionCreateParamsNonStreaming & `TranscriptionCreateParamsStreaming by @ishaan-jaff in #9455
New Contributors
- @andjsmi made their first contribution in #9326
- @azdolinski made their first contribution in #9374
- @SrilakshmiC made their first contribution in #9332
- @graysonchen made their first contribution in #9369
- @JamesGuthrie made their first contribution in #9366
- @Shiti made their first contribution in #9325
- @jollyolliel made their first contribution in #9290
- @hsaeed3 made their first contribution in #9451
Full Changelog: v1.63.12-nightly...v1.63.14.rc
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.63.14.rc
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Passed ✅ | 230.0 | 249.89683940656786 | 6.2068378536570386 | 0.0 | 1857 | 0 | 196.31636200006142 | 3500.2345190000597 |
| Aggregated | Passed ✅ | 230.0 | 249.89683940656786 | 6.2068378536570386 | 0.0 | 1857 | 0 | 196.31636200006142 | 3500.2345190000597 |
Contributors
graysonchen, Shiti, and 13 other contributors
Assets 4
v1.63.14-nightly
What's Changed
- Modify completion handler for SageMaker to use payload from
prepared_requestby @andjsmi in #9326 - Arize integration Fix by @nate-mar in #9338
- Fix get_llm_provider error + add model name to tpm/rpm cache key (enables wildcard models to work w/ usage-based-routing) by @krrishdholakia in #9355
- Litellm dev 03 18 2025 p1 v3 by @krrishdholakia in #9354
- [Bug Fix] Arize AI Logging Integration with LiteLLM by @ishaan-jaff in #9352
- build(model_prices_and_context_window.json): fix azure gpt-4o pricing by @krrishdholakia in #9361
- Contributor PR: Fix sagemaker too little data for content error by @krrishdholakia in #9335
- [Feat] - API - Allow using dynamic Arize AI Spaces on LiteLLM by @ishaan-jaff in #9353
- fix(common_utils.py): handle cris only model by @krrishdholakia in #9363
- docs(litellm_proxy): correct parameter assignment in litellm proxy docs by @colesmcintosh in #9375
- Feature flag checking LiteLLM_CredentialsTable by @krrishdholakia in #9376
- fix(lowest_tpm_rpm_v2.py): support batch writing increments to redis by @krrishdholakia in #9357
- Support 'prisma migrate' for db schema changes by @krrishdholakia in #9382
- Fix latency redis by @emerzon in #9387
- Revert "Fix latency redis" by @krrishdholakia in #9388
- build(model_prices_and_context_window.json): add o1-pro pricing by @krrishdholakia in #9397
- [Bug Fix] - Azure OpenAI - ensure SSL verification runs by @ishaan-jaff in #9341
- [Feat] - Allow building custom prompt management integration by @ishaan-jaff in #9384
- Litellm fix icons by @azdolinski in #9374
- [UI Improvement] Use local icons for model providers instead of downloading them by @ishaan-jaff in #9399
- fix(internal_user_endpoints.py): re-introduce upsert on user not found by @krrishdholakia in #9395
- docs: Include Phoenix Page into sidebar under observability by @SrilakshmiC in #9332
- fix(types/utils.py): support openai 'file' message type by @krrishdholakia in #9402
- feat: Add support for custom OPENROUTER_API_BASE via get_secret in co… by @graysonchen in #9369
- fix: VertexAI outputDimensionality configuration by @JamesGuthrie in #9366
- docs(vertexai): fix typo in required env variables by @Shiti in #9325
- Update perplexity.md by @jollyolliel in #9290
- fix: VoyageAI
prompt_tokenalways empty by @lucasra1 in #9260 - build(deps): bump litellm from 1.55.3 to 1.61.15 in /cookbook/litellm-ollama-docker-image by @dependabot in #9422
- [Feat] OpenAI o1-pro Responses API streaming support by @ishaan-jaff in #9419
- [Feat] Add OpenAI o1-pro support on Responses API by @ishaan-jaff in #9415
- [Docs - Draft] LiteLLM x MCP Interface by @ishaan-jaff in #9435
- support returning api-base on pass-through endpoints + consistently return 404 if team not found in DB by @krrishdholakia in #9439
- fix(handle_error.py): make cooldown error more descriptive by @krrishdholakia in #9438
- Consistent anthropic response_format streaming/non-streaming behaviour by @krrishdholakia in #9437
- New Azure Models (GPT-4.5-Preview, Mistral Small 3.1) by @emerzon in #9453
- Set max size limit to in-memory cache item - prevents OOM errors by @krrishdholakia in #9448
- fix(model_param_helper.py): update
_get_litellm_supported_transcription_kwargs()to use proper annotations fromTranscriptionCreateParamsNonStreaming& ``TranscriptionCreateParamsStreaming` by @hsaeed3 in #9451 - [Feat] LiteLLM x MCP Bridge - Use MCP Tools with LiteLLM by @ishaan-jaff in #9436
- fix(model_param_helper.py): update _get_litellm_supported_transcription_kwargs() to use proper annotations from TranscriptionCreateParamsNonStreaming & `TranscriptionCreateParamsStreaming by @ishaan-jaff in #9455
New Contributors
- @andjsmi made their first contribution in #9326
- @azdolinski made their first contribution in #9374
- @SrilakshmiC made their first contribution in #9332
- @graysonchen made their first contribution in #9369
- @JamesGuthrie made their first contribution in #9366
- @Shiti made their first contribution in #9325
- @jollyolliel made their first contribution in #9290
- @hsaeed3 made their first contribution in #9451
Full Changelog: v1.63.12-nightly...v1.63.14-nightly
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.63.14-nightly
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Passed ✅ | 210.0 | 225.80703369680768 | 6.282456357875219 | 0.0 | 1880 | 0 | 182.8801230000181 | 2803.2175499999994 |
| Aggregated | Passed ✅ | 210.0 | 225.80703369680768 | 6.282456357875219 | 0.0 | 1880 | 0 | 182.8801230000181 | 2803.2175499999994 |
Contributors
graysonchen, Shiti, and 13 other contributors
Assets 4
v1.63.12-nightly
What's Changed
- Fixes bedrock modelId encoding for Inference Profiles by @omrishiv in #9123
- Aim Security post-call guardrails support by @hxtomer in #8356
- Litellm dev 03 12 2025 contributor prs p2 by @krrishdholakia in #9216
- Support bedrock Application inference profiles + Support guardrails on streaming responses by @krrishdholakia in #9274
- v1.63.11-stable release notes by @ishaan-jaff in #9279
- Infer aws region from bedrock application profile id by @krrishdholakia in #9281
- feat: make masterkey secret configurable by @mknet3 in #9288
- fix(utils.py): Prevents final chunk w/ usage from being ignored by @krrishdholakia in #9314
- Update prompt_caching.md to fix typo by @afspies in #9317
- fix(redis_cache.py): add 5s default timeout by @krrishdholakia in #9322
- Support reading litellm proxy response cost header in sdk + support setting lower ssl security level by @krrishdholakia in #9330
- [Bug fix] Reset Budget Job by @ishaan-jaff in #9329
- fix(ollama/completions/transformation.py): pass prompt, untemplated o… by @krrishdholakia in #9333
- [UI] - Allow controlling default internal user settings on ui by @ishaan-jaff in #9328
- [Patch] - Allow disabling all spend updates / writes to DB by @ishaan-jaff in #9331
New Contributors
Full Changelog: v1.63.11-nightly...v1.63.12-nightly
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.63.12-nightly
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Passed ✅ | 240.0 | 266.99215766025395 | 6.081096542088128 | 0.0 | 1819 | 0 | 211.12568599994574 | 4206.960361000029 |
| Aggregated | Passed ✅ | 240.0 | 266.99215766025395 | 6.081096542088128 | 0.0 | 1819 | 0 | 211.12568599994574 | 4206.960361000029 |
Contributors
omrishiv, mknet3, and 4 other contributors