Litellm krrish staging 04 20 2026

[krrish-berri-2]

Relevant issues

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Screenshots / Proof of Fix

Type

🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test

Changes

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ yuneng-berri
❌ krrish-berri-2
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[gitguardian]

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
29203065	Triggered	JSON Web Token	26fcbc9	tests/test_litellm/proxy/test_litellm_pre_call_utils.py	View secret

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secret safely. Learn here the best practices.
3. Revoke and rotate this secret.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

[greptile-apps]

Greptile Summary

This PR introduces three new request-routing strategies — AutoRouter (semantic/embedding-based), ComplexityRouter (rule-based, <1ms), and QualityRouter (complexity tier → quality tier mapping with keyword override) — plus fixes the auto_routers hot-reload regression and lifts QualityRouter decisions into response headers.

Confidence Score: 5/5

Safe to merge; all open findings are P2 style/config issues that do not block the primary routing path.

The hot-reload regression (auto_routers not cleared) from the prior review is fixed, tests are comprehensive and mock-only, and the two remaining findings are a guard-ordering inefficiency and a broken example config — neither affects correctness of the routing logic itself.

litellm/proxy/new_secret_config.yaml (broken default model reference) and litellm/router.py init*_deployment methods (guard after construction).

Important Files Changed

Filename	Overview
litellm/router_strategy/auto_router/auto_router.py	New AutoRouter using semantic_router to classify messages and route to configured deployments; logic and fallback paths look correct.
litellm/router_strategy/quality_router/quality_router.py	New QualityRouter with lazy tier index, keyword override, and complexity-based routing; tiebreak logic is well tested, but the side-index dicts are never cleared before index rebuild.
litellm/router.py	Hot-reload fix (auto_routers now cleared in set_model_list), QualityRouter headers lifted into set_response_headers, but all three init_*_deployment methods check for duplicates after constructing the router object.
litellm/proxy/_new_secret_config.yaml	Example config references non-existent model "small-model" as the complexity router's default_model, which would cause routing failures for MEDIUM/REASONING tier requests.
tests/test_litellm/router_strategy/test_quality_router.py	Comprehensive mock-only tests covering tier index, resolution, keyword override, tiebreaking, and decision metadata; no real network calls detected.
tests/test_litellm/router_strategy/test_auto_router.py	Tests for AutoRouter._extract_text_from_messages are solid; routing hook tests are marked skip-beta and have the known RouteChoice isinstance issue (surfaced in a prior review).
litellm/router_strategy/quality_router/config.py	Clean Pydantic config models for QualityRouter with sensible defaults and extra='allow' for forward-compatibility.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Incoming Request] --> B{async_pre_routing_hook}
    B --> C{model in auto_routers?}
    C -- Yes --> D[AutoRouter\nSemanticRouter embedding lookup]
    D --> E[RouteChoice.name or default_model]
    C -- No --> F{model in complexity_routers?}
    F -- Yes --> G[ComplexityRouter\nRule-based scoring]
    G --> H[classify → SIMPLE/MEDIUM/COMPLEX/REASONING]
    H --> I[get_model_for_tier]
    F -- No --> J{model in quality_routers?}
    J -- Yes --> K{keyword match?}
    K -- Yes --> L[Keyword override\nhighest quality_tier wins]
    K -- No --> M[ComplexityRouter.classify\n→ complexity_to_quality mapping]
    M --> N[_resolve_model_for_quality_tier\nexact → round-up → round-down → default]
    J -- No --> O[Normal routing]
    E --> P[PreRoutingHookResponse]
    I --> P
    L --> P
    N --> P
    P --> Q[Router selects deployment\nset_response_headers adds x-litellm-quality-router-* headers]

Loading

_{Reviews (3): Last reviewed commit: "style: apply black formatting to websear..." | Re-trigger Greptile}

[greptile-apps]

auto_routers not reset on hot-reload

set_model_list resets quality_routers and complexity_routers to prevent stale state after hot-reload, but auto_routers is never cleared. Any set_model_list call that re-registers an existing auto_router/ deployment will hit the guard in init_auto_router_deployment and raise a ValueError, breaking hot-reload. Fix: add self.auto_routers = {} alongside the other resets.

[greptile-apps]

Tier index not invalidated on dynamic model additions

_tier_to_models_cache is built lazily on first access and never cleared. Models added via add_deployment after the cache is populated are silently excluded from quality routing until set_model_list is called.