Skip to content

feat: add LiteLLM as provider preset#64

Open
RheagalFire wants to merge 1 commit into
PurpleDoubleD:masterfrom
RheagalFire:feat/add-litellm-provider
Open

feat: add LiteLLM as provider preset#64
RheagalFire wants to merge 1 commit into
PurpleDoubleD:masterfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire

Copy link
Copy Markdown

Summary

Add LiteLLM as a provider preset, giving users one-click access to 100+ LLM providers (OpenAI, Anthropic, Azure, Bedrock, Vertex, Groq, Ollama, etc.) through the LiteLLM proxy.

Since LiteLLM proxy speaks OpenAI-compatible protocol, it reuses the existing openai provider with zero new provider code - just a preset entry alongside vLLM, LM Studio, Groq, etc.

Changes

  • src/api/providers/types.ts - added litellm preset to PROVIDER_PRESETS (baseUrl: http://localhost:4000/v1, providerId: openai, isLocal: true)

Tests

Tests (18/18 pass):

npx vitest run src/api/__tests__/provider-types.test.ts

 Test Files  1 passed (1)
      Tests  18 passed (18)
   Duration  157ms

All preset validation tests pass (unique IDs, valid providerIds, local presets have no placeholder, unique ports).

Live E2E (LiteLLM proxy -> Claude Sonnet via Azure AI Foundry):

Model discovery via /v1/models:

Models: anthropic/claude-sonnet-4-6

Chat completion via /v1/chat/completions:

Response: 4
Model: claude-sonnet-4-6
Tokens: prompt=20, completion=5

Streaming via /v1/chat/completions?stream=true:

Streamed: Hi|!

All three endpoints verified end-to-end through the same OpenAI-compatible path that the existing OpenAIProvider class uses.

Example usage

In the app: Settings > Providers > select "LiteLLM" from the preset dropdown. Models are auto-discovered via /v1/models.

Start the LiteLLM proxy:

pip install litellm
litellm --model anthropic/claude-sonnet-4-6
# Proxy runs on http://localhost:4000

Python usage example (LiteLLM SDK):

import litellm

response = litellm.completion(
    model="anthropic/claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Hello!"}],
    drop_params=True,
)
print(response.choices[0].message.content)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant