Skip to content

Xata Agent ‐ Use with LiteLLM

Jump to bottom
Steffen Siering edited this page Apr 23, 2025 · 1 revision

This guide walks you through the integration of the Xata Agent with LiteLLM.

Step 1: configure LiteLLM

The Xata Agent fetches all available models from LiteLLM. The builtin models will be ignored.

The agent will only use models that support the 'chat' mode. The model_info.mode field should be explicitly set to 'chat' in the model configuration when integrating with Ollama or LM Studio.

The Xata Agent supports custom LiteLLM settings:

  • model_info.xata_agent.model_id (optional): Model ID used within the agent. Use when the underlying model is versioned or multiple types exist and you want to be able to change the actual model being used without changing the model ID in the agent. Having a stable model ID ensures that scheduled monitoring jobs continue to work properly.

Example:

model_list:
  - model_name: Claude 3.7 Sonnet
    litellm_params:
      model: anthropic/claude-3-7-sonnet-20250219
    model_info:
      mode: chat
      xata_agent:
        model_id: anthropic:claude-3-7-sonnet

The agent will use the litellm_params.model if model_id is not set. In this case the / will be replaced with a :.

  • model_info.xata_agent.priority: Model priority used when fallback or alias configurations overlap. The model with the higher priority will be chosen. Default priority is 1.

  • model_info.xata_agent.private: Boolean flag to mark the model as private. The model can be used internally as fallback or alias model, but it will not be selectable to users via the UI.

  • model_info.xata_agent.group_fallback (optional): Fallback model for common model group. The model ID uses the schema group:model. The group_fallback allows you to set the current model as fallback for a common model group. This gives the Agent a fallback model in case a model has been removed from the LiteLLM configuration.

Example:

model_list:
  - model_name: GPT-4o
    litellm_params:
      model: openai/gpt-4o
    model_info:
      xata_agent:
        group_fallback: openai
  • model_info.xata_agent.alias (optional): Model alias IDs. The agent supports some builtin aliases one might want to configure:
    • chat: chat alias configures the default chat model to use.
    • title: model used to create a chat title from the initial conversation.
    • summary: internal summarizer model

Example:

model_list:
  - model_name: GPT-4o
    litellm_params:
      model: openai/gpt-4o
    model_info:
      xata_agent:
        group_fallback: openai
        alias:
          - chat
  - model_name: GPT-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
    model_info:
      xata_agent:
        private: true
        alias:
          - title
          - summary

Step 2. Configure Agent to use LiteLLM

Set environment variables LITELLM_BASE_URL (e.g. "http://localhost:4000") and LITELLM_API_KEY.

Step 3. Start LiteLLM and Xata agent

The agent will fetch the list of available models when running a scheduled monitor or when opening a chat. The default builtin models will be ignored by the Agent.

Example configurations

Configure internal models for title/summary retrieval

model_list:
  ...
  - model_name: title
    litellm_params:
      model: openai/gpt-4o
    model_info:
      xata_agent:
        model_id: title
        private: true

  - model_name: summary
    litellm_params:
      model: openai/gpt-4o
    model_info:
      xata_agent:
        model_id: summary
        private: true

Configure default 'chat' model

model_list:
  ...
  - model_name: GPT-4o
    litellm_params:
      model: openai/gpt-4o
    model_info:
      xata_agent:
        alias:
          - chat

Configure custom "latest" model ID for versioned model ID

  - model_name: Claude 3.7 Sonnet
    litellm_params:
      model: anthropic/claude-3-7-sonnet-20250219
    model_info:
      mode: chat
      xata_agent:
        model_id: anthropic:claude-3-7-sonnet

Configure group fallback

We use the model ID format <group>:<model>. We parse out the 'group' part to find group fallbacks. The group fallback is the model we will use for unknown IDs. The following configuration will fallback to "GPT-4o".

model_list:
  - model_name: GPT-4o
    litellm_params:
      model: openai/gpt-4o
    model_info:
      xata_agent:
        group_fallback: openai

  - model_name: GPT-4 Turbo
    litellm_params:
      model: openai/gpt-4-turbo

Use LiteLLM with LM Studio

  1. Add custom LiteLLM configuration (use 'import' to read the default configuration file)
model_list:
  - model_name: Mistral Nemo
    litellm_params:
      model: lm_studio/mistral-nemo-instruct-2407
      api_base: os.environ/LM_STUDIO_API_BASE
      api_key: "test"   # by default LM_STUDIO has no api key configured, add dummy key
    model_info:
      mode: chat
      max_tokens: 4096
      input_cost_per_token: 0
      output_cost_per_token: 0
      xata_agent:
        model_id: studio:mistral-nemo

        # overwrite default chat model
        default: ['chat']
        priority: 10

Clone this wiki locally