Support for other backends?

[ujh]

I am a big fan of raix, so thank you for that. I use it extensively in my side project, where I can use ChatGPT. But at work I am in a B2B environment in Europe, where that is just out of the question. I am playing around with AWS Bedrock, which offers a somewhat unified abstraction over the models using the Converse API. They do have a Ruby gem that we are using, but it's just so much off a hassle. Especially when writing out the tool calls.

Before I start digging into raix to figure out how to add support for that I would be interested in knowing:

1. if that's something you're even interested
2. how hard that would be and where the best place would be for it. I do wonder if we could get away with a BedrockPretendsToBeOpenAI class that you pass in instead of the client from the ruby-openai project?

[obie]

I'm looking into this feature. Having pluggable LLM client adapters would be a nice improvement to the current implementation. Let me explore some approaches for this.

[obie]

Proposed Design for Pluggable LLM Client Adapters

I've analyzed the current implementation and here's a comprehensive approach to refactor the conditional OpenAI/OpenRouter logic into a clean adapter pattern:

Current Implementation Analysis

The current implementation uses conditional statements to switch between providers:

• Provider selection based on openai parameter
• Separate methods: openai_request and openrouter_request
• Provider-specific parameter handling scattered throughout
• Difficult to add new providers without modifying core logic

Proposed Adapter Pattern Design

1. Base Adapter Interface

module Raix
  module Adapters
    class Base
      attr_reader :client, :config

      def initialize(client, config = {})
        @client = client
        @config = config
      end

      # Core interface methods that all adapters must implement
      def chat_completion(messages:, model:, **params)
        raise NotImplementedError
      end

      def supports_streaming?
        false
      end

      def supports_tools?
        true
      end

      def supports_response_format?
        true
      end

      protected

      # Common parameter transformation logic
      def prepare_messages(messages)
        messages
      end

      def prepare_parameters(params)
        params
      end
    end
  end
end

2. Provider-Specific Adapters

module Raix
  module Adapters
    class OpenAI < Base
      def chat_completion(messages:, model:, **params)
        prepared_params = prepare_parameters(params, model)
        
        response = client.chat(
          parameters: {
            model: model,
            messages: prepare_messages(messages),
            **prepared_params
          }
        )
        
        transform_response(response)
      end

      def supports_streaming?
        true
      end

      private

      def prepare_parameters(params, model)
        transformed = params.dup
        
        # OpenAI-specific transformations
        if model.start_with?("o") && params[:temperature]
          transformed.delete(:temperature)
        end
        
        unless params[:prediction]
          transformed[:max_completion_tokens] = transformed.delete(:max_tokens)
        end
        
        if params[:stream]
          transformed[:stream_options] = { include_usage: true }
        end
        
        transformed
      end
    end

    class OpenRouter < Base
      def chat_completion(messages:, model:, **params)
        prepared_params = prepare_parameters(params)
        
        response = client.complete(
          messages: prepare_messages(messages),
          model: model,
          **prepared_params
        )
        
        transform_response(response)
      rescue Faraday::ServerError
        retry_with_backoff { super }
      end

      private

      def prepare_parameters(params)
        transformed = params.dup
        
        # OpenRouter doesn't support max_completion_tokens
        if transformed[:max_completion_tokens]
          transformed[:max_tokens] = transformed.delete(:max_completion_tokens)
        end
        
        # Handle provider-specific params
        if params[:response_format] && \!supports_native_json_mode?
          transformed[:provider] ||= {}
          transformed[:provider][:require_parameters] = true
        end
        
        transformed
      end
    end
  end
end

3. Adapter Registry

module Raix
  class AdapterRegistry
    class << self
      def register(name, adapter_class, client_factory = nil)
        adapters[name] = { class: adapter_class, client_factory: client_factory }
      end

      def get(name)
        adapter_info = adapters[name]
        raise "Unknown adapter: #{name}" unless adapter_info
        
        client = if adapter_info[:client_factory]
                   adapter_info[:client_factory].call
                 else
                   Raix.configuration.send("#{name}_client")
                 end
        
        adapter_info[:class].new(client)
      end

      private

      def adapters
        @adapters ||= {}
      end
    end
  end
end

4. Refactored ChatCompletion Module

module Raix
  module ChatCompletion
    def chat_completion(prompt = nil, params: {}, **options)
      # Determine adapter
      adapter = determine_adapter(options)
      
      # Prepare messages
      messages = prepare_messages(prompt, options[:messages])
      
      # Delegate to adapter
      response = adapter.chat_completion(
        messages: messages,
        model: options[:model] || model,
        **params
      )
      
      # Handle response
      process_response(response, options)
    end

    private

    def determine_adapter(options)
      if options[:adapter]
        Raix::AdapterRegistry.get(options[:adapter])
      elsif options[:openai]
        Raix::AdapterRegistry.get(:openai)
      else
        Raix::AdapterRegistry.get(default_adapter)
      end
    end

    def default_adapter
      Raix.configuration.default_adapter || :openrouter
    end
  end
end

5. Usage Examples

# Using default adapter
class MyBot
  include Raix::ChatCompletion
  
  def generate
    chat_completion("Hello\!")
  end
end

# Explicitly choosing adapter
bot.chat_completion("Hello\!", adapter: :openai, model: "gpt-4")

# Registering custom adapter
Raix.configure do |config|
  config.register_adapter(:custom_llm, CustomAdapter, {
    api_key: "...",
    base_url: "..."
  })
end

# Adding Anthropic support
config.register_adapter(:anthropic, Raix::Adapters::Anthropic)
bot.chat_completion("Hello\!", adapter: :anthropic, model: "claude-3-opus")

Benefits of This Approach

1. Clean Separation: Each provider's logic is encapsulated in its own adapter
2. Easy Extension: Adding new providers just requires creating a new adapter class
3. Consistent Interface: All adapters follow the same interface
4. Configuration Flexibility: Can register adapters dynamically
5. Backward Compatibility: Can maintain the current openai: parameter behavior
6. Testability: Each adapter can be tested in isolation
7. Feature Discovery: Adapters can declare their capabilities (streaming, tools, etc.)

Migration Strategy

1. Implement adapter classes alongside existing code
2. Gradually move provider-specific logic into adapters
3. Maintain backward compatibility with current API
4. Deprecate old conditional approach in future version
5. Document adapter creation for community contributions

This design would make it much easier to add support for new LLM providers (Anthropic, Cohere, local models, etc.) while keeping the codebase clean and maintainable. The adapter pattern is a natural fit for this use case and would significantly improve the library's extensibility.

[ujh]

That is one idea. Do we really need to support being able to switch from one LLM to the next in the same agent? At least that is not how I am using it and that is also what open router's use case is, isn't it? If we were to statically decide per agent what backend it uses this could be much simpler. It's going to be complex enough if you take function calling into account, which the proposal does not (i.e. the structure of the tool call spec is probably different from tool to tool).

[ArtemXTech]

It would be great to integrate Google AI Studio API. It is great for prototyping in roast with generous free limits and fast models.
https://ai.google.dev/gemini-api/docs

Thank you!

[dersam]

Being able to use local LLMs with Ollama would be great for Roast to be able to run workflows without network access.