Skip to content

Latest commit

 

History

History
43 lines (29 loc) · 2.21 KB

ai-gateway.md

File metadata and controls

43 lines (29 loc) · 2.21 KB
description icon
AI Gateway documentation
webhook

AI Gateway

Spice provides a high-performance, OpenAI API-compatible AI Gateway optimized for managing and scaling large language models (LLMs). Additionally, Spice offers tools for Enterprise Retrieval-Augmented Generation (RAG), such as SQL query across federated datasets and an advanced search feature (see Search).

Spice supports full OpenTelemetry observability, enabling detailed tracking of data flows and requests for full transparency and easier debugging.

Supported Models​

Spice supports a variety of LLMs, including OpenAI, Azure OpenAI, Anthropic, Groq, Hugging Face, and more (see Model Providers for all supported models).

Core Features​

  • Custom Tools: Equip models with tools to interact with the Spice runtime.
  • System Prompts: Customize system prompts and override defaults for v1/chat/completion.

For detailed configuration and API usage, refer to the API Documentation.

Example: Configuring an OpenAI Compatible Model​

To use a language model hosted on OpenAI (or compatible), specify the openai path and model ID in from.

Example spicepod.yml:

models:
  - from: openai:gpt-4o-mini
    name: openai
    params:
      openai_api_key: ${ secrets:SPICE_OPENAI_API_KEY }

  - from: openai:llama3-groq-70b-8192-tool-use-preview
    name: groq-llama
    params:
      endpoint: https://api.groq.com/openai/v1
      openai_api_key: ${ secrets:SPICE_GROQ_API_KEY }

For details, see OpenAI (or Compatible) Language Models.