This repo has been archived in favour of https://github.com/DefangLabs/openai-access-gateway

Bedrock Sidecar

A proxy server that converts OpenAI API requests to AWS Bedrock requests, allowing you to use OpenAI-compatible clients with AWS Bedrock. The Defang Bedrock Sidecar is deployed alongside Defang services, and presents an OpenAI-compatible interface for AWS Bedrock.

Usage

Enable model access in Bedrock

1. Navigate to the Bedrock Model Access dashboard
2. Click the "Modify model access" button
3. Select the models you would like to use with Bedrock
4. Click the "Next" button at the bottom of the page
5. Click the "Submit" button

Update your compose file

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
    # Add this tag to set up integration with bedrock when deploying to aws
    x-defang-llm: true

Environment variables

• DEBUG: if set (to anything) will show debug logs
• MODEL_NAME_MAP: a JSON encoded map of model names to Bedrock model IDs
• PORT: the TCP port to listed on for HTTP API requests

Features

• Intercepts OpenAI API requests
• Converts OpenAI chat completion requests to AWS Bedrock format
• Converts AWS Bedrock responses back to OpenAI format
• Supports basic chat completion functionality

Limitations

• Currently only supports basic chat completion functionality
• Token counting is not implemented

Contributing

See our Contributing document.