@thiagoscodelerae I've got a branch that supports custom models and an open PR. If you setup via my branch you'll be all set:

Clone my branch, see feat: Enable Imported models #97 for changes

git clone -b imported-models https://github.com/sean-smith/bedrock-access-gateway
cd bedrock-access-gateway/

Build a new docker container, I use --platform linux/amd64 to ensure it works even if built on a apple silicon:

cd src/
docker buildx build --platform linux/amd64 -t bedrock-access-gateway .

Create an ECR repo called bedrock-access-gateway

Tag and push to ECR:

aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <account-id>.dkr.ecr.<region>.amazonaws.com
docker tag bedrock-access-gateway:latest <account-id>.dkr.ecr.<region>.amazonaws.com/bedrock-access-gateway:latest
docker push <account-id>.dkr.ecr.<region>.amazonaws.com/bedrock-access-gateway:latest

Create a stack with a custom ECR image, making sure to update the ECR image URI <account-id>.dkr.ecr.<region>.amazonaws.com/bedrock-access-gateway:latest and the ApiKeyParam if you didn't call it BedrockProxyAPIKey:

aws cloudformation create-stack --stack-name bedrock-access-gateway --template-body file://deployment/BedrockProxy.yaml --parameter ParameterKey=ApiKeyParam,ParameterValue=BedrockProxyAPIKey ParameterKey=EnableImportedModels,ParameterValue=true ParameterKey=ImageUri,ParameterValue=<account-id>.dkr.ecr.<region>.amazonaws.com/bedrock-access-gateway:latest
 --capabilities CAPABILITY_AUTO_EXPAND

Now you can call your custom model and it'll show up when you list custom models:

from openai import OpenAI

client = OpenAI()

# List available models
for model in client.models.list():
    print(model.id)

# call custom model by arn
completion = client.chat.completions.create(
    model="arn:aws:bedrock:us-west-2:<account-id>:imported-model/<model-id>",
    messages=[
        {
            "role": "user",
            "content": "Hello! please tell me a joke"
        }
    ],
)

Voila! See #97 for more details.

Question: Support for imported models? #99

Description

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions