Bedrock text completions

[sixlive]

[ChrisB-TL]

Following our conversation last night, I've had a look at this.

They have two APIs for most models:

• a model specific API, e.g. you can use Anthropic's; and
• the ConverseAPI, which is compatible with most but not all models.

Given that, I think what we should probably do is:

• have a Bedrock Converse provider; and
• add an option/config on Anthropic to point it at Bedrock - i.e. config to route all request to Bedrock, or the ability to do so at runtime.

I think we should do the latter as its an easy win and makes migrating or using Anthropic on both (e.g. due to rate limiting) easy.

Let me know if you have any notes on that approach. I will probably start working on this later in the week.

[ChrisB-TL]

As an update on this, I have both Anthropic and Converse APIs up and running on Bedrock:

• Anthropic API has parity with the Anthropic Prism provider, apart from documents and citations (not yet available on Bedrock); and
• Converse API has support for Text and Structured (Adapted), with support for caching (though currently in invite only early access) and documents. Not yet implemented images or tools. I'll add embeddings shortly.

[ChrisB-TL]

I've now added image and tool support for the Converse API.

Embeddings wise, the Converse API does not support them. So we'll need to add an API schema for each Bedrock embeddings model we want to support. I've added support for Cohere initially as it seems to be the more popular one.

So it has a clean sweep on the supported modalities table!

@sixlive give me a shout when you are ready to make the repo public and I'll write up some docs to go in the main repo in the meantime.

[chrisnetonline]

@ChrisB-TL Any update on this? We are looking at switching from OpenAI to Sonnet via Bedrock.

[ChrisB-TL]

It is good to go, we just need to find a moment to get it release-ready. I'll catch up with @sixlive on it.

[chrisnetonline]

Thanks @ChrisB-TL!