Uses the new Microsoft.Extensions.AI preview to compare the relative performance of OpenAI, AzureAI and xAI, using the same client code.
Currently, comparison includes:
- OpenAI: gpt-4o, gpt-4o-mini
- AzureAI: gpt-4o, gpt-4o-mini
- xAI: grok-beta, grok-2-latest
Prompt:
IList<ChatMessage> prompt =
[
new ChatMessage(ChatRole.System, "You are a chatbot inspired by the Hitchhiker's Guide to the Galaxy."),
new ChatMessage(ChatRole.User, "What is the meaning of life, the universe, and everything?"),
];Results:
BenchmarkDotNet v0.14.0, Windows 11 (10.0.22631.4890/23H2/2023Update/SunValley3)
Intel Core i9-10900T CPU 1.90GHz, 1 CPU, 20 logical and 10 physical cores
.NET SDK 9.0.200
[Host] : .NET 8.0.13 (8.0.1325.6609), X64 RyuJIT AVX2
DefaultJob : .NET 8.0.13 (8.0.1325.6609), X64 RyuJIT AVX2
| Method | Client | Provider | Model | Mean | Error | StdDev | Median |
|---|---|---|---|---|---|---|---|
| Chat | aai-gpt-4o | Azure AI | gpt-4o | 20.170 s | 6.4958 s | 19.1529 s | 13.888 s |
| Chat | aai-gpt-4o-mini | Azure AI | gpt-4o-mini | 20.221 s | 5.3235 s | 15.6966 s | 18.061 s |
| Chat | oai-gpt-4o | OpenAI | gpt-4o | 2.387 s | 0.1910 s | 0.5540 s | 2.269 s |
| Chat | oai-gpt-4o-mini | OpenAI | gpt-4o-mini | 2.620 s | 0.1723 s | 0.4859 s | 2.573 s |
| Chat | xai-grok-2 | xAI | grok-2 | 1.876 s | 0.1525 s | 0.4447 s | 1.770 s |
| Chat | xai-grok-beta | xAI | grok-beta | 1.661 s | 0.1001 s | 0.2919 s | 1.655 s |
You can trivially update (and optionally send a PR) the results by running the benchmarks locally.
Simply fork the repository, clone it locally and run the .\update.ps1 script from a pwsh prompt.
The only requirement is to have jq installed and available in your PATH.