Track Token usage

[KurtP20]

Technical Feedback

In my Blazor App, I would like to track token usage in SemanticKernel and KernelMemory with streaming responses and document ingestion on a per-user basis for billing purposes.

Desired Outcome

Being able to track the detailed token usage on a per user basis, i.e. get the tokens after ImportWebPageAsync, SearchAsync, GetStreamingChatMessageContentsAsync, ... With the token count I want to be able to calculate the inference costs for billing purposes.

Current Workaround

I don't have one. Copilot suggested to intercept OpenTelemetry, but this is on a per-app basis. Maybe one could wrap all calls around Action to pass the user name to OT, I haven't tried.

[kinfey]

I would recommend using Azure API Management (APIM) to manage access to the Azure OpenAI Service API. This approach provides more effective management and retrieval of tokens for different users with each API call. I don't recommend directly using the native AOAI API approach, as this is unrelated to Semantic Kernel from my understanding.

Here are some helpful resources:

AI Gateway Sample
Leveraging API Management for OpenAI Applications

[KurtP20]

Thanks for pointing me to this solution, it seems to be the only way. I could implement it half way, i.e. I send the proper headers and see the usage on Azure. For me it's overkill and having to spend money on APIM only to track token usage in an enterprise grade framework (Semantic Kernel) is not so nice. And it was difficult to implement (microsoft/semantic-kernel#13120). And I still have to figure out my App can access the token usage from APIM/Application Insights - can somebody point me to an example please?