Missing data from OpenTelemetry output

[szatixx]

Hi!

I am trying to integrate Semantic Kernel with an ASP.NET Core app, and trace the app with the built in OpenTelemetry output. I was fallowing the official docs, but it only provides fraction of the data it should.

My output is like this:

Activity.TraceId: d6cdfcfe9b91389f78462af195684085
Activity.SpanId: cbc9b34229dfeb60
Activity.TraceFlags: Recorded
Activity.ParentSpanId: 0d93bc4473128707
Activity.DisplayName: invoke_agent review_agent
Activity.Kind: Internal
Activity.StartTime: 2025-04-16T20:58:03.4577425Z
Activity.Duration: 00:00:00.6668680
Activity.Tags:
gen_ai.operation.name: invoke_agent
gen_ai.agent.id: f888c700-c769-4f53-b958-0b1e14bb7da0
gen_ai.agent.name: review_agent
Instrumentation scope (ActivitySource):
Name: Microsoft.SemanticKernel.Diagnostics
Resource associated with Activity:
telemetry.sdk.name: opentelemetry
telemetry.sdk.language: dotnet
telemetry.sdk.version: 1.11.2
service.name: unknown_service:JokeTeller

All the powerfull data is missing, like the prompt, the token count etc.

Is there a settings to add them?

I am using .NET 9 and Microsoft.SemanticKernel 1.46.0 and Microsoft.SemanticKernel.Connectors.AzureAIInference 1.46.0-beta

Thanks
Attila

[sophialagerkranspandey]

Adding @TaoChenOSU

[TaoChenOSU]

Hi @szatixx!

Thank you for raising the question!

Did you set the following switch to true: https://review.learn.microsoft.com/en-us/semantic-kernel/concepts/enterprise-readiness/observability/telemetry-with-console?branch=main&tabs=Powershell-CreateFile%2CEnvironmentFile&pivots=programming-language-csharp#add-telemetry?

// Enable model diagnostics with sensitive data.
AppContext.SetSwitch("Microsoft.SemanticKernel.Experimental.GenAI.EnableOTelDiagnosticsSensitive", true);

[szatixx]

Hi @TaoChenOSU!

Yes I did. Here is my full source: https://gist.github.com/szatixx/550aa6696bd18a684f5cf19c12e7e6bc

This is newer version than I was using in my original post, but there are still missing data. Now I have successfully integrated the code with Application Insights, and the newer version of SK and Azure Inference are sending more data, so there was some improvements.

I am attaching some images of my Application Insights traces. If I am using Azure OpenAI, it looks good in the App Insights, but not in Azure AI Foundry.
With Azure Inference API its a different story, it sends less traces, and doesn't shows up in AI Foundry at all.

My company is developing a system using 8-10 agents to solve a task together. We need the AI Foundry style monitoring feature to efficiently truck and debug our production system. So while it is not fully working its a no go for us.

Also I was also surprised that the name of the agent even in the more or less working Azure OpenAI traces are missing. I see the type of the agent message (eg. system, user) and the message itself, but not the name of the agent. Since we are using many cooperating agents it is hard to "replay" the chat without knowing the name of the sender. Do you have plans to include the agent name in the telemetry?

Thanks
Attila

Inference API missing data

Azure Ai Foundry error

SK telemetry

[ChrisL-GU]

I've been struggling with this myself. Following the example provided in the answer above works. (@TaoChenOSU! the public facing URL is: https://learn.microsoft.com/en-us/semantic-kernel/concepts/enterprise-readiness/observability/telemetry-with-console#add-telemetry)

But when trying to replicate it using DI in a web app it seems it's more particular on how it is configured. Below are my code snippets. I do get traces, but I'm not seeing any metrics. I've spent hours trying to get it to work. I don't know what I'm missing.

AppContext.SetSwitch("Microsoft.SemanticKernel.Experimental.GenAI.EnableOTelDiagnosticsSensitive", true);

_ = services
    .AddOpenTelemetry()
    .UseAzureMonitor(options =>
    {
        options.ConnectionString = appInsightsConnectionString;
    })
    .WithTracing(builder =>
        builder
            .AddSource(Telemetry.SourceName)
            .AddSource("Microsoft.SemanticKernel*")
            .SetResourceBuilder(Telemetry.ResourceBuilder)
            .AddConsoleExporter()
    )
    .WithMetrics(builder => builder
        .AddMeter("Microsoft.SemanticKernel*")
        .SetResourceBuilder(Telemetry.ResourceBuilder)
        .AddConsoleExporter());
        
        services
            .AddKernel()
            .AddAzureOpenAIChatCompletion(
                deploymentName,
                endpoint,
                apiKey);

[Inject] public required IChatCompletionService ChatCompletionService { get; set; }

...

var response = ChatCompletionService?.GetStreamingChatMessageContentsAsync(chatHistory)