Tracing Sources configuration

[jesperkristensen]

IdentityServer uses a lot of activities to create trace spans. This can be very nice, but a very large percentage of our spans (and therefore cost related to storing them) come from IdentityServer, so we would like to reduce the level of detail it produces. This has already been made configurable using tracing sources as described on https://docs.duendesoftware.com/identityserver/diagnostics/otel/#tracing-sources but for our use cases, we have many sources which are difficult to keep track of, so we add them all using a wildcard .AddSource("*"). Unfortunately the OpenTelemetry SDK does not support negative wildcards, so we cannot tell it to e.g. "add everything except for IdentityServerConstants.Tracing.Stores". So now we have the choice between including these too verbose spans from IdentityServer, or remove the wildcard and try (and fail) to maintain a list of sources we want. It would be nice if we could configure this to only include IdentityServerConstants.Tracing.Basic from IdentityServer, and all sources from everywhere else.

[wcabus]

What does your OTel setup look like? Which service(s) are you using to ingest the telemetry data?
Because you may be able to filter out or limit the intake on the service level instead, or purge the data afterwards.

[jesperkristensen]

I don't know if the tools we use on the intake service side supports that, but they are controlled by different teams in our organization, so even if the tools support it, we would be unable to get the the configuration updated.

[ProgrammerAL]

@jesperkristensen Depending on your tooling, you may be able to do it at the collector level by setting config for the processor https://opentelemetry.io/docs/collector/transforming-telemetry. This is the recommended approach with OTel for performance reasons, it moves the processing out of your application.

If that isn't an option, you can filter data inside the application by using a custom processor. This can be put into an internal library are shared with the other teams.

Create the Processor:

using System.Collections.Frozen;
using System.Diagnostics;
using Duende.IdentityServer;
using OpenTelemetry;

namespace Aspire.ServiceDefaults;

internal class MySpecificExcludeProcessor : BaseProcessor<Activity>
{
	//Create a collection of strings to flag on
    private static readonly FrozenSet<string> ExcludedNames = FrozenSet.Create(
        StringComparer.OrdinalIgnoreCase,
        IdentityServerConstants.Tracing.Basic,
        IdentityServerConstants.Tracing.Cache,
		"Some.specific.source");

    public override void OnEnd(Activity activity)
    {
		//When the trace source matches a specific name, don't record it
        if (ExcludedNames.Any(x => activity.Source.Name.StartsWith(x)))
        {
            activity.ActivityTraceFlags &= ~ActivityTraceFlags.Recorded;
        }
    }
}

Initialize the processor for traces:

builder.Services.AddOpenTelemetry()
    .WithMetrics(...)
    .WithTracing(tracing =>
    {
    	...
    	//Add custom processor to remove specific sources
    	tracing.AddProcessor(new MySpecificExcludeProcessor());
    });

Both ways will filter out the trace sources before they are pushed to the OTel provider, potentially saving on ingress costs.

[jesperkristensen]

I noticed that .NET 11 is going to add a simpler way to implement this: https://learn.microsoft.com/en-us/dotnet/core/whats-new/dotnet-11/libraries#activity-tracing-configuration