Listening to event counters make full events to be written to the EventPipe

[verdie-g]

Description

A side-car app is listening to all well-known EventCounters of the main app using a similar code as this

Dictionary<string, string> eventCounterArguments = new() { ["EventCounterIntervalSec"] = "10" };
string[] providerNames = ["System.Runtime", "Microsoft.AspNetCore.Hosting", "Microsoft-AspNetCore-Server-Kestrel", "System.Net.Http", "System.Net.Sockets", "System.Net.NameResolution"];
var providers = providerNames
    .Select(n => new EventPipeProvider(n, EventLevel.Informational, 0, eventCounterArguments))
    .ToArray();
EventPipeSessionConfiguration config = new(providers, requestRundown: false, requestStacks: false);

DiagnosticsClient diagnosticsClient = new(pid);
var session = await diagnosticsClient.StartEventPipeSessionAsync(config, CancellationToken.None);
EventPipeEventSource source = new(session.EventStream);
source.Dynamic.All += evt =>
{
    if (evt.EventName == "EventCounters")
    {
        // ...
    }
};

I'm expecting that it would enable some code path that increment some counters and periodic counter events to be sent to the pipe. But in this perf flamegraph, you can see that outgoing HTTP requests seem to write an event which is not expected.

Same issue with incoming requests

Is there a way to only listen to HTTP counters without the full HTTP events?

Configuration

• Linux x64
• CentOS 8
• .NET 8

Regression?

Unknown.

Data

The full flamegraph contains too much sensitive info.

Workaround

We could use the OTEL metrics that I think expose the same data but we are years away from that in my company.

[noahfalk]

Is there a way to only listen to HTTP counters without the full HTTP events?

Instead of using the Informational level for those providers, try using EventLevel.Critical. Technically you still could get some non-EventCounters event that was logged at the Critical verbosity level but such events should be very rare.

var providers = providerNames
    .Select(n => new EventPipeProvider(n, EventLevel.Critical, 0, eventCounterArguments))

[verdie-g]

Nice, I'll give it a try. That should be enough until we move to OTEL metrics. I assume event counters ignore the EventLevel?

[noahfalk]

Yeah, the event is logged at level 'LogAlways'

runtime/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Tracing/EventCounter.cs

Line 122 in 0004083

EventSource.Write("EventCounters", new EventSourceOptions() { Level = EventLevel.LogAlways }, new CounterPayloadType(payload));

[verdie-g]

I was able to confirm that using EventLevel.Critical solves my problem

using System.Diagnostics.Tracing;
using Microsoft.Diagnostics.NETCore.Client;
using Microsoft.Diagnostics.Tracing;

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

app.MapGet("/", () => "Hello World!");

var t = app.RunAsync();

Dictionary<string, string> eventCounterArguments = new() { ["EventCounterIntervalSec"] = "10" };
string[] providerNames = ["Microsoft.AspNetCore.Hosting", "Microsoft-AspNetCore-Server-Kestrel"];
var providers = providerNames
    .Select(n => new EventPipeProvider(n, EventLevel.Critical, 0, eventCounterArguments))
    .ToArray();
EventPipeSessionConfiguration config = new(providers, requestRundown: false);
DiagnosticsClient diagnosticsClient = new(Environment.ProcessId);
var session = await diagnosticsClient.StartEventPipeSessionAsync(config, CancellationToken.None);
EventPipeEventSource source = new(session.EventStream);
source.Dynamic.All += evt =>
{
    if (evt.EventName == "EventCounters")
    {
        app.Logger.LogInformation("Some event counter");
    }
    else
    {
        app.Logger.LogError("Unexpected stuff {0}", evt.EventName);
    }
};
source.Process();

await t;