Support for Azure.Monitor.OpenTelemetry.AspNetCore?

[zyofeng]

Looks like ApplicationInsight SDK is going to be gradually phrased out in favor of Azure.Monitor.OpenTelemetry.AspNetCore.
https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-dotnet-migrate?tabs=aspnetcore

Im wondering what's the best approach to support various AzureMonitor exporters instead of TelemetryClient? Maybe this belong to Serilog.Sink.OpenTelemetry (allow exporters instead of endpoints?)

[nblumhardt]

Hi Mike, thanks for the note.

I think the best integration point for this is likely to be OpenTelemetry.Logs.LoggerProvider/Logger, which would allow a direct mapping from Serilog's LogEvent type, to the OTel SDK LogRecord type:

https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry.Api/Logs/Logger.cs#L61

Once handed off to the OTel SDK, the Azure Monitor distro would take care of the rest.

A Serilog.Sinks.OpenTelemetrySdk package would be the best place for this; if I have a chance I'll try spiking one out this week, but any investigation you can do around this would also be helpful. Cheers!

[nblumhardt]

Spiked: https://github.com/serilog/serilog-sinks-opentelemetrysdk

Mike, this one needs experimental OpenTelemetry SDK APIs so won't necessarily be production-ready anytime soon. But, it'd be great to know if it meets your needs for this, so that we can chat with the OTel SDK maintainers about the status of these APIs or a recommended alternative path forward.

[zyofeng]

Just tried it, but doesn't seem to be logging anything, see test below:

public class StandaloneLogTest(ITestOutputHelper output)
{
    [Fact]
    public void Test()
    {
        var services = new ServiceCollection();

        services
            .AddSerilog((sp, config) =>
                config
                    .MinimumLevel.Debug()
                    .WriteTo.Console()
                    .WriteTo.OpenTelemetrySdk(sp)
            )
            .AddOpenTelemetry()
            .UseAzureMonitor(x =>
                x.ConnectionString =
                    "InstrumentationKey=xxxxx");

        var sp = services.BuildServiceProvider(true);

        var logger = sp.GetRequiredService<ILogger<StandaloneLogTest>>();

        logger.LogInformation("Log: {@Request}",
            new TestRecord
            {
                SomeField = "Test FirstName",
                AnotherField = "Test Message"
            });
    }

    public class TestRecord
    {
        public string SomeField { get; set; } = null!;
        public string AnotherField { get; set; } = null!;
    }
}

[nblumhardt]

Thanks for taking a look! I'll dig in when I can, but I think there are a few possible problems.

First, you'll likely need a using keyword on the sp declaration, so that buffered events are flushed:

using var sp = services.BuildServiceProvider(true);

If this doesn't fix it, there's a chance UseAzureMonitor() is overriding the ILoggerFactory otherwise registered by Serilog.

You could try re-ordering those statements to work around it:

        services
            .AddOpenTelemetry()
                .UseAzureMonitor(x =>
                    x.ConnectionString = "InstrumentationKey=xxxxx")
            .AddSerilog((sp, config) =>
                config
                    .MinimumLevel.Debug()
                    .WriteTo.Console()
                    .WriteTo.OpenTelemetrySdk(sp)
            );

But, it might be worth constructing a test without the AddSerilog() bit, and just create a separate Serilog Logger to check that everything's hooked up correctly:

var services = new ServiceCollection();

services.AddOpenTelemetry()
    .UseAzureMonitor(x =>
        x.ConnectionString =
            "InstrumentationKey=xxxxx");

using var sp = services.BuildServiceProvider(true);

using var logger = new LoggerConfiguration()
    .WriteTo.OpenTelemetrySdk(services)
    .CreateLogger();

loggerInformation("Log: {@Request}",
    new TestRecord
    {
        SomeField = "Test FirstName",
        AnotherField = "Test Message"
    });

Finally, make sure that the version of the OpenTelemetry package is the 1.9.0-rc one (or another RC), the sink won't work with any RTM version.

Let me know how you go! 😎

[zyofeng]

alright we are getting somewhere, I was using 1.3.0-beta.3 version of Azure.Monitor.OpenTelemetry.AspNetCore which references RTM 1.11.2 of OpenTelemetry that has EXPOSE_EXPERIMENTAL_FEATURES disabled.
I've reverted to 1.2.0 version of Azure.Monitor.OpenTelemetry.AspNetCore and the implicit reference to Otel is now 1.9.0-rc.1

I've also changed the test code

        using var sp = new ServiceCollection()
            .AddLogging(x => x.AddOpenTelemetry(y => y.AddAzureMonitorLogExporter(z =>
                z.ConnectionString =
                    "InstrumentationKey=xxx")))
            .BuildServiceProvider(true);

        Log.Logger = new LoggerConfiguration()
            .MinimumLevel.Debug()
            .WriteTo.Console()
            .WriteTo.OpenTelemetrySdk(sp)
            .CreateLogger();
            
        var logger = sp.GetRequiredService<ILogger<StandaloneLogTest>>();

        logger.LogInformation("Log 1: {@Request}",
            new TestRecord
            {
                SomeField = "Test FirstName",
                AnotherField = "Test Message"
            });
        
        Log.Information("Log 2: {@Request}",
            new TestRecord
            {
                SomeField = "Test FirstName",
                AnotherField = "Test Message"
            });
    }

    public class TestRecord
    {
        public string SomeField { get; set; } = null!;
        public string AnotherField { get; set; } = null!;
    }

Got the following AI entry, looks like the properties are not getting expanded (which is expected since it's the default logger), but Log 2 is missing

[nblumhardt]

Figured it out:

The problem is that the Azure Monitor exporter (at the version we're using?) ignores LogRecord.Body - it assumes every message will come from an ILogger<T> and thus have a LogRecord.ILoggerData member with a non-null FormattedMessage property.

Unfortunately there's no extensibility point I can find that would allow the sink to set this, but it can be "fixed up" with:

class FixILoggerDataProcessor : BaseProcessor<LogRecord>
{
    public override void OnEnd(LogRecord data)
    {
        data.FormattedMessage ??= data.Body;
        base.OnEnd(data);
    }
}

If you add this to the OTel configuration:

using var sp = new ServiceCollection()
         .AddLogging(x => x.
             AddOpenTelemetry(y => y
                 .AddProcessor(new FixILoggerDataProcessor())
                 .AddConsoleExporter()
                 .AddAzureMonitorLogExporter(z =>
                 {
                     z.ConnectionString = "InstrumentationKey=...";
                 }
         )))
         .BuildServiceProvider(true);

There might be similar issues to discover around Exceptions.

The first step to get this resolved would be to check in with the OTel SDK folks about plans for the LoggerProvider API; if it's moving forward, then I think the bug would be with the Azure Monitor exporter - but until there's a supported non-ILogger way to generate events, the exporter behavior isn't problematic in itself.

[zyofeng]

This largely fixes the issue, some small issues with CustomProperties not expanding Attributes (instead just doing a tostring)

MessageTemplate shows as message_template.text

Destructurama does not work at all, but Serilog.Redaction works fine.

Definitely usable though

[nblumhardt]

Great, thanks. I found some discussion around mappings for the OTel AnyValue type, which should fix the issues displaying structured data.

Had some feedback on the OTel Slack that the APIs here have been sidelined until the upcoming "events" support is finished. Will stay tuned.

There's also an NLog target waiting for the same experimental support to stabilize: https://github.com/juliuskoval/NLog.Targets.OpenTelemetryProtocol

[zyofeng]

yeah I came across that repo as well and was confused why it didn't reference the bridge.

[sgryphon]

Microsoft now has a limited preview of direct OTLP ingestion into Azure Monitor / App Insights (which has been on their OTel roadmap): https://learn.microsoft.com/en-us/azure/azure-monitor/fundamentals/opentelemetry-protocol-ingestion

Once they have OTLP support, Serilog can just use direct OTLP export, rather than via the Otel SDK.