Configure Logger headers declaratively with JSON

[frank-ang]

Summary

Opentelemetry sink incorrectly ignores request headers when the Logger is configured declaratively using JSON ConfigurationBuilder.
In contrast, request headers are sent correctly only when the Logger is configured programmatically.

Steps to reproduce

1. Configure a header section of Serilog.Sinks.OpenTelemetry sink in the JSON configuration file.

    "WriteTo": [
      {
        "Name": "OpenTelemetry",
        "Args": { 
          "endpoint": "http://127.0.0.1:4318/v1/logs",
          "protocol": "HttpProtobuf",
          "headers": {
            "Authorization": "Bearer TOKEN_REDACTED"          
          }
        },
      }
    ]

2. Build the logger using JSON

var configuration = new ConfigurationBuilder()
    .AddJsonFile("appsettings.json")
    .Build();
Log.Logger = new LoggerConfiguration()
    .ReadFrom.Configuration(configuration)
    .CreateLogger();

3. Call the logger, Log.Information()

Expected Result.

• Authentication succeeds.
• Network packet capture should show headers are included in requests to OpenTelemetry endpoint.

Actual Result.

• Authentication fails.
  • Exception when using HttpProtobuf protocol: Exception while emitting periodic batch from Serilog.Sinks.PeriodicBatching.PeriodicBatchingSink: System.Net.Http.HttpRequestException: Response status code does not indicate success: 403 (Forbidden).
  • Exception when using Grpc protocol: Exception while emitting periodic batch from Serilog.Sinks.PeriodicBatching.PeriodicBatchingSink: Grpc.Core.RpcException: Status(StatusCode="Unauthenticated",
• Network packet capture (Wireshark) verifies headers are missing from requests to OpenTelemetry endpoint.

Potentially Related

• Add a ConfigurationMethod to Populate ResourceAttributes-dictionary from appsettings.json #107
• Configure ResourceAttributes via appsettings.json serilog-settings-configuration#398
• Headers, like ResourceAttributes, is an IDictionary data type

  serilog-sinks-opentelemetry/src/Serilog.Sinks.OpenTelemetry/Sinks/OpenTelemetry/OpenTelemetrySinkOptions.cs

  Line 63 in 3b04262

  public IDictionary<string, string> Headers { get; set; } = new Dictionary<string, string>();

[juarola]

@frank-ang @nblumhardt This seems to be working fine now, i'm sending "x-seq-apikey" header from appsettings to Seq without any issues.

"WriteTo:OpenTelemetry": {
  "Name": "OpenTelemetry",
  "Args": {
    "Endpoint": "__SeqBaseUrl__/ingest/otlp/v1/logs",
    "Protocol": "HttpProtobuf",
    "resourceAttributes": {
      "service.name": "redacted"
    },
    "headers": {
      "x-seq-apikey": "__SeqApiKey__"
    }
  }
},