Issue to send traces from a .NET Application running on kubernetes to Elastic APM with a .NET application

[mribeiro-clearsale]

Question

Hello everyone.

We currently use Elastic Observability's Elastic APM Agent for our .NET applications that run on k8s and we are migrating to OpenTelemetry, so we can be vendor agnostic and use auto-instrumentation, not requiring specific code changes for each programming language.

I'm using the demo as a basis for this and created an OpenTelemetry Collector, as shown below:

mode: daemonset

image:
  repository: otel/opentelemetry-collector-k8s

presets:
  # enables the k8sattributesprocessor and adds it to the traces, metrics, and logs pipelines
  kubernetesAttributes:
    enabled: true
  # enables the kubeletstatsreceiver and adds it to the metrics pipelines
  kubeletMetrics:
    enabled: true
  # Enables the filelogreceiver and adds it to the logs pipelines
  logsCollection:
    enabled: true
## The chart only includes the loggingexporter by default
## If you want to send your data somewhere you need to
## configure an exporter, such as the otlpexporter
config:
  receivers:
    otlp:
      protocols:
        http:
          cors:
            allowed_origins:
              - http://*
              - https://*
  exporters:
    debug:
      verbosity: detailed
      sampling_initial: 5
      sampling_thereafter: 200        
    otlp/elastic:        
      endpoint: ###.apm.us-central1.gcp.cloud.es.io:443      
      compression: none
      headers:
        Authorization: "Bearer ###"    
  processors:
    batch: {}
    k8sattributes:
      extract:
        metadata:
        - k8s.namespace.name
        - k8s.deployment.name
        - k8s.statefulset.name
        - k8s.daemonset.name
        - k8s.cronjob.name
        - k8s.job.name
        - k8s.node.name
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.pod.start_time
      passthrough: false
      pod_association:
      - sources:
        - from: resource_attribute
          name: k8s.pod.ip
      - sources:
        - from: resource_attribute
          name: k8s.pod.uid
      - sources:
        - from: connection
    memory_limiter:
      check_interval: 5s
      limit_percentage: 80
      spike_limit_percentage: 25
    resource:
      attributes:
      - action: insert
        from_attribute: k8s.pod.uid
        key: service.instance.id
      - action: insert
        from_attribute: service.namespace
        key: service.namespace
      - action: insert
        from_attribute: service.name
        key: service.name
      - action: insert
        value: dev
        key: deployment.environment
  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [k8sattributes, memory_limiter, resource, batch]
        exporters: [debug, otlp/elastic]
      metrics:
        receivers: [otlp]
        processors: [k8sattributes, memory_limiter, resource, batch]
        exporters: [debug, otlp/elastic]
      logs:
        receivers: [otlp]
        processors: [k8sattributes, memory_limiter, resource, batch]
        exporters: [otlp/elastic, debug]

The logs are actually being exported, as I can see in Log Explorer, but the service.name, service.namespace and deployment.environment are not. In fact, I don't know if this is a problem with traces not appearing, apparently only pod logs are being exported, traces and metrics are not.

Here are the environments variables:

        - name: OTEL_RESOURCE_ATTRIBUTES
          value: "service.name=#serviceName#,service.namespace=#namespace#"
        - name: OTEL_SERVICE_NAME
          value: #serviceName#
        - name: OTEL_EXPORTER_OTLP_PROTOCOL
          value: http/protobuf
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: https://#####.apm.us-central1.gcp.cloud.es.io:443
        - name: OTEL_EXPORTER_OTLP_HEADERS
          value: "Authorization=Bearer ####"
        - name: OTEL_TRACES_EXPORTER
          value: otlp               
        - name: OTEL_METRICS_EXPORTER
          value: otlp 
        - name: OTEL_LOGS_EXPORTER
          value: otlp 

Application Code:

builder.Logging
    .AddOpenTelemetry(options => options.AddOtlpExporter())
    .AddConsole();

builder.WebHost.ConfigureKestrel(options => { options.AddServerHeader = false; });

builder.Configuration
    .AddSystemsManager(builder.Configuration["AWS:ParameterStore:Path"])
    .Build();

builder.Services.AddDependencyInjection(builder.Configuration);

builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService(serviceName: builder.Configuration["Otel:ServiceName"], serviceNamespace: builder.Configuration["Otel:Namespace"]))
    .WithTracing(tracerBuilder => tracerBuilder
        .AddAspNetCoreInstrumentation()
        .AddGrpcClientInstrumentation()
        .AddHttpClientInstrumentation()
        .AddAWSInstrumentation()
        .AddOtlpExporter()
        .AddConsoleExporter())
    .WithMetrics(meterBuilder => meterBuilder
        .AddProcessInstrumentation()
        .AddRuntimeInstrumentation()
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter()
        .AddConsoleExporter());

Someone had the same problem and could give me some advices?

[rogercoll]

Thanks for the detailed explanation.

I think the issue could be because the OTEL_SERVICE_NAME environment variable value is not correctly parsed in the .NET application builder. If you are already providing those environment variables to the application, the .NET SDK should use them to automatically populate the resource service attributes. See: https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/docs/trace/customizing-the-sdk/README.md#resource

Could you try removing the manual service name and namespace configuration?

    .ConfigureResource(r => r.AddService(serviceName: builder.Configuration["Otel:ServiceName"], serviceNamespace: builder.Configuration["Otel:Namespace"]))

The cart service example application is a good example of auto resource creation from the environment variables:

• Resource
• Env vars

If the previous proposed solution does not work either, could you try setting a string value instead of reading from environment variables? (to discard any potential issue in Elasticsearch)

    .ConfigureResource(r => r.AddService(serviceName: "MyService"))

(Similar example but with a default value: https://github.com/open-telemetry/opentelemetry-dotnet/blob/d8ce51b1b482e7432da0e7498fdc1052eb2ca807/examples/AspNetCore/Program.cs#L39)