AddOtlpExporter never sends data to Collector

[SpirTBBX]

Hello everyone.

I've got a little problem while using OpenTelemetry on my app C# app (with EF Core and WebAPI).

Environment:

• C# app on windows (using .net 7)
• Containers running on WSL2 Ubuntu VM with network mirrored mode.
• CSPROJ:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net7.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.OpenApi" Version="7.0.19" />
    <PackageReference Include="OpenTelemetry" Version="1.11.1" />
    <PackageReference Include="OpenTelemetry.Exporter.Console" Version="1.11.1" />
    <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.11.1" />
    <PackageReference Include="OpenTelemetry.Exporter.Prometheus.AspNetCore" Version="1.9.0-beta.2" />
    <PackageReference Include="OpenTelemetry.Exporter.Zipkin" Version="1.11.1" />
    <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.11.1" />
    <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.11.0" />
    <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.11.0" />
    <PackageReference Include="Serilog.Sinks.Grafana.Loki" Version="8.3.0" />
    <PackageReference Include="Swashbuckle.AspNetCore" Version="6.5.0" />
  </ItemGroup>

  <ItemGroup>
    <Folder Include="Logs\" />
  </ItemGroup>

</Project>

I've configured the app to export the data to Otlp collector (running on docker) like this:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracerProviderBuilder =>
    {
        tracerProviderBuilder
            .SetResourceBuilder(resourceBuilder)
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation();

        foreach (var source in activitySources)
        {
            tracerProviderBuilder.AddSource(source);
        }

        tracerProviderBuilder.AddConsoleExporter();

        tracerProviderBuilder.AddOtlpExporter(options => 
        {
            options.Endpoint = new Uri("http://localhost:4318/v1/traces");
            options.Protocol = OtlpExportProtocol.HttpProtobuf;
        });
    })

Collector config:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  debug:
    verbosity: detailed
  zipkin:
    endpoint: "http://172.17.0.1:9411/api/v2/spans"
    tls:
      insecure: true
  otlphttp/loki:
    endpoint: "http://172.17.0.1:3100/otlp"
    tls:
      insecure: true
  otlphttp/zipkin:
    endpoint: "http://172.17.0.1:9411"
    tls:
      insecure: true

service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [debug, otlphttp/loki]
    traces:
      receivers: [otlp]
      exporters: [debug, zipkin, otlphttp/zipkin]

I've left the docker compose file attached to the message, in case somebody needs to look at it.

Anyways, using wireshark, on the wsl2 vm, I can't capture any traffic with a POST to /v1/traces

However, if I change from OtlpExporter to ZipkinExporter and leave the OTLP Collector address and path there, I can see a /v1/traces request but I get a 400 BAD Request response (from C# app to the Collector Container). If the address of ZipKin exporter on the C# app is http://localhost:9411/api/v2/spans, it works fine.

From what I've understood so far, based on my testings:
Cases:

• OtlpExporter with http://localhost:4318/v1/traces doesn't show logs on the otel-collector and there's no traffic on wireshark
• ZipKin exporter with the above url shows traffic and the collector replies with 400 bad request
• ZipKin exporter with http://localhost:9411/api/v2/spans works like a charm (since the app on windows sends directly to zipkin)
• Making a request, directly from the C# app, using HttpClient to http://localhost:4318/v1/traces, with content type json and a well-formatted body, allows collector to receive the data.

Attempts with other envs:

• I've attempted to run the collector on windows (in case it could be a networking issue) but got the same behaviour.

Docker compose file:

services:
  zipkin:
    image: openzipkin/zipkin
    container_name: zipkin
    ports:
      - "9411:9411"
    user: "1000:1000"
    environment:
      - MYSQL_DB=zipkin
      - MYSQL_USER=zipkin
      - MYSQL_HOST=172.17.0.1
      - MYSQL_PASS=zipkin
      - STORAGE_TYPE=mysql
  grafana:
    image: grafana/grafana
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - ./grafana-data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    user: "1000:1000"
  loki:
    image: grafana/loki:latest
    container_name: loki
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki-config/local-config.yaml:/etc/loki/local-config.yaml
      - ./loki-data:/tmp/loki
    user: "1000:1000"
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    network_mode: host
#    ports:
#      - "9090:9090"
    volumes:
      - ./prometheus-data:/prometheus
      - ./prometheus-config:/etc/prometheus
    user: "1000:1000"
  otel-collector:
    image: otel/opentelemetry-collector:latest
    container_name: otel-collector
#    network_mode: host
    volumes:
      - ./otlp-config/otel-config.yaml:/etc/otel-config.yaml
    command: [ "--config", "/etc/otel-config.yaml" ]
    ports:
      - "4317:4317"
      - "4318:4318"
    depends_on:
      - loki

[SpirTBBX]

The ZipkinExporter bad request error was fixed by adding a zipkin receiver and by specifying it on the traces receivers' pipeline.

The OtlpExporter, however, doesn't work. Tried with both HTTP & GRPC, with and without the path of /v1/logs

[TimothyMothra]

Hi @SpirTBBX,
I don't have any ideas on your specific problem, but I can share some troubleshooting steps you can try.

You can troubleshoot the OTLP exporter by using the Aspire Dashboard. We're still working to get Aspire instructions added to this repo, but you can review a draft on this old PR: https://github.com/open-telemetry/opentelemetry-dotnet/pull/5779/files#diff-e68906f22281657b76471ca80113351485d8e6df5caeab9cc2d9f0f3005c3f83

You're looking for this command:
docker run --rm -it -p 18888:18888 -p 4317:18889 -p 4318:18890 -d -e DOTNET_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS=true --name aspire-dashboard mcr.microsoft.com/dotnet/aspire-dashboard:latest

If the Aspire dashboard works, then you can trust that the OTLP Exporter is working correctly.
Next steps would be to go to the Collector repo and ask them for help troubleshooting.

If the Aspire dashboard doesn't work then please reply here.

[SpirTBBX]

After some testing, I can tell the issue came from WSL2.

I had my instance running in mirrored mode so that I could have both linux and windows using the "localhost" host to communicate. This caused issues only for the OTLP exporter, for some reason. It doesn't really make sense to me, since Zipkin worked perfectly (with the WSL from win to linux).

I think this should be added into the docs stating that WSL2 may cause some issues. I don't know if it could be a problem caused by the network mode I'm using, or from WSL itself.

Anyways, thanks for the help and thanks for all the suggestions!

EDIT: Running Collector and the .NET app on the same OS works like a charm. I've attempted the following cenarios:

• Collector and .NET on windows, containers for telemetry on WSL2: works
• Collector, .NET and containers for telemetry on WSL2: works
• Collector on Linux, .NET on OS Windows (or the way around): doesn't work