Map more activity tags to Google Trace labels

[EricStG]

Issue with OpenTelemetry.Exporter.Stackdriver

List of all OpenTelemetry NuGet
packages and version that you are
using (e.g. OpenTelemetry 1.0.2):

• OpenTelemetry 1.3.0
• OpenTelemetry.Exporter.Stackdriver

Runtime version (e.g. net462, net48, netcoreapp3.1, net6.0 etc. You can
find this information from the *.csproj file): net6.0

Is this a feature request or a bug?

• Feature Request
• Bug

What is the expected behavior?

Google Trace has a long list of predefined labels. While the exporter currently supports some, I'd be great to add the missing ones, especially /error/* and /stacktrace

What is the actual behavior?

Some of the values are being mapped (mostly around /http/*), but more could be mapped

Additional Context

N/A

[trenslow]

I'm not sure if this is the right place, but it would be great if someone would have a look at the following while adding the missing Google Trace labels:

I'm using both AspNetCoreInstrumentation (inbound requests) as well as HttpClientInstrumentation (outbound requests). When I look at the traces on Google Trace, both the inbound and outbound requests don't integrate well with what Google is expecting.

AspNetCoreInstrumentation example:

Here the status code is missing. Also it would be nice to see what fills in the dynamic value in the route, e.g. v1/media/1234567-abcdefg, but that actually comes up in the labels under /http/target (omitted for security reasons).

HttpClientInstrumentation example:

Here, the status code is also missing, but also the HTTP Route info is missing, so I guess it's defaulting to filling it out with the Http method. I would prefer to see the HTTP target/host in that place, like in the AspNetCore instrumentation library.

Right now I can work with the AspNetCore traces, but the HttpClient traces are difficult to work with because I can only get insights across the Http methods, and not e.g. the hosts that my .NET service is calling (Google storage, IdP, etc.).

Is this a configuration issue on my side? As far as I could tell I don't have any ability to configure the instrumentation libraries besides telling them which requests to (not) trace or adding custom metadata.

Does it have anything to do with running my service within Istio? My guess would be no, since Istio tries its best to propagate trace context/metadata, and also my service is talking directly to the Google Trace API via the Stackdriver Exporter.

Would it be improved by adding the predefined Google Trace labels to the Span metadata?

Sorry for the long-winded comment, I just see a lot of potential for these libraries at my org. so I'm trying my best to fully understand them and contribute to the project.

[Kielek]

@trenslow, I think that it will be great to test your application also with another exporter for debugging purposes.
Consider to use https://www.nuget.org/packages/OpenTelemetry.Exporter.Console if you have access to stdout or https://www.nuget.org/packages/OpenTelemetry.Exporter.OpenTelemetryProtocol with OpenTelemetry collector.

In both cases you should be able to see all tags assigned to Activity/Span. Part of them could be vary from what you see on the Stackdriver side due to manual mapping

opentelemetry-dotnet-contrib/src/OpenTelemetry.Exporter.Stackdriver/Implementation/ActivityExtensions.cs

Lines 28 to 38 in e4a61e4

	private static Dictionary<string, string> labelsToReplace = new Dictionary<string, string>
	{
	{ "component", "/component" },
	{ "http.method", "/http/method" },
	{ "http.host", "/http/host" },
	{ "http.status_code", "/http/status_code" },
	{ "http.user_agent", "/http/user_agent" },
	{ "http.target", "/http/target" },
	{ "http.url", "/http/url" },
	{ "http.route", "/http/route" },
	};

What is more, there were breaking changes in the latest OTel RC release in context of attributes.
If you are using it it could work in unexpected way https://github.com/open-telemetry/opentelemetry-dotnet/releases/tag/core-1.4.0-rc.1

[trenslow]

So to provide some context for my comment above, I'm running the following versions:

<PackageReference Include="OpenTelemetry.Exporter.Stackdriver" Version="1.0.0-beta.4" />
<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.0.0-rc9.9" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc9.9" />
<PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.0.0-rc9.9" />

This would be the trace/spans created the same way as the Stackdriver traces from my last comment. Here you can see that the DisplayName lines up with what Stackdriver shows.

Activity.TraceId:          821e50df8c357f6a837cb3b3d1d7428c
Activity.SpanId:           299cf076720b022e
Activity.TraceFlags:           Recorded
Activity.ActivitySourceName: Microsoft.AspNetCore
Activity.DisplayName: v1/media/{mediaId}
Activity.Kind:        Server
Activity.StartTime:   2022-12-15T14:09:22.6591684Z
Activity.Duration:    00:00:02.3537586
Activity.Tags:
    http.host: localhost:5000
    http.method: GET
    http.scheme: http
    http.target: /v1/media/media-7370-A
    http.url: http://localhost:5000/v1/media/media-7370-A
    http.flavor: 1.1
    http.user_agent: PostmanRuntime/7.29.2
    http.route: v1/media/{mediaId}
    http.status_code: 200

This is the outgoing request which created the second image from above. Here you can also see how the DisplayName aligns with the name of the Trace in Stackdriver.

Activity.TraceId:          821e50df8c357f6a837cb3b3d1d7428c
Activity.SpanId:           115f16c676a3059d
Activity.TraceFlags:           Recorded
Activity.ParentSpanId:    299cf076720b022e
Activity.ActivitySourceName: System.Net.Http
Activity.DisplayName: HTTP GET
Activity.Kind:        Client
Activity.StartTime:   2022-12-15T14:09:24.6146772Z
Activity.Duration:    00:00:00.2286496
Activity.Tags:
    http.scheme: https
    http.method: GET
    http.host: storage.googleapis.com
    http.url: https://storage.googleapis.com/storage/v1/<omitted>
    http.flavor: 1.1
    http.status_code: 200
Resource associated with Activity:
    service.name: <omitted>
    service.instance.id: b1430611f8ec

Looking here, it seems like the DisplayName is going to the Summary Table Name field, which Google expects to be service_name.call_name. In my case I guess that would be the service.name tag and http.url combined with a '.' in between. But that's hard as the package developer, since the service.name and service.instance.id fields were added by me.

Maybe there's some more conventional way to build the Summary Table Name field? I had a look at the breaking changes of core-1.4.0-rc.1 and I believe they're unrelated to this issue.

[trenslow]

A small update: I noticed that there were newer versions of the libraries above and I updated to the latest, now:

<PackageReference Include="OpenTelemetry.Exporter.Stackdriver" Version="1.0.0-beta.4" />
<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.4.0-rc.1" /> (updated)
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc9.10" /> (updated)
<PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.0.0-rc9.10" /> (updated)

which didn't change the behavior I'm experiencing.