Description
Bug Report
OpenTelemetry NuGet packages:
- OpenTelemetry.Instrumentation.AspNetCore 1.0.0-rc9.4
- OpenTelemetry 1.3.0
- OpenTelemetry.Exporter.OpenTelemetryProtocol 1.3.0
Other relevant NuGet packages:
- Grpc.AspNetCore 2.47.0
- Google.Protobuf 3.21.1
Runtime version: net6.0
Background:
Our product is acting as otel trace collector, using the standard trace.proto files, and traces from different applications are being sent to our collector gRPC endpoint.
Symptom
When our gRPC endpoint is being called no gRPC tags are added to the top level span (server kind) by the OpenTelemetry.Instrumentation.AspNetCore instrumentation library.
observe the following tags from Jaeger:
span-name: /opentelemetry.proto.collector.trace.v1.TraceService/Export
- http.flavor 2.0
- http.host localhost:5050
- http.method POST
- http.scheme http
- http.status_code 200
- http.target /opentelemetry.proto.collector.trace.v1.TraceService/Export
- http.url http://localhost:5050/opentelemetry.proto.collector.trace.v1.TraceService/Export
- http.user_agent grpc-dotnet/2.43.0 (.NET 6.0.1; CLR 6.0.1; net6.0; osx; arm64)
- internal.span.format proto
- otel.library.name OpenTelemetry.Instrumentation.AspNetCore
- otel.library.version 1.0.0.0
- span.kind server
What is the expected behavior?
additional RPC tags should be added to this span, at least grpc.method tag should be added
What did you expect to see?
all gRPC related tags under this span
What is the actual behavior?
I found out the following code gives different results if getting into the NET6_0_OR_GREATER section or not.
the activity returned by using the HttpContext.Features is not the same activity that returned by using the actifity.current inside the while loop.
when getting the activity from the HttpContext.Features, the activity that is being updated with the grpc.method tag is the wrong one because HttpInListener makes the new activity as a "sibling" of the activity created by Asp.Net Core (check the next section)
//HttpContextServerCallContext.cs
private Activity? GetHostActivity()
{
#if NET6_0_OR_GREATER
// Feature always returns the host activity
var feature = HttpContext.Features.Get<IHttpActivityFeature>();
if (feature != null)
{
return feature.Activity;
}
#endif
// If feature isn't available, or not supported, then fallback to Activity.Current.
var activity = Activity.Current;
while (activity != null)
{
// We only want to add gRPC metadata to the host activity
// Search parent activities in case a new activity was started in middleware before gRPC endpoint is invoked
if (string.Equals(activity.OperationName, GrpcServerConstants.HostActivityName, StringComparison.Ordinal))
{
return activity;
}
activity = activity.Parent;
}
return null;
}
And this happened only in a specific condition in the OpenTelemetry.Instrumentation.AspNetCore under the
HttpInListener.cs, when entering into the condition that creates a new activity
// Ensure context extraction irrespective of sampling decision
var request = context.Request;
var textMapPropagator = Propagators.DefaultTextMapPropagator;
if (textMapPropagator is not TraceContextPropagator)
{
var ctx = textMapPropagator.Extract(default, request, HttpRequestHeaderValuesGetter);
if (ctx.ActivityContext.IsValid()
&& ctx.ActivityContext != new ActivityContext(activity.TraceId, activity.ParentSpanId, activity.ActivityTraceFlags, activity.TraceStateString, true))
{
// Create a new activity with its parent set from the extracted context.
// This makes the new activity as a "sibling" of the activity created by
// Asp.Net Core.
Activity newOne = new Activity(ActivityOperationName);
newOne.SetParentId(ctx.ActivityContext.TraceId, ctx.ActivityContext.SpanId, ctx.ActivityContext.TraceFlags);
newOne.TraceStateString = ctx.ActivityContext.TraceState;
newOne.SetTag("IsCreatedByInstrumentation", bool.TrueString);
// Starting the new activity make it the Activity.Current one.
newOne.Start();
// Set IsAllDataRequested to false for the activity created by the framework to only export the sibling activity and not the framework activity
activity.IsAllDataRequested = false;
Console.Write("\nHttpRequestIn: new created activity: "+ newOne.SpanId+"\n");
activity = newOne;
}
Baggage.Current = ctx.Baggage;
}