Duplicated spans using ASP.NET controllers

[matteopey]

Describe the bug
I'm not sure if this is a bug or is by design, so I'm opening this issue to have some guidance on how to solve this.
I have an application that is using .NET Core 6. The controllers in this applications are implemented using the ControllerBase class from ASP.NET.
The result on Datadog is that inside the trace two spans are shown, even though the controller path and the request made is only one.
The service entry is of type aspnet_core.request the inner one is of type aspnet_core_mvc.request.
Instead, using the new routing like this app.MapGet("/api/mapget", () => "Hello!"); only one trace is shown.

To Reproduce
I setup Datadog with an agent running on the host machine on Docker with APM enabled and the application is running on the same host, integrating the .NET tracer as explained in the /apm/intro guide on Datadog.

Expected behavior
One span is shown given the single request and the single controller.

Screenshots
Controllers

MapGet

Runtime environment (please complete the following information):

• Automatic instrumentation
• Tracer version: 2.29.0
• OS: Windows 10 on WSL2. Happening in production on K8S cluster.
• CLR: >= .NET Core 3.1

Additional context
Noticed this issue in our production systems, but is easily reproducible with a simple application locally.

[andrewlock]

Hi @matteopey, the behaviour you're seeing is expected when you use minimal APIs. There is no controller/Razor Page with minimal APIs so the MVC request aspect doesn't apply here, and there's no additional span to create. The root span contains details about the selected endpoint.

Can I take it from this issue that you would like there to be an additional span for the endpoint execution? If so, what information is it that you'd like to obtain? Is it the duration of the endpoint relative to the whole request? Or is there some additional information you would hope to see?

We've had some feedback that having two spans per request is problematic for some people, so it would be good to know if you prefer the two-span setup we have. Thanks!

[matteopey]

Hi @andrewlock I probably explained myself badly.
The behavior of the minimal API is the one that I think is correct.
What I would prefer is to have for the controllers only one span per request instead of the two currently, so exactly the same behavior as the minimal APIs. I'm therefore aligned with the feedback you received from the other people.
For us having two span is creating too much overhead in some places of our applications, where we see on Datadog hundreds of spans while instead they are half of that.

[lpodolak]

I have also mixed feelings about these duplicated spans. I think they are the source of problems that I observe:

I discovered that it matters at which stage of processing we enhance spans with custom metadata using the span.SetTag() method:

1. If we do it in middleware, before request is processed, the metadata lands on the first span. These custom data are indexed and I can search/filter by them on the Datadog UI
2. If I do it later however, e.g. by attaching from middlware to context.Response.OnStarting() or even earlier - in asp.net core Filters, then these metadata land on the 2nd span. Unfortunately it is also not indexed, which means I can't filter/search for these data.

Recently we migrated our old Datadog.Tracer & the Datadog.Trace from version 1.27 to latest: 2.30. We don't have older data, but we're sure that we could filter/search by our custom fields added in a way #2 before the upgrade.

@andrewlock Do you have any idea how we could make these fields from the span #2 be indexed so that we can filter/search for them?
Or maybe you could get back to an idea of single-span per controller action processing?

[lpodolak]

Here's a screenshot with our custom metadata landing on the 2nd span and so - being not searchable anymore.

[rjune]

Describe the bug I'm not sure if this is a bug or is by design, so I'm opening this issue to have some guidance on how to solve this. I have an application that is using .NET Core 6. The controllers in this applications are implemented using the ControllerBase class from ASP.NET. The result on Datadog is that inside the trace two spans are shown, even though the controller path and the request made is only one. The service entry is of type aspnet_core.request the inner one is of type aspnet_core_mvc.request. Instead, using the new routing like this app.MapGet("/api/mapget", () => "Hello!"); only one trace is shown.

To Reproduce I setup Datadog with an agent running on the host machine on Docker with APM enabled and the application is running on the same host, integrating the .NET tracer as explained in the /apm/intro guide on Datadog.

Expected behavior One span is shown given the single request and the single controller.

Screenshots Controllers

MapGet

Runtime environment (please complete the following information):

• Automatic instrumentation
• Tracer version: 2.29.0
• OS: Windows 10 on WSL2. Happening in production on K8S cluster.
• CLR: >= .NET Core 3.1

Additional context Noticed this issue in our production systems, but is easily reproducible with a simple application locally.

I don't know if you're still having this issue but, if you set DD_TRACE_ROUTE_TEMPLATE_RESOURCE_NAMES_ENABLED=false then the "duplicate" span goes away.
https://ddtrace.readthedocs.io/en/stable/configuration.html

[andrewlock]

I don't know if you're still having this issue but, if you set DD_TRACE_ROUTE_TEMPLATE_RESOURCE_NAMES_ENABLED=false then the "duplicate" span goes away.

It's true that this goes away, but you also lose rich template names, the resource names become inconsistent, and you're using an old, deprecated mode.

I took another look at our options here recently, and the big trouble with offering a single-span mode is you lose a bunch of information related to how your app is behaving.

For example, looking at the screen shots above, the mvc span gives you interesting extra information, mostly around timings and the source of any child spans. We can still preserve the tags that are recorded, i.e. information about the controller/action, but we lose all the timing information. There doesn't seem to be a great solution to that either.

Another issue is when the middleware pipeline is re-executed, for example when an error occurs, and the pipeline redirects to an error controller. Currently, both of those MVC executions show up as separate spans nested in the root span, each with their own tags (and timings). If we switch to a single root span, there's a question about how we can preserve that information - we can tag an array of controller+action pairs that were invoked, but it's all a bit of a mess.

Which is all to say that we're still well aware of the difficulty here, we're considering our options, but haven't worked out the fine details yet I'm afraid.

[rjune]

I don't know if you're still having this issue but, if you set DD_TRACE_ROUTE_TEMPLATE_RESOURCE_NAMES_ENABLED=false then the "duplicate" span goes away.

It's true that this goes away, but you also lose rich template names, the resource names become inconsistent, and you're using an old, deprecated mode.

I took another look at our options here recently, and the big trouble with offering a single-span mode is you lose a bunch of information related to how your app is behaving.

For example, looking at the screen shots above, the mvc span gives you interesting extra information, mostly around timings and the source of any child spans. We can still preserve the tags that are recorded, i.e. information about the controller/action, but we lose all the timing information. There doesn't seem to be a great solution to that either.

Another issue is when the middleware pipeline is re-executed, for example when an error occurs, and the pipeline redirects to an error controller. Currently, both of those MVC executions show up as separate spans nested in the root span, each with their own tags (and timings). If we switch to a single root span, there's a question about how we can preserve that information - we can tag an array of controller+action pairs that were invoked, but it's all a bit of a mess.

Which is all to say that we're still well aware of the difficulty here, we're considering our options, but haven't worked out the fine details yet I'm afraid.

I totally get it, that isn't going to work for everybody. We don't use a lot of middleware and so the extra span that refers to just the endpoint method as opposed to the actual entry point into the API it's pretty useless for us. But if you've got a lot of middleware classes then you will lose some information about what's happening there.
I didn't see it mentioned anywhere and wanted to make sure you at least aware of it so you had the option

[matteopey]

Hi @rjune I missed your comment. I'm trying now and it looks like is working as expected.
We are also not using much the middleware part so is fine for now.

Thank you very much.

[dario-hd]

Hi! We are currently removing our old custom instrumentation to leverage the standard APM and Continuous Profiler and are experiencing the same. We want to keep things as simple as possible, and since the additional span doesn't provide much more information (all the meaningful attributes are on the parent span), we would like to remove it. Is it expected to have both spans? Is there a way to disable the apsnet_core_mvc.request span?

[subria11]

The same question, how to disable the apsnet_core_mvc.request span? Tracer version 3.8.0.

[andrewlock]

There is currently no way to disable the aspnet_core_mvc.request span. It's something we've looked into several times, but the reality is that it provides value to many customers, it would be a big breaking change to remove it, and making the span optional would add significant complexity to an already complex area of the code base. That's not to say we won't investigate again in the future of course.

One thing I'd like to highlight is that in the Datadog UI you can now run trace queries, so it's not a requirement for tags to be on the root tag for you to be able to search them. I'm wondering if this would address many of the problems/reasons for wanting to remove the span?

[subria11]

Traces are duplicated. This makes it difficult to find problems.

Yes, we can set up a corresponding filter on traces,

but it is difficult to tell our business guys about this.

[rjune]

There is currently no way to disable the aspnet_core_mvc.request span. It's something we've looked into several times, but the reality is that it provides value to many customers, it would be a big breaking change to remove it, and making the span optional would add significant complexity to an already complex area of the code base. That's not to say we won't investigate again in the future of course.

One thing I'd like to highlight is that in the Datadog UI you can now run trace queries, so it's not a requirement for tags to be on the root tag for you to be able to search them. I'm wondering if this would address many of the problems/reasons for wanting to remove the span?

When did that get disabled? Up near the top of this thread I posted a way to do the thing these people are asking about and you agreed that it would work. Though it wasn't recommended because you lose information specifically regarding middleware layers

[jlegan]

@andrewlock you can find the details of this issue, along with the referenced images in ticket https://help.datadoghq.com/hc/en-us/requests/2075312

We are coming up on another renewal and the most significant limitation/issue with the Datadog offering persists to this day. In advance of our renewal, we need a meaningful dialog and a documented workaround, solution or plan moving forward on how to address this issue. We are not the only company that has expressed these concerns. The most recent example we could find, which has updates as recent as 8 weeks ago can be found here: #4093 (comment). This latest issue thread has a proposed solution that, if I am being frank, makes the issue worse, not better in the sense that it demonstrates that Datadog is aware of the issue, has chosen not to resolve it or give the customers the necessary tools to resolve it themselves. I have a lot of respect for Andrew. I have leaned on his expertise in Microsoft's stack more than once throughout my career. I know that his explanation/intention in the thread is rooted in the value the current paradigm offers. That being said, the power to solve this challenge for your customers is within your control.

Attached to this ticket are all of the diagrams highlighting the same issue discussed in #4093 (comment). However, I am providing a significant amount of detail in the attached screenshots, which have callouts for where the problem occurs, the challenges it poses, the cases where it can be worked around and the very real cases it cannot and where Datadog APM fails us, causing us to either spend significant engineering effort on externally correlating data, using third party tools or simply not being able to have the insight we need into our own data.

At the root of the issue is the current design, which Andrew talks about in the thread, of the aspnet_core.request and aspnet_core_mvc.request span split. I completely understand and appreciate the value this brings as it differentiates the pre-request processing (i.e. middleware) and the controller processing and I am not necessarily advocating giving a customer a way to disable this and merge them. Andrew explains why this is not viable in the thread and I agree with his assessment. However, there is a viable solution here that has not been pursued by Datadog and which again we jump through hoops to solve for ourselves. The Datadog.Trace library, which is used by customers to add critical span details that allow for automations, reporting, etc. in the portal does not allow customers to, either by default or by choice, add span tags to the entry span. In the case of ASP.NET Core, this would be aspnet_core.request. When AddSpanTag is executed from within a trace context after the aspnet_core_mvc.request span has been generated, any and all span tags added, prior to that span context moving out of scope, can only be added to the aspnet_core_mvc.request span. There is no way to propagate them to the service entry span. If this was possible, the over 100 custom span tags we add to span contexts throughout our platform would all be designated as such. This simple flexibility given to customers would eliminate all of the extra engineering effort of getting the necessary span tags into the service entry (aspnet_request.core) span.

Our solution to this limitation has varied over the years. We have a significant amount of duplicated span tags that get created by dedicated middleware to ensure they are always available on the service entry span. The same method adds them again as an action filter or as part of response generation while the aspnet_core_mvc.request span is still in scope. This works but only when the span tags are known at the start of a request. Examples of this are data points derived from a claims principal or a url/path/query string. In other cases we will bypass directly adding span tags and instead add them through abstraction to the HttpContext.Items collection so that they can be read out in middleware post-controller request processing and added at that time. We have used a solution similar to how Nick at Datalust (Seq/Serilog) approached it and saw effective results. However, we were batching requests via sinks and that carried with it other baggage. Finally we have looked into modifying our cluster/Kubernetes agents to "copy" tags based on certain key words but have put that work aside for now in favor of other solutions.

All of this is to say, the current design has well known limitations, these cause real world problems for real customers and there is at least one viable solution within your control which would not break the existing paradigm, will not reduce the value or fideltity of the data and could be implemented in a way that does not violate data integrity rules. Allowing customers to propagate span tags to their parents, all the way to service entry, regardless of current nesting. Simply preventing child span tag propagation from overwriting parent tags that already exist, or even going as far as assigning a protected status to certain tags or top level tag roots (i.e. http) could be leveraged to protect customers from unwittingly modifying service entry span tag integrity.

The possible solution mentioned in #4093 (comment) is to use Trace Queries (https://docs.datadoghq.com/tracing/trace_explorer/trace_queries/). While these are a cool idea, there are many limitations to them. I will not go into too much detail here, but I will say they are both insanely complex for anyone without a deep understanding of the APM and traces to even write. They are also not available where they matter most. I have included a screenshot of a widget, in a dashboard. This widget required precisely the types of workarounds mentioned above to simply be able to write the query for and display the business critical data. This is data with 100% retention filters placed on it, which it is unclear whether trace queries use custom retention filters since the docs explicitly call out head and intelligent sampling but omit references to custom retention filters, but even if they do, you cannot use trace queries for the vast majority of use cases outside of the APM Explorer. This makes it a viable solution for some surgical querying, if you know enough about the application architecture, the limitations of the APM in .NET and how and where span tags get added and whether it was done in middleware or after the controller picked up the request... That sentence alone should highlight why trace queries are not a viable solution for this problem. They solve a problem and are another tool in my toolbag and I love it. However, it does not address this long standing issue.

After I submit this ticket, I will cross post this to Github and tag Andrew in the issue mentioned above. I will also send it to our account team. With the pending renewal, I want to leverage this to force a discussion. This can be private through the ticket/account team and a scheduled meeting. It can also be public, via Github, either in the current issue or another one if that is preferred. I hope you will help us close this gap for your customers, including us.