diff --git a/docs/core/diagnostics/distributed-tracing-builtin-activities.md b/docs/core/diagnostics/distributed-tracing-builtin-activities.md index be61a20ca3c3b..80e87f16ffb17 100644 --- a/docs/core/diagnostics/distributed-tracing-builtin-activities.md +++ b/docs/core/diagnostics/distributed-tracing-builtin-activities.md @@ -11,6 +11,9 @@ This is a reference for distributed tracing [activities](xref:System.Diagnostics ## System.Net activities +> [!TIP] +> For a comprehensive guide about collecting and reporting `System.Net` traces, see [Networking distributed traces in .NET](../../fundamentals/networking/telemetry/metrics.md). + ### HTTP client request and report the HTTP client request activity following the recommendations defined in OpenTelemetry [HTTP Client Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/http/http-spans/#http-client). It describes the HTTP request sent by 's send overloads during the time interval the underlying handler completes the request. Completing the request includes the time up to reading response headers from the network stream. It doesn't include the time spent reading the response body. may retry requests, for example, on connection failures or HTTP version downgrades. Retries are not reported as separate *HTTP client request* activities. diff --git a/docs/core/diagnostics/media/aspire-starter.png b/docs/core/diagnostics/media/aspire-starter.png new file mode 100644 index 0000000000000..8c2ed32474478 Binary files /dev/null and b/docs/core/diagnostics/media/aspire-starter.png differ diff --git a/docs/core/diagnostics/media/grafana-http11-connections.png b/docs/core/diagnostics/media/grafana-http11-connections.png new file mode 100644 index 0000000000000..0887f8c43c8ed Binary files /dev/null and b/docs/core/diagnostics/media/grafana-http11-connections.png differ diff --git a/docs/core/diagnostics/snippets/OTel-Prometheus-Grafana-Jaeger/csharp/Program.cs b/docs/core/diagnostics/snippets/OTel-Prometheus-Grafana-Jaeger/csharp/Program.cs index b595839d566ad..b4738789cf6af 100644 --- a/docs/core/diagnostics/snippets/OTel-Prometheus-Grafana-Jaeger/csharp/Program.cs +++ b/docs/core/diagnostics/snippets/OTel-Prometheus-Grafana-Jaeger/csharp/Program.cs @@ -41,6 +41,9 @@ // Metrics provides by ASP.NET Core in .NET 8 .AddMeter("Microsoft.AspNetCore.Hosting") .AddMeter("Microsoft.AspNetCore.Server.Kestrel") + // Metrics provided by System.Net libraries + .AddMeter("System.Net.Http") + .AddMeter("System.Net.NameResolution") .AddPrometheusExporter()); // Add Tracing for ASP.NET Core and our custom ActivitySource and export to Jaeger @@ -88,6 +91,24 @@ app.MapGet("/NestedGreeting", SendNestedGreeting); // +// +app.MapGet("/ClientStress", async Task (ILogger logger, HttpClient client) => +{ + string[] uris = ["http://example.com", "http://httpbin.org/get", "https://example.com", "https://httpbin.org/get"]; + await Parallel.ForAsync(0, 50, async (_, ct) => + { + string uri = uris[Random.Shared.Next(uris.Length)]; + + try + { + await client.GetAsync(uri, ct); + logger.LogInformation($"{uri} - done."); + } + catch { logger.LogInformation($"{uri} - failed."); } + }); + return "Sent 50 requests to example.com and httpbin.org."; +}); +// #if !AZURE_MONITOR // @@ -99,7 +120,7 @@ app.Run(); // -async Task SendGreeting(ILogger logger) +async Task SendGreeting(ILogger logger) { // Create a new Activity scoped to the method using var activity = greeterActivitySource.StartActivity("GreeterActivity"); diff --git a/docs/fundamentals/networking/telemetry/includes/aspire-service-defaults.md b/docs/fundamentals/networking/telemetry/includes/aspire-service-defaults.md new file mode 100644 index 0000000000000..653c293128789 --- /dev/null +++ b/docs/fundamentals/networking/telemetry/includes/aspire-service-defaults.md @@ -0,0 +1,22 @@ +The Aspire Service Defaults project provides an easy way to configure OTel for ASP.NET projects, *even if not using the rest of .NET Aspire* such as the AppHost for orchestration. The Service Defaults project is available as a project template via Visual Studio or `dotnet new`. It configures OTel and sets up the OTLP exporter. You can then use the [OTel environment variables](https://github.com/open-telemetry/opentelemetry-dotnet/tree/c94c422e31b2a5181a97b2dcf4bdc984f37ac1ff/src/OpenTelemetry.Exporter.OpenTelemetryProtocol#exporter-configuration) to configure the OTLP endpoint to send telemetry to, and provide the resource properties for the application. + +The steps to use *ServiceDefaults* outside .NET Aspire are: + +1. Add the *ServiceDefaults* project to the solution using Add New Project in Visual Studio, or use `dotnet new`: + + ```dotnetcli + dotnet new aspire-servicedefaults --output ServiceDefaults + ``` + +1. Reference the *ServiceDefaults* project from your ASP.NET application. In Visual Studio, select **Add** > **Project Reference** and select the **ServiceDefaults** project" +1. Call the OpenTelemetry setup function `ConfigureOpenTelemetry()` as part of your application builder initialization. + + ``` csharp + var builder = WebApplication.CreateBuilder(args) + builder.ConfigureOpenTelemetry(); // Extension method from ServiceDefaults. + var app = builder.Build(); + app.MapGet("/", () => "Hello World!"); + app.Run(); + ``` + +For a full walkthrough, see [Example: Use OpenTelemetry with OTLP and the standalone Aspire Dashboard](../../../../core/diagnostics/observability-otlp-example.md). diff --git a/docs/fundamentals/networking/telemetry/includes/aspire-telemetry-overview.md b/docs/fundamentals/networking/telemetry/includes/aspire-telemetry-overview.md new file mode 100644 index 0000000000000..4ee8511a97430 --- /dev/null +++ b/docs/fundamentals/networking/telemetry/includes/aspire-telemetry-overview.md @@ -0,0 +1,7 @@ +A simple way to collect traces and metrics in ASP.NET applications is to use [.NET Aspire](/dotnet/aspire/get-started/aspire-overview). .NET Aspire is a set of extensions to .NET to make it easy to create and work with distributed applications. One of the benefits of using .NET Aspire is that telemetry is built in, using the OpenTelemetry libraries for .NET. + +The default project templates for .NET Aspire contain a `ServiceDefaults` project. Each service in the .NET Aspire solution has a reference to the Service Defaults project. The services use it to set up and configure OTel. + +The Service Defaults project template includes the OTel SDK, ASP.NET, HttpClient, and Runtime Instrumentation packages. These instrumentation components are configured in the [Extensions.cs](https://github.com/dotnet/aspire/blob/main/src/Aspire.ProjectTemplates/templates/aspire-servicedefaults/Extensions.cs) file. To support telemetry visualization in Aspire Dashboard, the Service Defaults project also includes the OTLP exporter by default. + +Aspire Dashboard is designed to bring telemetry observation to the local debug cycle, which enables developers to ensure that the applications are producing telemetry. The telemetry visualization also helps to diagnose those applications locally. Being able to observe the calls between services is as useful at debug time as in production. The .NET Aspire dashboard is launched automatically when you F5 the `AppHost` Project from Visual Studio or `dotnet run` the `AppHost` project from command line. diff --git a/docs/fundamentals/networking/telemetry/media/aspire-connection_setup-thumb.png b/docs/fundamentals/networking/telemetry/media/aspire-connection_setup-thumb.png new file mode 100644 index 0000000000000..0bfb650cb0e0e Binary files /dev/null and b/docs/fundamentals/networking/telemetry/media/aspire-connection_setup-thumb.png differ diff --git a/docs/fundamentals/networking/telemetry/media/aspire-connection_setup.png b/docs/fundamentals/networking/telemetry/media/aspire-connection_setup.png new file mode 100644 index 0000000000000..4658fc75e116b Binary files /dev/null and b/docs/fundamentals/networking/telemetry/media/aspire-connection_setup.png differ diff --git a/docs/fundamentals/networking/telemetry/media/aspire-httpclient-get-thumb.png b/docs/fundamentals/networking/telemetry/media/aspire-httpclient-get-thumb.png new file mode 100644 index 0000000000000..61d44bda516a9 Binary files /dev/null and b/docs/fundamentals/networking/telemetry/media/aspire-httpclient-get-thumb.png differ diff --git a/docs/fundamentals/networking/telemetry/media/aspire-httpclient-get.png b/docs/fundamentals/networking/telemetry/media/aspire-httpclient-get.png new file mode 100644 index 0000000000000..7f41b8bd3f1ed Binary files /dev/null and b/docs/fundamentals/networking/telemetry/media/aspire-httpclient-get.png differ diff --git a/docs/fundamentals/networking/telemetry/media/aspire-metrics-thumb.png b/docs/fundamentals/networking/telemetry/media/aspire-metrics-thumb.png new file mode 100644 index 0000000000000..96e86ea04d7e0 Binary files /dev/null and b/docs/fundamentals/networking/telemetry/media/aspire-metrics-thumb.png differ diff --git a/docs/fundamentals/networking/telemetry/media/aspire-metrics.png b/docs/fundamentals/networking/telemetry/media/aspire-metrics.png new file mode 100644 index 0000000000000..173b4acf55b0c Binary files /dev/null and b/docs/fundamentals/networking/telemetry/media/aspire-metrics.png differ diff --git a/docs/fundamentals/networking/telemetry/media/aspire-starter.png b/docs/fundamentals/networking/telemetry/media/aspire-starter.png new file mode 100644 index 0000000000000..8c2ed32474478 Binary files /dev/null and b/docs/fundamentals/networking/telemetry/media/aspire-starter.png differ diff --git a/docs/fundamentals/networking/telemetry/media/connection-link.png b/docs/fundamentals/networking/telemetry/media/connection-link.png new file mode 100644 index 0000000000000..9ebdb6297a9e4 Binary files /dev/null and b/docs/fundamentals/networking/telemetry/media/connection-link.png differ diff --git a/docs/fundamentals/networking/telemetry/media/dotnet-counters.png b/docs/fundamentals/networking/telemetry/media/dotnet-counters.png new file mode 100644 index 0000000000000..bdc0521cc2b32 Binary files /dev/null and b/docs/fundamentals/networking/telemetry/media/dotnet-counters.png differ diff --git a/docs/fundamentals/networking/telemetry/media/grafana-connections.png b/docs/fundamentals/networking/telemetry/media/grafana-connections.png deleted file mode 100644 index 91f032b45ed1f..0000000000000 Binary files a/docs/fundamentals/networking/telemetry/media/grafana-connections.png and /dev/null differ diff --git a/docs/fundamentals/networking/telemetry/metrics.md b/docs/fundamentals/networking/telemetry/metrics.md index 4cf7e0ec49b2e..4e525a9d66a4a 100644 --- a/docs/fundamentals/networking/telemetry/metrics.md +++ b/docs/fundamentals/networking/telemetry/metrics.md @@ -8,27 +8,31 @@ ms.date: 11/14/2023 # Networking metrics in .NET -[Metrics](../../../core/diagnostics/metrics.md) are numerical measurements reported over time. They are typically used to monitor the health of an app and generate alerts. +[Metrics](../../../core/diagnostics/metrics.md) are numerical measurements reported over time. They're typically used to monitor the health of an app and generate alerts. Starting with .NET 8, the `System.Net.Http` and the `System.Net.NameResolution` components are instrumented to publish metrics using .NET's new [System.Diagnostics.Metrics API](../../../core/diagnostics/metrics.md). These metrics were designed in cooperation with [OpenTelemetry](https://opentelemetry.io/) to make sure they're consistent with the standard and work well with popular tools like [Prometheus](https://prometheus.io/) and [Grafana](https://grafana.com/). -They are also [multi-dimensional](../../../core/diagnostics/metrics-instrumentation.md#multi-dimensional-metrics), meaning that measurements are associated with key-value pairs called tags (a.k.a. attributes or labels) that allow data to be categorized for analysis. +They're also [multi-dimensional](../../../core/diagnostics/metrics-instrumentation.md#multi-dimensional-metrics), meaning that measurements are associated with key-value pairs called tags (also known as attributes or labels). Tags enable the categorization of the measurement to help analysis. > [!TIP] > For a comprehensive list of all built-in instruments together with their attributes, see [System.Net metrics](../../../core/diagnostics/built-in-metrics-system-net.md). ## Collect System.Net metrics -There are two parts to using metrics in a .NET app: +To take advantage of the built-in metrics instrumentation, a .NET app needs to be configured to collect these metrics. This typically means transforming them for external storage and analysis, for example, to monitoring systems. -* **Instrumentation:** Code in .NET libraries takes measurements and associates these measurements with a metric name. .NET and ASP.NET Core include many built-in metrics. -* **Collection:** A .NET app configures named metrics to be transmitted from the app for external storage and analysis. Some tools might perform configuration outside the app using configuration files or a UI tool. +There are several ways to collect networking metrics in .NET. -This section demonstrates various methods to collect and view System.Net metrics. +- For a quick overview using a simple, self-contained example, see [Collect metrics with dotnet-counters](#collect-metrics-with-dotnet-counters). +- For **production-time** metrics collection and monitoring, you can use [Grafana with OpenTelemetry and Prometheus](#view-metrics-in-grafana-with-opentelemetry-and-prometheus) or [Azure Monitor Application Insights](../../../core/diagnostics/observability-applicationinsights.md). However, these tools might be inconvenient to use at development time because of their complexity. +- For **development-time** metrics collection and troubleshooting, we recommend using [.NET Aspire](#collect-metrics-with-net-aspire), which provides a simple but extensible way to kickstart metrics and distributed tracing in your application and to diagnose issues locally. +- It's also possible to [reuse the Aspire Service Defaults](#reuse-service-defaults-project-without-net-aspire-orchestration) project without the Aspire orchestration, which is a handy way to introduce the OpenTelemetry tracing and metrics configuration APIs into your ASP.NET project. -### Example app +### Collect metrics with dotnet-counters -For the sake of this tutorial, create a simple app that sends HTTP requests to various endpoints in parallel. +[`dotnet-counters`](../../../core/diagnostics/dotnet-counters.md) is a cross-platform command line tool for ad-hoc examination of .NET metrics and first-level performance investigation. + +For the sake of this tutorial, create an app that sends HTTP requests to various endpoints in parallel. ```dotnetcli dotnet new console -o HelloBuiltinMetrics @@ -39,118 +43,102 @@ Replace the contents of `Program.cs` with the following sample code: :::code language="csharp" source="snippets/metrics/Program.cs" id="snippet_ExampleApp"::: -### View metrics with dotnet-counters - -[`dotnet-counters`](../../../core/diagnostics/dotnet-counters.md) is a cross-platform performance monitoring tool for ad-hoc health monitoring and first-level performance investigation. +Make sure `dotnet-counters` is installed: ```dotnetcli dotnet tool install --global dotnet-counters ``` -When running against a .NET 8+ process, `dotnet-counters` enables the instruments defined by the `--counters` argument and displays the measurements. It continuously refreshes the console with the latest numbers: +Start the HelloBuiltinMetrics app. -```console -dotnet-counters monitor --counters System.Net.Http,System.Net.NameResolution -n HelloBuiltinMetrics +```dotnetcli +dotnet run -c Release ``` -### View metrics in Grafana with OpenTelemetry and Prometheus - -#### Overview - -[OpenTelemetry](https://opentelemetry.io/): +Start `dotnet-counters` in a separate CLI window and specify the process name and the meters to watch, then press a key in the HelloBuiltinMetrics app so it starts sending requests. As soon as measurements start landing, `dotnet-counters` continuously refreshes the console with the latest numbers: -- Is a vendor-neutral, open-source project supported by the [Cloud Native Computing Foundation](https://www.cncf.io/). -- Standardizes generating and collecting telemetry for cloud-native software. -- Works with .NET using the .NET metric APIs. -- Is endorsed by [Azure Monitor](/azure/azure-monitor/app/opentelemetry-overview) and many APM vendors. - -This tutorial shows one of the integrations available for OpenTelemetry metrics using the OSS [Prometheus](https://prometheus.io/) and [Grafana](https://grafana.com/) projects. The metrics data flow consists of the following steps: +```console +dotnet-counters monitor --counters System.Net.Http,System.Net.NameResolution -n HelloBuiltinMetrics +``` -1. The .NET metric APIs record measurements from the example app. -1. The OpenTelemetry library running in the app aggregates the measurements. -1. The Prometheus exporter library makes the aggregated data available via an HTTP metrics endpoint. 'Exporter' is what OpenTelemetry calls the libraries that transmit telemetry to vendor-specific backends. -1. A Prometheus server: +![`dotnet-counters` output](media/dotnet-counters.png) - - Polls the metrics endpoint. - - Reads the data. - - Stores the data in a database for long-term persistence. Prometheus refers to reading and storing data as *scraping* an endpoint. - - Can run on a different machine. +### Collect metrics with .NET Aspire -1. The Grafana server: +[!INCLUDE[Aspire Telemetry Overview](./includes/aspire-telemetry-overview.md)] - - Queries the data stored in Prometheus and displays it on a web-based monitoring dashboard. - - Can run on a different machine. +#### Quick walkthrough -#### Configure the example app to use OpenTelemetry's Prometheus exporter +1. Create a **.NET Aspire 9 Starter App** by using `dotnet new`: -Add a reference to the OpenTelemetry Prometheus exporter to the example app: + ```dotnetcli + dotnet new aspire-starter-9 --output AspireDemo + ``` -```dotnetcli -dotnet add package OpenTelemetry.Exporter.Prometheus.HttpListener --prerelease -``` + Or in Visual Studio, create a new project and select the **.NET Aspire 9 Starter App** template: -> [!NOTE] -> This tutorial uses a pre-release build of OpenTelemetry's Prometheus support available at the time of writing. + ![Create a .NET Aspire 9 Starter App in Visual Studio](media/aspire-starter.png) -Update `Program.cs` with OpenTelemetry configuration: +1. Open `Extensions.cs` in the `ServiceDefaults` project, and scroll to the `ConfigureOpenTelemetry` method. Notice the `AddHttpClientInstrumentation()` call subscribing to the networking meters. -:::code language="csharp" source="snippets/metrics/Program.cs" id="snippet_PrometheusExporter" highlight="5-8"::: + :::code language="csharp" source="snippets/tracing/ConnectionTracingDemo.ServiceDefaults/Extensions.cs" id="snippet_Metrics" highlight="4"::: -In the preceding code: + Note that on .NET 8+, `AddHttpClientInstrumentation()` can be replaced by manual meter subscriptions: -- `AddMeter("System.Net.Http", "System.Net.NameResolution")` configures OpenTelemetry to transmit all the metrics collected by the built-in `System.Net.Http` and `System.Net.NameResolution` meters. -- `AddPrometheusHttpListener` configures OpenTelemetry to expose Prometheus' metrics HTTP endpoint on port `9184`. + ```csharp + .WithMetrics(metrics => + { + metrics.AddAspNetCoreInstrumentation() + .AddMeter("System.Net.Http") + .AddMeter("System.Net.NameResolution") + .AddRuntimeInstrumentation(); + }) + ``` -> [!NOTE] -> This configuration differs for ASP.NET Core apps, where metrics are exported with `OpenTelemetry.Exporter.Prometheus.AspNetCore` instead of `HttpListener`. See the [related ASP.NET Core example](/aspnet/core/log-mon/metrics/metrics#create-the-starter-app). +1. Run the `AppHost` project. This should launch the Aspire Dashboard. -Run the app and leave it running so measurements can be collected: +1. Navigate to the Weather page of the `webfrontend` app to generate an `HttpClient` request towards `apiservice`. Refresh the page several times to send multiple requests. -```dotnetcli -dotnet run -``` +1. Return to the Dashboard, navigate to the **Metrics** page, and select the `webfrontend` resource. Scrolling down, you should be able to browse the built-in `System.Net` metrics. -#### Set up and configure Prometheus + [![Networking metrics in Aspire Dashboard](media/aspire-metrics-thumb.png)](media/aspire-metrics.png#lightbox) -Follow the [Prometheus first steps](https://prometheus.io/docs/introduction/first_steps/) to set up a Prometheus server and confirm it is working. +For more information on .NET Aspire, see: -Modify the *prometheus.yml* configuration file so that Prometheus scrapes the metrics endpoint that the example app is exposing. Add the following highlighted text in the `scrape_configs` section: +- [Aspire Overview](/dotnet/aspire/get-started/aspire-overview) +- [Telemetry in Aspire](/dotnet/aspire/fundamentals/telemetry) +- [Aspire Dashboard](/dotnet/aspire/fundamentals/dashboard/explore) -:::code language="yaml" source="snippets/metrics/prometheus.yml" highlight="31-99"::: +### Reuse Service Defaults project without .NET Aspire orchestration -#### Start prometheus +[!INCLUDE[Aspire Service Defaults](./includes/aspire-service-defaults.md)] -1. Reload the configuration or restart the Prometheus server. -1. Confirm that OpenTelemetryTest is in the UP state in the **Status** > **Targets** page of the Prometheus web portal. -![Prometheus status](~/docs/core/diagnostics/media/prometheus-status.png) +### View metrics in Grafana with OpenTelemetry and Prometheus -1. On the Graph page of the Prometheus web portal, enter `http` in the expression text box and select `http_client_active_requests`. -![http_client_active_requests](~/docs/fundamentals/networking/telemetry/media/prometheus-search.png) - In the graph tab, Prometheus shows the value of the `http.client.active_requests` counter that's emitted by the example app. - ![Prometheus active requests graph](~/docs/fundamentals/networking/telemetry/media/prometheus-active-requests.png) +To see how to connect an example app with Prometheus and Grafana, follow the walkthrough in [Using OpenTelemetry with Prometheus, Grafana, and Jaeger](../../../core/diagnostics/observability-prgrja-example.md). -#### Show metrics on a Grafana dashboard +In order to stress `HttpClient` by sending parallel requests to various endpoints, extend the example app with the following endpoint: -1. Follow the [standard instructions](https://prometheus.io/docs/visualization/grafana/#installing) to install Grafana and connect it to a Prometheus data source. +:::code language="csharp" source="../../../core/diagnostics/snippets/OTel-Prometheus-Grafana-Jaeger/csharp/Program.cs" id="Snippet_ClientStress"::: -1. Create a Grafana dashboard by selecting the **+** icon on the top toolbar then selecting **Dashboard**. In the dashboard editor that appears, enter **Open HTTP/1.1 Connections** in the **Title** box and the following query in the PromQL expression field: +Create a Grafana dashboard by selecting the **+** icon on the top toolbar then selecting **Dashboard**. In the dashboard editor that appears, enter **Open HTTP/1.1 Connections** in the **Title** box and the following query in the PromQL expression field: ``` sum by(http_connection_state) (http_client_open_connections{network_protocol_version="1.1"}) ``` -![Grafana HTTP/1.1 Connections](~/docs/fundamentals/networking/telemetry/media/grafana-connections.png) +Select **Apply** to save and view the new dashboard. It displays the number of active vs idle HTTP/1.1 connections in the pool. -1. Select **Apply** to save and view the new dashboard. It displays the number of active vs idle HTTP/1.1 connections in the pool. +[![HTTP/1.1 Connections in Grafana](../../../core/diagnostics/media/grafana-http11-connections.png)](../../../core/diagnostics/media/grafana-http11-connections.png#lightbox) ## Enrichment -*Enrichment* is the addition of custom tags (a.k.a. attributes or labels) to a metric. This is useful if an app wants to add a custom categorization to dashboards or alerts built with metrics. +*Enrichment* is the addition of custom tags (also known as attributes or labels) to a metric. This is useful if an app wants to add a custom categorization to dashboards or alerts built with metrics. The [`http.client.request.duration`](../../../core/diagnostics/built-in-metrics-system-net.md#metric-httpclientrequestduration) instrument supports enrichment by registering callbacks with the . Note that this is a low-level API and a separate callback registration is needed for each `HttpRequestMessage`. A simple way to do the callback registration at a single place is to implement a custom . -This will allow you to intercept and modify the requests before they are forwarded to the inner handler and sent to the server: +This allows you to intercept and modify the requests before they're forwarded to the inner handler and sent to the server: :::code language="csharp" source="snippets/metrics/Program.cs" id="snippet_Enrichment"::: @@ -165,7 +153,7 @@ If you're working with [`IHttpClientFactory`](../../../core/extensions/httpclien ## `IMeterFactory` and `IHttpClientFactory` integration HTTP metrics were designed with isolation and testability in mind. These aspects are supported by the use of , which enables publishing metrics by a custom instance in order to keep Meters isolated from each other. -By default, all metrics are emitted by a global internal to the `System.Net.Http` library. This behavior can be overriden by assigning a custom instance to or . +By default, a global is used to emit all metrics. This internal to the `System.Net.Http` library. This behavior can be overridden by assigning a custom instance to or . > [!NOTE] > The is `System.Net.Http` for all metrics emitted by `HttpClientHandler` and `SocketsHttpHandler`. @@ -173,7 +161,7 @@ By default, all metrics are emitted by a global and assigns it to the primary handler it creates internally. > [!NOTE] -> Starting with .NET 8, the method automatically calls to initialize the metrics services and register the default implementation with . The default caches instances by name, meaning that there will be one with the name `System.Net.Http` per . +> Starting with .NET 8, the method automatically calls to initialize the metrics services and register the default implementation with . The default caches instances by name, meaning that there's one with the name `System.Net.Http` per . ### Test metrics @@ -187,7 +175,7 @@ Metrics are [more feature-rich](../../../core/diagnostics/compare-metric-apis.md Nevertheless, as of .NET 8, only the `System.Net.Http` and the `System.Net.NameResolutions` components are instrumented using Metrics, meaning that if you need counters from the lower levels of the stack such as `System.Net.Sockets` or `System.Net.Security`, you must use EventCounters. -Moreover, there are some semantical differences between Metrics and their matching EventCounters. +Moreover, there are some semantic differences between Metrics and their matching EventCounters. For example, when using `HttpCompletionOption.ResponseContentRead`, the [`current-requests` EventCounter](../../../core/diagnostics/available-counters.md) considers a request to be active until the moment when the last byte of the request body has been read. Its metrics counterpart [`http.client.active_requests`](../../../core/diagnostics/built-in-metrics-system-net.md#metric-httpclientactive_requests) doesn't include the time spent reading the response body when counting the active requests. diff --git a/docs/fundamentals/networking/telemetry/overview.md b/docs/fundamentals/networking/telemetry/overview.md index fb88a019e58cc..46a447c5392f5 100644 --- a/docs/fundamentals/networking/telemetry/overview.md +++ b/docs/fundamentals/networking/telemetry/overview.md @@ -8,9 +8,10 @@ ms.date: 10/18/2022 # Networking telemetry in .NET -The .NET networking stack is instrumented at various layers. .NET gives you the option to collect accurate timings throughout the lifetime of an HTTP request using metrics, event counters, and events. +The .NET networking stack is instrumented at various layers. .NET gives you the option to collect accurate timings throughout the lifetime of an HTTP request using metrics, distributed tracing, event counters, and events. -- **[Networking metrics](metrics.md)**: Starting with .NET 8, the HTTP and the name resolution (DNS) components are instrumented using the modern [System.Diagnostics.Metrics API](../../../core/diagnostics/metrics.md). These metrics were designed in cooperation with [OpenTelemetry](https://opentelemetry.io/). +- **[Networking metrics](metrics.md)**: Starting with .NET 8, the HTTP and the name resolution (DNS) components are instrumented using the modern [System.Diagnostics.Metrics API](../../../core/diagnostics/metrics.md). These metrics were designed in cooperation with [OpenTelemetry](https://opentelemetry.io/) and can be exported to various monitoring tools. +- **[Distributed tracing](tracing.md)**: `HttpClient` is instrumented to emit [distributed tracing](../../../core/diagnostics/distributed-tracing.md) activities (also known as spans). - **[Networking events](events.md)**: Events provide debug and trace information with accurate timestamps. - **[Networking event counters](event-counters.md)**: All networking components are instrumented to publish real-time performance metrics using the EventCounters API. diff --git a/docs/fundamentals/networking/telemetry/snippets/metrics/Program.cs b/docs/fundamentals/networking/telemetry/snippets/metrics/Program.cs index e490835f1b02a..d74c10cca6217 100644 --- a/docs/fundamentals/networking/telemetry/snippets/metrics/Program.cs +++ b/docs/fundamentals/networking/telemetry/snippets/metrics/Program.cs @@ -17,38 +17,15 @@ await Parallel.ForAsync(0, Random.Shared.Next(20), async (_, ct) => { string uri = uris[Random.Shared.Next(uris.Length)]; - byte[] bytes = await client.GetByteArrayAsync(uri, ct); - await Console.Out.WriteLineAsync($"{uri} - received {bytes.Length} bytes."); + try + { + byte[] bytes = await client.GetByteArrayAsync(uri, ct); + await Console.Out.WriteLineAsync($"{uri} - received {bytes.Length} bytes."); + } + catch { await Console.Out.WriteLineAsync($"{uri} - failed."); } }); } // -#elif snippet_PrometheusExporter -// -using OpenTelemetry.Metrics; -using OpenTelemetry; -using System.Net; - -using MeterProvider meterProvider = Sdk.CreateMeterProviderBuilder() - .AddMeter("System.Net.Http", "System.Net.NameResolution") - .AddPrometheusHttpListener(options => options.UriPrefixes = new string[] { "http://localhost:9184/" }) - .Build(); - -string[] uris = ["http://example.com", "http://httpbin.org/get", "https://example.com", "https://httpbin.org/get"]; -using HttpClient client = new() -{ - DefaultRequestVersion = HttpVersion.Version20 -}; - -while (!Console.KeyAvailable) -{ - await Parallel.ForAsync(0, Random.Shared.Next(20), async (_, ct) => - { - string uri = uris[Random.Shared.Next(uris.Length)]; - byte[] bytes = await client.GetByteArrayAsync(uri, ct); - await Console.Out.WriteLineAsync($"{uri} - received {bytes.Length} bytes."); - }); -} -// #elif snippet_Enrichment // using System.Net.Http.Metrics; diff --git a/docs/fundamentals/networking/telemetry/snippets/tracing/ConnectionTracingDemo.ServiceDefaults/ConnectionTracingDemo.ServiceDefaults.csproj b/docs/fundamentals/networking/telemetry/snippets/tracing/ConnectionTracingDemo.ServiceDefaults/ConnectionTracingDemo.ServiceDefaults.csproj new file mode 100644 index 0000000000000..5d08206bbd6f2 --- /dev/null +++ b/docs/fundamentals/networking/telemetry/snippets/tracing/ConnectionTracingDemo.ServiceDefaults/ConnectionTracingDemo.ServiceDefaults.csproj @@ -0,0 +1,22 @@ + + + + net9.0 + enable + enable + true + + + + + + + + + + + + + + + diff --git a/docs/fundamentals/networking/telemetry/snippets/tracing/ConnectionTracingDemo.ServiceDefaults/Extensions.cs b/docs/fundamentals/networking/telemetry/snippets/tracing/ConnectionTracingDemo.ServiceDefaults/Extensions.cs new file mode 100644 index 0000000000000..eb6a5822eb0d8 --- /dev/null +++ b/docs/fundamentals/networking/telemetry/snippets/tracing/ConnectionTracingDemo.ServiceDefaults/Extensions.cs @@ -0,0 +1,125 @@ +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Diagnostics.HealthChecks; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Diagnostics.HealthChecks; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.ServiceDiscovery; +using OpenTelemetry; +using OpenTelemetry.Metrics; +using OpenTelemetry.Trace; + +namespace Microsoft.Extensions.Hosting; + +// Adds common .NET Aspire services: service discovery, resilience, health checks, and OpenTelemetry. +// This project should be referenced by each service project in your solution. +// To learn more about using this project, see https://aka.ms/dotnet/aspire/service-defaults +public static class Extensions +{ + public static TBuilder AddServiceDefaults(this TBuilder builder) where TBuilder : IHostApplicationBuilder + { + builder.ConfigureOpenTelemetry(); + + builder.AddDefaultHealthChecks(); + + builder.Services.AddServiceDiscovery(); + + builder.Services.ConfigureHttpClientDefaults(http => + { + // Turn on resilience by default + http.AddStandardResilienceHandler(); + + // Turn on service discovery by default + http.AddServiceDiscovery(); + }); + + // Uncomment the following to restrict the allowed schemes for service discovery. + // builder.Services.Configure(options => + // { + // options.AllowedSchemes = ["https"]; + // }); + + return builder; + } + + public static TBuilder ConfigureOpenTelemetry(this TBuilder builder) where TBuilder : IHostApplicationBuilder + { + builder.Logging.AddOpenTelemetry(logging => + { + logging.IncludeFormattedMessage = true; + logging.IncludeScopes = true; + }); + + builder.Services.AddOpenTelemetry() + // + .WithMetrics(metrics => + { + metrics.AddAspNetCoreInstrumentation() + .AddHttpClientInstrumentation() + .AddRuntimeInstrumentation(); + }) + // + // + .WithTracing(tracing => + { + tracing.AddAspNetCoreInstrumentation() + // Instead of using .AddHttpClientInstrumentation() + // .NET 9 allows to add the ActivitySources directly. + .AddSource("System.Net.Http") + // Add the experimental connection tracking ActivitySources using a wildcard. + .AddSource("Experimental.System.Net.*"); + }); + // + + builder.AddOpenTelemetryExporters(); + + return builder; + } + + + private static TBuilder AddOpenTelemetryExporters(this TBuilder builder) where TBuilder : IHostApplicationBuilder + { + var useOtlpExporter = !string.IsNullOrWhiteSpace(builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"]); + + if (useOtlpExporter) + { + builder.Services.AddOpenTelemetry().UseOtlpExporter(); + } + + // Uncomment the following lines to enable the Azure Monitor exporter (requires the Azure.Monitor.OpenTelemetry.AspNetCore package) + //if (!string.IsNullOrEmpty(builder.Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"])) + //{ + // builder.Services.AddOpenTelemetry() + // .UseAzureMonitor(); + //} + + return builder; + } + + public static TBuilder AddDefaultHealthChecks(this TBuilder builder) where TBuilder : IHostApplicationBuilder + { + builder.Services.AddHealthChecks() + // Add a default liveness check to ensure app is responsive + .AddCheck("self", () => HealthCheckResult.Healthy(), ["live"]); + + return builder; + } + + public static WebApplication MapDefaultEndpoints(this WebApplication app) + { + // Adding health checks endpoints to applications in non-development environments has security implications. + // See https://aka.ms/dotnet/aspire/healthchecks for details before enabling these endpoints in non-development environments. + if (app.Environment.IsDevelopment()) + { + // All health checks must pass for app to be considered ready to accept traffic after starting + app.MapHealthChecks("/health"); + + // Only health checks tagged with the "live" tag must pass for app to be considered alive + app.MapHealthChecks("/alive", new HealthCheckOptions + { + Predicate = r => r.Tags.Contains("live") + }); + } + + return app; + } +} diff --git a/docs/fundamentals/networking/telemetry/tracing.md b/docs/fundamentals/networking/telemetry/tracing.md new file mode 100644 index 0000000000000..2f35ba5742287 --- /dev/null +++ b/docs/fundamentals/networking/telemetry/tracing.md @@ -0,0 +1,140 @@ +--- +title: Networking tracing +description: Learn how to consume .NET networking Tracing. +author: samsp-msft +ms.author: samsp +ms.date: 10/4/2024 +--- + +# Distributed tracing in System.Net libraries + +[Distributed tracing](../../../core/diagnostics/distributed-tracing.md) is a diagnostic technique that helps engineers localize failures and performance issues within applications, especially those that are distributed across multiple machines or processes. This technique tracks requests through an application by correlating together work done by different components and separating it from other work the application might be doing for concurrent requests. For example, a request to a typical web service might be first received by a load balancer and then forwarded to a web server process, which then makes several queries to a database. Distributed tracing allows engineers to distinguish if any of those steps failed and how long each step took. It can also log messages produced by each step as it ran. + +The tracing system in .NET is designed to work with OpenTelemetry (OTel), and uses OTel to export the data to monitoring systems. Tracing in .NET is implemented using the APIs, where a unit of work is represented by the class, which corresponds to an OTel [span](https://opentelemetry.io/docs/concepts/signals/traces/#spans). OpenTelemetry defines an industry-wide standard naming scheme for spans (activities) together with their attributes (tags), known as [semantic conventions](https://opentelemetry.io/docs/concepts/semantic-conventions). The .NET telemetry uses existing semantic conventions wherever possible. + +> [!NOTE] +> The terms *span* and *activity* are synonymous in this article. In context of .NET code, they refer to a instance. Don't confuse the OTel span with . + +> [!TIP] +> For a comprehensive list of all built-in activities together with their tags/attributes, see [Built-in activities in .NET](../../../core/diagnostics/distributed-tracing-builtin-activities.md). + +## Instrumentation + +To emit traces, the libraries are [instrumented](../../../core/diagnostics/distributed-tracing-instrumentation-walkthroughs.md#add-basic-instrumentation) with built-in sources, which create objects to track the work performed. Activities are only created if there are listeners subscribed to the . + +The built-in instrumentation evolved with .NET versions. + +- On .NET 8 and earlier, the instrumentation is limited to the creation of an empty [HTTP client request activity](../../../core/diagnostics/distributed-tracing-builtin-activities.md#http-client-request). This means that users have to rely on the [`OpenTelemetry.Instrumentation.Http`](https://github.com/open-telemetry/opentelemetry-dotnet-contrib/tree/1ca05685cbad63d3fa813b9cab49be341048e69e/src/OpenTelemetry.Instrumentation.Http#httpclient-and-httpwebrequest-instrumentation-for-opentelemetry) library to populate the activity with the information (for example, tags) needed to emit useful traces. +- .NET 9 extended the instrumentation by emitting the name, status, exception info, and the most important tags according to the OTel [HTTP client semantic conventions](https://opentelemetry.io/docs/specs/semconv/http/http-spans/#http-client) on the HTTP client request activity. This means that on .NET 9+, the `OpenTelemetry.Instrumentation.Http` dependency can be omitted, unless more advanced features like [enrichment](#enrichment) are required. +- .NET 9 also introduced [experimental connection tracing](#experimental-connection-tracing), adding new activities across the `System.Net` libraries to support diagnosing connection issues. + +## Collect System.Net traces + +At the [lowest level](../../../core/diagnostics/distributed-tracing-collection-walkthroughs.md#collect-traces-using-custom-logic), trace collection is supported via the method, which registers objects containing user-defined logic. + +However, as an application developer, you would likely prefer to rely on the rich ecosystem built upon the features provided by the [OpenTelemetry .NET SDK](https://opentelemetry.io/docs/languages/net/) to collect, export, and monitor traces. + +- To get a fundamental understanding on trace collection with OTel, see our guide on [collecting traces using OpenTelemetry](../../../core/diagnostics/distributed-tracing-collection-walkthroughs.md#collect-traces-using-opentelemetry). +- For **production-time** trace collection and monitoring, you can use OpenTelemetry with [Prometheus, Grafana, and Jaeger](../../../core/diagnostics/observability-prgrja-example.md) or with [Azure Monitor and Application Insights](../../../core/diagnostics/observability-applicationinsights.md). However, these tools are quite complex and might be inconvenient to use at development time. +- For **development-time** trace collection and monitoring, we recommend using [.NET Aspire](#collect-traces-with-net-aspire) which provides a simple but extensible way to kickstart distributed tracing in your application and to diagnose issues locally. +- It's also possible to [reuse the Aspire Service Defaults](#reuse-service-defaults-project-without-net-aspire-orchestration) project without the Aspire orchestration. This is a handy way to introduce and configure OpenTelemetry tracing and metrics in your ASP.NET projects. + +### Collect traces with .NET Aspire + +[!INCLUDE[Aspire Telemetry Overview](./includes/aspire-telemetry-overview.md)] + +[![Aspire Dashboard](../../../core/diagnostics/media/aspire-dashboard-thumb.png)](../../../core/diagnostics/media/aspire-dashboard.png#lightbox) + +For more information on .NET Aspire, see: + +- [Aspire Overview](/dotnet/aspire/get-started/aspire-overview) +- [Telemetry in Aspire](/dotnet/aspire/fundamentals/telemetry) +- [Aspire Dashboard](/dotnet/aspire/fundamentals/dashboard/explore) + +### Reuse Service Defaults project without .NET Aspire Orchestration + +[!INCLUDE[Aspire Service Defaults](./includes/aspire-service-defaults.md)] + +## Experimental connection tracing + +When troubleshooting `HttpClient` issues or bottlenecks, it might be crucial to see where time is being spent when sending HTTP requests. Often, the problem occurs during HTTP connection establishment, which typically breaks down to DNS lookup, TCP connection, and TLS handshake. + +.NET 9 introduced experimental connection tracing adding an `HTTP connection setup` span with three child spans representing the DNS, TCP, and TLS phases of the connection establishment. The HTTP part of the connection tracing is implemented within , meaning that the activity model has to respect the underlying connection pooling behavior. + +> [!NOTE] +> In , connections and requests have independent lifecycles. A [pooled connection](../http/httpclient-guidelines.md#pooled-connections) can live for a long time and serve many requests. When making a request, if there's no connection immediately available in the connection pool, the request is added to a request queue to wait for an available connection. There's no direct relationship between waiting requests and connections. The connection process might have started when another connection became available for use, in which case the freed connection is used. As a result, the `HTTP connection setup` span isn't modeled as a child of the `HTTP client request` span; instead, span links are used. + +.NET 9 introduced the following spans to enable collecting detailed connection information: + +| Name | | Description | +|---|---|---| +| [`HTTP wait_for_connection`](../../../core/diagnostics/distributed-tracing-builtin-activities.md#http-client-request-wait-for-connection-experimental) | `Experimental.System.Net.Http.Connections` | A child span of the `HTTP client request` span that represents the time interval the request is waiting for an available connection in the request queue. | +| [`HTTP connection_setup`](../../../core/diagnostics/distributed-tracing-builtin-activities.md#http-connection-setup-experimental) | `Experimental.System.Net.Http.Connections` | Represents the establishment of the HTTP connection. A separate trace root span with its own `TraceId`. `HTTP client request` spans might contain links to `HTTP connection_setup`. | +| [`DNS lookup`](../../../core/diagnostics/distributed-tracing-builtin-activities.md#dns-lookup-experimental) | `Experimental.System.Net.NameResolution` | DNS lookup performed by the class. | +| [`socket connect`](../../../core/diagnostics/distributed-tracing-builtin-activities.md#socket-connect-experimental) | `Experimental.System.Net.Sockets` | Establishment of a connection. | +| [`TLS handshake`](../../../core/diagnostics/distributed-tracing-builtin-activities.md#tls-handshake-experimental) | `Experimental.System.Net.Security` | TLS client or server handshake performed by . | + +> [!NOTE] +> The corresponding `ActivitySource` names start with the prefix `Experimental`, as these spans might be changed in future versions as we learn more about how well they work in production. + +These spans are too verbose for use 24x7 in production scenarios with high workloads - they're noisy and this level of instrumentation isn't normally needed. However, if you're trying to diagnose connection issues or get a deeper understanding of how network and connection latency is affecting your services, then they provide insight that's hard to collect by other means. + +When the `Experimental.System.Net.Http.Connections` ActivitySource is enabled, *the `HTTP client request` span contains a link to the `HTTP connection_setup` span corresponding to the connection serving the request*. As an HTTP connection can be long lived, this could result in many links to the connection span from each of the request activities. Some APM monitoring tools aggressively walk links between spans to build up their views, and so including this span can cause issues when the tools weren't designed to account for large numbers of links. + +The following diagram illustrates the behavior of the spans and their relationship: + +[![Connection spans over time.](media/connection-link.png)](media/connection-link.png#lightbox) + +### Walkthrough: Using the experimental connection tracing in .NET 9 + +This walkthrough uses a [.NET 9 Aspire Starter App](/dotnet/aspire/get-started/build-your-first-aspire-app) to demonstrate connection tracing, but it should be easy to set it up with [other monitoring tools](#collect-systemnet-traces) as well. The key step is to enable the ActivitySources. + +1. Create a **.NET Aspire 9 Starter App** by using `dotnet new`: + + ```dotnetcli + dotnet new aspire-starter-9 --output ConnectionTracingDemo + ``` + + Or in Visual Studio: + + ![Create a .NET Aspire 9 Starter App in Visual Studio](media/aspire-starter.png) + +1. Open `Extensions.cs` in the `ServiceDefaults` project, and edit the `ConfigureOpenTelemetry` method adding the ActivitySources for connection in the tracing configuration callback: + + :::code language="csharp" source="snippets/tracing/ConnectionTracingDemo.ServiceDefaults/Extensions.cs" id="snippet_ConnectionTracing"::: + +1. Start the solution. This should open the [.NET Aspire Dashboard](/dotnet/aspire/fundamentals/dashboard/overview). + +1. Navigate to the Weather page of the `webfrontend` app to generate an `HttpClient` request towards `apiservice`. + +1. Return to the Dashboard and navigate to the **Traces** page. Open the `webfrontend: GET /weather` trace. + + [![HttpClient Spans in Aspire Dashboard](media/aspire-httpclient-get-thumb.png)](media/aspire-httpclient-get.png#lightbox) + +When HTTP requests are made with the connection instrumentation enabled, you should see the following changes to the client request spans: + +- If a connection needs to be established, or if the app is waiting for a connection from the connection pool, then an additional [`HTTP wait_for_connection`](../../../core/diagnostics/distributed-tracing-builtin-activities.md#http-client-request-wait-for-connection-experimental) span is shown, which represents the delay for waiting for a connection to be made. This helps to understand delays between the `HttpClient` request being made in code, and when the processing of the request actually starts. In the previous image: + - The selected span is the HttpClient request. + - The span below represents the time the request spends waiting for a connection to be established. + - The last span in yellow is from the destination processing the request. +- The HttpClient span will have a link to the [`HTTP connection_setup`](../../../core/diagnostics/distributed-tracing-builtin-activities.md#http-connection-setup-experimental) span, which represents the activity to create the HTTP connection used by the request. + +[![Connection setup spans in Aspire Dashboard](media/aspire-connection_setup-thumb.png)](media/aspire-connection_setup.png#lightbox) + +[As mentioned previously](#experimental-connection-tracing), the [`HTTP connection_setup`](../../../core/diagnostics/distributed-tracing-builtin-activities.md#http-connection-setup-experimental) span is a separate span with its own `TraceId`, as its lifetime is independent from each individual client request. This span typically has child spans [`DNS lookup`](../../../core/diagnostics/distributed-tracing-builtin-activities.md#dns-lookup-experimental), (TCP) [`socket connect`](../../../core/diagnostics/distributed-tracing-builtin-activities.md#socket-connect-experimental), and [`TLS client handshake`](../../../core/diagnostics/distributed-tracing-builtin-activities.md#tls-handshake-experimental). + +## Enrichment + +In some cases, it's necessary to augment the existing `System.Net` tracing functionality. Typically this means injecting additional tags/attributes to the built-in activities. This is called *enrichment*. + +### Enrichment API in the OpenTelemetry instrumentation library + +To add additional tags/attributes to the HTTP client request activity, the simplest approach is to use the [`HttpClient` enrichment APIs](https://github.com/open-telemetry/opentelemetry-dotnet-contrib/blob/main/src/OpenTelemetry.Instrumentation.Http/README.md#enrich-httpclient-api) of the OpenTelemetry HttpClient and HttpWebRequest instrumentation library. This requires taking a dependency on the [`OpenTelemetry.Instrumentation.Http`](https://www.nuget.org/packages/OpenTelemetry.Instrumentation.Http) package. + +### Manual enrichment + +It's possible to implement the enrichment of the `HTTP client request` activity manually. For this you need to access in the code that is running in the scope of the request activity, before the activity is finished. This can be done by implementing an `IObserver` and subscribing it to to get callbacks for when networking activity is occurring. In fact, this is how the [OpenTelemetry HttpClient and HttpWebRequest instrumentation library](#enrichment-api-in-the-opentelemetry-instrumentation-library) is implemented. For a code example, see the subscription code in [`DiagnosticSourceSubscriber.cs`](https://github.com/open-telemetry/opentelemetry-dotnet-contrib/blob/bfdf02928ea7f958c481493bc1adc238472b9b9c/src/Shared/DiagnosticSourceSubscriber.cs#L44) and the underlying implementation in [HttpHandlerDiagnosticListener.cs](https://github.com/open-telemetry/opentelemetry-dotnet-contrib/blob/bfdf02928ea7f958c481493bc1adc238472b9b9c/src/OpenTelemetry.Instrumentation.Http/Implementation/HttpHandlerDiagnosticListener.cs) where the notifications are delegated to. + +## Need more tracing? + +If you have suggestions for other useful information that could be exposed via tracing, create a [dotnet/runtime issue](https://github.com/dotnet/runtime/issues/new). diff --git a/docs/fundamentals/toc.yml b/docs/fundamentals/toc.yml index f3c21c755d52a..fa95fe12107df 100644 --- a/docs/fundamentals/toc.yml +++ b/docs/fundamentals/toc.yml @@ -1162,6 +1162,9 @@ items: - name: Metrics href: networking/telemetry/metrics.md displayName: networking metrics + - name: Distributed tracing + href: networking/telemetry/tracing.md + displayName: networking distributed tracing - name: Events href: networking/telemetry/events.md displayName: networking events