Inconsistent metrics being gathered from ASP.NET Core applications

[joaofbantunes]

Description

Disclaimer: no idea if I should file this here or in the ASP.NET Core repo, but hopefully if it's not the right place, you can direct me there.

I was trying to gather some metrics using dotnet monitor, in order to feed them into Prometheus. While doing this, working with Docker Compose, I noticed that I wasn't able to get all the Kestrel related metrics (as seen here).

When running directly on Windows though, all the metrics I expected were there. To try to get to the bottom of it, removing any possible specificities of my application, ran the same tests using just the result of dotnet new web. On Windows, everything looked fine, but running it on WSL, I got the same issues as I saw in the container.

To try to create a repro to share with you, went down a rabbit hole of testing different scenarios, getting very inconsistent results:

Running on Windows

To run on Windows, executed:

• dotnet new web -o Default && cd Default && dotnet run --environment Production --urls http://*:8000
• dotnet-monitor collect --no-auth --urls "http://*:52323" --configuration-file-path "./monitor-config.json"
  (you can find the monitor-config.json file here)

The output of this run can be seen here.

Running on WSL

To run on WSL, executed exactly the same as above, with the results available here.

Running in a container

To run on Docker, executed:

• docker run --rm -it -p 8000:80 --mount "source=dotnet-tmp,target=/tmp" mcr.microsoft.com/dotnet/sdk:7.0-alpine /bin/sh -c "dotnet new web -o App && cd App && dotnet run -c Release --environment Production --urls http://*:80"
• docker run -e DotnetMonitor_Metrics__Providers__0__ProviderName="Microsoft-AspNetCore-Server-Kestrel" -e DotnetMonitor_Metrics__Providers__1__ProviderName="Microsoft.AspNetCore.Hosting" -e DotnetMonitor_Metrics__Providers__2__ProviderName="System.Runtime" -it --rm -p 52323:52323 --mount "source=dotnet-tmp,target=/tmp" mcr.microsoft.com/dotnet/monitor collect --urls http://*:52323 --no-auth

The output of this run can be seen here.

Running the ASP.NET Core sample app in a container

Also remembered to try the ASP.NET Core sample app:

• docker run --rm -it -p 8000:80 --mount "source=dotnet-tmp,target=/tmp" mcr.microsoft.com/dotnet/samples:aspnetapp
• docker run -e DotnetMonitor_Metrics__Providers__0__ProviderName="Microsoft-AspNetCore-Server-Kestrel" -e DotnetMonitor_Metrics__Providers__1__ProviderName="Microsoft.AspNetCore.Hosting" -e DotnetMonitor_Metrics__Providers__2__ProviderName="System.Runtime" -it --rm -p 52323:52323 --mount "source=dotnet-tmp,target=/tmp" mcr.microsoft.com/dotnet/monitor collect --urls http://*:52323 --no-auth

The output of this run can be seen here.

As you can see, the various results are inconsistent, not at all what I was expecting. Don't if it's a bug, or an environmental thing, but I have no idea what might be the issue.

Configuration

Windows information: Windows 11 Pro, Version 22H2, OS build 22621.1105, Experience Windows Feature Experience Pack 1000.22638.1000.0

Ubuntu version in WSL: Ubuntu 20.04.5 LTS

Docker Desktop version: 4.16.3

.NET Version on Windows: 7.0.200-preview.22628.1

.NET Version in Ubuntu: 7.0.102

Other information

I also tried on macOS, and had similar results to Windows.

[kkeirstead]

Hey João, thanks for providing so much info for the repro! Quick question:

In your "Running in a container" example, it looks like you're running on top of the SDK image (as opposed to running on top of the ASP.NET runtime). Is this representative of how you're running your actual application? While investigating, we did a Docker Multi-Stage Build that created a new app using docker new web App on top of the SDK image, then copied the published folder on top of the runtime image. Doing this, we produced the metrics that are missing from the container example.

If that's not the issue, feel free to provide some more details for how you're building your actual image and we can dig a little deeper. Thanks!

[joaofbantunes]

Hi @kkeirstead,

In your "Running in a container" example, it looks like you're running on top of the SDK image (as opposed to running on top of the ASP.NET runtime). Is this representative of how you're running your actual application?

No, it was just me trying to create the simplest possible repro 🙂. In any case, should it matter? Even if running on the SDK image, which would be similar to running locally, I'd expect the same metrics to be reported.

I tried to create a slightly more complete and realistic sample, you can find it here.

It's composed of an API and a Worker project, where the Worker is hammering the API. You should be able to run it quickly with Docker Compose, then access the API metrics on http://localhost:6001/metrics.

I added a couple of sample outputs from the metrics endpoint, one from running it locally on Windows, the other when running with Docker Compose. As you can see, the Windows output includes more information, namely when we get to microsoftaspnetcoreserverkestrel_*.