Runtime metrics not generated

[WeihanLi]

Description

When I try to use the dotnet-monitor as a sidecar container, the runtime metrics is not generated

I could only get the following metrics:

It seemed only have microsoftaspnetcorehosting related provider? Not sure if I'm wrong with some config

Configuration

deployment yaml https://github.com/WeihanLi/SparkTodo/blob/82ba4ac7493afcb476d096cf11f482e9a297003a/sparktodo-api-k8s-deploy.yaml#L20

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sparktodo-api
  labels:
    app: sparktodo-api
spec:
  replicas: 1
  revisionHistoryLimit: 0
  selector:
    matchLabels:
      app: sparktodo-api
  minReadySeconds: 0
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1

  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "52323"
      labels:
        app: sparktodo-api
    
    spec:
      containers:
        - name: sparktodo-api
          image: weihanli/sparktodo-api:latest
          imagePullPolicy: Always
          resources:
            requests:
              memory: "64Mi"
              cpu: "20m"
            limits:
              memory: "128Mi"
              cpu: "50m"
          env:
          - name: ASPNETCORE_URLS
            value: http://+:80
          - name: DOTNET_DiagnosticPorts
            value: /diag/port
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /health
              port: 80
            initialDelaySeconds: 60
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /health
              port: 80
            initialDelaySeconds: 60
            periodSeconds: 30
          volumeMounts:
          - mountPath: /diag
            name: diagvol
          - mountPath: /dumps
            name: dumpsvol
        - name: monitor
          image: mcr.microsoft.com/dotnet/monitor:6.0.0
          # DO NOT use the --no-auth argument for deployments in production
          args: [ "--no-auth" ]
          imagePullPolicy: Always
          ports:
            - containerPort: 52323
          env:
          - name: DOTNETMONITOR_DiagnosticPort__ConnectionMode
            value: Listen
          - name: DOTNETMONITOR_DiagnosticPort__EndpointName
            value: /diag/port
          - name: DOTNETMONITOR_Storage__DumpTempFolder
            value: /dumps
          - name: DOTNETMONITOR_Urls
            value: "http://*:52323"
          volumeMounts:
          - mountPath: /diag
            name: diagvol
          - mountPath: /dumps
            name: dumpsvol
          resources:
            requests:
              cpu: 50m
              memory: 32Mi
            limits:
              cpu: 100m
              memory: 256Mi
          securityContext:
            capabilities:
              add: ["SYS_PTRACE"]
      volumes:
      - name: diagvol
        emptyDir: {}
      - name: dumpsvol
        emptyDir: {}

[WeihanLi]

I try to download the trace data, and there's runtime counters data in the trace

[johan-bjerling]

I'm experiencing the same issue when I run dotnet-monitor as a sidecar in AKS.

I'm able add other providers, e.g. System.Net.Http and Microsoft-AspNetCore-Server-Kestrel and I get metrics for them.
I even tried to add System.Runtime manually, both with IncludeDefaultProviders set to true and to false, but to no avail.

[jander-msft]

Hey Folks, make sure that you are using the latest dotnet-monitor version (this is easily done if you are using a floating tag e.g. 6 or 6-alpine, you may need to configure your deployment to always pull the image in order to get a newer version for these floating tags). We released an update in August that may contain a fix for this issue:

• Release 6.2.2: https://github.com/dotnet/dotnet-monitor/releases/tag/v6.2.2
• Relevant Change: [release/6.2] Fix duplicated values for matric group #2177

[johan-bjerling]

Hey Folks, make sure that you are using the latest dotnet-monitor version (this is easily done if you are using a floating tag e.g. 6 or 6-alpine, you may need to configure your deployment to always pull the image in order to get a newer version for these floating tags). We released an update in August that may contain a fix for this issue:

• Release 6.2.2: https://github.com/dotnet/dotnet-monitor/releases/tag/v6.2.2
• Relevant Change: [release/6.2] Fix duplicated values for matric group #2177

@jander-msft Just tried this again with the latest version of dotnet-monitor running in a side-car in AKS with default providers. But I can't seem to get any metrics other than these I'm afraid:

/app # dotnet-monitor --version
6.2.2+8abeb94c15ee4175d7078a3f768ed1ef15032bd8
/app # curl http://localhost:52323/metrics
# HELP microsoftaspnetcorehosting_requests_per_second Request Rate
# TYPE microsoftaspnetcorehosting_requests_per_second gauge
microsoftaspnetcorehosting_requests_per_second 6 1663747451248
microsoftaspnetcorehosting_requests_per_second 8 1663747456248
microsoftaspnetcorehosting_requests_per_second 7 1663747461248
# HELP microsoftaspnetcorehosting_total_requests Total Requests
# TYPE microsoftaspnetcorehosting_total_requests gauge
microsoftaspnetcorehosting_total_requests 307 1663747451248
microsoftaspnetcorehosting_total_requests 315 1663747456248
microsoftaspnetcorehosting_total_requests 322 1663747461248
# HELP microsoftaspnetcorehosting_current_requests Current Requests
# TYPE microsoftaspnetcorehosting_current_requests gauge
microsoftaspnetcorehosting_current_requests 0 1663747451248
microsoftaspnetcorehosting_current_requests 0 1663747456248
microsoftaspnetcorehosting_current_requests 0 1663747461248
# HELP microsoftaspnetcorehosting_failed_requests Failed Requests
# TYPE microsoftaspnetcorehosting_failed_requests gauge
microsoftaspnetcorehosting_failed_requests 1 1663747451248
microsoftaspnetcorehosting_failed_requests 1 1663747456248
microsoftaspnetcorehosting_failed_requests 1 1663747461248

[xsoheilalizadeh]

@johan-bjerling I'm using following configuration and it works for me.

- name: monitor
  securityContext:
    runAsUser: 1000
  image: mcr.microsoft.com/dotnet/monitor:6-alpine
  args: [ "--no-auth" ]
  imagePullPolicy: IfNotPresent
  ports:
    - name: metrics
      containerPort: 52323
  env:
    - name: DOTNETMONITOR_DiagnosticPort__ConnectionMode
      value: "Listen"
    - name: DOTNETMONITOR_DiagnosticPort__EndpointName
      value: "/diag/port.sock"
    - name: DOTNETMONITOR_Storage__DumpTempFolder
      value: "/diag/dumps"
    - name: DOTNETMONITOR_Urls
      value: http://+:52323
    - name: DefaultProcess__Filters__0__Key 
      value: "ProcessName"
    - name: DefaultProcess__Filters__0__Value 
      value: "dotnet"
    - name: DotnetMonitor_Metrics__Providers__0__ProviderName
      value: "Microsoft-AspNetCore-Server-Kestrel"     
    - name: DotnetMonitor_Metrics__Providers__1__ProviderName
      value: "Microsoft.AspNetCore.Http.Connections"    
    - name: DotnetMonitor_Metrics__Providers__2__ProviderName
      value: "System.Net.Http"    
    - name: DotnetMonitor_Metrics__Providers__3__ProviderName
      value: "System.Net.NameResolution"    
    - name: DotnetMonitor_Metrics__Providers__4__ProviderName
      value: "System.Net.Security"    
    - name: DotnetMonitor_Metrics__Providers__5__ProviderName
      value: "System.Net.Sockets"    
  volumeMounts:
    - mountPath: /diag
      name: diagvol
  resources:
    requests:
      memory: "32Mi"
      cpu: "50m"
    limits:
      memory: "256Mi"
      cpu: "250m"    

[johan-bjerling]

@xsoheilalizadeh Yep, I can get all of those metrics to work for me as well. Unfortunately neither adding "System.Runtime" explicitly or relying on it implicitly being added via the default providers seem to work.

[jander-msft]

@xsoheilalizadeh Yep, I can get all of those metrics to work for me as well. Unfortunately neither adding "System.Runtime" explicitly or relying on it implicitly being added via the default providers seem to work.

Could you provide the full configuration of your dotnet-monitor instance? If you exec into the dotnet-monitor container and execute dotnet-monitor config show, that would provide the configuration as understood by the running instance.

#2730 may be related

[johan-bjerling]

@jander-msft sure, here's the full config:

{
  "urls": "http://localhost:52323",
  "Kestrel": ":NOT PRESENT:",
  "Templates": ":NOT PRESENT:",
  "CollectionRuleDefaults": ":NOT PRESENT:",
  "GlobalCounter": {
    "IntervalSeconds": "5"
  },
  "CollectionRules": ":NOT PRESENT:",
  "CorsConfiguration": ":NOT PRESENT:",
  "DiagnosticPort": {
    "ConnectionMode": "Listen",
    "DeleteEndpointOnStartup": "true",
    "EndpointName": "/diag/port.sock"
  },
  "Metrics": {
    "Enabled": "True",
    "Endpoints": "http://localhost:52325",
    "IncludeDefaultProviders": "True",
    "MetricCount": "3",
    "Providers": [
      {
        "ProviderName": "Microsoft-AspNetCore-Server-Kestrel"
      },
      {
        "ProviderName": "Microsoft.AspNetCore.Http.Connections"
      },
      {
        "ProviderName": "System.Net.Http"
      },
      {
        "ProviderName": "System.Net.NameResolution"
      },
      {
        "ProviderName": "System.Net.Security"
      },
      {
        "ProviderName": "System.Net.Sockets"
      },
      {
        "ProviderName": "System.Runtime"
      }
    ]
  },
  "Storage": {
    "DumpTempFolder": "/diag/dumps"
  },
  "DefaultProcess": {
    "Filters": [
      {
        "Key": "ProcessId",
        "ProcessName": "dotnet",
        "Value": "1"
      }
    ]
  },
  "Logging": {
    "Console": {
      "FormatterName": "json",
      "FormatterOptions": {
        "IncludeScopes": "True",
        "TimestampFormat": "yyyy-MM-ddTHH:mm:ss.fffffffZ",
        "UseUtcTimestamp": "true"
      }
    },
    "EventLog": {
      "LogLevel": {
        "Default": "Information",
        "Microsoft": "Warning",
        "Microsoft.Diagnostics": "Information",
        "Microsoft.Hosting.Lifetime": "Information"
      }
    },
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning",
      "Microsoft.Diagnostics": "Information",
      "Microsoft.Diagnostics.Monitoring.WebApi.OutputStreamResult": "Warning",
      "Microsoft.Hosting.Lifetime": "Information"
    }
  },
  "Authentication": ":NOT PRESENT:",
  "Egress": {
    "Properties": {
      "MonitorBlobAccountKey": ":REDACTED:"
    },
    "AzureBlobStorage": {
      "MonitorBlob": {
        "AccountUri": "redacted",
        "BlobPrefix": "dotnetmetrics",
        "ContainerName": "storeapi",
        "CopyBufferSize": ":NOT PRESENT:",
        "QueueName": ":NOT PRESENT:",
        "QueueAccountUri": ":NOT PRESENT:",
        "SharedAccessSignature": ":NOT PRESENT:",
        "AccountKey": ":NOT PRESENT:",
        "SharedAccessSignatureName": ":NOT PRESENT:",
        "AccountKeyName": "MonitorBlobAccountKey",
        "ManagedIdentityClientId": ":NOT PRESENT:"
      }
    },
    "FileSystem": ":NOT PRESENT:"
  }

[jander-msft]

Couple of thoughts here:

• Is anyone using OpenTelemetry within their application to gather metrics? It might be possible that OpenTelemetry is setting a counter interval that's different that what dotnet-monitor is using; dotnet-monitor will filter out any counters that do not match the configured global counter interval (its default value is 5).
• Is your application trimmed? Someone reported that trimmed applications are not reporting System.Runtime counters: dotnet-counters: Missing System.Runtime-Counters when publishing a trimmed app runtime#79648

[johan-bjerling]

Couple of thoughts here:

• Is anyone using OpenTelemetry within their application to gather metrics? It might be possible that OpenTelemetry is setting a counter interval that's different that what dotnet-monitor is using; dotnet-monitor will filter out any counters that do not match the configured global counter interval (its default value is 5).
• Is your application trimmed? Someone reported that trimmed applications are not reporting System.Runtime counters: dotnet-counters: Missing System.Runtime-Counters when publishing a trimmed app runtime#79648

My app was not trimmed. It was however using OpenTelemetry. But as we couldn't get the dotnet-monitor sidecar to work quite how we wanted it, that's now been removed. So I don't have an easy way of testing this again without OpenTelemetry.

I suspect I won't have the time in the near future to try this out, but I will report back if I do!
Thanks for the ideas though @jander-msft, hopefully they'll be of help to others!

[Krishnadas-KP]

I have the same problem. Unable to get system.runtime metrics in the /metrics endpoint. This is my config

{
  "urls": "http://localhost:52323",
  "Kestrel": ":NOT PRESENT:",
  "Templates": ":NOT PRESENT:",
  "CollectionRuleDefaults": ":NOT PRESENT:",
  "GlobalCounter": {
    "IntervalSeconds": "5"
  },
  "CollectionRules": ":NOT PRESENT:",
  "CorsConfiguration": ":NOT PRESENT:",
  "DiagnosticPort": {
    "ConnectionMode": "Listen",
    "DeleteEndpointOnStartup": "true"
  },
  "InProcessFeatures": ":NOT PRESENT:",
  "Metrics": {
    "Enabled": "True",
    "Endpoints": "http://\u002B:52325",
    "IncludeDefaultProviders": "True",
    "MetricCount": "3"
  },
  "Storage": {
    "DefaultSharedPath": "/diag"
  },
  "DefaultProcess": {
    "Filters": [
      {
        "Key": "ProcessId",
        "Value": "1"
      }
    ]
  },
  "Logging": {
    "Console": {
      "FormatterName": "json",
      "FormatterOptions": {
        "IncludeScopes": "True",
        "TimestampFormat": "yyyy-MM-ddTHH:mm:ss.fffffffZ",
        "UseUtcTimestamp": "true"
      }
    },
    "EventLog": {
      "LogLevel": {
        "Default": "Information",
        "Microsoft": "Warning",
        "Microsoft.Diagnostics": "Information",
        "Microsoft.Hosting.Lifetime": "Information"
      }
    },
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning",
      "Microsoft.Diagnostics": "Information",
      "Microsoft.Hosting.Lifetime": "Information"
    }
  },
  "Authentication": ":NOT PRESENT:",
  "Egress": ":NOT PRESENT:"

I am using version 7.0.2 as a sidecar in a pod in AKS.