Kubernetes probe fails when creating a dump

[ckotzbauer]

Description

I tried to add the monitor-container as sidecar to my ASP.NET Core application. The main-container in the deployment also has Kubernetes livenessProbe and readinessProbe configured. When creating a dump with the monitor (calling the /dump) endpoint, the main-process of the application is suspended as stated in the docs. This causes the probes to fail, so Kubernetes will restart the application before the dump-creation succeeds.

Is there any configuration I can try to make this work? I cannot remove the probes in a production application. 😉
If this is not possible with dotnet-monitor at the moment, is there another way to create a dump or do other analysis to a running Kubernetes pod without exec to the running container and run some preinstalled tools/scripts?

Configuration

ASP.NET Core version: 3.1.17 (based on mcr.microsoft.com/dotnet/core/aspnet:3.1-alpine)
Dotnet Monitor: 5.0.0-preview.6.21370.3+9c1714943e349b437ca3adeb8f4f8cb0df1355b3
OS: Alpine (both application and monitor)
Arch: x64
Env: Bare-Metal Kubernetes-Cluster (no cloud-provider)

Kubernetes Deployment YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: feature-diagnostics-api
spec:
  selector:
    matchLabels: {}
  template:
    spec:
      containers:
      - image: our.aspnet-core-application:latest
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /configuration/getVersion
            port: http
            scheme: HTTP
          initialDelaySeconds: 90
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: api
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        resources:
          limits:
            cpu: 1500m
            memory: 1Gi
          requests:
            cpu: 250m
            memory: 400Mi
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /configuration/getVersion
            port: http
            scheme: HTTP
          initialDelaySeconds: 90
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        securityContext:
          capabilities:
            drop:
            - ALL
          privileged: false
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - mountPath: /tmp
          name: diagnostics
      - args:
        - --urls
        - https://*:52323
        env:
        - name: DotnetMonitor_Metrics__Enabled
          value: "false"
        - name: DotnetMonitor_Kestrel__Certificates__Default__Password
          valueFrom:
            secretKeyRef:
              key: pass
              name: monitorcert
        - name: DotnetMonitor_Kestrel__Certificates__Default__Path
          value: /etc/aspnet/customcert/cert.pfx
        - name: DotnetMonitor_ApiAuthentication__ApiKeyHashType
          value: SHA256
        - name: DotnetMonitor_ApiAuthentication__ApiKeyHash
          valueFrom:
            secretKeyRef:
              key: ApiAuthentication__ApiKeyHash
              name: monitorkey
        image: mcr.microsoft.com/dotnet/monitor:5.0.0-preview.6-alpine
        imagePullPolicy: IfNotPresent
        name: monitor-sidecar
        ports:
        - containerPort: 52323
          name: http
          protocol: TCP
        securityContext:
          capabilities:
            drop:
            - ALL
          privileged: false
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - mountPath: /tmp
          name: diagnostics
        - mountPath: /etc/aspnet/customcert
          name: customcert
      volumes:
      - emptyDir: {}
        name: diagnostics
      - name: customcert
        secret:
          defaultMode: 420
          secretName: monitorcert

Regression?

Not tested.

Other information

Nothing, please ask if some info is missing.

[gabbersepp]

A short update here. We finally were able to create full memory dumps within a few seconds. Christian had the problems mentioned above when he uses heap dumps (default value of dotnet-monitor /dump endpoint). After switching to full dumps it only takes a few seconds.
But nevertheless it would be nice if there are some official best practices. I think the time the dump takes will depend on the overall workload. As we have not migrated the production systems into our cluster we currently can not make any assumptions how long it will take in the future

[jaya-rama]

Description

I tried to add the monitor-container as sidecar to my ASP.NET Core application. The main-container in the deployment also has Kubernetes livenessProbe and readinessProbe configured. When creating a dump with the monitor (calling the /dump) endpoint, the main-process of the application is suspended as stated in the docs. This causes the probes to fail, so Kubernetes will restart the application before the dump-creation succeeds.

Is there any configuration I can try to make this work? I cannot remove the probes in a production application. 😉
If this is not possible with dotnet-monitor at the moment, is there another way to create a dump or do other analysis to a running Kubernetes pod without exec to the running container and run some preinstalled tools/scripts?

Configuration

ASP.NET Core version: 3.1.17 (based on mcr.microsoft.com/dotnet/core/aspnet:3.1-alpine)
Dotnet Monitor: 5.0.0-preview.6.21370.3+9c1714943e349b437ca3adeb8f4f8cb0df1355b3
OS: Alpine (both application and monitor)
Arch: x64
Env: Bare-Metal Kubernetes-Cluster (no cloud-provider)

Kubernetes Deployment YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: feature-diagnostics-api
spec:
  selector:
    matchLabels: {}
  template:
    spec:
      containers:
      - image: our.aspnet-core-application:latest
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /configuration/getVersion
            port: http
            scheme: HTTP
          initialDelaySeconds: 90
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: api
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        resources:
          limits:
            cpu: 1500m
            memory: 1Gi
          requests:
            cpu: 250m
            memory: 400Mi
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /configuration/getVersion
            port: http
            scheme: HTTP
          initialDelaySeconds: 90
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        securityContext:
          capabilities:
            drop:
            - ALL
          privileged: false
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - mountPath: /tmp
          name: diagnostics
      - args:
        - --urls
        - https://*:52323
        env:
        - name: DotnetMonitor_Metrics__Enabled
          value: "false"
        - name: DotnetMonitor_Kestrel__Certificates__Default__Password
          valueFrom:
            secretKeyRef:
              key: pass
              name: monitorcert
        - name: DotnetMonitor_Kestrel__Certificates__Default__Path
          value: /etc/aspnet/customcert/cert.pfx
        - name: DotnetMonitor_ApiAuthentication__ApiKeyHashType
          value: SHA256
        - name: DotnetMonitor_ApiAuthentication__ApiKeyHash
          valueFrom:
            secretKeyRef:
              key: ApiAuthentication__ApiKeyHash
              name: monitorkey
        image: mcr.microsoft.com/dotnet/monitor:5.0.0-preview.6-alpine
        imagePullPolicy: IfNotPresent
        name: monitor-sidecar
        ports:
        - containerPort: 52323
          name: http
          protocol: TCP
        securityContext:
          capabilities:
            drop:
            - ALL
          privileged: false
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - mountPath: /tmp
          name: diagnostics
        - mountPath: /etc/aspnet/customcert
          name: customcert
      volumes:
      - emptyDir: {}
        name: diagnostics
      - name: customcert
        secret:
          defaultMode: 420
          secretName: monitorcert

Regression?

Not tested.

Other information

Nothing, please ask if some info is missing.

Hey any update on this??