CPU Utilization Spikes on Node only when Dozzle is Running

[mitchnemirov]

Check for existing issues

• Completed

How is Dozzle deployed?

Swarm Mode

Dozzle version

8.12.4

✅ Command used to run Dozzle

services:
  app:
    image: amir20/dozzle:latest
    deploy:
      mode: global
      labels:
        - traefik.enable=true
        - traefik.docker.network=proxy
        - traefik.http.routers.dozzle.rule=Host(`logs.docker.domain`)
        - traefik.http.routers.dozzle.entrypoints=websecure
        - traefik.http.routers.dozzle.tls=true
        - traefik.http.routers.dozzle.tls.certresolver=letsencrypt
        - traefik.http.services.dozzle.loadbalancer.server.port=8080
    environment:
      - DOZZLE_MODE=swarm
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - proxy
      - dozzle

networks:
  proxy:
    external: true
  dozzle:
    driver: overlay

Describe the performance issue you are seeing

Possibly related to #3140

Shortly after the Dozzle service starts up across all our nodes, CPU usage spikes on one of them (most likely the leader at that time). CPU usage grows until the node is basically unusable, and causes services running on the affected node to flatline and/or timeout. The only solution is to stop Dozzle, drain and restart the docker.socket service on the affected node, then bring the node back to active.

How many containers, hosts, agents, etc. are you using?

We have three nodes running in Docker Swarm mode, with Dozzle running in swarm mode with deployment set to global. We have anywhere from 30-50 containers running at a time, and have recently cleaned up inactive images and containers.

Screenshots, videos, or logs

No response

[amir20]

Can you help investigate a few things?

1. Run as a single node on the host that is experiencing spikes. We are trying to determine if this is related to swarm mode or just Dozzle.
2. Issue System becomes irresponsive, 100% resource utilization if high container count #3140 concerned having many inactive containers. With 30 to 50, that is small compared to what others are running. I doubt it’s the same issue.
3. Enable debug mode, and the next time it spikes, go to <host>/debug. Please send me the goroutines and heap.
4. How much memory is Dozzle using when it spikes? And does it only happen when you use Dozzle in the browser or even before that?

I'll run in swarm mode with 300 containers when I get a chance and see if I can reproduce it.

[amir20]

I tested with 200 containers. Dozzle was running smoothly using about ~40MB. Can't say I can reproduce this. Will close if I don't hear anything.

[mitchnemirov]

@amir20 We have a maintenance window next week where I can replicate the issue and perform debug collection. I have little info immediately otherwise, as I had to disable the Dozzle stack to prevent further issues with our prod environment.

Memory usage was relatively low/stable across the board, it was just host CPU that was being spiked indefinitely. If it helps, the process that was pegging the CPU was /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock, but immediately subsided once Dozzle was turned off.

[amir20]

That's strange. I would expect the Dozzle CPU usage to be high as well.

Enabling debug logs would be helpful. The only thing that could cause this issue is if you are running thousands of containers. However, I'll wait until you have debug enabled. I wonder if Dozzle is getting stuck in a loop, which could be overloading Docker.