Stale stats accumulate over time

[jsgosh]

🔍 Check for existing issues

• Completed

How is Dozzle deployed?

Agents

📦 Dozzle version

8.14.6

✅ Command used to run Dozzle

docker run --name dozzle-agent --rm --detach -v /var/run/docker.sock:/var/run/docker.sock -p 7007:7007 amir20/dozzle:latest agent --no-analytics

🐛 Describe the bug / provide steps to reproduce it

After a week of uptime (and many docker-compose stacks having been brought up/down) the number of containers and their stats appear inflated compared to reality. For example, the dashboard shows ~1000 containers when there are less than 500. Specific containers that have ceased to exist may also remain in the lists. This drift can be corrected by replacing each running Dozzle agent.

It has been noticeable for several releases now, but since updating resets everything I wanted to wait and see if the latest version was still affected. Finally I decided to report the issue anyway.

💻 Environment

Client: Docker Engine - Community
 Version:    28.5.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.29.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.40.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 480
  Running: 459
  Paused: 0
  Stopped: 21
 Images: 84
 Server Version: 28.5.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b98a3aace656320842a23f4a392a33f46af97866
 runc version: v1.3.0-0-g4ca628d1
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-86-generic
 Operating System: Ubuntu 24.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 186.3GiB
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

📜 Dozzle logs are required for debugging purposes. You may need to enable debug mode. See https://dozzle.dev/guide/debugging.

No response

📸 If applicable, add screenshots to help explain your bug

No response

[amir20]

I think I am having a hard time understanding this. Something doesn't make sense because no one else has reported this issue. I would expect for others to see it.

For example, the dashboard shows ~1000 containers when there are less than 500. Specific containers that have ceased to exist may also remain in the lists. This drift can be corrected by replacing each running Dozzle agent.

Are you saying there are more containers in Dozzle than they actually exist?

What state are they containers in? If you click on them what happens?

I would need debug logs to investigate more.

It has been noticeable for several releases now, but since updating resets everything I wanted to wait and see if the latest version was still affected. Finally I decided to report the issue anyway.

This logic hasn't changed for a long time now. So I don't think it was introduced recently.

In addition to debug logs, if you can run docker events when containers are being destroyed would be good.

[jsgosh]

Are you saying there are more containers in Dozzle than they actually exist?

Yes. I'll try to provide more information next time.

If you click on them what happens?

The log viewer showed a placeholder of some kind.

In addition to debug logs, if you can run docker events when containers are being destroyed would be good.

I'll run in debug mode until it happens again. Logging the docker events contextually is a trickier problem, but I can solve that later because it doesn't require restarting any Dozzle containers.

[amir20]

I'll run in debug mode until it happens again. Logging the docker events contextually is a trickier problem, but I can solve that later because it doesn't require restarting any Dozzle containers.

I just mostly care about the events that are emitted by Docker. It might be something I am ignoring. But debug logs would show this too.

[amir20]

Haven't heard back for a while. Going to convert this to a discussion. I'll need more info to debug. Make sure you have debug logs.

[jsgosh]

46f7fb3f809a is a container that no longer exists.

Output snipped from docker logs dozzle-agent 2>&1 | grep 46f7fb3f809a:

{"level":"debug","version":"v8.14.7","id":"46f7fb3f809a","time":"1970-01-01T14:25:12Z","message":"Finding container"}
... following three lines repeat every 10 seconds up to this point ...
{"level":"debug","version":"v8.14.7","event":"exec_create: curl --fail -sLo /dev/null https://localhost:20613/_health","id":"46f7fb3f809a","time":"1970-01-01T14:26:13Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","event":"exec_start: curl --fail -sLo /dev/null https://localhost:20613/_health","id":"46f7fb3f809a","time":"1970-01-01T14:26:13Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","event":"exec_die","id":"46f7fb3f809a","time":"1970-01-01T14:26:13Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","container":"46f7fb3f809a","host":"[redacted]","time":"1970-01-01T14:26:14Z","message":"starting to stream stats"}
... following three lines repeat every 10 seconds up to this point ...
{"level":"debug","version":"v8.14.7","event":"exec_create: curl --fail -sLo /dev/null https://localhost:20613/_health","id":"46f7fb3f809a","time":"1970-01-02T01:57:15Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","event":"exec_start: curl --fail -sLo /dev/null https://localhost:20613/_health","id":"46f7fb3f809a","time":"1970-01-02T01:57:15Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","event":"exec_die","id":"46f7fb3f809a","time":"1970-01-02T01:57:15Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","container":"46f7fb3f809a","host":"[redacted]","error":"context canceled","time":"1970-01-02T01:57:20Z","message":"stopping to stream stats"}
... following three lines repeat every 10 seconds up to this point ...
{"level":"debug","version":"v8.14.7","event":"exec_create: curl --fail -sLo /dev/null https://localhost:20613/_health","id":"46f7fb3f809a","time":"1970-01-02T08:28:11Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","event":"exec_start: curl --fail -sLo /dev/null https://localhost:20613/_health","id":"46f7fb3f809a","time":"1970-01-02T08:28:11Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","event":"exec_die","id":"46f7fb3f809a","time":"1970-01-02T08:28:11Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","container":"46f7fb3f809a","host":"[redacted]","time":"1970-01-02T08:28:17Z","message":"starting to stream stats"}
... following three lines repeat every 10 seconds up to this point ...
{"level":"debug","version":"v8.14.7","event":"exec_create: curl --fail -sLo /dev/null https://localhost:20613/_health","id":"46f7fb3f809a","time":"1970-01-02T11:07:25Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","event":"exec_start: curl --fail -sLo /dev/null https://localhost:20613/_health","id":"46f7fb3f809a","time":"1970-01-02T11:07:25Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","event":"exec_die","id":"46f7fb3f809a","time":"1970-01-02T11:07:25Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","event":"exec_create: curl --fail -sLo /dev/null https://localhost:20613/_health","id":"46f7fb3f809a","time":"1970-01-02T12:20:13Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","event":"exec_start: curl --fail -sLo /dev/null https://localhost:20613/_health","id":"46f7fb3f809a","time":"1970-01-02T12:20:13Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","event":"exec_die","id":"46f7fb3f809a","time":"1970-01-02T12:20:13Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","id":"46f7fb3f809a","since":"1970-01-01T00:00:00Z","stdType":"all","host":"[redacted]","time":"1970-01-08T11:25:24Z","message":"Streaming logs for container"}

Events like "stop", "die", and "destroy" appear in the log for other container IDs but not this one. "oom" does not appear at all, and the nearest "kill" events are 3+ hours before/after the final health check. There are 1252 total lines logged at the timestamp corresponding to "1970-01-02T12:20:13Z".

A noticeable hour-long gap in all events is oddly bracketed by another container's healthcheck with no intervening messages:

{"level":"debug","version":"v8.14.7","event":"exec_create: check.sh ","id":"b215b671262e","time":"1970-01-02T11:07:30Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","event":"exec_start: check.sh ","id":"b215b671262e","time":"1970-01-02T11:07:30Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","event":"exec_die","id":"b215b671262e","time":"1970-01-02T12:20:13Z","message":"received container event"}

check.sh is another simple curl, not any shenanigans, as demonstrated by the end of that container's life a day later:

... following three lines repeat every 10 seconds up to this point ...
{"level":"debug","version":"v8.14.7","event":"exec_create: check.sh ","id":"b215b671262e","time":"1970-01-03T09:32:34Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","event":"exec_start: check.sh ","id":"b215b671262e","time":"1970-01-03T09:32:34Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","event":"exec_die","id":"b215b671262e","time":"1970-01-03T09:32:34Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","event":"kill","id":"b215b671262e","time":"1970-01-03T09:32:40Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","event":"stop","id":"b215b671262e","time":"1970-01-03T09:32:40Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","event":"die","id":"b215b671262e","time":"1970-01-03T09:32:40Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","id":"b215b671262e","time":"1970-01-03T09:32:40Z","message":"container died"}
{"level":"debug","version":"v8.14.7","event":"destroy","id":"b215b671262e","time":"1970-01-03T09:32:41Z","message":"received container event"}
{"level":"debug","version":"v8.14.7","id":"b215b671262e","time":"1970-01-03T09:32:41Z","message":"container destroyed"}

The dashboard counts 637 containers, contradicting the output of docker system info. This discrepancy only increases over time.

Server:
 Containers: 489
  Running: 489
  Paused: 0
  Stopped: 0
 Images: 85

[jsgosh]

I am going to create a VM locally with 500 containers. Then every minutes I am going to create a container and destroy it. I'll check back in a couple hours to see that would reproduce it.

That sounds like a reasonable test, though I suggest ensuring the containers also produce exec_create, exec_start, and exec_die events every 10 seconds or sooner. I don't think this issue is actually "over time" though time was needed for passive manual detection.

I’d like to clarify the main issue: I don’t see a kill event for 46f7fb3f809a.

Yes, I was not able to find any terminating events for that container. I suspect that if I could, then Dozzle would not still be listing/counting it. Docker docs for docker system events denote a rolling limit of 256 entries; perhaps the number of events produced by application health checks can randomly flood Docker's API buffer faster than Dozzle's polling rate.

Please let me know if you can provide the full log.

I don't feel comfortable sharing detailed application logs when the number of entries exceeds my ability to manually review all information, but I did try filtering the data from multiple angles before assembling the excerpts above.

Notably, on the day corresponding to 1970-01-02 all entries within several hours of that final healthcheck contained the id property. The only messages appearing in conjunction with that property are as follows:

• Finding container
• container health status changed
• received container event
• container died
• container destroyed

Excluding entries containing id over the whole day, there is a recurring pattern of the following messages:

• Listing containers
• Fetching host
• starting to stream stats
• stopping to stream stats

That last one is sometimes associated with a GET "http:///var/run/docker.sock/v1.51/containers/<id>/stats?stream=1": context canceled error, otherwise just context canceled or EOF.

One message shows up 24 times throughout the day, none proximate to the critical hour.

• container renamed

Having excluded all of the above, the remaining messages each appear once, 3+ hours before the final healthcheck.

• stopped container stats collector
• starting to listen to docker events

I left an agent running over the weekend and its dashboard counter climbed to 1000+ containers as most active containers were rotated. If there's no easy fix I'm content to work around the issue by scripting a daily Dozzle restart.

[amir20]

Hi there,

I wish I had better news. My test on my VM has been running all day yesterday and today. I started 400 containers. I created an additional container for Dozzle agent. Then I created a Dozzle UI and connected to the agent. I created a new container and killed it every 5 minutes. At all times, Dozzle reported between 401 and 402 containers. Which is to be expected.

Thanks for being patient! ❤️ I have been going through the code a lot looking for edge cases. I haven't been able to find anything that would explain this behavior.

I may have a workaround though! See my responses below for the rest.

exec_create, exec_start, and exec_die events every 10 seconds or sooner.

This might be an idea. Not all my containers had health. Perhaps that is flooding it.

Docker docs for docker system events denote a rolling limit of 256 entries

This could be true. But I pick events up as fast as possible. If this is true, then perhaps Dozzle should poll every minute. I would want to avoid this because list all containers is an expensive operation. Perhaps adding exec_create with health would reproduce this.

I don't feel comfortable sharing detailed application logs when the number of entries exceeds my ability to manually

I am looking for any error. I assume there isn't if you didn't find it which means the store manager for Dozzle is working as expected. You'd see "docker store unexpectedly disconnected" at error.

I did notice something interesting though. As you said, there was a log for stopping to stream stats.

{"level":"debug","version":"v8.14.7","container":"46f7fb3f809a","host":"[redacted]","error":"context canceled","time":"1970-01-02T01:57:20Z","message":"stopping to stream stats"}

This typically happens when the container is stopped. I could use this event to also check if the container has stopped and update the internal database (container_store). But this is perplexing because I would expect the event to be received in both places.

So tldr; I have no idea why the store is not being updated. Even more confusing 😵‍💫 why has no one else reported this. There are plenty of people using Dozzle with 200+ containers. This makes me wonder if it's a bug with Dozzle or Docker 🤔

The only workaround solution could be to also look at the container event when the stats stream has ended.

What do you think? Can you think of any reason why this is only happening for you?

As next steps:

1. I am going to try adding health to 400+ containers and flood the store.
2. If I still can't reproduce it then I can try to rely on the stat collector ending as a fallback signal.

[amir20]

Alright testing now! 500+ containers with

{"level":"debug","version":"v8.14.9","event":"exec_create: /bin/sh -c ps aux | grep -v grep | grep echo || exit 1","id":"e12724829b3e","time":"2025-11-24T19:07:14Z","message":"received container event"}

If the bug exists where docker events is being flooded then over time 503 containers should increase.

[amir20]

Well, no luck. I let it run all day with 503 containers and health checks. I couldn't reproduce any of the issues. Everything worked smoothly, and Dozzle was using about 5%.

I'm out of ideas. I think the only way to fix it is by using the fallback plan I described above. However, it will bother me since I can't reproduce the problem. Something is different about your setup and I can't figure out why.

[jsgosh]

However, it will bother me since I can't reproduce the problem.

Sorry to bother you.

Something is different about your setup and I can't figure out why.

It is mysterious. I have Docker barely customized with only an expanded IP range and a nominal cgroup-parent (not imposing any configuration) in daemon.json. Maybe a quirk of my Compose definition or application workload can be blamed.

If someone has a similar problem they might find this discussion. Thank you for looking into it with me.