How to configure Healthcheck

[danutzzzzz]

Hi,

Im getting the following error randomly, how can i configure the polling/timeout values?

Health check exceeded timeout (5s)

[qdm12]

Where do you get this message from?

[qdm12]

Can you share your full logs? I'm unclear on what's the console you refer to. The healthcheck shouldn't time out, and its time values are configurable by your docker config system (not configurable by the running program within the container)

[danutzzzzz]

sorry ive been away..

When i referred to console i meant from the console logs that portainer captures... im still getting the issue with the warnings but now im not seeing that error in the logs, portainer and netdata are still alerting me though.. its the only container getting this issue

[qdm12]

Thanks for sharing more info! What are the container logs or unhealthy reason (find it with docker inspect)

[danutzzzzz]

                    "Start": "2024-03-01T12:40:39.902029651Z",
                    "End": "2024-03-01T12:40:45.095112254Z",
                    "ExitCode": -1,
                    "Output": "Health check exceeded timeout (5s)"

[danutzzzzz]

docker inspect ddns-updater
[
    {
        "Id": "6bdecf4d928fd3b1ee88c8edee09c4fec5703c2eea008a1a1e56237eb8cbc115",
        "Created": "2024-03-03T03:53:17.742918611Z",
        "Path": "/updater/app",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 5141,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2024-03-03T03:53:24.665510549Z",
            "FinishedAt": "0001-01-01T00:00:00Z",
            "Health": {
                "Status": "unhealthy",
                "FailingStreak": 2,
                "Log": [
                    {
                        "Start": "2024-03-03T12:29:54.827276771Z",
                        "End": "2024-03-03T12:29:55.268640642Z",
                        "ExitCode": 0,
                        "Output": ""
                    },
                    {
                        "Start": "2024-03-03T12:30:55.280689499Z",
                        "End": "2024-03-03T12:30:55.696660506Z",
                        "ExitCode": 0,
                        "Output": ""
                    },
                    {
                        "Start": "2024-03-03T12:31:55.708718214Z",
                        "End": "2024-03-03T12:31:56.035984353Z",
                        "ExitCode": 0,
                        "Output": ""
                    },
                    {
                        "Start": "2024-03-03T12:32:56.056296554Z",
                        "End": "2024-03-03T12:33:01.227891476Z",
                        "ExitCode": -1,
                        "Output": "Health check exceeded timeout (5s)"
                    },
                    {
                        "Start": "2024-03-03T12:34:01.244164763Z",
                        "End": "2024-03-03T12:34:06.435969489Z",
                        "ExitCode": -1,
                        "Output": "Health check exceeded timeout (5s)"
                    }
                ]
            }
        },
        "Image": "sha256:adc3d0c9a6562980102c65eabca4e754285946b1aa144da12d90add79d706539",
        "ResolvConfPath": "/mnt/ssd1/docker/docker-images/containers/6bdecf4d928fd3b1ee88c8edee09c4fec5703c2eea008a1a1e56237eb8cbc115/resolv.conf",
        "HostnamePath": "/mnt/ssd1/docker/docker-images/containers/6bdecf4d928fd3b1ee88c8edee09c4fec5703c2eea008a1a1e56237eb8cbc115/hostname",
        "HostsPath": "/mnt/ssd1/docker/docker-images/containers/6bdecf4d928fd3b1ee88c8edee09c4fec5703c2eea008a1a1e56237eb8cbc115/hosts",
        "LogPath": "/mnt/ssd1/docker/docker-images/containers/6bdecf4d928fd3b1ee88c8edee09c4fec5703c2eea008a1a1e56237eb8cbc115/6bdecf4d928fd3b1ee88c8edee09c4fec5703c2eea008a1a1e56237eb8cbc115-json.log",
        "Name": "/ddns-updater",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/mnt/ssd1/docker/docker-data/DDNS-Updater/data:/updater/data:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "bridge-new",
            "PortBindings": {
                "8000/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "8001"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                0,
                0
            ],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": [],
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/mnt/ssd1/docker/docker-images/overlay2/a8118db8ad36111647f6ebc49ee9293ae95624651ba48de13465ac7a9791bacc-init/diff:/mnt/ssd1/docker/docker-images/overlay2/0e4ffb4f67d46c421ee9b0f171d0b0e151b786f5b1d34e9b108f62a86f56f8da/diff",
                "MergedDir": "/mnt/ssd1/docker/docker-images/overlay2/a8118db8ad36111647f6ebc49ee9293ae95624651ba48de13465ac7a9791bacc/merged",
                "UpperDir": "/mnt/ssd1/docker/docker-images/overlay2/a8118db8ad36111647f6ebc49ee9293ae95624651ba48de13465ac7a9791bacc/diff",
                "WorkDir": "/mnt/ssd1/docker/docker-images/overlay2/a8118db8ad36111647f6ebc49ee9293ae95624651ba48de13465ac7a9791bacc/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/mnt/ssd1/docker/docker-data/DDNS-Updater/data",
                "Destination": "/updater/data",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "ddns-updater",
            "Domainname": "",
            "User": "1000:1000",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "8000/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "cpus=1.0",
                "memory-swap=100m",
                "BACKUP_DIRECTORY=/updater/data/backups",
                "HTTP_TIMEOUT=10s",
                "user=0",
                "TZ=Europe/London",
                "PERIOD=5m",
                "memory=100m",
                "BACKUP_PERIOD=72h0m",
                "memory-reservation=75m",
                "cpus-shares=256",
                "LISTENING_ADDRESS=:8000",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "CONFIG=",
                "UPDATE_COOLDOWN_PERIOD=5m",
                "PUBLICIP_FETCHERS=all",
                "PUBLICIP_HTTP_PROVIDERS=all",
                "PUBLICIPV4_HTTP_PROVIDERS=all",
                "PUBLICIPV6_HTTP_PROVIDERS=all",
                "PUBLICIP_DNS_PROVIDERS=all",
                "PUBLICIP_DNS_TIMEOUT=3s",
                "DATADIR=/updater/data",
                "RESOLVER_ADDRESS=",
                "RESOLVER_TIMEOUT=5s",
                "LISTENING_PORT=8000",
                "ROOT_URL=/",
                "LOG_LEVEL=info",
                "LOG_CALLER=hidden",
                "SHOUTRRR_ADDRESSES="
            ],
            "Cmd": null,
            "Healthcheck": {
                "Test": [
                    "CMD",
                    "/updater/app",
                    "healthcheck"
                ],
                "Interval": 60000000000,
                "Timeout": 5000000000,
                "StartPeriod": 10000000000,
                "Retries": 2
            },
            "Image": "qmcgaw/ddns-updater:v2.5.0",
            "Volumes": null,
            "WorkingDir": "/",
            "Entrypoint": [
                "/updater/app"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "0157295188eab693ed6e3329e29c5b39a2254976b23cb889af5c63e848c1c8f9",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "",
                "com.docker.compose.image": "sha256:adc3d0c9a6562980102c65eabca4e754285946b1aa144da12d90add79d706539",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "docker-compose",
                "com.docker.compose.project.config_files": "/mnt/ssd1/docker/docker-compose/docker-compose.yml",
                "com.docker.compose.project.working_dir": "/mnt/ssd1/docker/docker-compose",
                "com.docker.compose.replace": "623a622aa9bb8984d1959c0ac841361c2c1105a93a2f10955f593327eabe4eb6",
                "com.docker.compose.service": "ddns-updater",
                "com.docker.compose.version": "2.16.0",
                "org.opencontainers.image.authors": "[email protected]",
                "org.opencontainers.image.created": "2023-06-07T09:11:38Z",
                "org.opencontainers.image.description": "Universal DNS updater with WebUI",
                "org.opencontainers.image.documentation": "https://github.com/qdm12/ddns-updater",
                "org.opencontainers.image.revision": "a6f72d9",
                "org.opencontainers.image.source": "https://github.com/qdm12/ddns-updater",
                "org.opencontainers.image.title": "ddns-updater",
                "org.opencontainers.image.url": "https://github.com/qdm12/ddns-updater",
                "org.opencontainers.image.version": "v2.5.0"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "47ec3b7c6d1e300adddc2dfedc7cafec833f65a74f65c57322efcb87a77e8b58",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "8000/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "8001"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "8001"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/47ec3b7c6d1e",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "bridge-new": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "6bdecf4d928f",
                        "ddns-updater"
                    ],
                    "NetworkID": "74c87577f031b6f754156b220d1823b9e2462cddfc30ea1f96b1d6c4ed67180d",
                    "EndpointID": "ea93174886be9da8d5c81d2d8440bba2d82fb28438b39cfaf8f616539e4ed482",
                    "Gateway": "172.15.0.1",
                    "IPAddress": "172.15.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:0f:00:02",
                    "DriverOpts": null
                }
            }
        }
    }
]

[danutzzzzz]

Any thoughts?

to note i realised i was using an older version and updated to the latest but the issue still persists.

[qdm12]

Health check exceeded timeout seems to happen after a few successful health checks (exit code 0), and the message comes from your docker daemon and not from the ddns-updater program. Maybe your Machine is overloaded sometimes?
Try increasing the healthcheck timeout (check docker compose documentation online)?

[danutzzzzz]

I'll look at the docker compose documentation but this is the only container on this VM/Host that is reporting any issues. There are no bottlenecks or capacity issues.

[danutzzzzz]

its only been a day but since adding the default compose values for healthcheck im no longer getting these alerts.

[qdm12]

adding the default compose values for healthcheck

What do you mean by that? Did you increase the healthcheck timeout? 🤔