Automatically restart unhealthy containers

[Vahor]

Coolify already has a healthcheck feature. But currently it's only used to inform traefik whether to route requests to the app.

This feature would require:

• On advanced panel, add a checkbox saying we want to automatically restart unhealthy service. (on docker-compose I'd say it could be per service)
• When coolify status check reports that the service is unhealthy, it restart it (clean stop, start)
• It might start a restart loop, but if we can have notifications for when an app is unhealthy and restarted this won't be an issue as we'll be able to manually stop the service if needed.

The requirement is to use the healthcheck feature and not only check if the container is present or not . (it can be both, restart if the container is not present or unhealthy)

[nuxwin]

@Vahor

A little bash script on your host should do the job (no tested). You need run it in background.

#!/bin/bash

sleep 60

while true; do
  # Get the list of unhealthy containers that have a healthcheck defined
  unhealthy_containers=$(docker ps --filter "health=unhealthy" --format "{{.ID}} {{.Names}}" | while read container_id container_name; do
    # Check if the container has a healthcheck defined by inspecting its health status
    if [ "$(docker inspect --format='{{.State.Health.Status}}' "$container_id")" != "none" ]; then
      echo "$container_id"
    fi
  done)

  if [ -n "$unhealthy_containers" ]; then
    echo "Restarting unhealthy containers: $unhealthy_containers"
    docker restart $unhealthy_containers
  fi

  # Wait before checking again
  sleep 60
done

Basically put, every minute, the script check for container status (only those providing healthcheck will be processed for restart. You could add logic for notification inside this script of course.

[kravetsone]

Please i want it out of the box...

[Cinzya]

Do you actually have an app that needs a restart to become healthy again? The majority of the time, a unhealthy container is caused by misconfigured or a bug in your app. So a restart wouldn't really fix this without manual changes required to the config or your code.

You should know if your app crashes, coolify will actually perform a restart. But a unhealthy container doesn't necessarily mean that there was a hard crash.

[kravetsone]

i agree that if i need restart when unhealthy probably i do something wrong. But on other hand i have no time to properly debug&fix this issue so optional feature restart-when-unhealthy is great thing to temporal workaround

[Domysee]

For example, my use-case is OpenPanel, the API container sometimes gets into an unhealthy state and a restart fixes it. Since this is 3rd party software there's not much I can do (within reasonable time), so the auto-restart of unhealthy containers would be godsent.

[filipeportes]

The default restartPolicy for a Kubernetes Pod is Always, so for me It is quite surprising that a unhealthy container will just go down and not even try to restart in Coolify

[Cinzya]

@filipeportes Coolify sets the docker restart policy to unless-stopped but Dockers restart policy only fires when the main process in the container either does clean exit, crashes, OOM kill (exit 137), SIGTERM (143), etc.
When a container is unhealthy that means the underlying process has a problem but is still running, thus it's not triggering a restart.

[banjo]

Looking for this as well. I have a quite big app in production that sometimes stop working whilst running. Not sure why, but it will take some time to fix it.

Currently, I have to use a third party service to find out that it is not working and then restart it manually which is not very convenient.

[robbedgdev]

Please add this, it would be convenient