Plugin is trying to check the cluster status after draining the nodes

[patademahesh]

Hi @jsiebens

First of all, thank you so much for creating this plugin.

We are testing this plugin to scale droplets in DigitalOcean. The droplet is getting scaled as per the defined policy but after scaling down and draining the node the plugin starts giving the below warning continuously. The plugin is actually trying to monitor the health of the drained droplet and also the droplet is not getting removed from DigitalOcean.

2022-09-30T13:55:08.537Z [WARN] external_plugin.do-droplets: node pool status readiness check failed: error="node c92595ac-08f2-7686-75fc-5fb90e29f478 is ineligible" timestamp=2022-09-30T13:55:08.537Z

Scaling policy:

scaling "app-redis-slave" {
  enabled = true
  min     = 1
  max     = 2

  policy {
    cooldown            = "1m"
    evaluation_interval = "10s"

    check "redis_jobs_in_progess" {
      source = "prometheus"
      query  = "sum(nomad_nomad_job_summary_queued{exported_job=~\"app\",task_group=\"redis-slave\"} + nomad_nomad_job_summary_running{exported_job=~\"app\",task_group=\"redis-slave\"}) OR on() vector(0)"

      strategy "pass-through" {}
    }

    target "do-droplets" {
      name = "app-redis-slave"
      region = "xxx1"
      size = "s-1vcpu-1gb"
      snapshot_id = "XXXXXX"
      user_data = "local/do-user_data.sh"
      tags = "hashi-stack,job:app,group:redis-slave"

      datacenter             = "do"
      node_drain_deadline    = "15m"
      node_drain_ignore_system_jobs = true
      node_selector_strategy = "empty_ignore_system"
    }
  }
}