Improve Nomad UI interface: Show correct count of job instances

[EugenKon]

Proposal

On the command line we see that Nomad want to create one additional job, but from Nomad UI we see there are 42 jobs.

It is not clear what is going on: why numbers differ?

More screenshots

[philrenaud]

Hi @EugenKon ! This receives the groupCountSum from our backend. In rough terms, for service jobs, this tallies up all the group.count for all groups in the job.

Your screenshots are showing a single group here. Can you confirm whether or not there are more groups than the one you're showing?

[EugenKon]

wi knecht is only one group inside this job.

[philrenaud]

That's very curious; would you mind passing me the job spec? Relevant parts for me would be count and scaling rules, but I wonder if there's something non-obvious to me here.

[EugenKon]

variable "min_count" {
  type    = number
  default = 1
}

variable "max_count" {
  type    = number
  default = 10
}

job "wi-knecht" {
  region      = "planitar"
  datacenters = ["plntr_dc"]
  type        = "service"

  priority = 30

  update {
    max_parallel      = 1
    min_healthy_time  = "30s"
    healthy_deadline  = "10m"
    progress_deadline = "11m"
    health_check      = "checks"
    canary            = 0
    auto_revert       = true
  }

  group "wi-knecht-group" {
    count = "${var.min_count}"

    scaling {
      enabled = true
      min     = "${var.min_count}"
      max     = "${var.max_count}"
      policy {
        cooldown            = "15m"
        evaluation_interval = "5m"

        check "high-cpu-usage" {
          source       = "nomad-apm"
          query        = "avg_cpu-allocated"
          query_window = "3m"
          group        = "cpu-usage"

          strategy "threshold" {
            lower_bound           = 70
            delta                 = 2
            within_bounds_trigger = 1
          }
        }

        check "low-cpu-usage" {
          source       = "nomad-apm"
          query        = "avg_cpu-allocated"
          query_window = "3m"
          group        = "cpu-usage"

          strategy "threshold" {
            upper_bound           = 30
            delta                 = -1
            within_bounds_trigger = 1
          }
        }

        check "high-memory-usage" {
          source       = "nomad-apm"
          query        = "avg_memory-allocated"
          query_window = "3m"
          group        = "memory-usage"

          strategy "threshold" {
            lower_bound           = 70
            delta                 = 2
            within_bounds_trigger = 1
          }
        }

        check "low-memory-usage" {
          source       = "nomad-apm"
          query        = "avg_memory-allocated"
          query_window = "3m"
          group        = "memory-usage"

          strategy "threshold" {
            upper_bound           = 30
            delta                 = -1
            within_bounds_trigger = 1
          }
        }
      }
    }

    restart {
      attempts = 5
      delay    = "15s"
      interval = "3m"
      mode     = "delay"
    }

    task "cleanup-task" {
    }

    task "post-stop-cleanup" {
    }

    task "wi-knecht-task" {
      constraint {
        attribute = "${node.class}"
        operator  = "regexp"
        value     = "worker"
      }

      driver = "docker"

      service {
        tags     = ["wi-knecht"]
        name     = "wi-knecht"
        provider = "consul"
      }

    }
  }
}

[philrenaud]

@EugenKon I gave things a shot with your job spec; even after scaling events I wasn't able to see a difference in jobs list allocation counts.

I wonder if this perhaps has something to do with caching. Do you see the issue after refresh?

[EugenKon]

I was in private mode. There should not be caching.

I am not sure: our cluster does not experience HA right now or this is belongs to the upgraded Nomad to v1.9. Let me couple of weeks to observe this. I'll put feed back after that.

[EugenKon]

I do not see the same issue, but I still see some divergences:

CLI tells that we have 4 instances and 4 healthy, but actually we have only 3 instances. The page was refreshed without cache and tried in incognito mode.

[tgross]

To be clear, you have 4 allocations healthy, not 3. That's what the CLI is telling you, and that's getting the data directly from the Nomad state store without caching. The only way that would be inaccurate is if it's changed on the client and the client hasn't pushed the update to the server yet (which happens in batches in <1s). The web UI screenshot you're showing shows 4 allocations: 3 healthy and 1 "unplaced", which suggests the data is somehow lagging behind the server.