New dynamic host volume never gets picked up by scheduler

[smdesmet]

I am on nomad 1.10.4 running on debian 12

My setup is a 3 node cluster, running some jobs where my patroni job has a 'patroni' dynamic host volume on each of the 3 nodes. These are visible in the 'Storage' of the nomad UI. I have limited this patroni job to 3 instances.

The scenario I am executing is: I want to replace one of the nodes.

• add additional node: this works, and some of my jobs start an additional allocation on the new node
• disable and drain one of the original nodes
• remove the dynamic volumes from this node: I first look if there are any claims and remove those first, then I remove the volumes
• remove the node
• create the 'patroni' volume on the new node, so a new allocation can use it

All of these steps work, but then I end up with this:

And this never gets resolved.

However, when I completely remove the job and recreate it, it immediately uses the new volume(and the 2 already existing volumes).

[jrasell]

Hi @smdesmet and thanks for raising this issue. Could you provide details of the job specification you are running when encountering this problem? This will help me look into this a bit easier, thanks.

[smdesmet]

below is a striped version of our minio job(it is a lot simpler than the patroni job), but it exhibits the same problem. In the 'Storage' tab I can see the additional volume, but it doesn't get allocations and it shows the same error as above(missing compatible host volume)

locals {
  max_instances           = 3
  registration_proxy_port = 1001
}

job "minio" {
  datacenters = ["*"]
  type        = "service"

  update {
      max_parallel      = 1
      health_check      = "task_states"
      min_healthy_time  = "10s"
      healthy_deadline  = "5m"
      progress_deadline = "2562047h"
      auto_revert       = false
      auto_promote      = false
      canary            = 0
      stagger           = "30s"
  }

  group "minio" {
    update {
      health_check = "checks"
    }
    scaling {
      enabled = true
      min     = 1
      max     = local.max_instances

      policy {
        evaluation_interval = "5s"
        cooldown            = "10s"

        check "client-count" {
          source = "nomad-client-count"
          query  = "Status == 'ready' && SchedulingEligibility == 'eligible'  && Meta.is_external == 'false'"
          strategy "pass-through" {}
        }
      }
    }

    restart {
      attempts = 1
      delay    = "8s"
      interval = "20m"
      mode     = "fail"
    }

    reschedule {
      delay          = "8s"
      delay_function = "constant"
      max_delay      = "8s"
      unlimited      = true
    }

    constraint {
      operator = "distinct_hosts"
      value    = "true"
    }

    network {
      mode = "bridge"

    }

    volume "minio" {
      type            = "host"
      read_only       = false
      source          = "minio"
      sticky          = true
      access_mode     = "single-node-writer"
      attachment_mode = "file-system"
    }

    task "minio" {
      driver = "docker"
      volume_mount {
        read_only   = false
        destination = "/data"
        volume      = "minio"
      }

      config {
        image   = "minio:latest"
        command = "server"
        args = [
          "--address", "0.0.0.0:9000",
          "--console-address", "0.0.0.0:9001",
          "/data"
        ]
      }

      resources {
        cpu    = 500
        memory = 1024
      }
    }
  }

  group "replicated-proxy" {
    # run a maximum of 2 instances if nodes are available
    scaling {
      enabled = true
      min     = 1
      max     = 2

      policy {
        evaluation_interval = "5s"
        cooldown            = "10s"

        check "client-count" {
          source = "nomad-client-count"
          query  = "Status == 'ready' && SchedulingEligibility == 'eligible'  && Meta.is_external == 'false'"
          strategy "pass-through" {}
        }
      }
    }

    restart {
      attempts = 3
      delay    = "8s"
      interval = "20m"
      mode     = "fail"
    }

    reschedule {
      delay          = "8s"
      delay_function = "constant"
      max_delay      = "8s"
      unlimited      = true
    }

    constraint {
      operator = "distinct_hosts"
      value    = "true"
    }

    network {
      mode = "bridge"
      port "haproxy_metrics" {}
      port "haproxy_health" {}
    }

    task "minio-haproxy" {
      driver = "docker"
      config {
        image = "haproxy:latest"
        args  = ["-f", "/local/haproxy.conf"]
      }
      template {
        change_mode   = "signal"
        change_signal = "SIGHUP"
        destination   = "/local/haproxy.conf"
        data          = <<-EOTPL
                global
                    maxconn 100

                defaults
                    log global
                    mode tcp
                    retries 2
                    timeout client 30m
                    timeout connect 4s
                    timeout server 30m
                    timeout check 2s

                listen stats
                    mode http
                    bind *:{{ env "NOMAD_PORT_haproxy_metrics" }}
                    http-request use-service prometheus-exporter if { path /metrics }
                    stats enable
                    stats uri /

                listen health_check
                    bind *:{{ env "NOMAD_PORT_haproxy_health" }}
                    mode http
                    monitor-uri /health
                    option dontlognull

                listen inbound
                    bind 127.0.0.1:9000
                    mode tcp
                    balance leastconn
                    option httpchk GET /minio/health/live
                    http-check send hdr Host localhost
                    http-check expect status 200
                    server minio1 127.0.0.1:19000 check inter 2s
                    server minio2 127.0.0.1:19001 check inter 2s
                    server minio3 127.0.0.1:19002 check inter 2s
                EOTPL
      }

      resources {
        cpu    = 120
        memory = 128
      }
    }
  }
}

[jrasell]

Thanks for the additional detail @smdesmet, this has helped me reproduce this locally which I believe matches your initial walkthrough.

I reproduced it with:

$ nomad version
Nomad v1.11.0-dev
BuildDate 2025-10-27T08:25:16Z
Revision c6079293ff3e5d11a66a9c0084a37cd4c656e2ff

job spec

job "minio" {
  datacenters = ["*"]
  type        = "service"

  update {
      max_parallel      = 1
      health_check      = "task_states"
      min_healthy_time  = "10s"
      healthy_deadline  = "5m"
      progress_deadline = "2562047h"
      auto_revert       = false
      auto_promote      = false
      canary            = 0
      stagger           = "30s"
  }

  group "minio" {

    count = 3

    update {
      health_check = "checks"
    }

    restart {
      attempts = 1
      delay    = "8s"
      interval = "20m"
      mode     = "fail"
    }

    reschedule {
      delay          = "8s"
      delay_function = "constant"
      max_delay      = "8s"
      unlimited      = true
    }

    constraint {
      operator = "distinct_hosts"
      value    = "true"
    }

    volume "minio" {
      type            = "host"
      read_only       = false
      source          = "minio"
      sticky          = true
      access_mode     = "single-node-writer"
      attachment_mode = "file-system"
    }

    task "minio" {
      driver = "docker"
      volume_mount {
        read_only   = false
        destination = "/data"
        volume      = "minio"
      }

      config {
        image   = "ubuntu:24.04"
        command = "sleep"
        args    = ["infinity"]
      }
    }
  }
}

volume spec

namespace = "default"
name      = "minio"
type      = "host"
plugin_id = "mkdir"
node_id   = "1bd20c54-db80-9c69-8138-6342afbf1c33"

capability {
  access_mode     = "single-node-writer"
  attachment_mode = "file-system"
}

I create a host volume via the mkdir plugin on each of my three clients:

$ nomad volume status
Dynamic Host Volumes
ID        Name   Namespace  Plugin ID  Node ID   Node Pool  State
223bbaef  minio  default    mkdir      38bf1788  default    ready
90228913  minio  default    mkdir      08cddf7d  default    ready
a6439858  minio  default    mkdir      1bd20c54  default    ready

I then registered the job, which started successfully:

$ nomad run test.nomad.hcl
==> 2025-10-31T08:36:18Z: Monitoring evaluation "628853b5"
    2025-10-31T08:36:18Z: Evaluation triggered by job "minio"
    2025-10-31T08:36:18Z: Evaluation within deployment: "23fd07ab"
    2025-10-31T08:36:18Z: Allocation "03e3f933" created: node "08cddf7d", group "minio"
    2025-10-31T08:36:18Z: Allocation "5894c676" created: node "38bf1788", group "minio"
    2025-10-31T08:36:18Z: Allocation "cdb35bee" created: node "1bd20c54", group "minio"
    2025-10-31T08:36:18Z: Evaluation status changed: "pending" -> "complete"
==> 2025-10-31T08:36:18Z: Evaluation "628853b5" finished with status "complete"
==> 2025-10-31T08:36:18Z: Monitoring deployment "23fd07ab"
  ✓ Deployment "23fd07ab" successful

    2025-10-31T08:36:30Z
    ID          = 23fd07ab
    Job ID      = minio
    Job Version = 0
    Status      = successful
    Description = Deployment completed successfully

    Deployed
    Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
    minio       3        3       3        0          2318-02-10T07:36:29Z

I drained and removed a single node from the cluster. The job was placed into a state I would expect:

$ nomad status minio
ID            = minio
Name          = minio
Submit Date   = 2025-10-31T08:36:18Z
Type          = service
Priority      = 50
Datacenters   = *
Namespace     = default
Node Pool     = default
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost  Unknown
minio       1       0         2        0       1         0     0

Placement Failure
Task Group "minio":
  * Constraint "distinct_hosts": 2 nodes excluded by filter

Latest Deployment
ID          = 23fd07ab
Status      = successful
Description = Deployment completed successfully

Deployed
Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
minio       3        3       3        0          2318-02-10T07:36:29Z

Allocations
ID        Node ID   Task Group  Version  Desired  Status    Created    Modified
03e3f933  08cddf7d  minio       0        run      running   2m58s ago  2m48s ago
5894c676  38bf1788  minio       0        stop     complete  2m58s ago  13s ago
cdb35bee  1bd20c54  minio       0        run      running   2m58s ago  2m48s ago

I then added a new node into the cluster, created a volume on it, then tried to force an evaluation of the job:

$ nomad job eval minio
==> 2025-10-31T08:42:29Z: Monitoring evaluation "49ba1b0d"
    2025-10-31T08:42:29Z: Evaluation triggered by job "minio"
    2025-10-31T08:42:30Z: Evaluation within deployment: "23fd07ab"
    2025-10-31T08:42:30Z: Evaluation status changed: "pending" -> "complete"
==> 2025-10-31T08:42:30Z: Evaluation "49ba1b0d" finished with status "complete" but failed to place all allocations:
    2025-10-31T08:42:30Z: Task Group "minio" (failed to place 1 allocation):
      * Constraint "missing compatible host volumes": 1 nodes excluded by filter
      * Constraint "distinct_hosts": 2 nodes excluded by filter
    2025-10-31T08:42:30Z: Evaluation "17caeb8a" waiting for additional capacity to place remainder
==> 2025-10-31T08:42:30Z: Monitoring deployment "23fd07ab"
  ✓ Deployment "23fd07ab" successful

    2025-10-31T08:42:30Z
    ID          = 23fd07ab
    Job ID      = minio
    Job Version = 0
    Status      = successful
    Description = Deployment completed successfully

    Deployed
    Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
    minio       3        3       3        0          2318-02-10T07:36:29Z

The missing compatible host volumes error is coming from the scheduler feasibility checker. This indicates that the node is not passing the host volumes check. I'll get this moved onto our backlog.