Limit the CPU usage of the container through resources.cpu, and the container will be concentrated on running on a certain core

[lzytaro]

Nomad version

Nomad v1.10.0
BuildDate 2025-04-09T16:40:54Z
Revision e26a2bd

Operating system and Environment details

[root@node1 mnt]# uname -a
Linux node1 5.14.0-503.35.1.el9_5.x86_64+rt #1 SMP PREEMPT_RT Thu Apr 3 11:59:14 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
[root@node1 mnt]# cat /etc/os
os-release ostree/
[root@node1 mnt]# cat /etc/os-release
NAME="Rocky Linux"
VERSION="9.5 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.5"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.5 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
VENDOR_NAME="RESF"
VENDOR_URL="https://resf.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.5"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.5"

Issue

Limit the CPU usage of the container through resources.cpu, and the container will be concentrated on running on a certain core

Reproduction steps

1. The client configures the CPU to reserve 0 to 31 cores
2. Use the same resources.cpu configuration to start 20 containers (more are also acceptable)
3. Monitor the occupancy of each CPU core

Expected Result

It is hoped that the load can be evenly distributed among the 32-191 cores. Or the cores parameter supports decimal configuration, such as 0.2, which can limit each core to running a maximum of 5 containers

Actual Result

The CPU load is concentrated on Core 32. If more containers are created, the load on Core 32 will reach 100%, and the command to start docker containers will execute very slowly

Job file (if appropriate)

[root@node1 mnt]# cat hcl/plc-20.nomad.hcl

job "plc-20" {
  type        = "service"

  task "plc-20" {
    driver = "docker"

    config {
      cpu_hard_limit = true
      image        = "vplc:1.2"
      entrypoint   = ["/opt/autostart.sh"]
      network_mode = "macvlan0" 
      ipv4_address = "10.30.191.20"
      args         = ["/bin/bash"]
      privileged   = true
      volumes = [
        "/mnt/new_data/plc-20:/App"
      ]
    }

    resources {
      cpu = 1000
      memory = 512   # MB
    }

    user = "root"
  }
}

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

5月 12 18:47:41 node1 nomad[33418]: 2025-05-12T18:47:41.559+0800 [INFO] client.alloc_runner.task_runner: restarting task: alloc_id=e1f4fa23-a295-d2bc-f769-22c6edcfe730 task=plc-20 reason="Restart within policy" delay=17.216233921s
5月 12 18:47:41 node1 nomad[33418]: 2025-05-12T18:47:41.559+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=e1f4fa23-a295-d2bc-f769-22c6edcfe730 task=plc-20 type=Restarting msg="Task restarting in 17.216233921s" failed=false
5月 12 18:47:58 node1 nomad[33418]: 2025-05-12T18:47:58.867+0800 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=1494b1a0a18983e0734d05a94e629073d0567ff1df4fa26ff625d8bfaea1ae38
5月 12 18:47:59 node1 nomad[33418]: 2025-05-12T18:47:59.258+0800 [INFO] client.driver_mgr.docker: started container: driver=docker container_id=1494b1a0a18983e0734d05a94e629073d0567ff1df4fa26ff625d8bfaea1ae38
5月 12 18:47:59 node1 nomad[33418]: 2025-05-12T18:47:59.285+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=e1f4fa23-a295-d2bc-f769-22c6edcfe730 task=plc-20 type=Started msg="Task started by client" failed=false
5月 12 19:28:09 node1 nomad[33418]: 2025-05-12T19:28:09.339+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=52cda26d-9eda-3b8d-05d1-70154255fb28 task=plc-21 type=Received msg="Task received by client" failed=false
5月 12 19:28:09 node1 nomad[33418]: 2025-05-12T19:28:09.340+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=52cda26d-9eda-3b8d-05d1-70154255fb28 task=plc-21 type="Task Setup" msg="Building Task Directory" failed=false
5月 12 19:28:09 node1 nomad[33418]: 2025-05-12T19:28:09.487+0800 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=85f6bf16281679445db20e4692dbc5014ec12abebdf658da5a1530f171e1bcdf
5月 12 19:28:09 node1 nomad[33418]: 2025-05-12T19:28:09.993+0800 [INFO] client.driver_mgr.docker: started container: driver=docker container_id=85f6bf16281679445db20e4692dbc5014ec12abebdf658da5a1530f171e1bcdf
5月 12 19:28:10 node1 nomad[33418]: 2025-05-12T19:28:10.025+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=52cda26d-9eda-3b8d-05d1-70154255fb28 task=plc-21 type=Started msg="Task started by client" failed=false
5月 12 19:28:21 node1 nomad[33418]: 2025-05-12T19:28:21.474+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=76ad2123-2cc9-3fc9-1f89-733da428b700 task=plc-22 type=Received msg="Task received by client" failed=false
5月 12 19:28:21 node1 nomad[33418]: 2025-05-12T19:28:21.475+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=76ad2123-2cc9-3fc9-1f89-733da428b700 task=plc-22 type="Task Setup" msg="Building Task Directory" failed=false
5月 12 19:28:21 node1 nomad[33418]: 2025-05-12T19:28:21.616+0800 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=accfce7eabc836bbce8162161573d1d1dff586cb190255ae3a3cf7a6e0f00dd5
5月 12 19:28:22 node1 nomad[33418]: 2025-05-12T19:28:22.006+0800 [INFO] client.driver_mgr.docker: started container: driver=docker container_id=accfce7eabc836bbce8162161573d1d1dff586cb190255ae3a3cf7a6e0f00dd5
5月 12 19:28:22 node1 nomad[33418]: 2025-05-12T19:28:22.037+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=76ad2123-2cc9-3fc9-1f89-733da428b700 task=plc-22 type=Started msg="Task started by client" failed=false
5月 12 19:28:33 node1 nomad[33418]: 2025-05-12T19:28:33.613+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=9ddbbda3-e960-2cc4-6359-0a480f2a3ad5 task=plc-23 type=Received msg="Task received by client" failed=false
5月 12 19:28:33 node1 nomad[33418]: 2025-05-12T19:28:33.614+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=9ddbbda3-e960-2cc4-6359-0a480f2a3ad5 task=plc-23 type="Task Setup" msg="Building Task Directory" failed=false
5月 12 19:28:33 node1 nomad[33418]: 2025-05-12T19:28:33.758+0800 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=2ba5f85c5723dbeee9f3c19ced85de8e469cb4e541d2c0c104b22108806cc102
5月 12 19:28:34 node1 nomad[33418]: 2025-05-12T19:28:34.097+0800 [INFO] client.driver_mgr.docker: started container: driver=docker container_id=2ba5f85c5723dbeee9f3c19ced85de8e469cb4e541d2c0c104b22108806cc102
5月 12 19:28:34 node1 nomad[33418]: 2025-05-12T19:28:34.129+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=9ddbbda3-e960-2cc4-6359-0a480f2a3ad5 task=plc-23 type=Started msg="Task started by client" failed=false
5月 12 19:28:45 node1 nomad[33418]: 2025-05-12T19:28:45.752+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=a9bae878-1025-463d-35ca-5abd6ff149f7 task=plc-24 type=Received msg="Task received by client" failed=false
5月 12 19:28:45 node1 nomad[33418]: 2025-05-12T19:28:45.754+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=a9bae878-1025-463d-35ca-5abd6ff149f7 task=plc-24 type="Task Setup" msg="Building Task Directory" failed=false
5月 12 19:28:45 node1 nomad[33418]: 2025-05-12T19:28:45.902+0800 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=3e24a7483778fb157c28e0c4be9efc77c84e60d777151f741060ce3fe365a032
5月 12 19:28:46 node1 nomad[33418]: 2025-05-12T19:28:46.410+0800 [INFO] client.driver_mgr.docker: started container: driver=docker container_id=3e24a7483778fb157c28e0c4be9efc77c84e60d777151f741060ce3fe365a032
5月 12 19:28:46 node1 nomad[33418]: 2025-05-12T19:28:46.442+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=a9bae878-1025-463d-35ca-5abd6ff149f7 task=plc-24 type=Started msg="Task started by client" failed=false
5月 12 19:28:57 node1 nomad[33418]: 2025-05-12T19:28:57.884+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=52801ece-66d2-36c2-5ab2-6add6868015f task=plc-25 type=Received msg="Task received by client" failed=false
5月 12 19:28:57 node1 nomad[33418]: 2025-05-12T19:28:57.886+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=52801ece-66d2-36c2-5ab2-6add6868015f task=plc-25 type="Task Setup" msg="Building Task Directory" failed=false
5月 12 19:28:58 node1 nomad[33418]: 2025-05-12T19:28:58.034+0800 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=c8d968a6dff113f7de18bf907e168cb4e51a030c5d7263b97bc0351013c89cc7
5月 12 19:28:58 node1 nomad[33418]: 2025-05-12T19:28:58.396+0800 [INFO] client.driver_mgr.docker: started container: driver=docker container_id=c8d968a6dff113f7de18bf907e168cb4e51a030c5d7263b97bc0351013c89cc7
5月 12 19:28:58 node1 nomad[33418]: 2025-05-12T19:28:58.428+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=52801ece-66d2-36c2-5ab2-6add6868015f task=plc-25 type=Started msg="Task started by client" failed=false
5月 12 19:29:10 node1 nomad[33418]: 2025-05-12T19:29:10.016+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=d260aee0-3368-448c-6935-f0e4a2ad4361 task=plc-26 type=Received msg="Task received by client" failed=false
5月 12 19:29:10 node1 nomad[33418]: 2025-05-12T19:29:10.018+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=d260aee0-3368-448c-6935-f0e4a2ad4361 task=plc-26 type="Task Setup" msg="Building Task Directory" failed=false
5月 12 19:29:10 node1 nomad[33418]: 2025-05-12T19:29:10.168+0800 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=3a8c64ed78ae6d1e4723e0db0ce99aa92f4654efc8220570124d14ceb340c31e
5月 12 19:29:10 node1 nomad[33418]: 2025-05-12T19:29:10.610+0800 [INFO] client.driver_mgr.docker: started container: driver=docker container_id=3a8c64ed78ae6d1e4723e0db0ce99aa92f4654efc8220570124d14ceb340c31e
5月 12 19:29:10 node1 nomad[33418]: 2025-05-12T19:29:10.642+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=d260aee0-3368-448c-6935-f0e4a2ad4361 task=plc-26 type=Started msg="Task started by client" failed=false
5月 12 19:29:22 node1 nomad[33418]: 2025-05-12T19:29:22.159+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=b4333149-cb18-f593-2447-b96c27d02912 task=plc-27 type=Received msg="Task received by client" failed=false
5月 12 19:29:22 node1 nomad[33418]: 2025-05-12T19:29:22.161+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=b4333149-cb18-f593-2447-b96c27d02912 task=plc-27 type="Task Setup" msg="Building Task Directory" failed=false
5月 12 19:29:22 node1 nomad[33418]: 2025-05-12T19:29:22.327+0800 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=9e61b2fc948a0787268910da846c89c57ca39694be16582be7fa94a95f7f36fe
5月 12 19:29:22 node1 nomad[33418]: 2025-05-12T19:29:22.698+0800 [INFO] client.driver_mgr.docker: started container: driver=docker container_id=9e61b2fc948a0787268910da846c89c57ca39694be16582be7fa94a95f7f36fe
5月 12 19:29:22 node1 nomad[33418]: 2025-05-12T19:29:22.729+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=b4333149-cb18-f593-2447-b96c27d02912 task=plc-27 type=Started msg="Task started by client" failed=false
5月 12 19:29:34 node1 nomad[33418]: 2025-05-12T19:29:34.298+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=7ec2248d-0c68-4813-875c-4a6764121438 task=plc-28 type=Received msg="Task received by client" failed=false
5月 12 19:29:34 node1 nomad[33418]: 2025-05-12T19:29:34.300+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=7ec2248d-0c68-4813-875c-4a6764121438 task=plc-28 type="Task Setup" msg="Building Task Directory" failed=false
5月 12 19:29:34 node1 nomad[33418]: 2025-05-12T19:29:34.443+0800 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=7f4f01f3f2f84b452f097a922d854bab81d5fbcac08215938941cbdd2c5169f6
5月 12 19:29:34 node1 nomad[33418]: 2025-05-12T19:29:34.792+0800 [INFO] client.driver_mgr.docker: started container: driver=docker container_id=7f4f01f3f2f84b452f097a922d854bab81d5fbcac08215938941cbdd2c5169f6
5月 12 19:29:34 node1 nomad[33418]: 2025-05-12T19:29:34.824+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=7ec2248d-0c68-4813-875c-4a6764121438 task=plc-28 type=Started msg="Task started by client" failed=false
5月 12 19:29:46 node1 nomad[33418]: 2025-05-12T19:29:46.437+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=b357f101-30f5-25c4-c2f4-f309cdaf475f task=plc-29 type=Received msg="Task received by client" failed=false
5月 12 19:29:46 node1 nomad[33418]: 2025-05-12T19:29:46.439+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=b357f101-30f5-25c4-c2f4-f309cdaf475f task=plc-29 type="Task Setup" msg="Building Task Directory" failed=false
5月 12 19:29:46 node1 nomad[33418]: 2025-05-12T19:29:46.601+0800 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=da260d4ef374d7d0e54e62725afd5da1c845e53252800e3e447e1575f2b35c10
5月 12 19:29:47 node1 nomad[33418]: 2025-05-12T19:29:47.106+0800 [INFO] client.driver_mgr.docker: started container: driver=docker container_id=da260d4ef374d7d0e54e62725afd5da1c845e53252800e3e447e1575f2b35c10
5月 12 19:29:47 node1 nomad[33418]: 2025-05-12T19:29:47.138+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=b357f101-30f5-25c4-c2f4-f309cdaf475f task=plc-29 type=Started msg="Task started by client" failed=false
5月 12 19:29:58 node1 nomad[33418]: 2025-05-12T19:29:58.576+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=fad4e3df-60b6-c376-4fa4-3ba0fb499535 task=plc-30 type=Received msg="Task received by client" failed=false
5月 12 19:29:58 node1 nomad[33418]: 2025-05-12T19:29:58.577+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=fad4e3df-60b6-c376-4fa4-3ba0fb499535 task=plc-30 type="Task Setup" msg="Building Task Directory" failed=false
5月 12 19:29:58 node1 nomad[33418]: 2025-05-12T19:29:58.720+0800 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=5433e436510f4986ab5dbfa6cc3f2326c749850cedf89f8f50303dd26893636d
5月 12 19:29:59 node1 nomad[33418]: 2025-05-12T19:29:59.202+0800 [INFO] client.driver_mgr.docker: started container: driver=docker container_id=5433e436510f4986ab5dbfa6cc3f2326c749850cedf89f8f50303dd26893636d
5月 12 19:29:59 node1 nomad[33418]: 2025-05-12T19:29:59.234+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=fad4e3df-60b6-c376-4fa4-3ba0fb499535 task=plc-30 type=Started msg="Task started by client" failed=false
5月 12 19:30:10 node1 nomad[33418]: 2025-05-12T19:30:10.716+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=7d4ea535-a4a7-0164-14d4-403ba51e9c6d task=plc-31 type=Received msg="Task received by client" failed=false
5月 12 19:30:10 node1 nomad[33418]: 2025-05-12T19:30:10.717+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=7d4ea535-a4a7-0164-14d4-403ba51e9c6d task=plc-31 type="Task Setup" msg="Building Task Directory" failed=false
5月 12 19:30:10 node1 nomad[33418]: 2025-05-12T19:30:10.865+0800 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=b589c7e3cc5b13cfcbdefc94f00657b1254bc742416e5a88a8842baf360e6975
5月 12 19:30:11 node1 nomad[33418]: 2025-05-12T19:30:11.295+0800 [INFO] client.driver_mgr.docker: started container: driver=docker container_id=b589c7e3cc5b13cfcbdefc94f00657b1254bc742416e5a88a8842baf360e6975
5月 12 19:30:11 node1 nomad[33418]: 2025-05-12T19:30:11.327+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=7d4ea535-a4a7-0164-14d4-403ba51e9c6d task=plc-31 type=Started msg="Task started by client" failed=false
5月 12 19:30:22 node1 nomad[33418]: 2025-05-12T19:30:22.854+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=2a26a95a-f69e-74dd-5260-0212a04970e0 task=plc-32 type=Received msg="Task received by client" failed=false
5月 12 19:30:22 node1 nomad[33418]: 2025-05-12T19:30:22.856+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=2a26a95a-f69e-74dd-5260-0212a04970e0 task=plc-32 type="Task Setup" msg="Building Task Directory" failed=false
5月 12 19:30:22 node1 nomad[33418]: 2025-05-12T19:30:22.996+0800 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=1b53f03d188a397d6dd91b02887435a2e3ecde095e219ebd7a6d119ba9148a9e
5月 12 19:30:23 node1 nomad[33418]: 2025-05-12T19:30:23.392+0800 [INFO] client.driver_mgr.docker: started container: driver=docker container_id=1b53f03d188a397d6dd91b02887435a2e3ecde095e219ebd7a6d119ba9148a9e
5月 12 19:30:23 node1 nomad[33418]: 2025-05-12T19:30:23.425+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=2a26a95a-f69e-74dd-5260-0212a04970e0 task=plc-32 type=Started msg="Task started by client" failed=false
5月 12 19:30:34 node1 nomad[33418]: 2025-05-12T19:30:34.988+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=498ddc0a-0468-fd33-4f41-686d2fae8cec task=plc-33 type=Received msg="Task received by client" failed=false
5月 12 19:30:34 node1 nomad[33418]: 2025-05-12T19:30:34.990+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=498ddc0a-0468-fd33-4f41-686d2fae8cec task=plc-33 type="Task Setup" msg="Building Task Directory" failed=false
5月 12 19:30:35 node1 nomad[33418]: 2025-05-12T19:30:35.132+0800 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=529fcc1b6de8c5099a1cdae57ad7ce5d1b87b8850c7ea76834988ea7c956107a
5月 12 19:30:35 node1 nomad[33418]: 2025-05-12T19:30:35.609+0800 [INFO] client.driver_mgr.docker: started container: driver=docker container_id=529fcc1b6de8c5099a1cdae57ad7ce5d1b87b8850c7ea76834988ea7c956107a
5月 12 19:30:35 node1 nomad[33418]: 2025-05-12T19:30:35.642+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=498ddc0a-0468-fd33-4f41-686d2fae8cec task=plc-33 type=Started msg="Task started by client" failed=false
5月 12 19:30:47 node1 nomad[33418]: 2025-05-12T19:30:47.121+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=73224580-7013-3770-c42a-eaa64ecf8e8a task=plc-34 type=Received msg="Task received by client" failed=false
5月 12 19:30:47 node1 nomad[33418]: 2025-05-12T19:30:47.122+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=73224580-7013-3770-c42a-eaa64ecf8e8a task=plc-34 type="Task Setup" msg="Building Task Directory" failed=false
5月 12 19:30:47 node1 nomad[33418]: 2025-05-12T19:30:47.267+0800 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=9df1cc9633b9f1db68189b4e10c230dafc655639f92d99656040fbb44d315793
5月 12 19:30:47 node1 nomad[33418]: 2025-05-12T19:30:47.705+0800 [INFO] client.driver_mgr.docker: started container: driver=docker container_id=9df1cc9633b9f1db68189b4e10c230dafc655639f92d99656040fbb44d315793
5月 12 19:30:47 node1 nomad[33418]: 2025-05-12T19:30:47.738+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=73224580-7013-3770-c42a-eaa64ecf8e8a task=plc-34 type=Started msg="Task started by client" failed=false
5月 12 19:30:59 node1 nomad[33418]: 2025-05-12T19:30:59.261+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=b8fcd197-7264-5ed8-a661-79b5d69c83b2 task=plc-35 type=Received msg="Task received by client" failed=false
5月 12 19:30:59 node1 nomad[33418]: 2025-05-12T19:30:59.262+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=b8fcd197-7264-5ed8-a661-79b5d69c83b2 task=plc-35 type="Task Setup" msg="Building Task Directory" failed=false
5月 12 19:30:59 node1 nomad[33418]: 2025-05-12T19:30:59.407+0800 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=d8a6112818201e6cccb0b11ce3864c1da86aa132bc08511aa7ff3d8858278bf6
5月 12 19:30:59 node1 nomad[33418]: 2025-05-12T19:30:59.798+0800 [INFO] client.driver_mgr.docker: started container: driver=docker container_id=d8a6112818201e6cccb0b11ce3864c1da86aa132bc08511aa7ff3d8858278bf6
5月 12 19:30:59 node1 nomad[33418]: 2025-05-12T19:30:59.829+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=b8fcd197-7264-5ed8-a661-79b5d69c83b2 task=plc-35 type=Started msg="Task started by client" failed=false
5月 12 19:31:11 node1 nomad[33418]: 2025-05-12T19:31:11.396+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=98bba444-93d4-01e9-653d-5a95d804caa1 task=plc-36 type=Received msg="Task received by client" failed=false
5月 12 19:31:11 node1 nomad[33418]: 2025-05-12T19:31:11.397+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=98bba444-93d4-01e9-653d-5a95d804caa1 task=plc-36 type="Task Setup" msg="Building Task Directory" failed=false
5月 12 19:31:11 node1 nomad[33418]: 2025-05-12T19:31:11.558+0800 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=45577ad131e442a3023dd0a43c525e35798c1aff83527ac1a9fb04bdb3d3c996
5月 12 19:31:12 node1 nomad[33418]: 2025-05-12T19:31:12.011+0800 [INFO] client.driver_mgr.docker: started container: driver=docker container_id=45577ad131e442a3023dd0a43c525e35798c1aff83527ac1a9fb04bdb3d3c996
5月 12 19:31:12 node1 nomad[33418]: 2025-05-12T19:31:12.041+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=98bba444-93d4-01e9-653d-5a95d804caa1 task=plc-36 type=Started msg="Task started by client" failed=false
5月 12 19:31:23 node1 nomad[33418]: 2025-05-12T19:31:23.535+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=13a1c808-a671-118d-cfc0-beec977895c5 task=plc-37 type=Received msg="Task received by client" failed=false
5月 12 19:31:23 node1 nomad[33418]: 2025-05-12T19:31:23.537+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=13a1c808-a671-118d-cfc0-beec977895c5 task=plc-37 type="Task Setup" msg="Building Task Directory" failed=false
5月 12 19:31:23 node1 nomad[33418]: 2025-05-12T19:31:23.679+0800 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=f8d6a5c09741221a6f403f9cd114978b75389ac34c63f60bbfd022015c63288a
5月 12 19:31:24 node1 nomad[33418]: 2025-05-12T19:31:24.097+0800 [INFO] client.driver_mgr.docker: started container: driver=docker container_id=f8d6a5c09741221a6f403f9cd114978b75389ac34c63f60bbfd022015c63288a
5月 12 19:31:24 node1 nomad[33418]: 2025-05-12T19:31:24.127+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=13a1c808-a671-118d-cfc0-beec977895c5 task=plc-37 type=Started msg="Task started by client" failed=false
5月 12 19:31:35 node1 nomad[33418]: 2025-05-12T19:31:35.674+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=c6ec3432-f8d4-1467-891f-26e7e4a9d29d task=plc-38 type=Received msg="Task received by client" failed=false
5月 12 19:31:35 node1 nomad[33418]: 2025-05-12T19:31:35.676+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=c6ec3432-f8d4-1467-891f-26e7e4a9d29d task=plc-38 type="Task Setup" msg="Building Task Directory" failed=false
5月 12 19:31:35 node1 nomad[33418]: 2025-05-12T19:31:35.823+0800 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=022653acb8da79b31d2253ac73fccd6c8f310f9221e108b8e3a71d648d2ac5f3
5月 12 19:31:36 node1 nomad[33418]: 2025-05-12T19:31:36.200+0800 [INFO] client.driver_mgr.docker: started container: driver=docker container_id=022653acb8da79b31d2253ac73fccd6c8f310f9221e108b8e3a71d648d2ac5f3
5月 12 19:31:36 node1 nomad[33418]: 2025-05-12T19:31:36.232+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=c6ec3432-f8d4-1467-891f-26e7e4a9d29d task=plc-38 type=Started msg="Task started by client" failed=false
5月 12 19:31:47 node1 nomad[33418]: 2025-05-12T19:31:47.813+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=45855cc0-0fc2-58eb-cb9d-28c7ef463442 task=plc-39 type=Received msg="Task received by client" failed=false
5月 12 19:31:47 node1 nomad[33418]: 2025-05-12T19:31:47.814+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=45855cc0-0fc2-58eb-cb9d-28c7ef463442 task=plc-39 type="Task Setup" msg="Building Task Directory" failed=false
5月 12 19:31:47 node1 nomad[33418]: 2025-05-12T19:31:47.960+0800 [INFO] client.driver_mgr.docker: created container: driver=docker container_id=167e5be08de7ecf7b7eafd2f156ce2ef2a6a2d5cd5e9c21dac21847c235bf2ec
5月 12 19:31:48 node1 nomad[33418]: 2025-05-12T19:31:48.415+0800 [INFO] client.driver_mgr.docker: started container: driver=docker container_id=167e5be08de7ecf7b7eafd2f156ce2ef2a6a2d5cd5e9c21dac21847c235bf2ec
5月 12 19:31:48 node1 nomad[33418]: 2025-05-12T19:31:48.447+0800 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=45855cc0-0fc2-58eb-cb9d-28c7ef463442 task=plc-39 type=Started msg="Task started by client" failed=false

[pkazmierczak]

Hi @lzytaro, thanks for reporting. I've been running some tests and, sadly, I can't reproduce this. I run Nomad 1.10 on Ubuntu 24.04, with Docker 26.1.3. I adapted your jobspec to make sure I'm testing things I know use as many cores as told. This is my Dockerfile:

from ubuntu:latest

run apt update && apt -y install stress

entrypoint ["/usr/bin/stress"]

If I tarball the image and use it locally, like this:

job "plc-20" {
  type = "service"

  task "plc-20" {
    driver = "docker"

    config {
      cpu_hard_limit = true
      load           = "/home/ubuntu/stress/stress-1.0.0.tar"
      image          = "stress:1.0.0"
      args           = ["--cpu", "8"]
      privileged     = true
    }

    resources {
      cpu    = 1000
      memory = 512 # MB
    }

    user = "root"
  }
}

and run the job, I can see 8 cores being used. This leads me to believe there may be something with your docker image that prevents it from using multiple cores? Are you able to docker run your image and see it paralellize correctly?

[pkazmierczak]

Oh now that I read the issue description again, I might've misread. Your issue is that with multiple containers they all end up using the same core. How do you spawn multiple containers? Do you increase the job count? Do you have multiple jobs that use the same image?

[lzytaro]

Oh now that I read the issue description again, I might've misread. Your issue is that with multiple containers they all end up using the same core. How do you spawn multiple containers? Do you increase the job count? Do you have multiple jobs that use the same image?

Hello! I'm very glad to receive your reply. I generated 100 job files and created containers using the same image. I'm not sure if it's because my host machine is configured with CPU isolation. I isolated cores 0 to 31 in the kernel boot file, so that system tasks can only run on cores 0 to 31 by default, and cores 32 to 191 are used for business. These cores can only be used through the --cpuset-cpus parameter of taskset or docker. In fact, none of these are important. What matters is that the cores parameter can only support integers. My task has a relatively high requirement for real-time performance. I hope to bind up to 5 containers to the same core (for example, the cores are set to 0.2) to avoid real-time performance issues caused by the switching of tasks between cores.

[pkazmierczak]

I think maybe I'm confused. Sticking to Nomad-related issues: is the problem that the jobs you submit all get the same core? Is there any setting in your resources block that controls the number of cores?

[lzytaro]

I think maybe I'm confused. Sticking to Nomad-related issues: is the problem that the jobs you submit all get the same core? Is there any setting in your resources block that controls the number of cores?

Sorry, maybe I didn't express myself clearly. I'll try to describe the process in detail.
First, let me describe my hardware environment:

I have 10 Rocky Linux 9.5 servers with 192 cores. Cores 0 to 31 have been isolated, and cores 32 to 191 have been used for business processing. rt patches have been installed on the system.

My original requirement is:
Start 500 containers on each server. To ensure the real-time performance of each container, I want to bind each container to a specific core. Obviously, I cannot specify the specific core of each job in the job, because there may be different numbers of cores on different servers or other issues. So I want to use the resources.cores parameter in the job to automatically allocate cores. When I set resources.cores=1, the system can achieve the goal of binding one core to each container. However, in this way, only 160 containers can run on each server. However, my requirement is to have 500 containers run on each server, that is, every 3 to 4 containers are bound to the same core, so it cannot fully meet my needs. If the cores parameter supports the input of decimals or percentages, then when I configure it to 0.2 or 20%, cores 31 to 191 can run a total of 800 containers.

core 32: job 1,2,3,4,5
core 33: job 6,7,8,9,10
core 34: job 11,12,13,14,15
......

Since the cores parameter cannot set decimals, I can only try to cancel this configuration, give up some real-time performance, and let these 500 containers run randomly on cores 31 to 191. When I removed the cores parameter, I found that when starting the container, the load would be concentrated on Core 32, and cores 33 to 191 had basically no load. This also led to the docker run command becoming extremely laggy after creating about 40 containers, and it was almost impossible to start tasks later.

core 32: job 1,2,3,4,5,6,7,8,9,10........