Add ability to set additional CPU during allocation start (startup CPU Boost)

[habibimustafa]

Proposal

It would be nice to be able to set startup CPU boost to Nomad Job. Provides tasks with a temporary increase in CPU resources during their startup phase. Allow tasks with heavy initialization routines (such as application bootstrapping, data migration, or pre-warming caches) to run more efficiently and start faster without permanently over-provisioning CPU resources.

The feature could be implemented as an optional task configuration parameter (for example, startup_cpu_boost under resources stanza) that accepts a CPU boost factor or specifies a temporary CPU quota for a configurable duration (e.g., the first 10–30 seconds of task execution). Once the startup phase completes, the task would automatically revert to its normal CPU allocation.

Use-cases

1. Faster Application Bootstrapping:
   Applications with heavy initialization work (compiling assets, loading large configurations, pre-warming caches) can reduce startup latency significantly by temporarily boosting CPU resources.

2. Resource-Intensive Data Migrations:
   When a task performs initial data migrations or data processing on startup, a temporary CPU boost can decrease processing times and reduce overall service downtime.

3. Autoscaling Improvements:
   In environments where tasks are rapidly scaled up in response to demand, a startup boost ensures that new instances become fully operational more quickly, enhancing responsiveness and user experience.

4. Efficient Batch Processing:
   Batch jobs that require substantial computation at the beginning can complete their critical phases faster, leading to improved throughput and resource utilization without the need for permanently higher resource allocations.

Attempted Solutions

Manually edit spec and adjust resource, but:

• Nomad create new allocation, instead just update resource on existing allocation or task
• Inefficient for quick response or batch job

[tgross]

Hi @habibimustafa! This is a pretty cool idea, but the actual implementation details might be shockingly hairy 😁. The resources.cpu field really represents two different concepts in one:

• A unit of CPU requests for the scheduler to figure out which workloads can run on which nodes. This is measured in MHz because that gives Nomad a way to compare apples-to-apples across nodes that might have different amounts of CPU capacity.
• After the allocation is placed on a node, the task driver takes over and does... whatever it wants with that value! For example, the docker driver submits the value to the docker daemon, and then Docker maps that to a limit with logic that depends on which version of cgroups a host is using (ex. it'll get mapped to a ratio of cpu.weight under cgroups v2). A different task driver might map that to millicores, or even not place restrictions at all.

So any implementation of the feature you're looking for here is really specific to the task driver. Ex. we'd need to somehow tell Docker to change the cgroups limit on the container after the fact, which isn't an API that Docker supports (ref Container API). It might be possible to "reach in" to the cgroup after the task driver isolation has been created to change the cgroup value temporarily. That'd only work on Linux, of course.

That being said, this isn't impossible... we already have something similar in place for resources.cores (and NUMA aware scheduling). I'll mark this for roadmapping and further discussion, but I also want to set expectations that this is a fairly large lift.

Depending on the task driver, you might have some options already to help out the specific use case you have. For example, by default with the Docker task driver cpu_hard_limit = false, which is why you get a cpu.weight (under cgroups v2) and not a CPU quota.