nomad.client.allocs.cpu.total_ticks sometimes reporting as 10k

[seanamos]

Nomad version: 1.9.6

We have occasionally been seeing the nomad.client.allocs.cpu.total_ticks metric being reported as 10 000 for some allocations. This seems to happen when the allocation is being terminated, in our case by the nomad autoscaler.

As a result of this, it skews the average CPU usage for the job/task very high and it triggers alerts for us of high CPU usage and a scale up response. Ironically this is triggered during scale down because of low usage.

[tgross]

Hi @seanamos! It would help us track this behavior down to know if whether there's a specific task driver in use here. Did you start seeing this behavior on upgrade to 1.9.6 or do you know you saw it previous to that.

[seanamos]

The task driver in use in the docker driver.

The introduction of the issue does seem to line up with when we rolled out 1.9.6 (graph has a max rollup applied):

While I originally reported this as the metric being reported as 10k, looking back over our metrics, you can see that it sometimes reports as other large values (4.6k in particular).

[tgross]

Thanks @seanamos. I know we had some issue with the Docker stats collection but not that particular symptom. My colleague @Juanadelacuesta landed a patch a9e7166 that appears to have gone into the 1.10.0 release we shipped yesterday (there's no changelog for it, so I'll need to check on that). This is fixing a problem where the stats were dropping to zero because of a bug where the polling interval had to be exactly 1s to work. But I wonder if it might be causing the opposite as well, depending on how total ticks were being calculated.

I'm going to mark this for roadmapping, but once you've upgraded to 1.10.0 (after pre-prod testing, of course 😁 ), I'd be curious to know if that helps.

[seanamos]

Thanks for looking into this @tgross . I'll let you know my findings after our 1.10x upgrade. Its fortunately not a frequent issue, it occurs only a few times a month, so manageable for now.