VM stealtime>0, process.Percent(0) / cpu.Counts(true) is higher than 100%

[hitzhangjie]

Describe the bug
We have following code to update the per-cpu usage, it works well in normal machine.

Recently, one of our service instances keep reporting errors. It's deployed in a VM, and that time, the host report stealtime>0. And from our logging, we see the newv here is larger than 100%.

To Reproduce

	ncpu, err := cpu.Counts(true)
	if err != nil {
		logger.Error("load num of CPUs fail ... overload=false", zap.Error(err))
		return false
	}

	use, err := c.process.Percent(0)
	if err != nil {
		logger.Error("load process CPU usage fail ... overload=false", zap.Error(err))
		return false
	}

	oldv := c.percpu.Load()
	newv := use / float64(ncpu)

Expected behavior
process.Percent(0) / cpu.Counts(true) return the right cpu usage.

Environment (please complete the following information):

• Linux df-cnprod-set212-a-025-nj2 5.4.241-1-tlinux4-0023.1 #1 SMP Tue May 6 19:25:45 CST 2025 x86_64 x86_64 x86_64 GNU/Linux

Additional context
No.

Here's the log：

update per-cpu load {"process": 21012, "oldv": 51.060771011525944, "newv": 354.0590442937935}

And the logic above run every second.

[shirou]

Could you share the results of the following in your environment?

• runtime.NumCPU()
• cpu.Counts(true)
• cpu.Counts(false)

This might be caused by a mismatch between the CPU count seen by Go runtime and the actual logical CPU count in your VM.

And cloud you try to use runtime.NumCPU() instead of ncpu ?

use, _ := process.Percent(0)
newv := use / float64(runtime.NumCPU())

[hitzhangjie]

I expected runtime.NumCPU() and cpu.Counts(true) to both return 32. After migrating to the new host, the output remains 32 as expected.

Context & Observations: Our IaaS partner detected split-locks occurring on one of the VMs. These split-locks cause memory access to stall, triggering internal CPU retries. Consequently, system monitoring tools (like top or atop) show significantly higher CPU usage.

Regarding Steal Time: Our IaaS partner noted that steal time is roughly 0.0035% and suggested it is negligible. However, we observed that as steal time increased slightly, server performance trended downward. That said, we believe the primary cause of the high CPU usage is the split-lock issue.

My understanding is that CPU usage should not exceed 100% per core, and cpu.Counts(true) should consistently return 32 since it reads from /proc/cpuinfo and /proc/stat.

Unfortunately, because the VM has been migrated, I can no longer verify these files or re-run the tests on the original host.

---

Maybe it affects the time relevant logic, the time delta becomes smaller than expected. I'm not for sure.

[hitzhangjie]

Hypervisor and VM sync: tick catch up mechanism.

I'm not very familiar with this area; I basically asked AI about it. I am not 100% sure about this. Here's just some thoughts.

---

Assuming the formula is Overall CPU Usage = (t2/t1) x 100%,

t2 is calculated using CPU jiffies. AI mentioned that the update of jiffies is driven by clock interrupts.
In virtualization, t1 likely directly uses the host's clock source, kvm_clock, which should be accurate.

I'm not sure if this situation could occur when the host performance degrades: for example, if clock interrupts are not delivered to the VM in time, and they are all given to it at once when the VM is scheduled next. If so, the result maybe large than 100%.

[qinjinze]

There is indeed a false alarm on the virtual machine, with a display value of 0% or 100%

[shirou]

This is the expected behavior, consistent with Python's psutil which we reference in our implementation.

psutil explicitly documents that cpu_percent() can return values > 100% for multi-threaded processes across multiple cores, and does not clamp the result: see this comment

In your VM case with steal time / split-lock, the CPU jiffies can advance faster than wall-clock time, which causes the ratio to exceed the normal maximum. This is an accurate reflection of what the kernel is reporting.

If you need a clamped value in your application, you can cap it on the caller side:

use, _ := process.Percent(0)
perCPU := math.Min(use/float64(ncpu), 100.0)