Description
Hello,
I'm using incus 6.0.3 on TrueNAS Scale (Now community edition) 25.04 Beta 1. I setup reporting by setting
core.metrics_address to 8444 and disabling authentication for the metrics endpoint as described in the docs. I then read the data through prometheus and grafana.
The problem:
I have created a debian VM (with the netinstall iso, not the linuxcontainer.org image) and gave it 4Gb of RAM. When starting the VM the metrics reported seem to be wrong. Both the metrics API endpoint and the incus top command.
This is what incus top shows:
+-----------------+-------------+---------+---------+
| INSTANCE NAME | CPU TIME(S) | MEMORY | DISK |
+-----------------+-------------+---------+---------+
| debian-test | 25.20 | | |
and this is the reported memory of the VM through the API:
incus_memory_MemAvailable_bytes{name="debian-test",project="default",type="virtual-machine"} 1.8446744073692858e+19
That would mean that there are 4 Exobytes of free memory.
The CPU time seems to also be reported as negative on some cores:
incus_cpu_seconds_total{cpu="4",mode="user",name="debian-test",project="default",type="virtual-machine"} -0.14
The MemAvailable metric does not change depending on how much memory is actually used by the VM. The cpu_seconds_total does however change over time. I am not sure why though. I tried to test if CPU usage impacts it but loading a benchmark on the VM does not change the graph significantly. Which is also weird since normally it should go up to close to 100%. Node exporter for example shows that the host CPU gets a utilization close to 100% but the incus graph stays relatively flat.
This can lead to some weird graphs like these:
(Green is the debian VM) Query: sum by(name) (irate(incus_cpu_seconds_total{mode!="idle"}[$__rate_interval])) this then gets divided by the number of cores and multiplied by 100.
This is the memory:
(Green is the debian VM) Query: (incus_memory_MemTotal_bytes - incus_memory_MemFree_bytes - incus_memory_Cached_bytes)
These are the graphs when I run a benchmark inside the VM:
In this case the API returns this: (Only copied it for one Core, the others are basically the same)
incus_cpu_seconds_total{cpu="5",mode="iowait",name="debian-test",project="default",type="virtual-machine"} 0
incus_cpu_seconds_total{cpu="5",mode="irq",name="debian-test",project="default",type="virtual-machine"} 0
incus_cpu_seconds_total{cpu="5",mode="idle",name="debian-test",project="default",type="virtual-machine"} 0
incus_cpu_seconds_total{cpu="5",mode="nice",name="debian-test",project="default",type="virtual-machine"} 0
incus_cpu_seconds_total{cpu="5",mode="softirq",name="debian-test",project="default",type="virtual-machine"} 0
incus_cpu_seconds_total{cpu="5",mode="steal",name="debian-test",project="default",type="virtual-machine"} 0
incus_cpu_seconds_total{cpu="5",mode="system",name="debian-test",project="default",type="virtual-machine"} 19.77
incus_cpu_seconds_total{cpu="5",mode="user",name="debian-test",project="default",type="virtual-machine"} 2.79
Which honestly looks fine to me but I'm by no means an expert, maybe for the CPU part I misconfiguration something in the dashboard.
Activity