Please confirm
Distribution
Ubuntu
Distribution version
24.04
Output of snap list --all lxd core20 core22 core24 snapd
Name Version Rev Tracking Publisher Notes
core22 20260128 2339 latest/stable canonical✓ base,disabled
core22 20260225 2411 latest/stable canonical✓ base
core24 20260211 1499 latest/stable canonical✓ base,disabled
core24 20260317 1587 latest/stable canonical✓ base
lxd 6.7-12e2019 38450 6/stable canonical✓ disabled
lxd 6.7-d814d89 38768 6/stable canonical✓ -
snapd 2.73 25935 latest/stable canonical✓ snapd,disabled
snapd 2.74.1 26382 latest/stable canonical✓ snapd
System info
Kernel 6.8.0-100-generic
LXD 6.7
Instance log
n/a
Expected behavior
Used and Free should show actual memory consumption. This memory information has been made more widely visible now that it is also shown in the UI, which is where I noticed the data being incorrect.
Fix number one
https://github.com/canonical/lxd/blob/main/lxd/resources/memory.go#L222
The memory Used is calculated as memory.Used = info.Total - info.Free - info.Cached - info.Buffers
It should be memory.Used = info.Total - info.Free.
MemAvailable already takes reclaimability into account, it is currently being accounted twice for.
Fix number two
Grab TotalMemory from /proc/meminfo instead of /sys/devices/system/memory. These values are not the same. On my 48GiB system there is a 1.14GiB difference between the two due to kernel reservations; falsely adding ~1.14GiB to Used memory.
Actual behavior
Bug 1: Shmem included
lxc info --resources (and the UI that uses the same api endpoint) report inaccurate memory usage across all my nodes. Depending on the actual usage, Used can be either significantly lower or higher compared to actual system memory usage as reported by free -h, htop, and /proc/meminfo.
https://github.com/canonical/lxd/blob/main/lxd/resources/memory.go#L222
The memory Used is calculated as memory.Used = info.Total - info.Free - info.Cached - info.Buffers
The error here is that the Cached field in /proc/meminfo includes Shmem (tmpfs and shared memory), which is actively consumed by processes and instances, it is not reclaimable page cache. This causes LXD to treat potentially several GiB of real memory usage as free.
Bug 2: TotalMemory from Hardware instead of System
Another issue is TotalMemory being grabbed from /sys/devices/system/memory instead of /proc/memory. On my systems with 48GB memory, this shows a difference of 1.14GiB between the two values.
With the current calculations being done, this will incorrectly add 1.14GiB of memory to "Used".
Physical: 383 * 128 MiB = 49,024 MiB = 47.875 GiB
MemTotal: 49,010,100 kB = 49,010,100 / 1024 / 1024 = 46.74 GiB
cat /sys/devices/system/memory/block_size_bytes
8000000
grep -l "online" /sys/devices/system/memory/memory*/state | wc -l
383
grep MemTotal /proc/meminfo
MemTotal: 49010100 kB
Example data from 3 nodes:
dragonstone, a standalone LXD 6.7 node with high memory/shmem usage.
This node shows roughly 10GiB memory as free where it isn't.
grep -E "MemTotal|MemFree|MemAvailable|Buffers|^Cached|Shmem|SReclaimable" /proc/meminfo
MemTotal: 32569240 kB
MemFree: 683880 kB
MemAvailable: 10189176 kB
Buffers: 5796 kB
Cached: 20684916 kB # includes 11.15 GiB Shmem
Shmem: 11153292 kB
SReclaimable: 453568 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
root@dragonstone:/home/vos# lxc info --resources | grep Memory -A4
Memory:
Free: 20.37GiB # roughly 10 GiB higher because of Bug 1
Used: 11.63GiB # roughly 10 GiB too low
Total: 32.00GiB
root@dragonstone:/home/vos# free -h
total used free shared buff/cache available
Mem: 31Gi 21Gi 657Mi 10Gi 20Gi 9.7Gi
Swap: 0B 0B 0B
A cluster node with few VMs / low memory consumption:
vos@sapphire:~$ free -h
total used free shared buff/cache available
Mem: 46Gi 10Gi 23Gi 4.6Gi 17Gi 35Gi
Swap: 4.0Gi 0B 4.0Gi
vos@sapphire:~$ grep -E "MemTotal|MemFree|MemAvailable|Buffers|^Cached|Shmem|SReclaimable" /proc/meminfo
MemTotal: 49010100 kB
MemFree: 24982088 kB
MemAvailable: 37748060 kB
Buffers: 15744 kB
Cached: 17227808 kB
Shmem: 4829584 kB
SReclaimable: 960464 kB
ShmemHugePages: 8192 kB
ShmemPmdMapped: 0 kB
vos@sapphire:~$ lxc info --resources | grep Memory -A4
Memory:
Free: 40.26GiB
Used: 7.61GiB
Total: 47.88GiB
And finally, a cluster member that has zero VMs/LXCs running with basically zero shmem usage that clearly shows bug number 2:
vos@emerald:~$ free -h
total used free shared buff/cache available
Mem: 46Gi 5.9Gi 29Gi 17Mi 12Gi 40Gi
Swap: 4.0Gi 0B 4.0Gi
vos@emerald:~$ grep -E "MemTotal|MemFree|MemAvailable|Buffers|^Cached|Shmem|SReclaimable" /proc/meminfo
MemTotal: 49010352 kB # 46.74 GiB
MemFree: 30850612 kB
MemAvailable: 42828824 kB
Buffers: 11796 kB
Cached: 11750716 kB
Shmem: 17688 kB
SReclaimable: 841828 kB
ShmemHugePages: 8192 kB
ShmemPmdMapped: 0 kB
vos@emerald:~$ lxc info --resources | grep Memory -A4
Memory:
Free: 40.64GiB
Used: 7.24GiB # +- 1.14 GiB overstated vs free -h's 5.9Gi
Total: 47.88GiB # +-1.14 GiB larger than MemTotal, bug 2
Steps to reproduce
See my steps in Actual Behaviour.
Information to attach
Please confirm
Distribution
Ubuntu
Distribution version
24.04
Output of
snap list --all lxd core20 core22 core24 snapdSystem info
Kernel 6.8.0-100-generic
LXD 6.7
Instance log
n/a
Expected behavior
Used and Free should show actual memory consumption. This memory information has been made more widely visible now that it is also shown in the UI, which is where I noticed the data being incorrect.
Fix number one
https://github.com/canonical/lxd/blob/main/lxd/resources/memory.go#L222
The memory Used is calculated as
memory.Used = info.Total - info.Free - info.Cached - info.BuffersIt should be
memory.Used = info.Total - info.Free.MemAvailablealready takes reclaimability into account, it is currently being accounted twice for.Fix number two
Grab TotalMemory from /proc/meminfo instead of /sys/devices/system/memory. These values are not the same. On my 48GiB system there is a 1.14GiB difference between the two due to kernel reservations; falsely adding ~1.14GiB to Used memory.
Actual behavior
Bug 1: Shmem included
lxc info --resources(and the UI that uses the same api endpoint) report inaccurate memory usage across all my nodes. Depending on the actual usage,Usedcan be either significantly lower or higher compared to actual system memory usage as reported byfree -h,htop, and/proc/meminfo.https://github.com/canonical/lxd/blob/main/lxd/resources/memory.go#L222
The memory Used is calculated as
memory.Used = info.Total - info.Free - info.Cached - info.BuffersThe error here is that the Cached field in /proc/meminfo includes Shmem (tmpfs and shared memory), which is actively consumed by processes and instances, it is not reclaimable page cache. This causes LXD to treat potentially several GiB of real memory usage as free.
Bug 2: TotalMemory from Hardware instead of System
Another issue is TotalMemory being grabbed from /sys/devices/system/memory instead of /proc/memory. On my systems with 48GB memory, this shows a difference of 1.14GiB between the two values.
With the current calculations being done, this will incorrectly add 1.14GiB of memory to "Used".
Example data from 3 nodes:
dragonstone, a standalone LXD 6.7 node with high memory/shmem usage.This node shows roughly 10GiB memory as free where it isn't.
A cluster node with few VMs / low memory consumption:
And finally, a cluster member that has zero VMs/LXCs running with basically zero shmem usage that clearly shows bug number 2:
Steps to reproduce
See my steps in Actual Behaviour.
Information to attach
dmesg)lxc config show <instance> --expanded)/var/log/lxd/lxd.logor/var/snap/lxd/common/lxd/logs/lxd.log)--debug--debug(or uselxc monitorwhile reproducing the issue)