[beats receivers] Memory usage calculated by Fleet when using beats receivers is lower than expected

[cmacknz]

• Relates Memory usage for metrics monitoring components is not reported to Fleet #10274
• Relates [Fleet] Fix agent memory query kibana#174458

This test was done using 9.2.0-SNAPSHOT with with Elastic Agent self-monitoring using beats receivers and the system integration using the process runtime on Ubuntu 24.04.

systemctl status shows Elastic Agent using 521.3M of memory.

ubuntu@oriented-leafcutter:~/elastic-agent-9.2.0-SNAPSHOT-linux-arm64$ systemctl status elastic-agent
● elastic-agent.service - Elastic Agent is a unified agent to observe, monitor and protect your system.
     Loaded: loaded (/etc/systemd/system/elastic-agent.service; enabled; preset: enabled)
     Active: active (running) since Thu 2025-10-02 16:57:07 EDT; 4min 10s ago
   Main PID: 78275 (elastic-agent)
      Tasks: 32 (limit: 1059)
     Memory: 521.3M (peak: 609.0M)
        CPU: 4.087s
     CGroup: /system.slice/elastic-agent.service
             ├─78275 /opt/Elastic/Agent/elastic-agent
             ├─78291 /opt/Elastic/Agent/data/elastic-agent-9.2.0-SNAPSHOT-e917e0/elastic-agent otel --supervised --s>
             ├─78296 /opt/Elastic/Agent/data/elastic-agent-9.2.0-SNAPSHOT-e917e0/components/agentbeat filebeat -E se>
             └─78304 /opt/Elastic/Agent/data/elastic-agent-9.2.0-SNAPSHOT-e917e0/components/agentbeat metricbeat -E >

Fleet calculates agent as using 205M of memory:

The Elastic Agent metrics dashboards shows Fleet is correctly calculating the memory usage based on the information that it has, the values are just lower that what Linux reports via systemctl.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

ubuntu@oriented-leafcutter:~/elastic-agent-9.2.0-SNAPSHOT-linux-arm64$ systemctl status elastic-agent
● elastic-agent.service - Elastic Agent is a unified agent to observe, monitor and protect your system.
     Loaded: loaded (/etc/systemd/system/elastic-agent.service; enabled; preset: enabled)
     Active: active (running) since Thu 2025-10-02 16:57:07 EDT; 2h 4min ago
   Main PID: 78275 (elastic-agent)
      Tasks: 34 (limit: 1059)
     Memory: 472.1M (peak: 609.0M)
        CPU: 57.334s
     CGroup: /system.slice/elastic-agent.service
             ├─78275 /opt/Elastic/Agent/elastic-agent
             ├─78291 /opt/Elastic/Agent/data/elastic-agent-9.2.0-SNAPSHOT-e917e0/elastic-agent otel --supervised --s>
             ├─78296 /opt/Elastic/Agent/data/elastic-agent-9.2.0-SNAPSHOT-e917e0/components/agentbeat filebeat -E se>
             └─78304 /opt/Elastic/Agent/data/elastic-agent-9.2.0-SNAPSHOT-e917e0/components/agentbeat metricbeat -E >

Looking under the hood a bit:

ubuntu@oriented-leafcutter:~/elastic-agent-9.2.0-SNAPSHOT-linux-arm64$ cat /sys/fs/cgroup/system.slice/elastic-agent.service/memory.current
489504768

The memory.current appears to be the sum of anon, file, and kernel. It's not clear why this doesn't exactly match what systemctl status reports.

ubuntu@oriented-leafcutter:~/elastic-agent-9.2.0-SNAPSHOT-linux-arm64$ cat /sys/fs/cgroup/system.slice/elastic-agent.service/memory.stat
anon 352440320
file 131354624
kernel 5709824
kernel_stack 540672
pagetables 3751936
sec_pagetables 0
percpu 0
sock 0
vmalloc 8192
shmem 0
zswap 0
zswapped 0
file_mapped 120045568
file_dirty 16384
file_writeback 0
swapcached 0
anon_thp 0
file_thp 0
shmem_thp 0
inactive_anon 256704512
active_anon 95735808
inactive_file 100900864
active_file 30453760
unevictable 0
slab_reclaimable 804200
slab_unreclaimable 556936
slab 1361136
workingset_refault_anon 0
workingset_refault_file 675
workingset_activate_anon 0
workingset_activate_file 384
workingset_restore_anon 0
workingset_restore_file 371
workingset_nodereclaim 0
pgscan 73996
pgsteal 43994
pgscan_kswapd 73996
pgscan_direct 0
pgscan_khugepaged 0
pgsteal_kswapd 43994
pgsteal_direct 0
pgsteal_khugepaged 0
pgfault 174484
pgmajfault 1986
pgrefill 14700
pgactivate 1536
pgdeactivate 0
pglazyfree 0
pglazyfreed 0
zswpin 0
zswpout 0
zswpwb 0
thp_fault_alloc 0
thp_collapse_alloc 0
thp_swpout 0
thp_swpout_fallback 0

[cmacknz]

We report a number that agrees with the cgroup accounting but that's not what we report to Fleet. See memory.mem.usage.bytes in the agent stats.

● elastic-agent.service - Elastic Agent is a unified agent to observe, monitor and protect your system.
     Loaded: loaded (/etc/systemd/system/elastic-agent.service; enabled; preset: enabled)
     Active: active (running) since Thu 2025-10-02 16:57:07 EDT; 23h ago
   Main PID: 78275 (elastic-agent)
      Tasks: 35 (limit: 1059)
     Memory: 401.1M (peak: 609.0M)
        CPU: 11min 583ms
     CGroup: /system.slice/elastic-agent.service


ubuntu@oriented-leafcutter:~/diag$ curl localhost:6791/stats
{
  "beat": {
    "cgroup": {
      "cpu": {
        "id": "elastic-agent.service",
        "stats": {
          "periods": 0,
          "throttled": {
            "ns": 0,
            "periods": 0
          }
        }
      },
      "memory": {
        "id": "elastic-agent.service",
        "mem": {
          "usage": {
            "bytes": 420945920
          }
        }
      }
    },
    "memstats": {
      "gc_next": 45851954,
      "memory_alloc": 21569008,
      "memory_sys": 78733592,
      "memory_total": 2250815832,
      "rss": 115658752
    },

[cmacknz]

We seem to be reporting the memstats.memory_sys to Fleet as system.process.memory.size.

elastic-agent/internal/pkg/agent/application/monitoring/component/v1_monitor.go

Lines 1462 to 1467 in 92d5c1d

	// I should be able to see the Memory usage of Elastic Agent. Is the Elastic Agent using too much memory?
	map[string]interface{}{
	"from": "http.agent.beat.memstats.memory_sys",
	"to": "system.process.memory.size",
	},

[cmacknz]

I moved this to the 9.3 milestone, I don't think solving this is urgent right now that we have some higher priority bugs.

[cmacknz]

Moved to 9.4.0 milestone.