Linux Agent gets unhealthy on adding Linux integration.

[amolnater-qasource]

Kibana Build details:

VERSION: 8.17.0 BC1
BUILD: 80364
COMMIT: e3c75d19d796c366aedc5788960b2c6cc868014f

Artifact Link: https://staging.elastic.co/8.17.0-8031025a/downloads/beats/elastic-agent/elastic-agent-8.17.0-linux-x86_64.tar.gz

Host OS:
SLES15

Preconditions:

1. 8.17.0-BC1 Kibana cloud environment should be available.

Steps to reproduce:

1. Install Linux agent.
2. Add linux integration to this agent.
3. Observe agent gets unhealthy with errors in Linux integration:

Degraded
Error fetching data for metricset system.raid: failed to parse sysfs: no matches from path /sys/block

Degraded
Error fetching data for metricset linux.conntrack: error fetching conntrack stats: open /proc/net/stat/nf_conntrack: no such file or directory

Expected Result:
Linux Agent should remain healthy on adding Linux integration.

Screenshot:

Agent Logs:
elastic-agent-diagnostics-2024-11-27T08-39-33Z-00.zip

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mauri870]

@amolnater-qasource Can you confirm you have the conntrack module loaded on this system?

lsmod | grep conntrack

Also, was this system upgraded without a restart? This can cause failures sometimes.

[mauri870]

I tried deploying a SUSE 15 instance in Azure to debug this, I'm running the 8.17.0 version in the staging environment, I deployed the Beta Linux Metrics integration and couldn't find any errors from the agent:

$ uname -a
Linux mauri-suse 5.14.21-150500.33.66-azure #1 SMP PREEMPT_DYNAMIC Wed Sep 4 05:47:04 UTC 2024 (4885a53) x86_64 x86_64 x86_64 GNU/Linux
$ sudo elastic-agent status
┌─ fleet
│  └─ status: (HEALTHY) Connected
└─ elastic-agent
   └─ status: (HEALTHY) Running
azureuser@mauri-suse:~>

[mauri870]

Aha, after enabling Collect system metrics from Linux instances > Linux host raid metrics the agent went to a degraded state:

$ sudo elastic-agent status
┌─ fleet
│  └─ status: (HEALTHY) Connected
└─ elastic-agent
   ├─ status: (DEGRADED) 1 or more components/units in a degraded state
   └─ system/metrics-default
      ├─ status: (HEALTHY) Healthy: communicating with pid '3031'
      └─ system/metrics-default-system/metrics-system-7afde2f3-7310-4e56-8554-4847fa1c1567
         └─ status: (DEGRADED) Error fetching data for metricset system.raid: failed to parse sysfs: no matches from path /sys/block

[mauri870]

I just managed to reproduce the same issue on arch linux as well, so definitely not a SUSE issue.

[mauri870]

Could it be that we expect to find devices in raid mode, and if we find zero then we return an error? We should probably return nil in this case or treat the error accordingly.

https://github.com/elastic/beats/blob/42e25f7216862b6779c2e8a87a82c1ae30d9a6e1/metricbeat/module/system/raid/blockinfo/getdev.go#L46-L48

[pierrehilbert]

Wouldn't we have face this issue before if this is the root cause?
From what I can see, we didn't change anything there for a while.

Could it be a change in the newest version of Linux kernels that are disabling nf_conntrack by default?

[mauri870]

Does not seem to be the case. I confirmed that on SUSE and arch conntrack is loaded. SUSE is on the 5.14 kernel and Arch on 6.12, so I doubt this is related to the kernel itself.

cc @fearful-symmetry since you worked on the raid metrics.