Linux docker agent gets Unhealthy on adding linux integration.

[amolnater-qasource]

Kibana version: 8.7 BC6 Kibana cloud environment

Host OS:
Ubuntu 22 ARM64

Build details:
VERSION: 8.7 BC6
BUILD: 61051
COMMIT: 04ef24287f26854ad99a46ae983854c6184717cb

Preconditions:

1. 8.7 BC6 Kibana cloud environment should be available.
2. Docker setup should be done.

Steps to reproduce:

1. Install a docker agent using below command:

sudo docker run \
--env FLEET_ENROLL=1 \
--env FLEET_URL=<Fleet Server host URL> \
--env FLEET_ENROLLMENT_TOKEN=<enrollment token>
--rm docker.elastic.co/staging/elastic-agent:8.7.0-a7fb3750

2. Add linux integration to this policy and observe agent goes Unhealthy.

Note:

• All linux integration datasets should be enabled.

Expected Result:
Docker agent should remain healthy on adding linux integration.

Screen Recording:

Agents.-.Fleet.-.Elastic.-.Google.Chrome.2023-03-16.23-07-34.mp4

Logs:
elastic-agent-diagnostics-2023-03-16T17-37-58Z-00.zip
elastic-agent-diagnostics-2023-03-16T17-43-24Z-00.zip

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Secondary review for this ticket is Done

[jlind23]

@amolnater-qasource Looks like we faced some permission issues:
{"log.level":"error","@timestamp":"2023-03-16T17:33:32.349Z","message":"Error fetching data for metricset linux.pageinfo: error opening file: open /proc/pagetypeinfo: permission denied","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"linux/metrics-default","type":"linux/metrics"},"log":{"source":"linux/metrics-default"},"log.origin":{"file.line":256,"file.name":"module/wrapper.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

@fearful-symmetry does it ring a bell or should I ask the obs-service team to look at this specific integration first?

[fearful-symmetry]

@jlind23 There could be a few issues here; The original issue mentions docker, so it's possible that we need to set hostfs correctly and insure that /proc/pagetypeinfo is mounted into the container as /hostfs/proc/pagetypeinfo. It's also possible that /proc/pagetypeinfo does not exist on this particular OS at all.

[jlind23]

@amolnater-qasource could you please check what @fearful-symmetry said? On a side note, was this particular docker distribution working before now?

[amolnater-qasource]

Hi @fearful-symmetry @jlind23

Thank you for looking into this issue.

We observed /proc/pagetypeinfo is setup on the the using VM. Could you please confirm how we can check if it is mounted into the container?

Further, this issue was earlier observed during 8.5.0 SNAPSHOT testing, reported under #1454
However later this was working fine on 8.6 BC10.

Please let us know if we are missing anything here.
Thanks!

[jlind23]

@amolnater-qasource can't you ssh in this container and see if it is mounted? Are you relying on a different base docker image?

[fearful-symmetry]

Ah, brain skipped a beat, just noticed that it's actually a permissions error: /proc/pagetypeinfo: permission denied

I'm fairly certain that pagetypeinfo is one of those procfs files that's going to be the same as the host from within the container, which means it's not strictly necessary to mount it, and you can read from /proc/pagetypeinfo from within the container to monitor the host, but the permission error is a bit odd. Since the original issue mentions docker, my assumption is that there's a docker setup issue, and the metricbeat instance running in docker somehow doesn't have the proper permissions, or isn't running as root.

[amolnater-qasource]

Hi @jlind23

For testing the docker agent we followed below steps:

1. Setup Ubuntu 22.04 ARM architecture VM.
2. Installed Docker on the VM.
3. Directly ran below command:

sudo docker run \
--env FLEET_ENROLL=1 \
--env FLEET_URL=<Fleet Server host URL> \
--env FLEET_ENROLLMENT_TOKEN=<enrollment token>
--rm docker.elastic.co/staging/elastic-agent:8.7.0-a7fb3750

So, as per our understanding we aren't creating any new container for this and we are using this docker image for installing an agent.

Please let us know if we are missing anything here.
Thanks

[jlind23]

@fearful-symmetry would be great to have your eyes on this as soon as you have time to make sure this is not a regression we introduced in metricbeat.

[amolnater-qasource]

Hi Team,

We have revalidated this issue on latest 8.8 BC6 Kibana cloud environment and found it still reproducible.

Observations:

• Linux docker agent gets Unhealthy on adding linux integration.

Screenshot:

Logs:
elastic-agent-diagnostics-2023-05-19T08-18-10Z-00.zip

Build details:

VERSION: 8.8.0 BC6 Kibana cloud environment
BUILD: 63115
COMMIT: a4c256b39f7d1ee34abe61109a817ec7f5329009
Docker artifact: --rm docker.elastic.co/staging/elastic-agent:8.8.0-375abdf7 

Please let us know if anything else is required from our end.

Thanks!

[cmacknz]

This is a new error in the system metrics input:

- id: system/metrics-default
  state:
    state: 2
    message: 'Healthy: communicating with pid ''32'''
    units:
      ? unittype: 0
        unitid: system/metrics-default-system/metrics-system-aa6c87f0-f61c-11ed-b6d2-0b368c0c212a
      : state: 4
        message: '[failed to reloading inputs: 2 errors: Error creating runner from
          config: 1 error: error connecting to dbus: dial unix /var/run/dbus/system_bus_socket:
          connect: no such file or directory; Error creating runner from config: 1
          error: error connecting to dbus: error getting connection to system bus:
          dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory]'