[auditbeat] could not fetch events for auditd_manager on Debian 12 #47382
Description
The Auditd monitoring integration test was disabled in Elastic Agent CI following the latest Debian 12 VM update.
Filing an issue to track investigating a root cause. Logs from failing integration test:
=== Failed
--
| === FAIL: testing/integration/ess TestAuditdCorrectBinaries (unknown)
| tools.go:121: Creating enrollment API key...
| tools.go:112: Creating enrollment API key...
| fixture_install.go:200: [test TestAuditdCorrectBinaries] Inside fixture install function
| fixture_install.go:228: [test TestAuditdCorrectBinaries] Inside fixture installNoPkgManager function
| fetcher.go:102: Using existing artifact elastic-agent-9.1.7-SNAPSHOT-linux-x86_64.tar.gz
| fixture.go:311: Extracting artifact elastic-agent-9.1.7-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestAuditdCorrectBinaries4092918614
| fixture.go:329: Completed extraction of artifact elastic-agent-9.1.7-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestAuditdCorrectBinaries4092918614
| fixture.go:1011: Components were not modified from the fetched artifact
| fixture.go:684: >> running binary with: [/tmp/TestAuditdCorrectBinaries4092918614/elastic-agent-9.1.7-SNAPSHOT-linux-x86_64/elastic-agent install --force --non-interactive --url https://5826ee67d8f92844055b72356ab03816.fleet.us-west2.gcp.elastic-cloud.com:443 --enrollment-token aVQ4NEtwb0I5Q0gwakVwYTJ2aDY6Nzc4bHpRWmFhQjhkOFdOQVp0a0VSZw==]
| tools.go:112: >>> Enroll succeeded. Output: Installing in non-interactive mode.
| [ ===] Service Started [4s] Elastic Agent successfully installed, starting enrollment.
| [ ===] Waiting For Enroll... [4s] {"log.level":"info","@timestamp":"2025-10-28T09:49:37.249Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":536},"message":"Starting enrollment to URL: https://5826ee67d8f92844055b72356ab03816.fleet.us-west2.gcp.elastic-cloud.com:443/","ecs.version":"1.6.0"}
| {"log.level":"info","@timestamp":"2025-10-28T09:49:38.345Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":499},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
| {"log.level":"info","@timestamp":"2025-10-28T09:49:38.347Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).Execute","file.name":"cmd/enroll_cmd.go","file.line":317},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
| Successfully enrolled the Elastic Agent.
| [== ] Done [6s]
| Elastic Agent has been successfully installed.
| fixture.go:684: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
| tools.go:187: wanted fleet status to be HEALTHY, was STOPPED
| fixture.go:684: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
| tools.go:112: >>> Enrolled Agent ID: 555265a3-e5fc-40a5-ad18-c82a38e738c0
| fixture.go:684: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
| fixture_install.go:311: [test TestAuditdCorrectBinaries] Inside fixture cleanup function
| fixture_install.go:327: collecting diagnostics; test failed
| fixture.go:684: >> running binary with: [/opt/Elastic/Agent/elastic-agent diagnostics -f /opt/buildkite-agent/builds/bk-agent-prod-gcp-1761643596571755160/elastic/elastic-agent-extended-testing/build/diagnostics/TestAuditdCorrectBinaries-2025-10-28T10-00-01Z-diagnostics.zip]
| panic: Fail in goroutine after TestAuditdCorrectBinaries/TestBeatsMetrics has completed
|
| goroutine 993 [running]:
| testing.(*common).Fail(0xc00152c8c0)
| /opt/buildkite-agent/.asdf/installs/golang/1.24.9/go/src/testing/testing.go:988 +0xcb
| testing.(*common).Errorf(0xc00152c8c0, {0xdb6a8a9?, 0x2?}, {0xc001314860?, 0xcf202a0?, 0x13f0c4c0?})
| /opt/buildkite-agent/.asdf/installs/golang/1.24.9/go/src/testing/testing.go:1111 +0x5e
| github.com/stretchr/testify/assert.Fail({0xe9e4c60, 0xc00152c8c0}, {0xc0000c0be0, 0x47}, {0x0, 0x0, 0x0})
| /opt/buildkite-agent/.asdf/installs/golang/1.24.9/packages/pkg/mod/github.com/stretchr/[email protected]/assert/assertions.go:387 +0x370
| github.com/stretchr/testify/assert.NoError({0xe9e4c60, 0xc00152c8c0}, {0xe9e5140, 0xc000a52400}, {0x0, 0x0, 0x0})
| /opt/buildkite-agent/.asdf/installs/golang/1.24.9/packages/pkg/mod/github.com/stretchr/[email protected]/assert/assertions.go:1638 +0x125
| github.com/stretchr/testify/require.NoError({0xea23970, 0xc00152c8c0}, {0xe9e5140, 0xc000a52400}, {0x0, 0x0, 0x0})
| /opt/buildkite-agent/.asdf/installs/golang/1.24.9/packages/pkg/mod/github.com/stretchr/[email protected]/require/require.go:1398 +0xb0
| github.com/elastic/elastic-agent/testing/integration/ess.(*AuditDRunner).TestBeatsMetrics.func2()
| /opt/buildkite-agent/builds/bk-agent-prod-gcp-1761643596571755160/elastic/elastic-agent-extended-testing/testing/integration/ess/auditd_monitoring_test.go:118 +0x6ba
| github.com/stretchr/testify/assert.Eventually.func1()
| /opt/buildkite-agent/.asdf/installs/golang/1.24.9/packages/pkg/mod/github.com/stretchr/[email protected]/assert/assertions.go:1994 +0x23
| created by github.com/stretchr/testify/assert.Eventually in goroutine 882
| /opt/buildkite-agent/.asdf/installs/golang/1.24.9/packages/pkg/mod/github.com/stretchr/[email protected]/assert/assertions.go:2013 +0x21c
| auditd_monitoring_test.go:110: starting to ES for metrics at 2025-10-28T09:50:01.705739422Z
| auditd_monitoring_test.go:111:
| Error Trace: /opt/buildkite-agent/builds/bk-agent-prod-gcp-1761643596571755160/elastic/elastic-agent-extended-testing/testing/integration/ess/auditd_monitoring_test.go:111
| Error: Condition never satisfied
| Test: TestAuditdCorrectBinaries/TestBeatsMetrics
| Messages: could not fetch events for auditd_manager
| auditd_monitoring_test.go:105: executed at 2025-10-28T10:00:01.708106426Z: query: {"query":{"bool":{"must":[{"match":{"agent.id":"555265a3-e5fc-40a5-ad18-c82a38e738c0"}},{"exists":{"field":"auditd.summary.actor.primary"}}]}}}
|
| === FAIL: testing/integration/ess TestAuditdCorrectBinaries/TestBeatsMetrics (unknown)
| auditd_monitoring_test.go:110: starting to ES for metrics at 2025-10-28T09:50:01.705739422Z
| auditd_monitoring_test.go:111:
| Error Trace: /opt/buildkite-agent/builds/bk-agent-prod-gcp-1761643596571755160/elastic/elastic-agent-extended-testing/testing/integration/ess/auditd_monitoring_test.go:111
| Error: Condition never satisfied
| Test: TestAuditdCorrectBinaries/TestBeatsMetrics
| Messages: could not fetch events for auditd_manager
| auditd_monitoring_test.go:105: executed at 2025-10-28T10:00:01.708106426Z: query: {"query":{"bool":{"must":[{"match":{"agent.id":"555265a3-e5fc-40a5-ad18-c82a38e738c0"}},{"exists":{"field":"auditd.summary.actor.primary"}}]}}}