TestSensitiveLogsESExporter: "0" is not greater than or equal to "1"

[cmacknz]

Failing test case

TestSensitiveLogsESExporter

Error message

"0" is not greater than or equal to "1"

Build

https://buildkite.com/elastic/elastic-agent/builds/29872#019a4c92-3ee2-4582-bf3b-7d7aaf234a20

OS

Linux

Stacktrace and notes

=== RUN   TestSensitiveLogsESExporter
    fetcher.go:102: Using existing artifact elastic-agent-9.2.1-SNAPSHOT-linux-x86_64.tar.gz
    fixture.go:311: Extracting artifact elastic-agent-9.2.1-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestSensitiveLogsESExporter2014293634
    fixture.go:329: Completed extraction of artifact elastic-agent-9.2.1-SNAPSHOT-linux-x86_64.tar.gz to /tmp/TestSensitiveLogsESExporter2014293634
    fixture.go:1018: Components were not modified from the fetched artifact
    fixture_install.go:200: [test TestSensitiveLogsESExporter] Inside fixture install function
    fixture_install.go:228: [test TestSensitiveLogsESExporter] Inside fixture installNoPkgManager function
    fixture_install.go:247: >> running binary with: [/tmp/TestSensitiveLogsESExporter2014293634/elastic-agent-9.2.1-SNAPSHOT-linux-x86_64/elastic-agent install --force]
    fixture.go:812: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    fixture.go:812: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    fixture.go:812: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    fixture.go:812: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    fixture.go:812: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    beat_receivers_test.go:1253: 
        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-gcp-1762221254330026374/elastic/elastic-agent/testing/integration/ess/beat_receivers_test.go:1261
        	            				/opt/buildkite-agent/.asdf/installs/golang/1.24.9/go/src/runtime/asm_amd64.s:1700
        	Error:      	"0" is not greater than or equal to "1"
    beat_receivers_test.go:1253: 
        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-gcp-1762221254330026374/elastic/elastic-agent/testing/integration/ess/beat_receivers_test.go:1253
        	Error:      	Condition never satisfied
        	Test:       	TestSensitiveLogsESExporter
        	Messages:   	Expected at least 1 log, got 0
    fixture_install.go:311: [test TestSensitiveLogsESExporter] Inside fixture cleanup function
    fixture_install.go:327: collecting diagnostics; test failed
    fixture_install.go:789: >> running binary with: [/opt/Elastic/Agent/elastic-agent diagnostics -f /opt/buildkite-agent/builds/bk-agent-prod-gcp-1762221254330026374/elastic/elastic-agent/build/diagnostics/TestSensitiveLogsESExporter-2025-11-04T02-43-29Z-diagnostics.zip]
    fixture_install.go:740: >> running binary with: [/opt/Elastic/Agent/elastic-agent uninstall --force]
    fixture.go:1143: Dumping running processes in /opt/buildkite-agent/builds/bk-agent-prod-gcp-1762221254330026374/elastic/elastic-agent/build/diagnostics/TestSensitiveLogsESExporter-2025-11-04T02-43-29Z-ProcessDump-cleanup.json
    fixture.go:1434: Temporary directory "/tmp/TestSensitiveLogsESExporter2014293634" preserved for investigation/debugging
--- FAIL: TestSensitiveLogsESExporter (139.76s)
panic: runtime error: index out of range [0] with length 0 [recovered]
	panic: runtime error: index out of range [0] with length 0

goroutine 1760 [running]:
testing.tRunner.func1.2({0xd708240, 0xc00197a000})
	/opt/buildkite-agent/.asdf/installs/golang/1.24.9/go/src/testing/testing.go:1734 +0x21c
testing.tRunner.func1()
	/opt/buildkite-agent/.asdf/installs/golang/1.24.9/go/src/testing/testing.go:1737 +0x35e
panic({0xd708240?, 0xc00197a000?})
	/opt/buildkite-agent/.asdf/installs/golang/1.24.9/go/src/runtime/panic.go:792 +0x132
github.com/elastic/elastic-agent/testing/integration/ess.TestSensitiveLogsESExporter(0xc00150f880)
	/opt/buildkite-agent/builds/bk-agent-prod-gcp-1762221254330026374/elastic/elastic-agent/testing/integration/ess/beat_receivers_test.go:1266 +0x1d95
testing.tRunner(0xc00150f880, 0xe127648)
	/opt/buildkite-agent/.asdf/installs/golang/1.24.9/go/src/testing/testing.go:1792 +0xf4
created by testing.(*T).Run in goroutine 1
	/opt/buildkite-agent/.asdf/installs/golang/1.24.9/go/src/testing/testing.go:1851 +0x413

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[cmacknz]

The panic is likely because the assert.EventuallyWithT should be a require.EventuallyWithT

Nothing obvious jumps out from the logs, @khushijain21 take a look when you get a chance. Possibly this is just a timing problem with the query timeout being too short.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[swiatekm]

More failures, after the timeout increase:
https://buildkite.com/elastic/elastic-agent/builds/30747#019aa3d6-45e3-4281-b9ef-f382dc4b9026
https://buildkite.com/elastic/elastic-agent/builds/30746#019aa3d6-4ccd-47bd-acc7-66cfeccdb96b

[swiatekm]

There's a decent chance #11348 will help with this, although I haven't been able to confirm, as this test doesn't preserve diagnostics.