[Flaky Test]: TestAgentMetricsInput/agent – Condition never satisfied

[pchila]

Failing test case

TestAgentMetricsInput/agent
TestAgentMetricsInput/otel

Error message

"0" is not greater than "0" - Expected to find at least one document for metricset cpu in index .ds-metrics-system.cpu-gcztjmsagwwexlistvthvn9voqpxpp5021pbnsxudmy* and runtime "process", got 0

Build

https://buildkite.com/elastic/elastic-agent/builds/30752#019aa526-b6dc-4148-9ed4-ceef8ad853e7/4193-4508
https://buildkite.com/elastic/elastic-agent/builds/30752#019aa526-b6dc-4148-9ed4-ceef8ad853e7/4193-4577

OS

Linux

Stacktrace and notes

beat_receivers_test.go:557:
        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-aws-1763708434859356229/elastic/elastic-agent/testing/integration/ess/beat_receivers_test.go:568
        	            				/opt/buildkite-agent/.asdf/installs/golang/1.24.10/go/src/runtime/asm_arm64.s:1223
        	Error:      	"0" is not greater than "0"
        	Messages:   	docs count
    beat_receivers_test.go:557:
        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-aws-1763708434859356229/elastic/elastic-agent/testing/integration/ess/beat_receivers_test.go:557
        	Error:      	Condition never satisfied
        	Test:       	TestAgentMetricsInput/agent
        	Messages:   	Expected to find at least one document for metricset cpu in index .ds-metrics-system.cpu-gcztjmsagwwexlistvthvn9voqpxpp5021pbnsxudmy* and runtime "process", got 0

 beat_receivers_test.go:557:
        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-aws-1763708434859356229/elastic/elastic-agent/testing/integration/ess/beat_receivers_test.go:568
        	            				/opt/buildkite-agent/.asdf/installs/golang/1.24.10/go/src/runtime/asm_arm64.s:1223
        	Error:      	"0" is not greater than "0"
        	Messages:   	docs count
    beat_receivers_test.go:557:
        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-aws-1763708434859356229/elastic/elastic-agent/testing/integration/ess/beat_receivers_test.go:557
        	Error:      	Condition never satisfied
        	Test:       	TestAgentMetricsInput/otel
        	Messages:   	Expected to find at least one document for metricset cpu in index .ds-metrics-system.cpu-gcztjmsagwwexlistvthvn9voqpxpp5021pbnsxudmy* and runtime "otel", got 0

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[swiatekm]

Looks like the cause was an indexing problem in ES:

{"log.level":"warn","@timestamp":"2025-11-21T07:47:32.156Z","message":"Failed to index 11 events in last 10s: events were dropped! Look at the event log to view the event and cause.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"elasticsearch","log.origin":{"file.line":179,"file.name":"elasticsearch/client.go","function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.NewClient.func3"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

Interestingly, logs indexed just fine.

From the event log:

Validation Failed: 1: this action would add [2] shards, but this cluster currently has [999]/[1000] maximum normal shards open; for more information, see https://www.elastic.co/docs/deploy-manage/production-guidance/optimize-performance/size-shards?version=master#troubleshooting-max-shards-open

[swiatekm]

#11348 should have fixed this, so I'm closing it. We can reopen if we see it again.