[Flaky Test]: `TestStandaloneUpgradeRollback`– `agent state is not healthy: got 3`

[ebeahan]

Failing test case

TestStandaloneUpgradeRollback

Error message

Received unexpected error: agent state is not healthy: got 3

Build

https://buildkite.com/elastic/elastic-agent-extended-testing/builds/11202#019a9d67-7f0a-4df7-990d-c43fda22b519

OS

Linux

Stacktrace and notes

=== Failed
=== FAIL: testing/integration/ess TestStandaloneUpgradeRollback (312.16s)
    fetcher.go:102: Using existing artifact elastic-agent-9.2.1-linux-arm64.tar.gz
    fixture.go:338: Extracting artifact elastic-agent-9.2.1-linux-arm64.tar.gz to /tmp/TestStandaloneUpgradeRollback379987891
    fixture.go:356: Completed extraction of artifact elastic-agent-9.2.1-linux-arm64.tar.gz to /tmp/TestStandaloneUpgradeRollback379987891
    fixture.go:1045: Components were not modified from the fetched artifact
    fixture.go:888: >> running binary with: [/tmp/TestStandaloneUpgradeRollback379987891/elastic-agent-9.2.1-linux-arm64/elastic-agent version --binary-only --yaml]
    upgrade_rollback_test.go:83: Testing Elastic Agent upgrade from 9.2.1 to 9.3.0-SNAPSHOT...
    fetcher.go:102: Using existing artifact elastic-agent-9.3.0-SNAPSHOT-linux-arm64.tar.gz
    fixture.go:338: Extracting artifact elastic-agent-9.3.0-SNAPSHOT-linux-arm64.tar.gz to /tmp/TestStandaloneUpgradeRollback1588582945
    fixture.go:356: Completed extraction of artifact elastic-agent-9.3.0-SNAPSHOT-linux-arm64.tar.gz to /tmp/TestStandaloneUpgradeRollback1588582945
    fixture.go:1045: Components were not modified from the fetched artifact
    fixture.go:888: >> running binary with: [/tmp/TestStandaloneUpgradeRollback379987891/elastic-agent-9.2.1-linux-arm64/elastic-agent version --binary-only --yaml]
    fixture.go:888: >> running binary with: [/tmp/TestStandaloneUpgradeRollback1588582945/elastic-agent-9.3.0-SNAPSHOT-linux-arm64/elastic-agent version --binary-only --yaml]
    upgrader.go:245: installation of Elastic Agent will use --unprivileged as both start and end version support --unprivileged mode
    upgrader.go:280: Installing version "9.2.1"
    fixture_install.go:200: [test TestStandaloneUpgradeRollback] Inside fixture install function
    fixture_install.go:228: [test TestStandaloneUpgradeRollback] Inside fixture installNoPkgManager function
    fixture_install.go:247: >> running binary with: [/tmp/TestStandaloneUpgradeRollback379987891/elastic-agent-9.2.1-linux-arm64/elastic-agent install --force --non-interactive --unprivileged]
    fixture.go:839: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    upgrader.go:325: Upgrading from version "9.2.1-db3bfa212ae90d4172a030acd43f51aaca2ad87a" to version "9.3.0-SNAPSHOT-17cecc71c05f72c34004783689e743b84b112b16"
    upgrader.go:362: >> running binary with: [/opt/Elastic/Agent/elastic-agent upgrade 9.3.0-SNAPSHOT --source-uri file:///tmp/agent-upgrade-2347550202 --skip-verify]
    fixture.go:839: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    upgrader.go:384: waiting for upgrade watcher to start
    upgrader.go:389: upgrade watcher started
    upgrader.go:394: Checking upgrade details state while Upgrade Watcher is running
    fixture.go:888: >> running binary with: [/opt/Elastic/Agent/elastic-agent version --binary-only --yaml]
    fixture.go:839: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    fixture.go:839: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    upgrade_rollback_test.go:162: rollback finished, status is 'HEALTHY', cannot check UpgradeDetails
    fixture.go:839: >> running binary with: [/opt/Elastic/Agent/elastic-agent status --output json]
    upgrade_rollback_test.go:175:
        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-aws-1763577411160828303/elastic/elastic-agent-extended-testing/testing/integration/ess/upgrade_rollback_test.go:175
        	Error:      	Received unexpected error:
        	            	agent state is not healthy: got 3
        	Test:       	TestStandaloneUpgradeRollback

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[swiatekm]

This should've been fixed by #11257, we can reopen if it comes back.

[ebeahan]

I saw some instances in CI, but may they didn't include the fix from #11257.

I did open #11268 proactively if we end up needing.