Windows agent fall backs to 7.17.25 on upgrade to 8.16.0 BC2.

[amolnater-qasource]

Kibana Build details:

VERSION: 8.16.0 BC2
BUILD: 79434
COMMIT: 59220e984f2e3ca8b99fe904d077a5979f5f298d

Artifact: https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-7.17.25-windows-x86_64.zip

Preconditions:

1. 8.16.0-BC2 Kibana cloud environment should be available.
2. 7.17.25 Windows agent should be installed with System and Elastic Defend integration.

Steps to reproduce:

1. Navigate to Fleet>Agents tab.
2. Upgrade Windows 7.17.25 agent to 8.16.0.
3. Observe agent starts upgrading and falls back to 7.17.25.

What's working fine:

• Issue is not observed on Linux and MAC agent upgrades from 7.17.25>8.16.0 BC2.
  elastic-agent-diagnostics-2024-10-29T10-36-01Z-00.zip

• Issue is not observed on Windows agent upgrade from 8.15.3>8.16.0 BC2.
  elastic-agent-diagnostics-2024-10-29T10-41-37Z-00.zip

Expected Result:
Windows agent shouldn't fall back to 7.17.25 on upgrade to 8.16.0 BC2.

Logs:
elastic-agent-diagnostics-2024-10-29T10-23-14Z-00.zip
elastic-agent-diagnostics-2024-10-29T10-03-01Z-00.zip

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[amolnater-qasource]

@muskangulati-qasource Please review.

[muskangulati-qasource]

Secondary review is Done for this ticket!

[pierrehilbert]

Are you facing the same without Endpoint? And without system integration ?

[amolnater-qasource]

Hi @pierrehilbert

Thank you for looking into this issue.
Please find below testing details for the same:

Scenario	Status
Only System	Reproducible
Only Elastic Defend	Reproducible
No Integration	Reproducible

Logs:
elastic-agent-diagnostics-2024-10-29T11-54-00Z-00.zip
elastic-agent-diagnostics-2024-10-29T11-53-46Z-00.zip

No Integration Logs:
elastic-agent-diagnostics-2024-10-29T12-06-36Z-00.zip

Please let us know if anything else is required from our end.

Thanks!!

[pierrehilbert]

Just to confirm: 7.17.25 to 8.15.3 is working correctly?
I can see those errors:

{"log.level":"error","@timestamp":"2024-10-29T09:55:47.649Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2024-10-29T09:55:47Z - message: Application: [4b854a47-e861-4a66-8091-a1625e9c5ce8]: State changed to FAILED: failed verification of agent binary: 2 errors occurred:\n\t* hash for 'elastic-agent-8.16.0-windows-x86_64.zip' not found in 'C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-ef6504\\downloads\\elastic-agent-8.16.0-windows-x86_64.zip.sha512'\n\t* hash for 'elastic-agent-8.16.0-windows-x86_64.zip' not found\n\n - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-10-29T09:55:47.649Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":180},"message":"failed to dispatch actions, error: failed verification of agent binary: 2 errors occurred:\n\t* hash for 'elastic-agent-8.16.0-windows-x86_64.zip' not found in 'C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-ef6504\\downloads\\elastic-agent-8.16.0-windows-x86_64.zip.sha512'\n\t* hash for 'elastic-agent-8.16.0-windows-x86_64.zip' not found\n\n","ecs.version":"1.6.0"}

cc @ycombinator we will probably need someone from your team to investigate here

[amolnater-qasource]

@pierrehilbert Yes, agent upgraded successfully from 7.17.25>8.15.3.

Logs:
elastic-agent-diagnostics-2024-10-29T13-12-29Z-00.zip

Thank you!

[pierrehilbert]

So we can confirm this is coming from changes on 8.16, thx!

[cmacknz]

In the "no integration" diagnostics in #5872 (comment) the upgrade makes enough progress to start 8.16.0.

{"log.level":"info","@timestamp":"2024-10-29T11:57:06.065Z","log.origin":{"file.name":"application/application.go","file.line":67},"message":"Detecting execution mode","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-29T11:57:06.065Z","log.origin":{"file.name":"application/application.go","file.line":92},"message":"Agent is managed by Fleet","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-29T11:57:06.065Z","log.origin":{"file.name":"capabilities/capabilities.go","file.line":59},"message":"capabilities file not found in C:\\Program Files\\Elastic\\Agent\\capabilities.yml","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-29T11:57:06.574Z","log.logger":"composable.providers.docker","log.origin":{"file.name":"docker/docker.go","file.line":43},"message":"Docker provider skipped, unable to connect: protocol not available","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-29T11:57:06.576Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":62},"message":"Starting stats endpoint","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-29T11:57:06.576Z","log.origin":{"file.name":"application/managed_mode.go","file.line":299},"message":"Agent is starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-29T11:57:06.576Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":64},"message":"Metrics endpoint listening on: \\\\.\\pipe\\elastic-agent (configured: npipe:///elastic-agent)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-29T11:57:08.385Z","log.origin":{"file.name":"artifact/config.go","file.line":116},"message":"Source URI changed from \"https://staging.elastic.co/8.16.0-66e9e678/downloads/\" to \"https://staging.elastic.co/8.16.0-66e9e678/downloads/\"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-29T12:00:08.953Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2024-10-29T12:00:08Z - message: Application: [0e3c9dfd-0815-40d0-877e-c684b950e1ec]: State changed to UPDATING: Update to version '8.16.0' started - type: 'STATE' - sub_type: 'UPDATING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-29T12:00:08.953Z","log.origin":{"file.name":"upgrade/step_download.go","file.line":66},"message":"Downloading upgrade artifact","version":"8.16.0","source_uri":"https://staging.elastic.co/8.16.0-66e9e678/downloads/","drop_path":"","target_path":"C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-ef6504\\downloads","install_path":"C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-ef6504\\install","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-29T12:00:30.900Z","log.origin":{"file.name":"http/downloader.go","file.line":304},"message":"download from https://staging.elastic.co/8.16.0-66e9e678/downloads/beats/elastic-agent/elastic-agent-8.16.0-windows-x86_64.zip completed in 21 seconds @ 8.929MBps","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-29T12:00:31.126Z","log.origin":{"file.name":"http/downloader.go","file.line":304},"message":"download from https://staging.elastic.co/8.16.0-66e9e678/downloads/beats/elastic-agent/elastic-agent-8.16.0-windows-x86_64.zip.sha512 completed in Less than a second @ +InfYBps","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-29T12:00:37.067Z","log.origin":{"file.name":"application/managed_mode.go","file.line":328},"message":"Agent is stopped","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-29T12:00:37.078Z","log.logger":"reexec","log.origin":{"file.name":"reexec/reexec_windows.go","file.line":36},"message":"Running as Windows service Elastic Agent; triggering service restart","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-29T12:00:37.109Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":66},"message":"Stats endpoint (\\\\.\\pipe\\elastic-agent) finished: use of closed network connection","ecs.version":"1.6.0"}

[cmacknz]

If I look at the test report for the latest build of the 8.16 branch, there is a test for 7.17.25-SNAPSHOT to 8.16.0-SNAPSHOT that is passing for standalone agents, but not one covering those specific versions when Fleet managed. There is an assumption that this isn't necessary because there is not enough of a difference but perhaps that is wrong.

[cmacknz]

I tried this on a Windows VM and was able to look at the watcher logs:

{"log.level":"info","@timestamp":"2024-10-29T19:39:32.105Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.watchCmd","file.name":"cmd/watch.go","file.line":68},"message":"Upgrade Watcher started","process.pid":2780,"agent.version":"8.16.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-29T19:39:32.133Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.watchCmd","file.name":"cmd/watch.go","file.line":80},"message":"Loaded update marker &{Version: Hash:df4344 VersionedHome: UpdatedOn:2024-10-29 19:39:31.348391 +0000 UTC PrevVersion:7.17.25 PrevHash:ef6504 PrevVersionedHome: Acked:false Action:id: 1b045aa0-6a15-41d5-9445-bdfe718eca03, type: UPGRADE Details:<nil>}","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-29T19:39:32.486Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*AgentWatcher).Run","file.name":"upgrade/watcher.go","file.line":65},"message":"Agent watcher started","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-29T19:40:02.487Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*AgentWatcher).Run","file.name":"upgrade/watcher.go","file.line":133},"message":"Trying to connect to agent","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-10-29T19:40:02.490Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*AgentWatcher).Run","file.name":"upgrade/watcher.go","file.line":142},"message":"Failed connecting to running daemon: connection error: desc = \"transport: error while dialing: open \\\\\\\\.\\\\pipe\\\\PVofPrsCKolWs3UQ2cj1WCUSflrq2pW2.sock: The system cannot find the file specified.\"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-29T19:40:32.492Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*AgentWatcher).Run","file.name":"upgrade/watcher.go","file.line":133},"message":"Trying to connect to agent","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-10-29T19:40:32.494Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*AgentWatcher).Run","file.name":"upgrade/watcher.go","file.line":142},"message":"Failed connecting to running daemon: connection error: desc = \"transport: error while dialing: open \\\\\\\\.\\\\pipe\\\\PVofPrsCKolWs3UQ2cj1WCUSflrq2pW2.sock: The system cannot find the file specified.\"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-29T19:41:02.509Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*AgentWatcher).Run","file.name":"upgrade/watcher.go","file.line":133},"message":"Trying to connect to agent","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-10-29T19:41:02.511Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*AgentWatcher).Run","file.name":"upgrade/watcher.go","file.line":142},"message":"Failed connecting to running daemon: connection error: desc = \"transport: error while dialing: open \\\\\\\\.\\\\pipe\\\\PVofPrsCKolWs3UQ2cj1WCUSflrq2pW2.sock: The system cannot find the file specified.\"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-10-29T19:41:32.518Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*AgentWatcher).Run","file.name":"upgrade/watcher.go","file.line":133},"message":"Trying to connect to agent","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-10-29T19:41:32.519Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*AgentWatcher).Run","file.name":"upgrade/watcher.go","file.line":142},"message":"Failed connecting to running daemon: connection error: desc = \"transport: error while dialing: open \\\\\\\\.\\\\pipe\\\\PVofPrsCKolWs3UQ2cj1WCUSflrq2pW2.sock: The system cannot find the file specified.\"","ecs.version":"1.6.0"}