[Inconsistent]: Upgraded Fleet Server 8.18.0 BC3>9.0.0 BC3 goes offline permanently on upgraded self-managed kibana.

[amolnater-qasource]

Kibana Build details:

VERSION: 9.0.0-beta1 BC3
BUILD: 83575
COMMIT: a9ae718019d3909912f81e5d388ef597929071a1

Artifact: https://staging.elastic.co/8.18.0-16a1508a/downloads/beats/elastic-agent/elastic-agent-8.18.0-windows-x86_64.zip

Preconditions:

1. 8.18.0 BC3 Upgraded to 9.0.0 beta1 BC3 Kibana self-managed environment should be available.
2. 8.18.0 BC3 Fleet Server should be installed on 8.18.0 BC3.

Steps to reproduce:

1. Upgrade the 8.18.0 self-managed kibana to 9.0.0-beta1.
2. Upgrade 8.18.0 fleet server to latest 9.0.0-beta1.
3. Observe Fleet Server upgrades successfully and then goes permanently offline on upgrade.

Expected Result:
Upgraded Fleet Server 8.18.0 BC3>9.0.0 BC3 should remain Health on upgraded self-managed kibana.

NOTE:

• When we reinstalled the 8.18.0 Fleet server on upgraded self-managed 9.0.0, it upgraded successfully.
• The issue is only observed when it was installed on 8.18.0 BC3 build.
• [Update]: The issue is not consistently reproducible.

Logs:
elastic-agent-diagnostics-2025-02-14T10-02-11Z-00.zip

Agent json:
EC2AMAZ-ED5G7SB-agent-details.zip

Screenshot:

[amolnater-qasource]

@muskangulati-qasource Please review.

[muskangulati-qasource]

Secondary review is Done for this ticket!

[cmacknz]

            input-fleet-server-default-fleet-server-fleet_server-691879e0-d583-4022-bba5-61d8bb8dbd80:
                message: 'Error - could not start the HTTP server for the API: failed to listen on the named pipe \\.\pipe\UwGGXFL1il700DVAc6q-T-1Z9J1UjGMU.sock: open \\.\pipe\UwGGXFL1il700DVAc6q-T-1Z9J1UjGMU.sock: Access is denied.'
                state: 4
            output-fleet-server-default:
                message: 'Error - could not start the HTTP server for the API: failed to listen on the named pipe \\.\pipe\UwGGXFL1il700DVAc6q-T-1Z9J1UjGMU.sock: open \\.\pipe\UwGGXFL1il700DVAc6q-T-1Z9J1UjGMU.sock: Access is denied.'
                state: 4

Fleet Server not being able to create the named pipe it needs is for sure the problem here, but I'm not sure why.

[cmacknz]

@leehinman any ideas what would cause Fleet Server to get an Access is Denied error inconsistently trying to create the named pipe for its metrics server?

It's using the elastic-agent-libs makeListener not agent's SocketURLWithFallback

[cmacknz]

That this happens on upgrade makes me wonder if the previous fleet-server instance still has a reference to the pipe or something.

[amolnater-qasource]

Hi @cmacknz

While testing on 9.0rc1 self-managed kibana, we have observed that the issue is also inconsistently reproducible on direct installation of 9.0rc1 fleet-server.

Logs:

elastic-agent-diagnostics-2025-03-05T08-54-15Z-00.zip

Agent JSON:

EC2AMAZ-ED5G7SB-agent-details.zip

Build details:
VERSION: 9.0.0-rc1 BC1
BUILD: 83822
COMMIT: 07dc0afa460bddff658ee6d5342ad1f087fc766b
Artifact Link: https://staging.elastic.co/9.0.0-rc1-bd80e8a3/downloads/beats/elastic-agent/elastic-agent-9.0.0-rc1-windows-x86_64.zip

[UPDATE]:
The issue is more frequently reproducible on 9.0 rc1:

• While installing 9.0rc1 windows fleet-server.
• On restarting the Elastic Agent service for the installed fleet server.
  Logs when fleet server was restarted:
  elastic-agent-diagnostics-2025-03-05T10-36-06Z-00.zip
• Memory consumption reached 1 GB for the Fleet Server:
  

Please let us know if anything else is required.
Thanks!!

[michel-laterman]

Does this occur on other OS's or is it only reproducible on Windows?

[amolnater-qasource]

Hi @michel-laterman

We have installed fleet servers for Windows, linux and mac and observed that the log line: Unit state changed fleet-server-default (CONFIGURING->FAILED): Error - could not start the HTTP server for the API: failed to listen on the named pipe \\.\pipe\UwGGXFL1il700DVAc6q-T-1Z9J1UjGMU.sock: open \\.\pipe\UwGGXFL1il700DVAc6q-T-1Z9J1UjGMU.sock: Access is denied. is only observed on Windows Fleet server.

• So as per current tests, the issue is only reproducible on Windows.

Screen Recording:

EC2AMAZ-2JKJE0D.-.Agents.-.Fleet.-.Elastic.-.Google.Chrome.2025-03-12.14-34-30.mp4

Please let us know if anything else is required from our end.

Thanks!!