Preserve agent state when upgrading using package manager

[rdner]

Describe the enhancement:

Currently, we distribute Elastic Agent as a DEB and RPM package. However, we don't support upgrading the agent version using a package manager. Upgrading the agent using a package manager leads to abandoning the previous data/state folder and re-creating the whole state from scratch. Which is not the best user experience.

The only supported upgrade path for the agent at the moment is the elastic-agent upgrade command.

Describe a specific use case for the enhancement or feature:

Users running standalone agents on RPM or DEB driven OS.

What is the definition of done?

Upgrading an Elastic Agent DEB or RPM package using the suitable package manager preserves the existing state and does not lead to invalid behaviour for the agent.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[cmacknz]

Related #3784

[Maksimkovalenk]

Had a Case with where customer had this issue and asked to prioritize this issue to get updates via RPM properly implemented

[pierrehilbert]

I will let @leehinman / @blakerouse confirm here but I think upgrades are now working for DEB/RPM.

[simioa]

I have tested the RPM update from 8.15.0 to 8.15.1. Unfortunately, the run folder in which the state of the individual beats is persisted is not copied to the new agent folder. This only seems to work when upgrading via the built-in upgrade mechanism(https://github.com/blakerouse/elastic-agent/blob/ef69b58dd3169a7f05918cb07ab1ff7ce86cc36a/internal/pkg/agent/application/upgrade/upgrade.go#L477).
This function does not seem to be executed when updating the package via the package manager. In the Linux packaging templates I could not find a function that copies the run folder into the new agent folder.

Can you please check if my assumption is correct? This is currently preventing us from rolling out the elastic-agent on our systems without major effort

[kesslerm]

I can confirm this issue. The run folder is not copied in version 8.16.1, either, in the RPM package. Also, the RPM package does not honour the state of the elastic-agent.service. The service gets created and activated every time an installation or upgrade is performed. If the service was stopped or disabled before the upgrade it is re-enabled and run during the upgrade.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[nimarezainia]

Why is this issue isolated to just Standalone agents? what do we do with Fleet managed that is different in this regard? (latter also gets upgraded via the same method)

[cmacknz]

You can Fleet manage an agent installed via DEB or RPM, but you can't use the Fleet remote upgrade process with them.

The actual bug is we don't copy the state between versions like we do for upgrades triggered through Fleets, the directory structure of the DEB+RPM packages is different and it just wasn't implemented there.

I think this is independent of being standalone or Fleet managed.

[thomas-merz]

"Not supporting upgrading the agent version using a package manager" is also not the best user/admin experience 😢
I know some other "enterprise" companies that are also not able to provide packages/RPMs that can be updated. But removing and installing instead of upgrading or even running some "obscure" commands feel totally wrong for a user/admin of hundreds or thousand of linux machines and lead to total confusion when package manager says "you have version a.b.c installed", but in reality it's a.b.d! So please make your packages upgrade-able 🙏🏼 As a big company you should have the resources to hire someone (maybe some "Werkstudenten" from germany) to enable your company to make good packages. Thanks in advance.