Investigate using GoReleaser to package the Elastic Agent

[cmacknz]

For inspiration see https://github.com/open-telemetry/opentelemetry-collector-releases which includes packaging an MSI. This potentially allows eliminating https://github.com/elastic/elastic-stack-installers and including the MSI in the Windows tests here instead of relying on separate ones.

Goreleaser was previously explored in elastic/elastic-agent-shipper#73

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[v1v]

The APM Serverd moved to nfpm:

• https://github.com/elastic/apm-server/blob/main/packaging/nfpm.yml

See elastic/apm-server#8732

[jlind23]

@pchila ideally the first outcome here would be to get an estimate of how much time it would take us to get rid of Golang crossbuild in favor of this.

[pchila]

@jlind23 It really depends if we want a full substitution of all the artifacts currently created for elastic-agent or if we can keep the two packaging solutions side by side and then move packages type from the crossbuild+dev-tools to go-releaser one by one.

An iterative solution with some overlap would be my preference, starting with the .msi package that is currently packaged from https://github.com/elastic/elastic-stack-installers/ and then moving to the other packages but I will need to check how feasible that would be

[cmacknz]

There is no reason to do this in one big bang change instead of incrementally.

[jlind23]

Agree with Craig, let's try to avoid the big bang approach as much as we can. I don't have a strong opinion on where to start so I would trust your judgement here and start with stack installers.

[jlind23]

As discussed over slack we probably need to also support/build ppc compliant binaries. - elastic/golang-crossbuild#580

[cmacknz]

We don't need to build them in agent, but the cross build images need to support PPC to let others build their own PPC agents (which we do not offer support for but they are free to build on their own).

[pchila]

Stopped working on this issue a while ago but I forgot to report my findings.
All the code used for Goreleaser experiment is in a draft PR #10287

Quick summary:

• GoReleaser build is convenient for pure Go binaries but it requires something similar to golang-crossbuild once CGO is used (like in elastic-agent). For experimentation I quickly adapted a golang-crossbuild image by adding the goreleaser binary to it
• Packaging elastic-agent dependencies proved quite tedious: agent has multiple mechanisms for selecting and downloading dependencies and, in order to use our dependencies with GoReleaser, some extra mage targets had to be defined as hooks to download and extract dependency packages.
• GoReleaser pro supports packaging starting from prebuilt artifacts similar to what is currently done when packaging agent using binaries created by golang-crossbuild but I didn't really look deeply into that
• MSI packaging (one of the goal of this issue was to explore if GoReleaser could be a substitute for elastic-stack-installers) is supported by GoReleaser Pro and it requires msitools (and most likely Wix Toolset, but I didn't manage to test/verify that)

Observations and comments (in no particular order)

Overall GoReleaser looks like a convenient tool for projects that do not package too many external, dynamic dependencies where building, packaging, signing and releasing can all be handled by a single tool.

Elastic agent building, packaging and releasing is a more complex affair that involve multiple projects and teams.
Just the dependency collection of agent is difficult to represent in GoReleaser yaml and it requires additional code to be used as hooks.

Building Elastic Agent binary with GoReleaser is surprisingly similar to what is already done with golang-crossbuild, maybe we can take inspiration from https://github.com/goreleaser/goreleaser-cross but it does not look like a net upgrade.

Packaging of elastic-agent has recently been enhanced with #7602 and #7690 in order to package FIPS-capable elastic-agent artifacts and, as it stands, looks the more convenient solution for packaging now that is in a more declarative form.

As for Signing and Releasing, other teams are involved in that part of the cycle so we couldn't use GoReleaser mechanisms for those part of the lifecycle.

[cmacknz]

Thanks for this, it sounds like switching to GoReleaser overall is not necessarily a win, and we should try to make some time separately to see if we can use just the MSI packaging component or if that will lead us to a similar conclusion. There is a lot of value in being able to simplify away from elastic-stack-installers even if that is a one off use of GoReleaser.

[pchila]

@cmacknz If we want to test GoReleaser just for the .msi package generation, I would add also a test/experiment using only msitools to be able to gauge if we prefer having GoReleaser as a middleman or maybe integrate msitools directly in elastic-agent packaging process.

[swiatekm]

What I'm getting from this conclusion is that our build system is similar in complexity to a dedicated tool like goreleaser, and we should aspire to treat it with appropriate seriousness when it comes to testing and robustness.