Add cloud-defend to agent container images

[mjwolf]

What does this PR do?

Add cloud-defend to Agent container image artifacts.

Cloud-defend is a cloud-native, kubernetes product, so it only needs to be added container images, not the other installable package types.

This adds cloud-defend only to these image types: basic, ubi, wolfi, complete, and complete-wolfi.

Other image types are excluded; the new "elastic_agent_cloud_components" is now used for cloud and service images, and excludes cloud-defend. Slim and Edot images already do not include "elastic_agent_components", so they will not have cloud-defend.

Why is it important?

Cloud-defend is being added as a supported product in 9.3+, so it must be added to the elastic-agent container images.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

N/A

[mergify]

This pull request does not have a backport label. Could you fix it @mjwolf? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

If you want to reproduce what CI is doing locally try:

USE_PACKAGE_VERSION=true DOCKER_VARIANTS=basic EXTERNAL=true SNAPSHOT=true PACKAGES=docker PLATFORMS=linux/amd64 mage -v package

I get the same failure but I can't tell why

>>>>>>>>>>> Got packagename [cloud-defend-9.3.0-SNAPSHOT-linux-x86_64.tar.gz], looking for exact match
>>>>>>>>>>> Found exact match packageName for [main, 72eceeb9742bb683d135ab41fda2e5aeedbc3833]: {https://snapshots.elastic.co/9.3.0-eaf5f86d/downloads/cloud-defend/cloud-defend-9.3.0-SNAPSHOT-linux-x86_64.tar.gz https://snapshots.elastic.co/9.3.0-eaf5f86d/downloads/cloud-defend/cloud-defend-9.3.0-SNAPSHOT-linux-x86_64.tar.gz.sha512 https://snapshots.elastic.co/9.3.0-eaf5f86d/downloads/cloud-defend/cloud-defend-9.3.0-SNAPSHOT-linux-x86_64.tar.gz.asc tar x86_64 [linux]  {     }}
>>>>>>> Calculated directory to copy: [/Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-drop/.elastic-agent_flat/linux-x86_64.tar.gz/cloud-defend-9.3.0-SNAPSHOT-linux-x86_64]
> prepare to copy /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-drop/.elastic-agent_flat/linux-x86_64.tar.gz/cloud-defend-9.3.0-SNAPSHOT-linux-x86_64 into /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-drop/linux-x86_64.tar.gz
package ran for 16.885143s
Error: lstat /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-drop/.elastic-agent_flat/linux-x86_64.tar.gz/cloud-defend-9.3.0-SNAPSHOT-linux-x86_64: no such file or directory

[pchila]

https://snapshots.elastic.co/9.3.0-eaf5f86d/downloads/cloud-defend/cloud-defend-9.3.0-SNAPSHOT-linux-x86_64.tar.gz.sha512

@cmacknz @mjwolf
I tried downloading the cloud-defend DRA from snapshots.elastic.co/9.3.0-eaf5f86d/downloads/cloud-defend/cloud-defend-9.3.0-SNAPSHOT-linux-x86_64.tar.gz and I see that the inner directory structure has some differences from the package declaration in packages.yml

• There's a . directory at root (not sure how that translates when extracting, might disappear but it is still something that need to be fixed)
• The root directory name is cloud-defend-9.3.0-SNAPSHOT-linux-x86_64/cloud-defend-9.3.0-SNAPSHOT-amd64 (the first element is suspicious) so the rootDir does not match the template cloud-defend-{{.Version}}-{{.Platform}}

[mjwolf]

Hi @pchila, @cmacknz Thanks for looking into the cause of the build failure. I've fixed it now, by changing the directory name used in the cloud-defend artifacts.

I think this should be ready to review now

[mjwolf]

/test extended

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: ab822c1

Failed CI Steps

• Win2022:sudo:fleet
• Start ESS stack for FIPS integration tests

History

• 💔 Build #31666 failed ab822c1
• 💚 Build #31602 succeeded 77a713f
• 💛 Build #31579 was flaky 1df3b9d
• 💚 Build #31570 succeeded 8b4e0c6
• 💔 Build #31532 failed 2b79b9e

cc @mjwolf