Cleanup EC2 VMs created by FIPS integration tests #10002

ycombinator · 2025-09-17T01:50:47Z

What does this PR do?

This PR adds a new Buildkite pipeline and associated script to clean up any lingering EC2 VMs created by FIPS integration tests. The pipeline is executed every 4 hours.

Why is it important?

Elastic Agent CI runs FIPS integration tests. These integration tests create FIPS-compliant VMs in EC2 where the tests are actually executed. Sometimes these CI builds fail without cleaning up said VMs.

Checklist

I have read and understood the pull request guidelines of this project.
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool
I have added an integration test or an E2E test

Disruptive User Impact

How to test this PR locally

Related issues

Questions to ask yourself

How are we going to support this in production?
How are we going to measure its adoption?
How are we going to debug this?
What are the metrics I should take care of?
...

mergify · 2025-09-17T01:51:25Z

This pull request does not have a backport label. Could you fix it @ycombinator? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

ycombinator · 2025-09-22T19:00:33Z

.buildkite/scripts/steps/ec2-cleanup.sh

+export ACCOUNT_KEY_SECRET=$(vault kv get -field=client_email $VAULT_PATH)
+export ACCOUNT_SECRET=$(vault kv get -field=private_key $VAULT_PATH)
+export ACCOUNT_PROJECT_SECRET=$(vault kv get -field=project_id $VAULT_PATH)


Not sure if these are the correct fields in Vault for EC2? I copied these from the analogous gce-cleanup.sh file. @v1v?

I think we need to use a different approach here, and avoid accessing vault secrets at runtime - they are not masked.

The recommended approach is to use https://github.com/elastic/vault-secrets-buildkite-plugin/

However, I think we could use OIDC and avoid the situation of using static secrets:

https://github.com/elastic/oblt-aws-auth-buildkite-plugin

plus the OIDC configuration in the specific AWS account

v1v · 2025-09-22T19:26:48Z

catalog-info.yaml

        everyone:
          access_level: BUILD_AND_READ

+---


I normally suggest to create this catalog-info change in a first PR. Why? Buildkite integration with Terrazzo cannot be tested unless is merged.

So you could create a PR afterwards and point to your branch for testing purposes using the BK UI and validate changes work as expected during your PR development.

v1v · 2025-09-22T19:27:24Z

catalog-info.yaml

+        filter_enabled: true
+        # required by "build_pull_requests: true" when used with buildkite-pr-bot
+        filter_condition: >-
+          build.pull_request.id == null || (build.creator.name == 'elasticmachine' && build.pull_request.id != null)


If no BK PR bot, I think we can remove this section

v1v · 2025-09-22T19:27:32Z

catalog-info.yaml

+  description: Clean up stale EC2 instances
+  links:
+    - title: Pipeline
+      url: https://buildkite.com/elastic/elastic-agent


Suggested change

url: https://buildkite.com/elastic/elastic-agent

url: https://buildkite.com/elastic/elastic-agent-ec2-cleanup

v1v

I think the current implementation in the ingest-dev project could be used too. So we won't need this PR but only the changes from .buildkite/misc/ec2-cleanup.yml.

ycombinator · 2025-09-22T19:34:43Z

I think the current implementation in the ingest-dev project could be used too. So we won't need this PR but only the changes from .buildkite/misc/ec2-cleanup.yml.

Did you mean the ingest-dev repo? If so, what implementation are you referring to? I'm not finding anything that uses cloud-reaper under https://github.com/elastic/ingest-dev/tree/main/.buildkite.

v1v · 2025-09-22T19:47:15Z

I actually got confused with the ESS cleanup and https://github.com/elastic/elastic-agent/blob/main/.buildkite/pipeline.elastic-agent-gce-cleanup.yml. Somehow, I thought it was in the private repository.

I think using pipeline.elastic-agent-gce-cleanup.yml could be enough, it also runs every 4 hours, what do you think?

ycombinator · 2025-09-22T19:52:40Z

I think using pipeline.elastic-agent-gce-cleanup.yml could be enough, it also runs every 4 hours, what do you think?

The FIPS VMs are spun up in EC2, so how could we use the GCE cleanup script?

v1v · 2025-09-22T19:56:50Z

Good point, the naming is not meaningful in that case - I'm certainly getting blind after the long day, hence my bad with my suggestions 🙇

Although, can you elaborate on what you mean by 'spun up in EC2'? IIUC, those VMs are managed by the BK provisioner Gobld, hence we don't need to clean them up - they are destroyed automatically after the step finished (regardless of the success)

IIRC, the GCE cleanup was for the ogc when running ITs, but IIRC, that's not the case anymore

ycombinator · 2025-09-22T20:04:58Z

Although can you elaborate what you mean with spun up in EC2? IIUC, those VMs are managed by the BK provisioner Gobld

I linked to the line in my comment but basically the buildkite step has agents.provider: "aws". Doesn't this mean the VM used that step is provisioned in EC2?

IIRC, the GCE cleanup was for the ogc when running ITs, but IIRC, that's not the case anymore

Yeah, this was my initial thought too. The VMs should now be managed by Buildkite. However, I found that GCE VM cleanup pipeline is finding and cleaning up VMs some times (example). So it would seem something is still creating the OGC VMs and leaving them orphaned?

ycombinator · 2025-09-22T20:10:04Z

Although, can you elaborate on what you mean by 'spun up in EC2'? IIUC, those VMs are managed by the BK provisioner Gobld, hence we don't need to clean them up - they are destroyed automatically after the step finished (regardless of the success)

Okay, I see you edited your comment to add the note about automatic cleanup regardless of success of the step. In that case, it seems like we don't need this PR after all, which is great!

I do think we need to look into why the GCE cleanup pipeline is finding VMs sometimes but we can do that separately from this PR.

Given all of the above, I will close this PR without merging, unless there are any objections.

v1v · 2025-09-22T20:12:19Z

Given all of the above, I will close this PR without merging, unless there are any objections.

Thanks Shaunak for walking me through the specifics here 🙇

it would seem something is still creating the OGC VMs and leaving them orphaned?

I see names that are not elastic-agent but other things, name=ogc-linux-amd64-ubuntu-2404-fleet-endpoint-security-d2ce state=running ):

Maybe, developers run those ogc commands locally. I have not investigated much the state of the art for ogc in the ingest projects though, but I agree we can do it in another PR or GH issues discussion.

Thanks!

elastic-sonarqube · 2025-09-22T20:31:17Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2025-09-23T19:09:07Z

💔 Build Failed

Buildkite Build
Commit: 9ec46b5

Failed CI Steps

Packaging: Containers linux/amd64

cc @ycombinator

Add backstage catalog entry for cleanup pipeline

baca329

mergify bot assigned ycombinator Sep 17, 2025

ycombinator added 2 commits September 22, 2025 11:37

Adding scripts/config for EC2 VM cleanup

a29d7ce

Fix region

9ec46b5

ycombinator commented Sep 22, 2025

View reviewed changes

ycombinator marked this pull request as ready for review September 22, 2025 19:00

ycombinator requested a review from a team as a code owner September 22, 2025 19:00

ycombinator requested a review from v1v September 22, 2025 19:00

ycombinator added the backport-skip label Sep 22, 2025

v1v reviewed Sep 22, 2025

View reviewed changes

ycombinator closed this Sep 22, 2025

	url: https://buildkite.com/elastic/elastic-agent
	url: https://buildkite.com/elastic/elastic-agent-ec2-cleanup

Cleanup EC2 VMs created by FIPS integration tests #10002

Cleanup EC2 VMs created by FIPS integration tests #10002

Uh oh!

Conversation

ycombinator commented Sep 17, 2025

What does this PR do?

Why is it important?

Checklist

Disruptive User Impact

How to test this PR locally

Related issues

Questions to ask yourself

Uh oh!

mergify bot commented Sep 17, 2025

Uh oh!

ycombinator Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

v1v Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

v1v Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

v1v Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

v1v Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

v1v left a comment

Choose a reason for hiding this comment

Uh oh!

ycombinator commented Sep 22, 2025

Uh oh!

v1v commented Sep 22, 2025

Uh oh!

ycombinator commented Sep 22, 2025

Uh oh!

v1v commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ycombinator commented Sep 22, 2025

Uh oh!

ycombinator commented Sep 22, 2025

Uh oh!

v1v commented Sep 22, 2025

Uh oh!

elastic-sonarqube bot commented Sep 22, 2025

Quality Gate passed

Uh oh!

elasticmachine commented Sep 23, 2025

💔 Build Failed

Failed CI Steps

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

v1v commented Sep 22, 2025 •

edited

Loading