Skip to content

[EKS Scalability Testing] Implement workflows to execute large scale performance test - #1794

Merged
JayPolanco merged 3 commits into
mainfrom
eks-e2e-performance
Jul 30, 2025
Merged

[EKS Scalability Testing] Implement workflows to execute large scale performance test#1794
JayPolanco merged 3 commits into
mainfrom
eks-e2e-performance

Conversation

@JayPolanco

Copy link
Copy Markdown
Contributor

Description of the issue

As part of an initiative to improve our performance testing on large scale eks clusters, we need to implement workflows that scale up, install addon and execute test that runs on our 5k node cluster.

Description of changes

  1. eks-performance-cluster-scaling.yml - Responsible for scaling eks-performance cluster (5k node cluster).
    • Updated to also scale up leader node group dynamically. This node group has max of 1 node t3.xlarge
    • Scales up on Sunday 9am, scales down Monday 9pm
  2. eks-performance-cluster-addon-install.yml - Responsible for installing the addon upon successful completion of scaling workflow
  • adds the cwagent-target="true" label to all t3.medium instances in the cluster (non leader node). This ensures that cloudwatch-agent non leader pods are installed on these nodes
  • gets the hostname (ip-x-x-x-x..compute.internal) and substitutes it into the tag in the override file in the test repo under test/performance/eks/resources/leader_election_overrides/base-overrides.yml
  • installs the addon with helm, with the override file --values
  1. eks-performance-cluster-tests.yml - responsible for executing the test in the test repo (eks_performance_test.go)
    • Updated check-trigger to ensure tests are only ran if the cluster has active nodes
    • installs sample applications use kubectl (petclinic-sample-app in test repo)
    • cleans up once test is complete

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Cluster Scaling Workflow - https://github.com/aws/amazon-cloudwatch-agent/actions/runs/16512847744/job/46697932764
Addon Install Workflow - https://github.com/aws/amazon-cloudwatch-agent/actions/runs/16523571905/job/46731027008
Performance Test Workflow - https://github.com/aws/amazon-cloudwatch-agent/actions/runs/16526410774/job/46740785992

Requirements

Before commiting your code, please do the following steps.

  1. Run make fmt and make fmt-sh
  2. Run make lint

Integration Tests

To run integration tests against this PR, add the ready for testing label.

@JayPolanco
JayPolanco requested a review from a team as a code owner July 25, 2025 17:22
@JayPolanco JayPolanco changed the title [EKS Scalability Testing] [EKS Scalability Testing] Implement workflows to execute large scale performance test Jul 25, 2025
Comment thread .github/workflows/eks-performance-cluster-scaling.yml
TravisStark
TravisStark previously approved these changes Jul 28, 2025
Comment thread .github/workflows/eks-performance-cluster-scaling.yml
Comment thread .github/workflows/eks-performance-cluster-scaling.yml
Comment thread .github/workflows/eks-performance-cluster-addon-install.yml Outdated
@JayPolanco
JayPolanco force-pushed the eks-e2e-performance branch from 028a559 to 8a28ed4 Compare July 30, 2025 17:53
@JayPolanco
JayPolanco merged commit dae12ee into main Jul 30, 2025
17 checks passed
@JayPolanco
JayPolanco deleted the eks-e2e-performance branch July 30, 2025 17:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants