test: add AWS Fault Injection Service test for varying degrees of zonal failures #7838
Conversation
✅ Deploy Preview for karpenter-docs-prod canceled.
|
| env.CleanupObjects(common.CleanableObjects...) | ||
| env.EventuallyExpectNoLeakedKubeNodeLease() | ||
| env.ConsistentlyExpectNodeCount(">=", 0, time.Second*5) | ||
| env.ExpectActiveKarpenterPod() |
There was a problem hiding this comment.
We just check for an active Karpenter pod because a network failure in these tests can cause a crash/restart due to the underlying node failing its kubelet heartbeat.
Pull Request Test Coverage Report for Build 13660927806Details
💛 - Coveralls |
| ) | ||
| }) | ||
|
|
||
| func createExperimentTemplate(ctx context.Context, env *awsenv.Environment, targetAZ string, instances []ec2types.Instance, failurePercentage string) string { |
There was a problem hiding this comment.
What do you think about creating the experiments and roles in the github actions? We can keep these resources as part of the cluster setup. This will make it easier to manage these resources with the existing scripts for cleanup and monitoring
| ResourceType: aws.String("aws:ec2:instance"), | ||
| SelectionMode: aws.String("ALL"), | ||
| ResourceArns: lo.Map(targetInstances, func(id string, _ int) string { | ||
| return fmt.Sprintf("arn:aws:ec2:%s:%s:instance/%s", env.Region, env.ExpectAccountID(), id) |
There was a problem hiding this comment.
Can use tags instead of ids? This way we can have one experiment for many instances. This will also make it less leak the experiments
|
|
||
| func setupRoles(env *awsenv.Environment) { | ||
| // First, check if the FIS service-linked role exists | ||
| serviceRoleName := "AWSServiceRoleForFIS" |
There was a problem hiding this comment.
Can we just set up all the necessary role in the GH actions as part of cluster creation?
| }) | ||
| } | ||
|
|
||
| func getSubnetsInAZ(ctx context.Context, env *awsenv.Environment, targetAZ string) []string { |
There was a problem hiding this comment.
What do you tink about making this an env expectation?
| subnetARNs := getSubnetsInAZ(ctx, env, targetAZ) | ||
| By(fmt.Sprintf("Found %d subnets in AZ %s", len(subnetARNs), targetAZ)) | ||
|
|
||
| template := &fis.CreateExperimentTemplateInput{ |
There was a problem hiding this comment.
What do you think about adding the experiment resource to our resource monitoring dashboard?
Fixes #N/A
Description
).
Utilizes the AWS Fault Injection Service to test and observe Karpenter's behavior during zonal outages. This should serve as the scaffolding for future Fault Injection Service tests and can be expanded to included specific APIs (list of supported actions
The most challenging part of adding a test is getting the permissions correctly configured. There are managed policies
but actions may still fail. Double check CloudTrail if errors are encountered.
How was this change tested?
make e2etest
Does this change impact docs?
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.