fix: surface EC2 request rejections on the EC2NodeClass validation condition - #9454
fix: surface EC2 request rejections on the EC2NodeClass validation condition#9454somaz94 wants to merge 1 commit into
Conversation
|
Could a maintainer approve the workflow runs for this PR when you get a chance? CI, CI-TEST, CodeQL and DryRunGenPR have all been sitting in |
b83bab7 to
ed1e3ae
Compare
|
Rebased onto While rebasing I noticed this overlaps with #9360, which was opened about two weeks before mine and targets the same issue (#8148) and the same files. I'd rather not leave two PRs doing the same thing in your queue, so I'm happy to close this one if you'd prefer to take that one forward. One difference worth knowing before you decide: the comment on #9360 raises cache poisoning on transient, non-rate-limited API errors. This PR guards that case explicitly — Either way: the workflows on this PR have been at |
Fixes #8148
Description
validateRunInstancesAuthorizationreturnsfmt.Errorf("validating ec2:RunInstances authorization, %w", err)for every error that isn'tUnauthorizedOperationorDryRunOperation. But EC2 also rejects a dry run over the request itself, so anInvalidBlockDeviceMapping(root volume smaller than the AMI's snapshot) becomes a reconcile error that is only ever logged, and logged as if it were an authorization problem. The EC2NodeClass stays atValidationSucceeded=Unknown/AwaitingReconciliationand retries forever, which is what #8148 reports.validateCreateFleetAuthorizationandvalidateCreateLaunchTemplateAuthorizationhave the same shape.All three now surface an AWS API error on
ValidationSucceededwith the error's code and message instead of returning it:Two things worth flagging, since they go a little past the surfacing change:
IsServerErrornever matched an EC2 error. It testedapiErr.ErrorFault() == smithy.FaultServer, but EC2's generated deserializers build asmithy.GenericAPIErrorwithout settingFault, so it always readFaultUnknown. That is why the 503Unavailablein the issue also landed in the authorization bucket. It now falls back to the response status code viaretry.DefaultRetryableHTTPStatusCodes, so it tracks the SDK rather than hardcoding a range.IsServerErroris only called fromvalidation.go, so the blast radius stays in this file.A cached validation failure lives for
ValidationTTL(30 minutes) and drivesReady=False, so only errors a retry cannot fix should reach it. Transient codes are filtered out first by a newIsNonTerminalError, which reuses the SDK's own throttle and retryable code sets and addsAuthFailure,RequestExpired,PendingVerification, plus the two launch template not-found codes (a template can be garbage collected betweenEnsureAlland the dry run). Those requeue exactly as before.CreateLaunchTemplateisn't actually a dry run, so its message says "request" instead.How was this change tested?
go test ./pkg/errors/... ./pkg/controllers/...andgolangci-lint run(v2.12.2, the versionhack/toolchain.shpins) both pass locally against envtest 1.34.1.New coverage:
DescribeTableentries, one per validator, asserting the reason and the full condition messageEC2ThrottledExceptionrequeue without caching or failing validationpkg/errors/errors_test.go, which the package didn't have, coveringIsServerError,IsNonTerminalErrorandToAPIErrorMessageI also confirmed the two key specs fail when the fix is reverted.
Does this change impact docs?
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.