fix(custom-resources): retry CloudFormation response PUT in AwsCustomResource handler - #38429
Draft
aemada-aws wants to merge 4 commits into
Draft
fix(custom-resources): retry CloudFormation response PUT in AwsCustomResource handler#38429aemada-aws wants to merge 4 commits into
aemada-aws wants to merge 4 commits into
Conversation
…Resource handler The AwsCustomResource handler's respond() sent the CloudFormation response with a single, un-retried https PUT to the pre-signed S3 response URL and passed `resolve` directly as the response callback, so the HTTP status code was never inspected. A transient PUT failure or a non-2xx response was silently swallowed, so CloudFormation never received the response and waited out its ~1 hour timeout even though the function logged SUCCESS and exited cleanly. Extract the retry + exponential-backoff and status-code-checking HTTP logic (matching the provider framework runtime) into a shared module, lib/shared/http-response.ts, and use it from respond(): retry on network errors and >= 400 responses, treating only a successful response as success. sim: CFN-118294
Contributor
|
👋 It looks like your PR description follows the template but is missing a valid issue number in the first section. PRs without a linked issue will receive lower priority for review and merging. Please update the description to include a reference like |
aws-cdk-automation
requested changes
Jul 28, 2026
aws-cdk-automation
left a comment
Collaborator
There was a problem hiding this comment.
The pull request linter fails with the following errors:
❌ Fixes must contain a change to an integration test file and the resulting snapshot.
If you believe this pull request should receive an exemption, please comment and provide a justification. A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed, add Clarification Request to a comment.
…-retention handler too The log-retention handler's respond() had the same single, un-retried https PUT to the CloudFormation pre-signed S3 response URL as the AwsCustomResource handler, with no HTTP status-code check. Point it at the shared lib/shared/http-response.ts (withRetries + httpRequest) so both bundled handlers retry the response PUT consistently, and hoist the shared default retry options (DEFAULT_RESPONSE_RETRY_OPTIONS) into that module. The nodejs-entrypoint handler intentionally keeps its own copy: it is packaged with minifyAndBundle: false (copied verbatim, because it dynamically requires the user handler), so it cannot import a relative shared module. sim: CFN-118294
…s.ts Move lib/shared/http-response.ts to lib/utils.ts (and the test to test/utils.test.ts), and update the aws-custom-resource and log-retention handlers to import from '../../utils'. No behavior change. sim: CFN-118294
…utils.ts jsdoc sim: CFN-118294
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
Internal ref: CFN-118294 (no public issue linked yet — draft).
Reason for this change
The
AwsCustomResourcehandler'srespond()sends the CloudFormation response with a single, un-retriedhttpsPUT to the pre-signed S3 response URL, and passesresolvedirectly as the response callback so the HTTP status code is never inspected. A transient PUT failure or a non-2xx response is silently swallowed: CloudFormation never receives the response and waits out its ~1 hour timeout, even though the function already logged aSUCCESSpayload and exited cleanly. Thelog-retentionhandler has the identical un-retriedrespond()pattern and the same latent failure mode.Description of changes
Add a small shared module,
lib/shared/http-response.ts, mirroring the custom resource provider framework runtime:withRetries(exponential backoff with jitter) +DEFAULT_RESPONSE_RETRY_OPTIONS(5 attempts, 1s base).httpRequestconsumes the response stream, rejects on a>= 400status code (and on socket errors), and resolves only on a successful response.Both bundled handlers that PUT their own CloudFormation response now use it:
custom-resources/aws-custom-resource-handler(respond()).aws-logs/log-retention-handler(respond()).respond()now returnswithRetries(DEFAULT_RESPONSE_RETRY_OPTIONS, httpRequest)(requestOptions, responseBody)in both.Why not the
nodejs-entrypointhandler too? It is packaged withminifyAndBundle: false— the framework copies its singleindex.jsverbatim (it dynamicallyrequire()s the user handler, which esbuild can't bundle), so it cannot import a relative shared module and keeps its own equivalent copy. The two handlers changed here are esbuild-bundled, so the shared module is inlined at build time.Describe any new or updated permissions being added
None.
Description of how you validated changes
test/shared/http-response.test.ts:withRetries(success / retry-then-succeed / exhaustion) andhttpRequest(2xx resolves, >= 400 rejects, network error rejects).utils.test.tswithrespondtests (mockinghttps): single PUT on success, retry-then-succeed, rejection after exhausting retries.aws-sdk-v3-handler.test.tshttpsmock to supply a response object (statusCode: 200) now that the status code is inspected. Thelog-retentiontests usenock(reply200), so they pass unchanged.tsc -b+ framework esbuild bundle +cdk-lint) and the complete package jest suite (333 tests) pass locally.Checklist
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license.