fix(deployment): retry SNS/SQS resource-policy create on fresh-role propagation (#839) by go-to-k · Pull Request #843 · go-to-k/cdkd

go-to-k · 2026-06-13T13:48:16Z

Summary

Fixes a real cdkd deploy bug (issue #839) and adds a failure-seeking integration test that stresses four fresh-role IAM-propagation race edges in a single deploy.

Root cause (#839)

cdkd's fast SDK path creates an IAM role and then has a downstream resource consume it within roughly one second, before IAM has finished propagating the just-created role. CloudFormation never hits this because its own deployment latency lets IAM settle; cdkd does. For SNS TopicPolicy / SQS QueuePolicy whose policy document names a same-stack, just-created IAM role as Principal.AWS, the resource-policy PUT was issued before propagation completed and AWS rejected it:

SNS SetTopicAttributes rejects with Invalid parameter: Policy Error: PrincipalNotFound.
SQS SetQueueAttributes rejects the byte-for-byte-equivalent fresh-principal document with the less specific Invalid value for the parameter Policy.

Neither phrase was in cdkd's retry classifier, so the create failed permanently instead of retrying through the propagation window.

Fix

Adds two narrowly-anchored patterns to RETRYABLE_ERROR_MESSAGE_PATTERNS in src/deployment/retryable-errors.ts:

Policy Error: PrincipalNotFound (SNS wording)
Invalid value for the parameter Policy (SQS wording)

Both are anchored on the full vendor phrase so a genuinely malformed or non-existent principal (typo'd ARN, deleted role) still burns only the bounded retries before surfacing - they do not false-positive unrelated SNS/SQS parameter-validation errors. Unit tests in tests/unit/deployment/retryable-errors.test.ts assert both wire-format messages classify as retryable.

New integ: iam-propagation-stress

tests/integration/iam-propagation-stress/** - a race detector that creates FOUR brand-new IAM roles in ONE deploy, each consumed IMMEDIATELY by a DIFFERENT service so the DAG carries many independent race edges at once:

fresh Lambda exec role -> AWS::Lambda::Function (CreateFunction validates lambda.amazonaws.com can assume it)
fresh SFN role -> AWS::StepFunctions::StateMachine (CreateStateMachine validates the role; the SFN provider has no propagation retry of its own)
fresh EventBridge target role -> AWS::Events::Rule with an SFN target (PutTargets validates the rule can assume the role to StartExecution)
fresh principal role -> AWS::SQS::QueuePolicy + AWS::SNS::TopicPolicy (the resource-policy PUT validates the principal - this is the fix(deployment): SNS/SQS resource-policy create not retried on PrincipalNotFound (fresh-role propagation race) #839 edge)

Everything is cheap (no VPC, no NAT, no long-lived compute); the EventBridge rule is enabled: false on a 365-day schedule so it never fires but PutTargets still validates the fresh role at create time.

verify.sh assertion fixes

Post-destroy StepFunctions check now polls for DeleteStateMachine's async deletion - the API returns before the state machine is actually gone, so a single immediate describe-state-machine could falsely pass/fail.
Edge-2 SFN execution assertion retries start-execution -> describe-execution across the IAM-propagation window, because the fresh SFN role can take a moment to become assumable for the SFN-to-Lambda invoke grant.

Live validation against real AWS

This was validated GREEN end-to-end via /run-integ iam-propagation-stress before opening this PR (the "live-test the changed behavior" requirement is satisfied):

deploy clean across all four race edges
all four post-deploy consumer assertions OK
destroy reported 13 deleted / 0 errors
post-destroy orphan sweep empty (0 orphans)

The first real-AWS run is what surfaced the #839 edge-4 failure that this PR fixes.

Lessons (retrospective)

cdkd's fast SDK path beats IAM propagation for SNS/SQS resource policies on fresh roles - any "role created -> consumed within ~1s" edge is a latent failure that CloudFormation masks via deployment latency. The retry classifier must cover each consumer's specific rejection wording.
Post-deploy/destroy integ assertions must tolerate AWS async-delete windows (SFN DeleteStateMachine returns before the resource is gone) and IAM-propagation windows (an SFN execution invoking a Lambda on a fresh role) - assert with bounded polling/retry, not a single immediate probe.

Closes #839

…immediate-assume A new integ that maximizes the chance of catching an UNRETRIED IAM-propagation race on cdkd's fast SDK path. cdkd creates an IAM role and has a service assume it within ~1s, before IAM finishes propagating the role / its trust policy; CloudFormation tolerates this via deployment latency, cdkd does not. The race is handled narrowly today (RDS Enhanced Monitoring unprotected, and no integ deliberately stresses the breadth of fresh-role edges. CdkdIamPropagationStressExample creates four brand-new IAM roles, each consumed immediately by a different service in one deploy: - Lambda exec role -> AWS::Lambda::Function (CreateFunction) - SFN role -> AWS::StepFunctions::StateMachine (CreateStateMachine) - EventBridge target role -> AWS::Events::Rule SFN target (PutTargets) - fresh principal -> AWS::SQS::QueuePolicy + AWS::SNS::TopicPolicy No VPC / NAT / long-lived compute. The EventBridge rule is disabled on a 365-day schedule so it never fires, but PutTargets still validates the fresh target role at create time. The pass condition is: deploy SUCCEEDS. A deploy failure is a real cdkd finding (an unprotected consumer racing IAM propagation), so verify.sh prints which resource failed plus the error (deploy-log tail + cdkd events RESOURCE_FAILED lines + partial-state logicalId -> type map) for trivial triage, then still attempts destroy / cleanup. On success it asserts each role consumer works (invoke the Lambda + marker; start-execution -> poll to SUCCEEDED, proving the fresh SFN role and the SFN -> Lambda grant; list-targets-by-rule shows the SFN target bound to a role; queue/topic carry non-empty resource policies), then destroys and asserts the Lambda / state machine / rule / queue / topic / state file are each gone from AWS. verify.sh is BSD/macOS-portable (no grep -P / date -d), recovers the real deploy exit code from PIPESTATUS so a tee'd non-zero is not masked, traps aggressively (state + lock + deployment-events sidecar), and prints [verify] PASS only on full success. New scenario tag iam-fresh-role-immediate-assume in KNOWN_SCENARIOS (scripts/build-scenario-coverage-matrix.ts); coverage matrices regenerated; changelog + docs/testing.md entries added. Test-only change, no src edit. Not yet run against real AWS (a flaky failure there is the whole point).

…ropagation (#839) The iam-propagation-stress integ surfaced two deploy failures whose root cause is the same just-created-role IAM-propagation race already handled narrowly for RDS #794 / ECS #805 / Custom Resource #756: StressTopicPolicy: Invalid parameter: Policy Error: PrincipalNotFound StressQueuePolicy: Invalid value for the parameter Policy. Both the SNS TopicPolicy and the SQS QueuePolicy in the fixture name the same freshly-created PublisherRole ARN as Principal.AWS (verified in the synthesized template: the two policy documents are byte-for-byte the same shape). cdkd's fast SDK path issues the resource-policy PUT before IAM propagates the new role principal, so SNS rejects with the explicit PrincipalNotFound wording and SQS rejects with the less specific "Invalid value for the parameter Policy." -- both are the SAME race, not a malformed policy document. CloudFormation tolerates it via deployment latency; cdkd did not retry it. Add two narrowly-anchored patterns to RETRYABLE_ERROR_MESSAGE_PATTERNS so the existing withRetry absorbs the propagation window: - 'Policy Error: PrincipalNotFound' (SNS) - 'Invalid value for the parameter Policy' (SQS) Both anchors are specific enough that a genuinely malformed / non-existent principal still fails after the bounded retries. Unit tests assert the exact wire messages classify retryable, plus a negative guard that a permanently malformed policy without the race anchor stays non-retryable. No fixture change: both failures are confirmed races, so the fixture already exercises the real race path. Closes #839

…tions The Phase 1b edge-3 assertion fed the EventBridge rule's PHYSICAL ID directly to `aws events list-targets-by-rule --rule` and (post-destroy) `describe-rule --name`. cdkd's EventBridgeRuleProvider stores the rule ARN as the physical id (arn:<p>:events:<r>:<acct>:rule/<RuleName>), but those AWS CLI flags expect the bare rule NAME, not the ARN — so the call returned a ValidationException, the edge-3 target/role checks could not be satisfied, and the run aborted after edge 1 + edge 2 had already printed OK. Derive the bare rule name from the physical id by stripping everything up to and including the last `/` (`${RULE_PHYSICAL_ID##*/}`). A bare name (no `/`) passes through unchanged, and a custom-bus ARN (rule/<bus>/<name>) still yields the trailing rule name. The destroy-phase `describe-rule --name "${RULE_NAME}"` assertion now uses the correct name too. Edge 4 (SQS QueuePolicy + SNS TopicPolicy presence) was already correct and is unchanged.

…ress post-destroy assertion DeleteStateMachine is asynchronous: the state machine enters Status=DELETING and remains describable for a short window before AWS fully removes it (StateMachineDoesNotExist). The previous single-shot describe-state-machine assertion treated any successful describe as "still exists", racing AWS's async teardown and producing a false-positive "state machine still exists" failure. Replace it with a poll loop that tolerates Status=DELETING, treats a describe-failure / NotFound as gone, and fails fast on any non-DELETING status (delete never issued). cdkd's destroy was clean (13 deleted, 0 errors); this is a verify.sh assertion fix only, not a cdkd bug.

…the IAM-propagation window Wrap the edge-2 StepFunctions execution (start + poll) in an outer retry loop (up to 6 attempts, linear backoff) that starts a FRESH execution on the IAM-propagation authz signal (cause matching lambda:InvokeFunction / no identity-based policy allows / AccessDeniedException / not authorized to perform). Any non-authz terminal status fails immediately. Fixes an intermittent real-AWS false-positive: cdkd's fast SDK deploy returns before IAM propagates the SFN role's lambda:InvokeFunction grant to the SFN-assumed session, so the first execution started right after deploy can 403. CloudFormation never hits this because its slower finish lets IAM settle. cdkd's deploy is correct; this is a post-deploy runtime assertion that must tolerate the IAM propagation window.

…tion in iam-propagation-stress entry

…-at-timeout in iam-propagation-stress edge-2 Two review fixes for the iam-propagation-stress branch: - Add SQS QueuePolicy retry-classification boundary unit tests in retryable-errors.test.ts. The #839 SQS pattern `Invalid value for the parameter Policy` is intentionally broad (AWS emits it for any malformed SQS QueuePolicy). One test pins the accepted bounded-retry-then-surface tradeoff (a structurally-malformed QueuePolicy carrying that phrase classifies retryable); the other proves the substring match does not over-broaden to a different SQS error (InvalidAttributeValue) that lacks the phrase. Locks the contract so a future widening of the pattern is caught. - Handle a still-RUNNING execution at the inner-poll timeout in iam-propagation-stress edge-2. Previously a slow-but-not-failed execution left SM_STATUS=RUNNING, the cause query returned None, and the `*)` arm failed it immediately. Now a RUNNING status continues the outer retry loop (fresh execution after backoff) so a slow run gets more wall-clock across attempts; the outer loop stays capped at 6 and the post-loop guard fails only if it never SUCCEEDED.

go-to-k · 2026-06-13T14:25:06Z

Review complete: pr-code-reviewer + pr-test-reviewer ran (3-axis; spec-reviewer N/A for a bug fix without a design doc). Both findings fixed in 49d483f: (1) added SQS 'Invalid value for the parameter Policy' retry-classification boundary unit tests; (2) edge-2 SFN poll now handles RUNNING-at-timeout by continuing the bounded outer retry. Rebased on main.

## [0.221.1](v0.221.0...v0.221.1) (2026-06-13) ### Bug Fixes * **deployment:** retry SNS/SQS resource-policy create on fresh-role propagation ([#839](#839)) ([#843](#843)) ([05a4070](05a4070))

github-actions · 2026-06-13T14:29:27Z

🎉 This PR is included in version 0.221.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

go-to-k added 7 commits June 13, 2026 23:21

docs(changelog): reflect shipped #839 src fix + green real-AWS valida…

f00c0ca

…tion in iam-propagation-stress entry

go-to-k force-pushed the test/iam-propagation-stress branch from e23a822 to 49d483f Compare June 13, 2026 14:24

go-to-k merged commit 05a4070 into main Jun 13, 2026
5 checks passed

go-to-k deleted the test/iam-propagation-stress branch June 13, 2026 14:28

github-actions Bot added the released label Jun 13, 2026

Navigation Menu

Uh oh!

fix(deployment): retry SNS/SQS resource-policy create on fresh-role propagation (#839) - #843

fix(deployment): retry SNS/SQS resource-policy create on fresh-role propagation (#839)#843
go-to-k merged 7 commits into
mainfrom
test/iam-propagation-stress

go-to-k commented Jun 13, 2026

Uh oh!

go-to-k commented Jun 13, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jun 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

go-to-k commented Jun 13, 2026

Summary

Root cause (#839)

Fix

New integ: iam-propagation-stress

verify.sh assertion fixes

Live validation against real AWS

Lessons (retrospective)

Uh oh!

go-to-k commented Jun 13, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jun 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant