test(integ): add rollback-failure-injection integ (deploy-engine rollback on a rich stack + #808 fail events) - #836
Merged
Conversation
…ollback path Add a new integration test that exercises the cdkd deploy-engine ROLLBACK path on a RICH multi-resource stack. The only existing real-AWS rollback coverage is the trivial `basic` single-SQS CDKD_TEST_FAIL injection (one queue, nothing for rollback to delete); this fixture gives rollback real work by having several interdependent siblings COMPLETE before the failure. Fixture (CdkdRollbackFailureExample): VPC + Subnets + SecurityGroup + IAM Role + Lambda-in-VPC + SSM Parameter. The failure is self-contained (does NOT reuse the `basic` fixture's CDKD_TEST_FAIL plumbing): gated on ROLLBACK_INTEG_FAIL=true, an SQS Queue with an out-of-range messageRetentionPeriod (9999999; valid range [60, 1209600]) that AWS rejects on CreateQueue. The failing queue depends on the two fast siblings (IAM Role + SSM Parameter) so cdkd's event-driven DAG guarantees those are created before the queue is attempted, while the slow VPC/Lambda branch runs in parallel and is also rolled back. verify.sh (BSD/macOS-portable; real-rc capture + explicit `[verify] PASS`): 1. deploy with the fail flag ON exits non-zero 2. completed siblings rolled back (queried directly against AWS: SSM Parameter / SecurityGroup / VPC gone, no failing queue, no leftover hyperplane ENIs; cdkd state empty / no orphan) 3. #808 events captured the failure: RESOURCE_FAILED for AWS::SQS::Queue, ROLLBACK_STARTED + ROLLBACK_RESOURCE_SUCCEEDED, RUN_FINISHED result=FAILED 4. deploy with the flag OFF succeeds, destroy is clean (0 orphans) 5. EXIT trap aggressively sweeps any orphan VPC/ENI/SG/Role/Lambda/SSM/SQS plus the events sidecar on the failure path This is the deploy-engine rollback regression net and closes the failure-path follow-up the deployment-events fixture README explicitly deferred. Test-only: adds scenario tag `rollback-failure-injection` to KNOWN_SCENARIOS, regenerates the integ-coverage / scenario-coverage matrices, and notes the fixture in docs/testing.md + docs/changelog-cdkd.md. No src/ changes. refs #808
The clean-deploy step-5a assertion ("VPC created after a successful deploy")
filtered describe-vpcs by Name=tag:aws:cdk:path. cdkd's EC2 provider only
forwards template-supplied Tags and AWS reserves the aws: prefix, so cdkd
never sets aws:cdk:path on a VPC/SG/Role/Lambda. That filter always returned
empty: it falsely failed step 5a even though the clean deploy did create the
VPC (the run's destroy step deleted Vpc8378EB38), and it vacuously passed the
"gone" assertions (steps 3c/3d/7) plus the aggressive_cleanup Lambda/IAM
enumerators, masking any real orphan.
Fix: the fixture stack now applies a deterministic non-reserved tag
(cdkd:integ-fixture=rollback-failure-injection) to every resource via
cdk.Tags.of(this).add(...). cdkd reliably applies it (EC2 CreateTags /
IAM TagRole / Lambda TagResource), so find_fixture_vpcs / find_fixture_sgs
and the cleanup enumerators now filter on it. All three phases (rollback-gone,
clean-created, post-destroy-gone) are deterministic.
The rollback assertions (steps 2-3) and the #808 events assertions (step 4)
are unchanged.
go-to-k
force-pushed
the
test/rollback-failure-injection
branch
from
June 13, 2026 11:12
1940488 to
181e1f5
Compare
|
🎉 This PR is included in version 0.221.1 🎉 The release is available on: Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a real-AWS integ exercising the deploy-engine ROLLBACK path on a RICH multi-resource stack — previously rollback was only covered by the trivial
basicsingle-SQSCDKD_TEST_FAIL. Also asserts the #808 failure-path events (RESOURCE_FAILED/ROLLBACK_*/RUN_FINISHED result=FAILED) that were unit-only.Fixture (
CdkdRollbackFailureExample)VPC + NAT + public/private subnets + SecurityGroup + IAM Role + VPC-attached Lambda + SSM Parameter. A self-contained failing resource (gated on
ROLLBACK_INTEG_FAIL=true): an SQS Queue with an out-of-rangemessageRetentionPeriod,addDependency'd on the Role + Parameter so cdkd's event-driven DAG dispatches it only AFTER those siblings complete — guaranteeing rollback has already-created resources to delete.verify.sh asserts
cdkd:integ-fixturetag, NOTaws:cdk:pathwhich cdkd never applies); state reflects rollback.cdkd events --format json→RESOURCE_FAILEDfor AWS::SQS::Queue + ROLLBACK_* events + RUN_FINISHED result=FAILED.Real-AWS result
PASS — 17 deleted, 0 errors, 0 orphans (418s).
Note on the first run
First attempt FAILED on a TEST assertion bug: the step-5 VPC-existence check filtered on
tag:aws:cdk:path, which cdkd does not apply (AWS reserves theaws:prefix) → false "no VPC created" even though the clean deploy created + later destroyed the VPC. Fixed by tagging the stack with an owncdkd:integ-fixturetag and filtering on it (which also made the previously-vacuous cleanup enumerators real). The rollback + events feature itself passed on the first run.Coverage matrices regenerated.
refs #808