test(integ): add destroy-interrupt integ (#816 graceful SIGINT + #804 CR resume) - #835
Merged
Conversation
… CR replay fail-fast (#804) New tests/integration/destroy-interrupt fixture and verify.sh. Both the graceful-SIGINT destroy path (#816) and the Custom-Resource replay-after-partial-failure fail-fast (#804) shipped with unit + clean-destroy coverage only; this is their first real-AWS end-to-end verification. Fixture stack CdkdDestroyInterruptExample (16 synthesized resources): VPC + two isolated subnets + S3 gateway VPC endpoint + a VPC-attached Lambda (HandlerFn) backing a cdk.CustomResource (CrProbe) + four SSM parameters. The resource set makes a destroy span several seconds (VPC + subnets + SG + Lambda hyperplane ENI delete in order) so a mid-destroy SIGINT reliably lands during deletion, and the VPC-attached Lambda backing a CR is exactly the #804 shape (on a re-run the backing Lambda may already be gone). verify.sh (BSD/macOS-portable: no grep -P, no date -d, real exit codes captured to variables) runs four phases: 1. deploy clean. 2. first Ctrl-C: launch cdkd destroy --force in the background, poll its log for delete-loop evidence (bounded ~30s) before sending ONE kill -INT to cdkd's own SIGINT handler. On an interrupt that lands mid-destroy, assert the #816 contract: the drain notice is logged, the stack lock object is gone (released; pre-fix it stranded for its 30m TTL), and the state file is preserved (trimmed, still listing the not-yet-deleted resources). If the destroy finishes before the interrupt can land (a fast-account race) that is logged and accepted, falling through to the clean-end asserts rather than hard-failing. 3. re-run cdkd destroy --force to completion: assert it exits 0 (clean resume), finishes in < 180s (the #804 fail-fast: no ~10-minute GetFunction stall against the already-deleted backing Lambda), and the log carries no Pending / long-Lambda-waiter signature. 4. clean end-state: state + lock gone, backing Lambda gone, VPC gone (subnets / SG / ENI implicitly cleared, plus an explicit leftover-ENI scan), each captured SSM parameter physical id gone. New scenario tag destroy-interrupt in KNOWN_SCENARIOS (scripts/build-scenario-coverage-matrix.ts); the fixture .scenarios.json also tags custom-resource-async-poll + vpc-lambda-eni-release. Regenerated integ-coverage + scenario-coverage matrices and added a changelog note. No src/ change (integ-only). refs #816 #804
…and failure) The destroy-interrupt verify.sh failed with "stack lock object still present after graceful interrupt" against real AWS. Root cause was a TEST timing race, not a #816 cdkd gap: - The SIGINT-trigger grep matched "Resources to be deleted" (the PLAN banner), which destroy-runner.ts prints BEFORE acquiring the lock and BEFORE registering the #816 SIGINT handler (~L593, just before "Building dependency graph..."). The test sent SIGINT during lock-acquire, when Node's DEFAULT SIGINT behavior terminates the process before any handler runs, so the lock-releasing `finally` never executed and the lock was stranded. - The "OK: drain notice" line was a false positive: the grep "Interrupted|interrupt" also matches the stack name CdkdDestroyInterruptExample, so it passed even though the real drain notice was never logged (the process had already died). Fixes: - Trigger SIGINT only on a post-handler signal: a per-resource completion line ("<id> (<type>) deleted", anchored as "\) deleted") or "Building dependency graph..." (logged immediately after the handler is registered). The plan banner is no longer a trigger. Note the live-renderer "Deleting ..." label is suppressed on the non-TTY redirected log, so it is not a reliable signal. - Fallback SIGINT-anyway branch sends SIGINT only when the handler-registration signal is present; otherwise it warns and leaves the process alone (a pre-handler SIGINT would re-introduce the strand). - Drain-notice assertion now matches the actual text "finishing in-flight deletes" instead of a bare "interrupt". - Lock-release assertions use a bounded poll (wait_for_lock_release, ~5s) after the process exits, since lock release is an async S3 DeleteObject that can settle just after the node process exits. The PID/wait was already correct: the destroy is backgrounded with a plain redirect (not a pipe), so $! is the node PID and `wait` waits for that exact process. The lock S3 key the test checks (cdkd/<stack>/<region>/lock.json) matches LockManager.getLockKey. cdkd's #816 graceful-interrupt path is correct; no source change needed.
|
🎉 This PR is included in version 0.221.1 🎉 The release is available on: Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the missing real-AWS integ for the #816 graceful-SIGINT destroy path + the #804 CR-replay-after-partial-failure idempotency (both previously had only unit + clean-destroy coverage — the actual interrupt-and-resume was never exercised end-to-end).
Fixture (
CdkdDestroyInterruptExample)VPC (natGateways:0) + 2 isolated subnets + S3 gateway endpoint + SG + VPC-attached Lambda + a
CustomResourcewhose serviceToken is that Lambda (the #804 shape) + 4 SSM Parameters — enough that destroy spans several seconds (the SIGINT window).verify.sh asserts (robust SIGINT timing: polls the destroy log for a POST-handler signal — a per-resource delete line / "Building dependency graph" — before sending one
kill -INT; gracefully accepts + logs the race where destroy finishes first)GetFunctionstall (fail-fast); fully gone afterward (state, lock, backing Lambda, VPC, ENIs, SSM all deleted).Real-AWS result
PASS — 0 errors, 0 orphans (331s).
Note on the first run
The first attempt FAILED on a TEST timing bug (SIGINT fired on the PLAN banner, before the handler was registered → default-terminate → false "lock not released"; the "drain OK" was a grep false-positive matching the stack NAME). Fixed: trigger only on a post-handler signal, match the real drain text, bounded lock-release poll. Confirmed cdkd's #816 path is correct — the re-run shows the lock released gracefully and state preserved exactly as designed.
Coverage matrices regenerated.
refs #816 #804