Skip to content

testing.md

Latest commit

 

History

History
382 lines (316 loc) · 19 KB

testing.md

File metadata and controls

382 lines (316 loc) · 19 KB
Raw
description cdkd testing strategy (unit / integration / UPDATE / Rollback failure injection)
paths
tests/**

Testing Strategy

Unit Tests

  • tests/unit/**/*.test.ts
  • Uses Vitest
  • Import test APIs from 'vite-plus/test', never 'vitest' — vitest is a transitive dep bundled inside vite-plus, so a bare 'vitest' specifier resolves locally (pnpm hoisting) but fails CI's vp run typecheck:test with TS2307 (PR #1226). Copy the import line from a sibling test. Enforced by tests/unit/scripts/test-import-convention.test.ts (fails the local test run, naming the offending file).
  • Mocking: Mock AWS SDK with vi.mock()

Integration Tests

  • tests/integration/**
  • Uses actual AWS account
  • Environment variables: STATE_BUCKET, AWS_REGION
  • Examples verified with real AWS deployments (see tests/integration/ for full list)

verify.sh signal traps (mandatory)

A fixture that provisions real AWS resources must arm its cleanup trap on the signal paths too, in the exiting form:

trap cleanup EXIT
trap '(exit 130); cleanup; exit 130' INT
trap '(exit 143); cleanup; exit 143' TERM

trap cleanup EXIT INT TERM is NOT equivalent and must never be used: a bash signal handler returns to the interrupted point, so the script resumes the interrupted phase after cleanup and can exit 0 — reporting PASS while cleanup raced a still-live deploy. Omitting INT / TERM entirely leaks the stack on Ctrl-C or a harness timeout. Disarm with trap - EXIT INT TERM.

The (exit N) seed is load-bearing, not decoration. Many fixtures' cleanup opens with rc=$? and gates the whole teardown on it (if [ "${rc}" -eq 0 ]; then exit 0; fi). Inside a handler $? is the interrupted command's status, not the signal, so without the seed an interrupted run can see rc=0, skip the teardown entirely and exit 0 — the exact bug this convention exists to prevent. (exit N) sets $? to the signal's code, so rc=$? and ${1:-$?} cleanups both tear down correctly.

Enforced by tests/unit/scripts/integ-verify-signal-traps.test.ts (issue #1097); the user-facing writeup is in docs/testing.md.

verify.sh gone-probes (mandatory)

A destroy/leak assertion must never be a silenced blind probe: if aws <read-probe> ... >/dev/null 2>&1; then FAIL (and the inverse if ! aws ...; then <conclude gone>) read ANY failure (throttle, auth, network) as "gone" and silently pass the leak check (issue #1097 pattern 2). Route probes through the canonical helper block every affected fixture carries verbatim (source of truth: scripts/check-integ-probe-not-found.ts):

assert_gone "<leak description>" aws <service> <read-verb> [args...]
if ! gone_probe aws <service> <read-verb> [args...]; then ...still exists...; fi

gone_probe accepts ONLY the canonical not-found signature ('not ?found|no ?such|does ?not ?exist|non ?existent|\(404') and hard-FAILs on anything else. Probe state files via s3api head-object, never aws s3 ls (which exits 1 with empty output for "no keys"). Out of scope: mutation probes, fail-closed existence checks, pre-flight "already exists" guards, best-effort cleanup guards.

Two more spellings of the same defect are banned (issue #1120): capture-form fallbacks (N=$(aws <read-verb> ... 2>/dev/null || echo 0) / || true — a throttle reads as "0 remaining"; use a plain strict capture, or branch on gone_probe when not-found is legitimate) and silenced function wrappers (an exit-status wrapper fn() { aws ... >/dev/null 2>&1; } or a value wrapper with a swallow tail). Tail-less silenced captures/wrappers stay legal (set -e fails them loudly; for wrappers ONLY when the probe is the LAST command of the body), as does the strict stderr-capture idiom ($(cmd 2>&1 >/dev/null || true)).

Intermediate captures inside a value wrapper need || return 1: errexit is CLEARED inside $( ) command substitutions, so in a multi-statement wrapper called as V="$(fn)" an intermediate out="$(aws ...)" failure does not abort the body and the function exits 0 via its formatting tail; the explicit || return 1 propagates the probe error to the caller's set -e (local V=$(...) masks the status entirely; split declaration from assignment). A gone_probe-then-requery site must guard the requery against the TOCTOU race: canonical not-found on the requery is still "gone", anything else hard-fails. Best-effort cleanup is exempt via set +e[u] spans (bounded by the enclosing function) — mark cleanup helpers with set +eu in a SUBSHELL body (fn() { ( set +eu; ... ) }) instead of silencing probes, so calling them from a set +eu cleanup trap never re-arms strict mode mid-sweep. Enforced by tests/unit/scripts/integ-verify-probe-not-found.test.ts; user-facing writeup in docs/testing.md.

verify.sh CLI flags (mandatory)

Every flag a fixture passes must be declared on the subcommand it targets, not merely somewhere in src/cli/options.ts. The originating case (issue #1097): cdkd import --region died with error: unknown option '--region', so the import round-trip that fixture existed to exercise had never run once. --region IS declared in options.ts and IS accepted by ~10 sibling commands — import is the single one that never attaches it, so the flag looked right by analogy.

Two known traps when auditing by hand: --help omits hidden options (so help text is not decisive), and --region is NOT a no-op on the commands that DO accept it (it is the highest-precedence region source per cli-internals.md) — "cleaning up" deprecated --region flags would silently change region resolution.

Enforced by tests/unit/scripts/integ-cli-flags.test.ts, which walks the real Commander tree via buildProgram() (src/cli/program.ts) rather than --help or options.ts. A flag counts as accepted when the target command OR any ancestor declares it, matching Commander's own lookup. The check carries coverage floors (totals plus one per supported call shape), so a parser regression that stops seeing invocations fails loudly instead of passing vacuously -- two iterations of this lint were green while skipping most of the tree. The state-destroy-force-gate.sh hook remains the commit-time guard for the specific state destroy --force case; this lint generalizes it to every subcommand and also catches pre-existing occurrences the hook cannot see.

verify.sh version literals (mandatory)

Never hardcode a Lambda published-version literal in a fixture: version counters are monotonic per function/layer NAME and never reset, so a "${FN}:1" probe or a [ "${V}" != "2" ] assert passes only on the very first run in the account and fails every re-run with ResourceNotFoundException (issue #1324; third recurrence of the trap). Read version N from the live alias (--query 'FunctionVersion'), guard it numeric with a case pattern, and assert rotation as EXPECTED=$((N + 1)) — see codedeploy-lambda-deployment-group/verify.sh for the reference shape. Alias qualifiers (:live, :$LATEST), variable qualifiers, relative compares, and length(...) count queries stay legal; genuinely fixed versions (public cross-account layer ARNs) take # allow-version-literal: <reason>. Enforced by tests/unit/scripts/integ-verify-version-literals.test.ts (classifier: scripts/check-integ-version-literals.ts); user-facing writeup in docs/testing.md.

verify.sh must not call an aws verb the CLI does not have (mandatory)

A fixture must not call an aws <service> <verb> that is not a real AWS CLI subcommand. The trap is that such a verb can look entirely legitimate: the AWS CLI removes a set of operations from its command table (awscli/customizations/removals.py) that still exist in the API, so the SDKs, the API reference, and anything generated from them all offer it.

Originating case, verified 2026-08-09 against aws-cli/2.35.13: aws emr list-instance-groups, which forced two EMR fixtures to be rewritten. Its symptom was misleading —

Warning: Input is not a terminal (fd=0).
aws: [ERROR]: [Errno 22] Invalid argument

plus a hang without </dev/null, and --no-paginate --no-cli-pager did not help. That reads like an interactive "customization" and was first written up that way. That diagnosis was wrong. The actual cause:

  1. list-instance-groups is on the CLI's REMOVAL list — it is not an aws emr subcommand at all, and the CLI's own answer is Found invalid choice 'list-instance-groups'.
  2. The Errno 22 / hang is what cli_auto_prompt (on-partial in the maintainer's ~/.aws/config) does to ANY invalid-choice error: the CLI opens its interactive prompter, which cannot attach to a non-terminal stdin. AWS_CLI_AUTO_PROMPT=off makes the same call fail fast and legibly.

The corollary changes how you pick a replacement: the neighbouring verbs are usually fine. aws emr list-instance-fleets and aws emr list-instances are NOT removed and work non-interactively — so "the list-instance-* family is suspect" was the wrong generalization, and "assume any aws emr verb is suspect" over-blocks (the EMR fixtures rely on list-clusters / describe-cluster / modify-cluster-attributes / terminate-clusters). The unit of the defect is the (service, verb) pair.

Enforced by tests/unit/scripts/integ-aws-commands.test.ts (classifier: scripts/check-integ-aws-commands.ts) against the captured table tests/fixtures/aws-cli-removed-commands.json (refresh: vp run gen:aws-cli-removals). The table is a checked-in capture, not a live aws call, so the check is offline + deterministic — a checker that skips when its oracle is missing is the vacuous pass this file's checker rules forbid. Escape hatch: # allow-unavailable-aws-command: <reason> on the invocation's line or the line above.

Probe before you rely on an unfamiliar verb. AWS_CLI_AUTO_PROMPT=off aws <service> <verb> --help settles existence in under a second; running it against a bogus id settles behavior.

When the verb is unavailable, call the SDK directly rather than reaching for a different CLI verb that happens to work but returns less. The repo root already depends on every @aws-sdk/client-* cdkd uses, so a node --input-type=module -e one-liner from REPO_ROOT needs no extra install, and its response keys are the SDK's (PascalCase) shape:

REPO_ROOT="${PWD}/../../.."
list_instance_groups_json() { # $1 = cluster id -> JSON array of InstanceGroups
  ( cd "${REPO_ROOT}" && REGION="${REGION}" node --input-type=module -e "
import { EMRClient, ListInstanceGroupsCommand } from '@aws-sdk/client-emr';
const client = new EMRClient({ region: process.env.REGION });
const groups = [];
let marker;
do {
  const res = await client.send(
    new ListInstanceGroupsCommand({ ClusterId: process.argv[1], Marker: marker })
  );
  groups.push(...(res.InstanceGroups ?? []));
  marker = res.Marker;
} while (marker);
process.stdout.write(JSON.stringify(groups));
" "$1" ) || return 1
}

Two things in that shape are load-bearing, not decoration. The || return 1 propagates a node/SDK failure to the caller's set -e — without it an empty result silently satisfies a // empty-defaulted jq assertion (the gone-probe rule's failure mode, one layer up). And the Marker loop matches whatever pagination the provider under test does; a partial first page is a silent false pass. Reference implementations: tests/integration/emr-cluster/verify.sh and tests/integration/emr-instance-configs/verify.sh.

A pager invoked non-interactively is a SEPARATE route to a hang, so export AWS_PAGER="" near the top of a fixture is cheap insurance. This is a recommendation for NEW and affected fixtures, not a tree-wide invariant — most existing fixtures do not set it and are fine. tests/integration/emr-instance-configs/verify.sh is the reference.

Mechanically enforced since issue #1402 (see the lint named above). User-facing writeup in docs/testing.md.

verify.sh list readbacks must be order-insensitive (mandatory)

AWS does not preserve the submitted order of list-valued members on readback. An assertion that string-compares a joined list against the submitted order is flaky, and its failure message ACCUSES THE FIX — the worst kind of false negative. Verified 2026-08-09 on the lambda-esm-self-managed-kafka fixture: cdkd sent Endpoints.KAFKA_BOOTSTRAP_SERVERS = [b-1…, b-2…], list-event-source-mappings returned [b-2…, b-1…], and the assertion reported "issue #1384 NOT closed" while the fix was working perfectly.

Sort BOTH sides unless the list is genuinely order-significant:

--query "join(' ', sort(Path.To.List || \`[]\`))"

(The || \[]`` coalesce is the separate null-list guard the gone-probe rule covers — keep both.)

This is the integ-side twin of src/analyzer/drift-normalize.ts, which canonicalizes tag lists and resource-id/ARN arrays on BOTH comparison sides for exactly this reason. The same judgment call applies: a list that IS order-significant (DNS resolver lists, preference orders — see getDriftUnorderedPaths) must stay unsorted, because sorting it would HIDE a real regression.

NOT mechanically enforced — whether a given list is order-significant is a judgment call a lint cannot make, so this one stays a read-it-and-follow-it rule by design. User-facing writeup in docs/testing.md.

Fixture stateful L2s need an explicit removalPolicy (mandatory)

Stateful CDK L2 constructs (kinesis.Stream, dynamodb.Table/TableV2, s3.Bucket, logs.LogGroup, kms.Key, rds.DatabaseInstance/Cluster, efs.FileSystem, opensearchservice.Domain, ecr.Repository, cognito.UserPool, backup.BackupVault) default to RemovalPolicy.RETAIN -> DeletionPolicy: Retain in the template. Both CloudFormation and cdkd honor it, so a fixture that omits the policy leaks the resource on EVERY deploy/destroy cycle while destroy still reports success. Originating incident (issue #1326): the sqs-cloudwatch fixture's Kinesis Stream leaked 14 billed PROVISIONED streams across a month of us-west-2 benchmark runs; the lint then immediately found a second live case (log-pipeline).

Every instantiation of those constructs in tests/integration/*/{lib,bin} must do ONE of: pass an explicit removalPolicy (RETAIN included -- it has to be a decision, not a default), call applyRemovalPolicy(...) on the assigned variable/property in the same file, or carry an // allow-default-removal-policy: <reason> comment (for fixtures that intentionally exercise the default; the count is capped by the test). A props object passed as a same-file variable is resolved; a spread does NOT count -- restate the policy visibly.

Enforced by tests/unit/scripts/integ-fixture-removal-policy.test.ts (classifier: scripts/check-fixture-removal-policy.ts); user-facing writeup in docs/testing.md. L1 Cfn* constructs are out of scope (their template default is Delete).

A checker must prove it sees its input

When writing a lint or codegen that SCANS files (verify.sh scripts, templates, source), "0 violations" and "parsed nothing at all" produce the identical green result. Assert coverage explicitly: how many items were parsed, how many distinct kinds, and a floor per input SHAPE the parser claims to handle — not just a grand total.

This is not hypothetical. The #1097 CLI-flag lint shipped this defect twice while its suite was green:

  1. it ignored inline env prefixes, missing every CDKD_TEST_UPDATE=true node ... deploy invocation (46 of them — the UPDATE-mode deploys, i.e. the ones most worth checking);
  2. it required a literal cli.js in the node <script> token, so node "${LOCAL_DIST}" ... matched nothing — 135 of 195 fixtures contributed zero. Coverage was 36% of the tree.

Neither was found by reading the code or by tests passing. Both were found by instrumenting the checker to print what it actually parsed and reconciling that against an independently-grepped denominator. A third variant then appeared in the fix itself: an unanchored shape regex matched invocations that did not have the shape, so a total regression of that branch would still have cleared a > 0 check.

Practical rules:

  • before trusting a new checker, measure — count parsed items per shape and explain any gap against a rough independent count;
  • encode the measurement as assertions with real numeric floors, anchored so a near-miss cannot satisfy them;
  • aggregate floors alone are insufficient: one dead shape hides under them.

See tests/unit/scripts/integ-cli-flags.test.ts for the shape the assertions take.

A checker must also prove it FAILS — against real code

Coverage floors prove the checker SEES its input. They do not prove it still REJECTS a violation. Those are different failures, and the second one is what makes a green CI lie.

Synthetic unit fixtures cannot close it on their own, because a fixture encodes the author's mental model of the defect — so a checker and its tests can share the same blind spot and agree with each other. The only thing that proves rejection is introducing a REAL regression into the REAL tree and watching the checker exit non-zero.

This is not hypothetical. The update-wrap-coverage critic (#1269) shipped its first version with a passing suite that included a dedicated "flags an unguarded wrap" test. The test threw the typed error LEXICALLY inside the try. Real providers do not: the wrap is at the boundary and the throw lives in the delegated applyUpdate(). So against real code the check silently reported green and enforced nothing on exactly the providers it existed to protect. It was found by deleting the pass-through from the real LogsLogGroupProvider and observing rc=0 where rc=1 was required.

Practical rules:

  • for each CI-blocking verdict the checker can emit, introduce that violation into REAL repo code, confirm a non-zero exit naming the right target, then restore. Do this before trusting the checker, and record it in the PR;
  • treat a synthetic fixture that passes as necessary but never sufficient — when the real-code probe disagrees with it, the FIXTURE is usually wrong;
  • once a real-code probe finds a miss, add the shape it exercised as a synthetic regression test too, so the specific gap stays closed cheaply.

UPDATE Testing

  • Environment variable CDKD_TEST_UPDATE=true enables UPDATE test mode
  • Example: tests/integration/basic/lib/basic-stack.ts
  • Allows testing UPDATE operations without modifying code
  • JSON Patch (RFC 6902) verified working for S3, Lambda, IAM resources

Rollback Testing (failure injection)

  • Environment variable CDKD_TEST_FAIL=true injects a deliberately-failing resource (an AWS::SQS::Queue with an out-of-range MessageRetentionPeriod) into the basic stack
  • Verifies against real AWS that already-completed siblings get rolled back when one resource fails: CDKD_TEST_FAIL=true cdkd deploy CdkdBasicExample
  • After rollback, S3 and SSM Document should both be deleted and state file should be empty