docs(testing): record the two real-AWS verify.sh traps as repo rules, not memory

.claude/rules/testing.md

@@ -143,6 +143,106 @@ by `tests/unit/scripts/integ-verify-version-literals.test.ts` (classifier:
 `scripts/check-integ-version-literals.ts`); user-facing writeup in
 [docs/testing.md](../../docs/testing.md).
 
+### `verify.sh` must not call an AWS-CLI-CUSTOMIZED command (mandatory)
+
+Some `aws` subcommands are not plain API pass-throughs — the AWS CLI wraps them
+in an interactive customization. Run from a non-interactive shell (an integ
+`verify.sh`, a background task, CI) they print
+
+```text
+Warning: Input is not a terminal (fd=0).
+aws: [ERROR]: [Errno 22] Invalid argument
+```
+
+and, without a `</dev/null`, HANG instead of returning. Verified 2026-08-09 on
+`aws emr list-instance-groups`: `--no-paginate --no-cli-pager </dev/null` does
+NOT help. The `emr` family is the known offender (`create-cluster`, `ssh`,
+`socks`, the `list-instance-*` commands); assume any `aws emr` verb is suspect
+until proven otherwise, and re-probe when adding a fixture for a new service.
+
+**Probe before you rely on it.** Run the candidate against a bogus id with a
+hard timeout; a plain API command answers with a validation / not-found error
+in under a second, a customized one hangs or emits the `Errno 22` line above.
+
+**When the command is customized, call the SDK directly** rather than reaching
+for a different CLI verb that happens to work but returns less. The repo root
+already depends on every `@aws-sdk/client-*` cdkd uses, so a `node
+--input-type=module -e` one-liner from `REPO_ROOT` needs no extra install, and
+its response keys are the SDK's (PascalCase) shape:
+
+```bash
+REPO_ROOT="${PWD}/../../.."
+list_instance_groups_json() { # $1 = cluster id -> JSON array of InstanceGroups
+  ( cd "${REPO_ROOT}" && REGION="${REGION}" node --input-type=module -e "
+import { EMRClient, ListInstanceGroupsCommand } from '@aws-sdk/client-emr';
+const client = new EMRClient({ region: process.env.REGION });
+const groups = [];
+let marker;
+do {
+  const res = await client.send(
+    new ListInstanceGroupsCommand({ ClusterId: process.argv[1], Marker: marker })
+  );
+  groups.push(...(res.InstanceGroups ?? []));
+  marker = res.Marker;
+} while (marker);
+process.stdout.write(JSON.stringify(groups));
+" "$1" ) || return 1
+}
+```
+
+Two things in that shape are load-bearing, not decoration. The `|| return 1`
+propagates a node/SDK failure to the caller's `set -e` — without it an empty
+result silently satisfies a `// empty`-defaulted `jq` assertion (the
+gone-probe rule's failure mode, one layer up). And the `Marker` loop matches
+whatever pagination the provider under test does; a partial first page is a
+silent false pass. Reference implementations:
+`tests/integration/emr-cluster/verify.sh` and
+`tests/integration/emr-instance-configs/verify.sh`.
+
+A pager invoked non-interactively is a second route to the same hang, so
+`export AWS_PAGER=""` near the top of a fixture is cheap insurance. This is a
+recommendation for NEW and affected fixtures, not a tree-wide invariant — most
+existing fixtures do not set it and are fine, because the hang only bites the
+customized commands. `tests/integration/emr-instance-configs/verify.sh` is the
+reference.
+
+NOT mechanically enforced yet — a lint over `tests/integration/*/verify.sh` is
+tracked in issue
+[#1402](https://github.com/go-to-k/cdkd/issues/1402), so until then this is a
+read-it-and-follow-it rule. User-facing writeup in
+[docs/testing.md](../../docs/testing.md).
+
+### `verify.sh` list readbacks must be order-insensitive (mandatory)
+
+AWS does not preserve the submitted order of list-valued members on readback.
+An assertion that string-compares a joined list against the submitted order is
+flaky, and its failure message ACCUSES THE FIX — the worst kind of false
+negative. Verified 2026-08-09 on the `lambda-esm-self-managed-kafka` fixture:
+cdkd sent `Endpoints.KAFKA_BOOTSTRAP_SERVERS = [b-1…, b-2…]`,
+`list-event-source-mappings` returned `[b-2…, b-1…]`, and the assertion
+reported "issue #1384 NOT closed" while the fix was working perfectly.
+
+Sort BOTH sides unless the list is genuinely order-significant:
+
+```bash
+--query "join(' ', sort(Path.To.List || \`[]\`))"
+```
+
+(The `|| \`[]\`` coalesce is the separate null-list guard the gone-probe rule
+covers — keep both.)
+
+This is the integ-side twin of `src/analyzer/drift-normalize.ts`, which
+canonicalizes tag lists and resource-id/ARN arrays on BOTH comparison sides for
+exactly this reason. The same judgment call applies: a list that IS
+order-significant (DNS resolver lists, preference orders — see
+`getDriftUnorderedPaths`) must stay unsorted, because sorting it would HIDE a
+real regression.
+
+NOT mechanically enforced — whether a given list is order-significant is a
+judgment call a lint cannot make, so this one stays a read-it-and-follow-it
+rule by design. User-facing writeup in
+[docs/testing.md](../../docs/testing.md).
+
 ### Fixture stateful L2s need an explicit removalPolicy (mandatory)
 
 Stateful CDK L2 constructs (`kinesis.Stream`, `dynamodb.Table`/`TableV2`,

docs/testing.md

@@ -656,6 +656,61 @@ coverage floors per constructor-reference shape and per construct kind so a
 parser regression fails loudly rather than passing vacuously. Baseline
 2026-07-31: 523 fixture files scanned, 120 stateful-L2 instantiations.
 
+### Fixture convention: never call an AWS-CLI-customized command
+
+Some `aws` subcommands are wrapped in an interactive AWS CLI customization
+rather than being plain API pass-throughs. In a non-interactive shell — which
+is every `verify.sh` run — they print `Warning: Input is not a terminal
+(fd=0).` then die with `aws: [ERROR]: [Errno 22] Invalid argument`, and without
+a `</dev/null` they HANG instead of returning. Confirmed 2026-08-09 on `aws emr
+list-instance-groups`, where `--no-paginate --no-cli-pager </dev/null` did not
+help. The `emr` family is the known offender; treat any `aws emr` verb as
+suspect until you have probed it.
+
+Probe a candidate against a bogus id under a hard timeout before relying on it:
+a plain API command returns a validation / not-found error immediately, a
+customized one hangs or emits the `Errno 22` line.
+
+When a command is customized, call the AWS SDK directly from `verify.sh`
+instead of substituting a different CLI verb that happens to work but returns
+less data. The repo root already depends on every `@aws-sdk/client-*` cdkd
+uses, so a `node --input-type=module -e` one-liner run from the repo root needs
+no extra install. Two details in the reference shape are load-bearing: a
+`|| return 1` so an SDK failure reaches the caller's `set -e` (otherwise an
+empty result silently satisfies a `// empty`-defaulted `jq` assertion), and a
+pagination loop matching whatever the provider under test does (a partial first
+page is a silent false pass). See `tests/integration/emr-cluster/verify.sh` and
+`tests/integration/emr-instance-configs/verify.sh` for the full helper.
+
+A pager invoked non-interactively is a second route to the same hang, so
+`export AWS_PAGER=""` near the top of a fixture is cheap insurance. Treat this
+as a recommendation for new and affected fixtures rather than a tree-wide
+invariant — most existing fixtures do not set it and are fine, because the hang
+only bites the customized commands.
+`tests/integration/emr-instance-configs/verify.sh` is the reference.
+
+### Fixture convention: sort both sides of a list readback
+
+AWS does not preserve the submitted order of list-valued members when you read
+them back, so an assertion that string-compares a joined list against the order
+you sent is flaky — and because it fails on a correct implementation, its error
+message accuses the fix. Seen 2026-08-09 in the `lambda-esm-self-managed-kafka`
+fixture: cdkd submitted two Kafka bootstrap servers in one order,
+`list-event-source-mappings` returned them in the other, and the run reported
+"issue #1384 NOT closed" while the fix was working.
+
+Sort both sides in the query, unless the list is genuinely order-significant:
+
+```bash
+--query "join(' ', sort(Path.To.List || `[]`))"
+```
+
+This mirrors what `src/analyzer/drift-normalize.ts` does for the drift
+comparator — it canonicalizes tag lists and resource-id/ARN arrays on both
+sides for the same reason. The same caveat applies too: a list whose order
+carries meaning (DNS resolver lists, preference orders) must NOT be sorted,
+because sorting would hide a real regression rather than reveal one.
+
 ## 3. Deploy Using cdkd
 
 ```bash