Follow-up from PR #1401, which recorded the trap as a written rule
(.claude/rules/testing.md -> "verify.sh must not call an
AWS-CLI-CUSTOMIZED command"). That is rung 2 of the durable-rules ladder
(a checklist the doer must read). Rung 1 is a lint.
What to detect
An aws <service> <verb> invocation in tests/integration/*/verify.sh whose
verb is a known AWS-CLI customization rather than an API pass-through.
Non-interactively these print Warning: Input is not a terminal (fd=0). and
then die with [Errno 22] Invalid argument — or hang outright without a
</dev/null. Confirmed 2026-08-09 on aws emr list-instance-groups, where
--no-paginate --no-cli-pager </dev/null did not help.
Known offender family today: aws emr (create-cluster, ssh, socks, the
list-instance-* verbs). aws emr list-instances is NOT customized and is
used safely by tests/integration/emr-instance-configs/verify.sh, so the list
has to be per-verb, not per-service.
Why it was not built inline
The repo's own checker rules (.claude/rules/testing.md -> "A checker must
prove it sees its input" / "A checker must also prove it FAILS — against real
code") require per-shape coverage floors plus a real-code fail probe. The
calibration set today is a single command family, so the checker's value is
mostly forward-looking — worth doing, but not worth bundling into a docs PR.
Suggested shape
Extend the existing tests/unit/scripts/integ-cli-flags.test.ts parser (it
already walks every fixture's verify.sh and tokenizes invocations) rather
than writing a new scanner, so the coverage floors it already carries protect
this check too. Escape hatch: a # allow-customized-aws-command: <reason>
comment for a fixture that genuinely needs one and has proven it works.
Per the checker rules, before trusting it: introduce a real
aws emr list-instance-groups call into a real fixture, confirm the check
exits non-zero naming that fixture, then restore — and record that probe in
the PR.
Follow-up from PR #1401, which recorded the trap as a written rule
(
.claude/rules/testing.md-> "verify.shmust not call anAWS-CLI-CUSTOMIZED command"). That is rung 2 of the durable-rules ladder
(a checklist the doer must read). Rung 1 is a lint.
What to detect
An
aws <service> <verb>invocation intests/integration/*/verify.shwhoseverb is a known AWS-CLI customization rather than an API pass-through.
Non-interactively these print
Warning: Input is not a terminal (fd=0).andthen die with
[Errno 22] Invalid argument— or hang outright without a</dev/null. Confirmed 2026-08-09 onaws emr list-instance-groups, where--no-paginate --no-cli-pager </dev/nulldid not help.Known offender family today:
aws emr(create-cluster,ssh,socks, thelist-instance-*verbs).aws emr list-instancesis NOT customized and isused safely by
tests/integration/emr-instance-configs/verify.sh, so the listhas to be per-verb, not per-service.
Why it was not built inline
The repo's own checker rules (
.claude/rules/testing.md-> "A checker mustprove it sees its input" / "A checker must also prove it FAILS — against real
code") require per-shape coverage floors plus a real-code fail probe. The
calibration set today is a single command family, so the checker's value is
mostly forward-looking — worth doing, but not worth bundling into a docs PR.
Suggested shape
Extend the existing
tests/unit/scripts/integ-cli-flags.test.tsparser (italready walks every fixture's
verify.shand tokenizes invocations) ratherthan writing a new scanner, so the coverage floors it already carries protect
this check too. Escape hatch: a
# allow-customized-aws-command: <reason>comment for a fixture that genuinely needs one and has proven it works.
Per the checker rules, before trusting it: introduce a real
aws emr list-instance-groupscall into a real fixture, confirm the checkexits non-zero naming that fixture, then restore — and record that probe in
the PR.