fix(deployment): retry AWS throttling that surfaces as HTTP 400 by error name (#861)

Author: go-to-k

docs/_generated/integ-coverage.json

@@ -2291,6 +2291,7 @@
         "rollback-failure-injection",
         "sdk-ccapi-crossref",
         "tags-propagation",
+        "throttle-wide-dag",
         "update-replace",
         "vpc-lambda-cr-race"
       ],
@@ -2411,6 +2412,10 @@
           "l2",
           "literal"
         ],
+        "throttle-wide-dag": [
+          "l1",
+          "literal"
+        ],
         "update-replace": [
           "l2",
           "literal"
@@ -3320,6 +3325,7 @@
         "serverless-api",
         "sns-sqs-event",
         "tags-propagation",
+        "throttle-wide-dag",
         "update-policy-mutations"
       ],
       "signals": {
@@ -3395,6 +3401,10 @@
         "tags-propagation": [
           "l2"
         ],
+        "throttle-wide-dag": [
+          "l1",
+          "literal"
+        ],
         "update-policy-mutations": [
           "l2",
           "literal"
@@ -3565,6 +3575,7 @@
         "secrets-dynamic-ref",
         "state-info-command",
         "tags-propagation",
+        "throttle-wide-dag",
         "update-policy-mutations",
         "vpc-lookup"
       ],
@@ -3692,6 +3703,10 @@
           "l2",
           "literal"
         ],
+        "throttle-wide-dag": [
+          "l1",
+          "literal"
+        ],
         "update-policy-mutations": [
           "l2",
           "literal"

docs/_generated/integ-last-run.tsv

@@ -130,3 +130,4 @@ rollback-failure-injection	2026-06-13T11:11:55Z	PASS	418	verify.sh	deploy-engine
 deployment-events	2026-06-13T11:42:29Z	PASS	37	verify.sh	#831 +SNS/SSM not-found post-destroy; 0 err
 update-replace	2026-06-13T11:42:29Z	PASS	71	verify.sh	#831 +Lambda/IAM/SG not-found post-destroy; 0 err
 destroy-interrupt	2026-06-13T11:42:29Z	PASS	581	verify.sh	#831 +SG/IAM not-found post-destroy; #816+#804 verified; 0 orphan
+throttle-wide-dag	2026-06-14T04:35:46Z	PASS		verify.sh	bug15 throttle-retry validated deploy+destroy clean

docs/_generated/scenario-coverage.json

@@ -259,6 +259,10 @@
     {
       "tag": "vpc-lambda-eni-release",
       "description": "Lambda hyperplane ENI cleanup after DeleteFunction (5-30 min eventually consistent)."
+    },
+    {
+      "tag": "wide-dag-throttle-retry",
+      "description": "Wide (~100-resource: 80 SSM Parameters + 10 IAM Roles + 10 SNS Topics, 10-deep SSM Fn::Sub chain) single-stack burst deployed under a HIGH `--concurrency` to stress the concurrency limiter + event-driven DAG executor + throttle/retry classifier: a `TooManyRequests` / `Rate exceeded` / HTTP 429 during the burst must be RETRIED (deploy still succeeds) not fatal, the chained subset proves strict DAG ordering, and the destroy burst absorbs ~100 deletes with 0 orphans."
     }
   ],
   "fixtures": [
@@ -1161,6 +1165,13 @@
         "stack-level-tag-propagation-multitype"
       ]
     },
+    {
+      "name": "throttle-wide-dag",
+      "annotated": true,
+      "scenarios": [
+        "wide-dag-throttle-retry"
+      ]
+    },
     {
       "name": "update-policy-mutations",
       "annotated": true,
@@ -1706,6 +1717,13 @@
         "lambda",
         "vpc-lambda"
       ]
+    },
+    {
+      "scenario": "wide-dag-throttle-retry",
+      "description": "Wide (~100-resource: 80 SSM Parameters + 10 IAM Roles + 10 SNS Topics, 10-deep SSM Fn::Sub chain) single-stack burst deployed under a HIGH `--concurrency` to stress the concurrency limiter + event-driven DAG executor + throttle/retry classifier: a `TooManyRequests` / `Rate exceeded` / HTTP 429 during the burst must be RETRIED (deploy still succeeds) not fatal, the chained subset proves strict DAG ordering, and the destroy burst absorbs ~100 deletes with 0 orphans.",
+      "fixtures": [
+        "throttle-wide-dag"
+      ]
     }
   ],
   "orphanScenarios": [],

docs/changelog-cdkd.md

@@ -4,7 +4,7 @@
 
 Run `vp run scenario-coverage` to regenerate.
 
-**65 / 65 canonical scenarios** have at least one integ fixture exercising them. **132 / 151 integ fixtures** carry a `.scenarios.json` sidecar (with 0+ tags); the rest are un-annotated and contributor-reviewed below.
+**66 / 66 canonical scenarios** have at least one integ fixture exercising them. **133 / 152 integ fixtures** carry a `.scenarios.json` sidecar (with 0+ tags); the rest are un-annotated and contributor-reviewed below.
 
 ## How this is computed
 
@@ -26,7 +26,7 @@ This report is a visibility tool, not a commit-time gate. Many cdkd fixtures leg
 
 _None._ Every canonical scenario has at least one integ fixture tagged with it.
 
-## Per-scenario coverage (65 scenarios)
+## Per-scenario coverage (66 scenarios)
 
 | Scenario | Description | Integ Fixture(s) |
 |---|---|---|
@@ -95,6 +95,7 @@ _None._ Every canonical scenario has at least one integ fixture tagged with it.
 | `update-replace-breadth` | Second-deploy property mutation exercising BOTH cdkd update paths in one stack: in-place provider.update() (S3 versioning toggle / Lambda env+memory / IAM inline-policy edit / SecurityGroup ingress add — physical id unchanged) AND replacement (S3 BucketName change per the replacement-rules registry — new physical id, old resource cleaned up). Regression net for provider update() paths + #807 replacement propagation + #809 Cloud Control write-only-property UPDATE on non-ECS types. | [`update-replace`](../tests/integration/update-replace/) |
 | `vpc-lambda-cr-race` | Custom Resource invocation against a VPC Lambda mid-deploy (ENI-attach race window). | [`vpc-lambda-cr-race`](../tests/integration/vpc-lambda-cr-race/) |
 | `vpc-lambda-eni-release` | Lambda hyperplane ENI cleanup after DeleteFunction (5-30 min eventually consistent). | [`bench-cdk-sample`](../tests/integration/bench-cdk-sample/)<br>[`destroy-interrupt`](../tests/integration/destroy-interrupt/)<br>[`lambda`](../tests/integration/lambda/)<br>[`vpc-lambda`](../tests/integration/vpc-lambda/) |
+| `wide-dag-throttle-retry` | Wide (~100-resource: 80 SSM Parameters + 10 IAM Roles + 10 SNS Topics, 10-deep SSM Fn::Sub chain) single-stack burst deployed under a HIGH `--concurrency` to stress the concurrency limiter + event-driven DAG executor + throttle/retry classifier: a `TooManyRequests` / `Rate exceeded` / HTTP 429 during the burst must be RETRIED (deploy still succeeds) not fatal, the chained subset proves strict DAG ordering, and the destroy burst absorbs ~100 deletes with 0 orphans. | [`throttle-wide-dag`](../tests/integration/throttle-wide-dag/) |
 
 ## Un-annotated fixtures (19)
 

docs/integ-coverage.md

@@ -210,6 +210,8 @@ const KNOWN_SCENARIOS: Record<string, string> = {
   // ---- Multi-resource / broad-regression set ----
   'cross-cutting-deploy-destroy':
     'Broad real-AWS regression set (39+ resource VPC+NAT+CF+Lambda+SQS or comparable breadth). Refreshes the integ-broad gate.',
+  'wide-dag-throttle-retry':
+    'Wide (~100-resource: 80 SSM Parameters + 10 IAM Roles + 10 SNS Topics, 10-deep SSM Fn::Sub chain) single-stack burst deployed under a HIGH `--concurrency` to stress the concurrency limiter + event-driven DAG executor + throttle/retry classifier: a `TooManyRequests` / `Rate exceeded` / HTTP 429 during the burst must be RETRIED (deploy still succeeds) not fatal, the chained subset proves strict DAG ordering, and the destroy burst absorbs ~100 deletes with 0 orphans.',
 
   // ---- Intrinsic-function resolution patterns ----
   'intrinsics-torture':

docs/scenario-coverage.md

@@ -146,6 +146,23 @@ export const RETRYABLE_ERROR_MESSAGE_PATTERNS: readonly string[] = [
   // teardown lock and avoid retrying unrelated create-already-exists
   // conflicts that share the same exception name.
   'because it is in use',
+  // Throttling backstop: many AWS services surface a rate-limit rejection
+  // with the canonical "Rate exceeded" message (SSM PutParameter, STS,
+  // CloudWatch, API Gateway, etc.) and an HTTP 400 (NOT 429), so the status-
+  // code check below misses them. When cdkd dispatches a wide DAG at a high
+  // `--concurrency`, the create burst can exceed a per-service rate limit and
+  // AWS rejects the losers with `Rate exceeded. Ensure you have the high-
+  // throughput setting enabled ...`. The AWS SDK's own retry layer (3 fast
+  // attempts) is not enough to drain a large burst; cdkd's outer withRetry —
+  // with its longer 1s/2s/4s/8s backoff — spreads the remaining creates out
+  // until the rate window clears. "Rate exceeded" only ever means throttling,
+  // so a permanent failure cannot false-positive into the retry loop. This is
+  // a message-level backstop for the name-based throttle detection in
+  // isThrottlingError() (the ProvisioningError wrap preserves the SDK error's
+  // message string even when the original `.name` is one cause-link deeper).
+  // Surfaced by tests/integration/throttle-wide-dag (80 SSM parameters at
+  // --concurrency 40).
+  'Rate exceeded',
 ];
 
 /**
@@ -154,13 +171,58 @@ export const RETRYABLE_ERROR_MESSAGE_PATTERNS: readonly string[] = [
  */
 export const RETRYABLE_HTTP_STATUS_CODES: ReadonlySet<number> = new Set([429, 503]);
 
+/**
+ * AWS SDK v3 canonical throttling error names. Mirrors
+ * `@aws-sdk/service-error-classification`'s `THROTTLING_ERROR_CODES` — any
+ * error (or wrapped cause) whose `name` is one of these is a transient rate-
+ * limit rejection worth retrying with backoff. Detecting by NAME is more
+ * robust than by HTTP status because most AWS throttles surface as HTTP 400
+ * (not 429) with the throttling signal carried only in the error code / name
+ * (e.g. SSM `ThrottlingException` for the `Rate exceeded` message).
+ */
+export const THROTTLING_ERROR_NAMES: ReadonlySet<string> = new Set([
+  'BandwidthLimitExceeded',
+  'EC2ThrottledException',
+  'LimitExceededException',
+  'PriorRequestNotComplete',
+  'ProvisionedThroughputExceededException',
+  'RequestLimitExceeded',
+  'RequestThrottled',
+  'RequestThrottledException',
+  'SlowDown',
+  'ThrottledException',
+  'Throttling',
+  'ThrottlingException',
+  'TooManyRequestsException',
+  'TransactionInProgressException',
+]);
+
+/**
+ * Walk the error + its `.cause` chain (bounded) looking for an AWS SDK v3
+ * throttling error `name`. cdkd wraps the original AWS error in a
+ * `ProvisioningError`, so the throttling signal is typically one cause-link
+ * deep; the bounded walk also tolerates SDK errors that nest a `$response`/
+ * cause without exploding on a cyclic chain.
+ */
+function isThrottlingError(error: unknown): boolean {
+  let current: unknown = error;
+  for (let depth = 0; depth < 5 && current != null; depth++) {
+    const name = (current as { name?: unknown }).name;
+    if (typeof name === 'string' && THROTTLING_ERROR_NAMES.has(name)) return true;
+    current = (current as { cause?: unknown }).cause;
+  }
+  return false;
+}
+
 /**
  * Determine whether an AWS error should be retried.
  *
  * Checks (in order):
  *   1. HTTP status code on the error itself (`$metadata.httpStatusCode`)
  *   2. HTTP status code on a wrapped cause (`cause.$metadata.httpStatusCode`)
- *   3. Substring match against {@link RETRYABLE_ERROR_MESSAGE_PATTERNS}
+ *   3. Throttling error `name` on the error or any wrapped cause (most AWS
+ *      throttles are HTTP 400, not 429 — see {@link THROTTLING_ERROR_NAMES})
+ *   4. Substring match against {@link RETRYABLE_ERROR_MESSAGE_PATTERNS}
  */
 export function isRetryableTransientError(error: unknown, message: string): boolean {
   const metadata = (error as { $metadata?: { httpStatusCode?: number } }).$metadata;
@@ -171,5 +233,7 @@ export function isRetryableTransientError(error: unknown, message: string): bool
   const causeStatus = cause?.$metadata?.httpStatusCode;
   if (causeStatus !== undefined && RETRYABLE_HTTP_STATUS_CODES.has(causeStatus)) return true;
 
+  if (isThrottlingError(error)) return true;
+
   return RETRYABLE_ERROR_MESSAGE_PATTERNS.some((p) => message.includes(p));
 }

scripts/build-scenario-coverage-matrix.ts

@@ -0,0 +1,5 @@
+{
+  "scenarios": [
+    "wide-dag-throttle-retry"
+  ]
+}

src/deployment/retryable-errors.ts

@@ -0,0 +1,79 @@
+# throttle-wide-dag
+
+Failure-seeking integration test that stresses cdkd's **concurrency limiter**
+(`src/deployment/dag-executor.ts`), its **throttle / retry classifier**
+(`src/deployment/retryable-errors.ts` + `src/deployment/retry.ts`), and the
+**event-driven DAG executor** at SCALE.
+
+## Background
+
+cdkd dispatches every ready resource as soon as its dependencies complete,
+bounded only by `--concurrency` across the whole stack. A wide stack — many
+independent resources created in one burst — is the case most likely to trip
+an AWS-side rate limit (`TooManyRequestsException` / `Rate exceeded`, surfaced
+as HTTP 429). The retry helper classifies 429 (and 503) as transient and backs
+off (`1s -> 2s -> 4s -> 8s`, capped). This fixture exists to verify that a
+throttle during a large burst is RETRIED (the deploy still succeeds) rather
+than treated as fatal — and to surface DAG-scheduling or partial-failure bugs
+that only appear at scale.
+
+The resources are deliberately CHEAP, fast, and quota-friendly so the fixture
+can run at ~100 resources without cost or quota blow-ups. No VPC — every
+resource is a control-plane-only create.
+
+## Fixture
+
+`CdkdThrottleWideDagExample` (`lib/throttle-wide-dag-stack.ts`) — ~100 resources:
+
+| Type | Count | Role |
+| --- | --- | --- |
+| `AWS::SSM::Parameter` | 80 | Fast, high create rate -> most likely to throttle |
+| `AWS::IAM::Role` | 10 | Broadens the throttle surface to a second service |
+| `AWS::SNS::Topic` | 10 | Third service in the burst |
+
+### DAG shape
+
+- **Independent set (throttle pressure):** 70 of the 80 parameters + all 10
+  roles + all 10 topics have NO dependencies, so they form one large ready-set
+  the executor sheds across the `--concurrency` budget at once.
+- **Chained subset (DAG depth):** 10 parameters form a serial chain
+  `ChainParam0 -> ChainParam1 -> ... -> ChainParam9`. Each `ChainParam(K)` (K>=1)
+  embeds the previous parameter's name via `Fn::Sub`, creating an implicit Ref
+  edge. cdkd must serialize the chain in strict order while everything else
+  runs in parallel — a scheduling bug (dispatching a child before its parent
+  completes) would surface as a deploy ordering failure.
+
+## Automated run (`verify.sh`)
+
+Env: `AWS_REGION` (default `us-east-1`), `STATE_BUCKET` (required),
+`CDKD_CONCURRENCY` (default `40` — intentionally high vs. the `10` default to
+maximise throttle pressure).
+
+1. Install fixture deps (`pnpm install --ignore-workspace`).
+2. **Deploy** all ~100 resources with `--concurrency 40 --verbose`.
+   - Prints any `⏳ Retrying ... / TooManyRequests / Rate exceeded / 429`
+     activity observed (documents that the retry path was exercised when AWS
+     throttled — throttling is probabilistic, so a clean run is also valid).
+   - Asserts the deploy exited **0**. A non-zero exit on a throttle means the
+     classifier did NOT retry it -> a **real finding**; the throttle error is
+     printed.
+3. Asserts all resources reached AWS:
+   - cdkd state records exactly 100 resources.
+   - 80 SSM parameters under `/CdkdThrottleWideDagExample/` (paginated count).
+   - The deepest chain parameter (`/.../chain/9`) holds a `child-of-...`
+     `Fn::Sub` value -> the executor serialized the chain in DAG order.
+   - 10 IAM roles + 10 SNS topics exist.
+4. **Destroy** all ~100 resources with the same high `--concurrency` and assert
+   the delete burst also exits 0 (the destroy path must absorb ~100 deletes
+   without throttle-failing).
+5. Asserts **0 orphans**: 0 SSM parameters / 0 IAM roles / 0 SNS topics remain,
+   and the cdkd state file is gone.
+
+`verify.sh` is BSD-portable (no `grep -P`, no `date -d`), captures the real
+deploy/destroy exit codes, and ends with an explicit PASS line.
+
+Run via the skill: `/run-integ throttle-wide-dag`.
+
+## Scenario tag
+
+`wide-dag-throttle-retry` (see `.scenarios.json`).