fix(emr): rename ConfigurationProperties / StepProperties to the SDK Properties member

CFn spells the EMR application-configuration bag
`Configuration.ConfigurationProperties` and the step bag
`HadoopJarStepConfig.StepProperties`, while both SDK members are named
`Properties`. The EMR providers cast the CFn blobs straight to the SDK
types, and the AWS SDK v3 serializer drops unknown members - so every
EMR application configuration (spark-defaults / hive-site / yarn-site
...) silently vanished and the cluster came up unconfigured while cdkd
reported the deploy as successful. Step properties were dropped the
same way.

The value shapes already match (Record<string,string> for the former,
KeyValue[] for the latter - verified against the live CFn registry
schema), so both directions are pure key renames.

- new shared helper src/provisioning/emr-configuration.ts:
  toSdkConfigurations (recursive - Configurations nests into itself),
  toSdkStepConfigs, toSdkInstanceTypeConfigs
- wired at every forwarding site: EMRClusterProvider top-level
  Configurations / Steps, per-instance-group Configurations, per-fleet
  InstanceTypeConfigs; EMRInstanceGroupConfigProvider create;
  EMRInstanceFleetConfigProvider create + ModifyInstanceFleet update
- no inverse needed: Configurations / Steps / InstanceTypeConfigs are
  all in EMRClusterProvider.getDriftUnknownPaths and neither instance
  provider implements readCurrentState
- unit tests for the helper plus per-provider wiring tests, the four
  provider-level ones verified to fail without the fix
- emr-cluster fixture gains top-level + NESTED Configurations, a
  per-master-group Configurations block and a step with StepProperties;
  emr-instance-configs gains Configurations on the standalone TASK
  group. Both verify.sh files read the values back from AWS through a
  new SDK-based list_instance_groups_json helper (aws emr
  list-instance-groups is CLI-customized and unusable non-interactively)

Closes #1383

Author: go-to-k

.claude/rules/code-layout.md

@@ -77,6 +77,7 @@ paths:
 - **src/provisioning/register-providers.ts** - Shared provider registration (called from deploy.ts and destroy.ts)
 - **src/provisioning/data-delete-intent.ts** - Shared destroy data-guard intent helpers (issue #1340): `hasCdkAutoDeleteTag(properties, tagKey)` / `isTruthyCfnBoolean(value)` plus the CDK tag-key constants `S3_AUTO_DELETE_OBJECTS_TAG` (`aws-cdk:auto-delete-objects`, stamped by `autoDeleteObjects: true`) and `ECR_AUTO_DELETE_IMAGES_TAG` (`aws-cdk:auto-delete-images`). Consumed by `S3BucketProvider.delete` (auto-empty of a non-empty bucket only with the tag / `DeleteContext.forceDataDelete`), `S3DirectoryBucketProvider.delete` (issue #1344 — same gate; no CDK opt-in sugar exists for directory buckets, so plain destroy of a non-empty one fails with a manual-empty remediation), and `ECRProvider.delete` (`force: true` only with `EmptyOnDelete: true`, the tag, or `forceDataDelete`) — without an opt-in the AWS not-empty error surfaces like CloudFormation DELETE_FAILED. `DeleteContext.forceDataDelete` (src/provisioning/region-check.ts) is set ONLY by the deploy engine's replacement/recreate delete sites under `--force-stateful-recreation`. See the "Destroy data guards" section in docs/cli-reference.md and the DeleteContext contract note in .claude/rules/providers.md.
 - **src/provisioning/final-snapshot.ts** - `DeletionPolicy` / `UpdateReplacePolicy: Snapshot` support (issues #1352 / #1353 / #1354): `ATOMIC_FINAL_SNAPSHOT_TYPES` (RDS DBInstance / DBCluster, Neptune / DocDB clusters, ElastiCache CacheCluster — the delete call sites generate `buildFinalSnapshotIdentifier(physicalId, resourceType)` and thread it via `DeleteContext.finalSnapshotIdentifier`; each provider flips its delete from `SkipFinalSnapshot: true` to the API's atomic final-snapshot form; ONLY on the SDK route — a cc-api-routed atomic type is refused and `CloudControlProvider.delete` fail-closes on the field), `PRE_DELETE_SNAPSHOT_TYPES` + `createPreDeleteFinalSnapshot` dispatcher (all CC-routed: `AWS::EC2::Volume` via EC2 `CreateSnapshot` tagged `cdkd:final-snapshot-of`; `AWS::Redshift::Cluster` via `CreateClusterSnapshot`; `AWS::ElastiCache::ReplicationGroup` via ElastiCache `CreateSnapshot` — each waited to ready, idempotent reuse via the tag / the `finalSnapshotNamePrefix` name prefix across delete re-runs), `unsupportedFinalSnapshotError` / `ccRoutedFinalSnapshotError` refusals, and (issue #1366) `finalSnapshotMechanism(type, route)` / `refusesFinalSnapshot(type, route)` — the mechanism matrix as a PURE function, so the executor that ACTS on it and the `cdkd rollback` plan preview that DESCRIBES it read one source (issue #1368 extends that to the preview's STATE effect: a refused Snapshot delete no longer unwinds the record, since the next-older segment is classified against it). The two type sets are DISJOINT by construction — `finalSnapshotMechanism` tests the atomic set first, so a type in both would silently take the atomic arm and never reach the pre-delete snapshot; pinned in `final-snapshot.test.ts` alongside the union-equals-the-CFn-documented-list fence (re-homed there from the deleted `supportsFinalSnapshot` predicate, #1368). Consumed by the deploy engine (`prepareFinalSnapshotForDelete` — the shared gate for the DELETE branch AND the four replacement / recreate delete sites), `destroy-runner.ts`, and `rollback-executor.ts` — the latter twice: `rollbackFinalSnapshotId` for the delete-of-the-NEW-resource under `UpdateReplacePolicy` (honors only the atomic SDK-routed shape, plain-deletes otherwise — scope decision on #1354), and `prepareCreateRollbackFinalSnapshot` for a rolled-back CREATE under `DeletionPolicy` (the FULL matrix, refusing what it cannot snapshot — issue #1358). The engine's clients come from `DeployEngineOptions.finalSnapshotClients` (stack-region-pinned `AwsClients`, structurally a `PreDeleteSnapshotClients`), threaded on to `RollbackExecutorContext.finalSnapshotClients`; `--skip-final-snapshot` (deploy / destroy / state destroy / rollback, `skipFinalSnapshotOption` in `src/cli/options.ts` — deliberately NOT in the shared `destroyOptions` array `cdkd orphan` consumes) is the explicit data-loss opt-out.
+- **src/provisioning/emr-configuration.ts** - Shared CFn -> SDK shape converters for the `AWS::EMR::*` nested config blobs whose CFn key spelling diverges from `@aws-sdk/client-emr` (issue #1383): `toSdkConfigurations` (renames `Configuration.ConfigurationProperties` -> the SDK's `Properties` at EVERY `Configurations` nesting level), `toSdkStepConfigs` (`HadoopJarStepConfig.StepProperties` -> `Properties`), and `toSdkInstanceTypeConfigs` (per-instance-type nested `Configurations`). Both are pure key renames — the VALUE shapes already match (`Record<string,string>` / `KeyValue[]`, verified against the live CFn registry schema) — but the AWS SDK v3 serializer drops unknown members, so before the conversion every EMR application configuration (spark-defaults / hive-site / yarn-site ...) silently vanished while cdkd reported success. Consumed by `EMRClusterProvider` (top-level `Configurations` / `Steps`, per-group `Configurations`, per-fleet `InstanceTypeConfigs`), `EMRInstanceGroupConfigProvider` (create), and `EMRInstanceFleetConfigProvider` (create + the `ModifyInstanceFleet` update). No inverse is needed: `Configurations` / `Steps` / `InstanceTypeConfigs` are all declared in `EMRClusterProvider.getDriftUnknownPaths` and neither instance provider implements `readCurrentState`. Non-object / non-array inputs (an unresolved intrinsic) pass through untouched so AWS surfaces the real validation error. The `AWS::EMR::*` types are NOT yet in `NESTED_KEY_TARGETS` (`scripts/gen-nested-key-coverage.ts`) — critic target expansion is tracked in issue #1393.
 - **src/provisioning/ec2-termination-protection.ts** - Shared `--remove-protection` helper for `AWS::EC2::Instance`: `disableInstanceApiTermination()` (flip `DisableApiTermination` off, idempotent, errors swallowed at debug), `isTerminationProtectionPropagationError()` (matches the "may not be terminated. Modify its disableApiTermination" 400 from both `TerminateInstances` and the Cloud Control `DeleteResource` wrapper), and `TERMINATION_PROTECTION_MAX_ATTEMPTS`. Used by `EC2Provider.deleteInstance` (SDK path) and `CloudControlProvider.delete` (CC-API path — an instance routes through Cloud Control whenever its template trips the #614 silent-drop routing) so `--remove-protection` works regardless of which delete path the instance takes; the modify WRITE lags the delete READ, so both callers flip-off + retry the delete to close the propagation window. ALSO used by `ASGProvider.delete` (issue #796): an `AWS::AutoScaling::AutoScalingGroup` whose launch template sets `DisableApiTermination: true` launches instances that survive the group's `DeleteAutoScalingGroup(ForceDelete: true)` (ASG-level DeletionProtection + ForceDelete governs only the group + scale-in protection, not EC2-level termination protection), so under `--remove-protection` the provider enumerates the group's current instances and flips each one's `DisableApiTermination` off before the force delete — the ASG's own async terminate loop then absorbs the modify-WRITE propagation lag, so no per-instance delete retry is needed there. An ASG can ALSO route via Cloud Control when its template sets a silent-drop property such as `AvailabilityZoneIds` (#614 routing) — Cloud Control's `DeleteResource` cannot `ForceDelete` a protected ASG or clear its protection, so `CloudControlProvider.delete` detects `removeProtection === true && resourceType === 'AWS::AutoScaling::AutoScalingGroup'` and delegates to `new ASGProvider().delete(...)` (the single source of truth for protected-ASG deletion), keeping the SDK and CC routing paths behaviourally identical (issue #798; CDK's L2 emits `availabilityZones` names not `AvailabilityZoneIds`, so this CC path only fires for hand-written L1 / imported templates).
 - **src/provisioning/unsupported-types.ts** + **unsupported-types.generated.ts** - Pre-flight unsupported-type rejection. The `.generated.ts` ships the provider-coverage Tier 3 set (`ProvisioningType: NON_PROVISIONABLE`) into the runtime, codegen'd from `docs/_generated/provider-coverage.json` by `scripts/gen-unsupported-types.ts` (`vp run gen:unsupported-types`; CI fails on drift). The hand-written `.ts` adds `isNonProvisionable()` + `unsupportedTypeIssueUrl()`; both are consulted by `CloudControlProvider.isSupportedResourceType` (rejects Tier 3) and `ProviderRegistry.validateResourceTypes` (per-type error + issue link). The `--allow-unsupported-types` escape hatch routes named types through Cloud Control via `ProviderRegistry.allowUnsupportedTypes()`.
 - **src/provisioning/property-coverage.ts** + **property-coverage.generated.ts** - Pre-flight property-level rejection (parallel to unsupported-types but at top-level CFn property granularity). The `.generated.ts` ships per-Tier-1-type `{ handled, silentDrop }` records, codegen'd from `tests/fixtures/cfn-schemas/*.json` + each SDK provider's `handledProperties` / `unhandledByDesign` declarations by `scripts/gen-property-coverage.ts` (`vp run gen:property-coverage`; CI fails on drift; the codegen parses provider sources via the TypeScript Compiler API so no `dist/` bootstrap is needed). The hand-written `.ts` adds `getPropertyCoverage()` + `findSilentDropProperties()` + `unsupportedPropertyIssueUrl()`; all are consulted by `ProviderRegistry.validateResourceProperties` (per-resource per-property error + 1-click GitHub issue link + dedup'd re-run command). The `--allow-unsupported-properties` escape hatch (deploy only) routes named `<Type>:<Prop>` entries past the reject via `ProviderRegistry.allowUnsupportedProperties()`. Tier 2 (Cloud Control) types are intentionally NOT in the generated map — CC forwards the full property map to AWS, so no write-side silent drop is possible.

src/provisioning/emr-configuration.ts

@@ -0,0 +1,91 @@
+import type { Configuration, InstanceTypeConfig, StepConfig } from '@aws-sdk/client-emr';
+
+/**
+ * CFn -> SDK shape converters for the two `AWS::EMR::*` nested blobs whose CFn
+ * key spelling diverges from the `@aws-sdk/client-emr` model (issue #1383).
+ *
+ * | CFn key                                          | SDK member                     |
+ * | ------------------------------------------------ | ------------------------------ |
+ * | `Configuration.ConfigurationProperties`           | `Configuration.Properties`     |
+ * | `HadoopJarStepConfig.StepProperties`              | `HadoopJarStepConfig.Properties` |
+ *
+ * Both are plain renames — the VALUE shapes already match (`Record<string,string>`
+ * for the former, `KeyValue[]` for the latter, verified against the live CFn
+ * registry schema on 2026-08-09). The AWS SDK v3 serializer drops unknown
+ * members, so before this conversion every EMR application configuration
+ * (spark-defaults / hive-site / yarn-site ...) silently vanished while cdkd
+ * reported the deploy as successful.
+ *
+ * `Configurations` nests into itself, so the rename is applied at EVERY level.
+ * Non-object / non-array inputs (an unresolved intrinsic, a malformed template)
+ * pass through untouched so this layer never turns a bad template into a
+ * confusing crash — AWS surfaces the real validation error instead.
+ */
+
+function isRecord(value: unknown): value is Record<string, unknown> {
+  return typeof value === 'object' && value !== null && !Array.isArray(value);
+}
+
+function toSdkConfiguration(raw: unknown): Configuration {
+  if (!isRecord(raw)) return raw as Configuration;
+
+  const { ConfigurationProperties, Configurations, ...rest } = raw;
+  return {
+    ...rest,
+    ...(ConfigurationProperties !== undefined ? { Properties: ConfigurationProperties } : {}),
+    ...(Configurations !== undefined
+      ? { Configurations: toSdkConfigurations(Configurations) }
+      : {}),
+  } as Configuration;
+}
+
+/**
+ * CFn `Configurations` list -> SDK `Configuration[]`, renaming
+ * `ConfigurationProperties` -> `Properties` at every nesting level.
+ */
+export function toSdkConfigurations(value: unknown): Configuration[] | undefined {
+  if (value === undefined) return undefined;
+  if (!Array.isArray(value)) return value as Configuration[];
+  return value.map(toSdkConfiguration);
+}
+
+/**
+ * CFn `Steps` list -> SDK `StepConfig[]`, renaming
+ * `HadoopJarStep.StepProperties` -> `HadoopJarStep.Properties`.
+ */
+export function toSdkStepConfigs(value: unknown): StepConfig[] | undefined {
+  if (value === undefined) return undefined;
+  if (!Array.isArray(value)) return value as StepConfig[];
+
+  return value.map((step) => {
+    if (!isRecord(step)) return step as StepConfig;
+    const hadoopJarStep = step['HadoopJarStep'];
+    if (!isRecord(hadoopJarStep) || hadoopJarStep['StepProperties'] === undefined) {
+      return step as unknown as StepConfig;
+    }
+    const { StepProperties, ...restHadoopJarStep } = hadoopJarStep;
+    return {
+      ...step,
+      HadoopJarStep: { ...restHadoopJarStep, Properties: StepProperties },
+    } as unknown as StepConfig;
+  });
+}
+
+/**
+ * CFn `InstanceTypeConfigs` list -> SDK `InstanceTypeConfig[]`, converting each
+ * element's nested per-instance-type `Configurations`.
+ */
+export function toSdkInstanceTypeConfigs(value: unknown): InstanceTypeConfig[] | undefined {
+  if (value === undefined) return undefined;
+  if (!Array.isArray(value)) return value as InstanceTypeConfig[];
+
+  return value.map((instanceTypeConfig) => {
+    if (!isRecord(instanceTypeConfig) || instanceTypeConfig['Configurations'] === undefined) {
+      return instanceTypeConfig as InstanceTypeConfig;
+    }
+    return {
+      ...instanceTypeConfig,
+      Configurations: toSdkConfigurations(instanceTypeConfig['Configurations']),
+    } as InstanceTypeConfig;
+  });
+}

src/provisioning/providers/emr-cluster-provider.ts

@@ -32,6 +32,11 @@ import { getLogger } from '../../utils/logger.js';
 import { ProvisioningError, ResourceUpdateNotSupportedError } from '../../utils/error-handler.js';
 import { assertRegionMatch, type DeleteContext } from '../region-check.js';
 import { normalizeAwsTagsToCfn, resolveExplicitPhysicalId } from '../import-helpers.js';
+import {
+  toSdkConfigurations,
+  toSdkInstanceTypeConfigs,
+  toSdkStepConfigs,
+} from '../emr-configuration.js';
 import type {
   ResourceProvider,
   ResourceCreateResult,
@@ -258,13 +263,11 @@ export class EMRClusterProvider implements ResourceProvider {
         Applications: properties['Applications'] as
           | import('@aws-sdk/client-emr').Application[]
           | undefined,
-        Configurations: properties['Configurations'] as
-          | import('@aws-sdk/client-emr').Configuration[]
-          | undefined,
+        Configurations: toSdkConfigurations(properties['Configurations']),
         BootstrapActions: properties['BootstrapActions'] as
           | import('@aws-sdk/client-emr').BootstrapActionConfig[]
           | undefined,
-        Steps: properties['Steps'] as import('@aws-sdk/client-emr').StepConfig[] | undefined,
+        Steps: toSdkStepConfigs(properties['Steps']),
         KerberosAttributes: properties['KerberosAttributes'] as
           | import('@aws-sdk/client-emr').KerberosAttributes
           | undefined,
@@ -413,9 +416,7 @@ export class EMRClusterProvider implements ResourceProvider {
       Name: raw['Name'] as string | undefined,
       Market: raw['Market'] as import('@aws-sdk/client-emr').MarketType | undefined,
       BidPrice: raw['BidPrice'] as string | undefined,
-      Configurations: raw['Configurations'] as
-        | import('@aws-sdk/client-emr').Configuration[]
-        | undefined,
+      Configurations: toSdkConfigurations(raw['Configurations']),
       EbsConfiguration: raw['EbsConfiguration'] as
         | import('@aws-sdk/client-emr').EbsConfiguration
         | undefined,
@@ -435,9 +436,7 @@ export class EMRClusterProvider implements ResourceProvider {
       Name: raw['Name'] as string | undefined,
       TargetOnDemandCapacity: toNumber(raw['TargetOnDemandCapacity']),
       TargetSpotCapacity: toNumber(raw['TargetSpotCapacity']),
-      InstanceTypeConfigs: raw['InstanceTypeConfigs'] as
-        | import('@aws-sdk/client-emr').InstanceTypeConfig[]
-        | undefined,
+      InstanceTypeConfigs: toSdkInstanceTypeConfigs(raw['InstanceTypeConfigs']),
       LaunchSpecifications: raw['LaunchSpecifications'] as
         | import('@aws-sdk/client-emr').InstanceFleetProvisioningSpecifications
         | undefined,

src/provisioning/providers/emr-instance-fleet-config-provider.ts

@@ -13,6 +13,7 @@ import {
 import { getLogger } from '../../utils/logger.js';
 import { ProvisioningError, ResourceUpdateNotSupportedError } from '../../utils/error-handler.js';
 import { assertRegionMatch, type DeleteContext } from '../region-check.js';
+import { toSdkInstanceTypeConfigs } from '../emr-configuration.js';
 import type {
   ResourceProvider,
   ResourceCreateResult,
@@ -218,9 +219,7 @@ export class EMRInstanceFleetConfigProvider implements ResourceProvider {
       Name: properties['Name'] as string | undefined,
       TargetOnDemandCapacity: toNumber(properties['TargetOnDemandCapacity']),
       TargetSpotCapacity: toNumber(properties['TargetSpotCapacity']),
-      InstanceTypeConfigs: properties['InstanceTypeConfigs'] as
-        | import('@aws-sdk/client-emr').InstanceTypeConfig[]
-        | undefined,
+      InstanceTypeConfigs: toSdkInstanceTypeConfigs(properties['InstanceTypeConfigs']),
       LaunchSpecifications: properties['LaunchSpecifications'] as
         | import('@aws-sdk/client-emr').InstanceFleetProvisioningSpecifications
         | undefined,
@@ -288,9 +287,7 @@ export class EMRInstanceFleetConfigProvider implements ResourceProvider {
         ResizeSpecifications: properties['ResizeSpecifications'] as
           | import('@aws-sdk/client-emr').InstanceFleetResizingSpecifications
           | undefined,
-        InstanceTypeConfigs: properties['InstanceTypeConfigs'] as
-          | import('@aws-sdk/client-emr').InstanceTypeConfig[]
-          | undefined,
+        InstanceTypeConfigs: toSdkInstanceTypeConfigs(properties['InstanceTypeConfigs']),
       };
       await this.getClient().send(
         new ModifyInstanceFleetCommand({ ClusterId: clusterId, InstanceFleet: modify })

src/provisioning/providers/emr-instance-group-config-provider.ts

@@ -14,6 +14,7 @@ import {
 import { getLogger } from '../../utils/logger.js';
 import { ProvisioningError, ResourceUpdateNotSupportedError } from '../../utils/error-handler.js';
 import { assertRegionMatch, type DeleteContext } from '../region-check.js';
+import { toSdkConfigurations } from '../emr-configuration.js';
 import type {
   ResourceProvider,
   ResourceCreateResult,
@@ -220,9 +221,7 @@ export class EMRInstanceGroupConfigProvider implements ResourceProvider {
       Name: properties['Name'] as string | undefined,
       Market: properties['Market'] as import('@aws-sdk/client-emr').MarketType | undefined,
       BidPrice: properties['BidPrice'] as string | undefined,
-      Configurations: properties['Configurations'] as
-        | import('@aws-sdk/client-emr').Configuration[]
-        | undefined,
+      Configurations: toSdkConfigurations(properties['Configurations']),
       EbsConfiguration: properties['EbsConfiguration'] as
         | import('@aws-sdk/client-emr').EbsConfiguration
         | undefined,