Skip to content

troubleshooting.md

Latest commit

 

History

History
1197 lines (857 loc) · 35.9 KB

troubleshooting.md

File metadata and controls

1197 lines (857 loc) · 35.9 KB
Raw

cdkd Troubleshooting Guide

This document summarizes common issues when using cdkd and their solutions.

Table of Contents

  1. Lock Issues
  2. State Management Issues
  3. Deployment Errors
  4. Asset Publishing Issues
  5. Intrinsic Function Issues
  6. Permission Errors
  7. Performance Issues
  8. Orphaned Resources

Lock Issues

Issue: "Failed to acquire lock" Error

Symptoms

Error: Failed to acquire lock for stack 'MyStack' after 3 attempts.
Locked by: user@hostname:12345, operation: deploy

Causes

  • Another process is deploying the same stack
  • Previous process crashed and lock remains

Note: A first Ctrl-C during cdkd destroy / cdkd state destroy no longer strands the lock — the graceful-SIGINT handler (issue #816) finishes any in-flight delete, flushes the incremental state, and releases the lock before exiting non-zero. A re-run resumes immediately without waiting out the lock TTL. cdkd deploy behaves the same way on a first Ctrl-C: in-flight operations finish, partial state is saved, a rollback journal is recorded, and the lock is released before the non-zero exit.

A second Ctrl-C force-quits immediately (exit 130) without waiting for the in-flight delete. Because the force-quit path cannot run the normal lock-release cleanup, it fires a best-effort (un-awaited) lock release AND prints the exact recovery command to stderr:

Force-quit: stack lock may not be released. If the next run reports a lock, run: cdkd force-unlock MyStack

The best-effort release usually lands before the process dies, so most force-quits leave no lock; if a subsequent run reports a lock, run the printed cdkd force-unlock <stackName> (or the steps below) to clear it. A leftover lock therefore means an ungraceful kill (SIGKILL, a force-quit whose best-effort release did not complete, or a crash).

Solutions

1. Check if another process is running

# Check lock information
aws s3api get-object \
  --bucket ${STATE_BUCKET} \
  --key cdkd/MyStack/us-east-1/lock.json \
  /dev/stdout

# Example output:
# {
#   "owner": "goto@macbook:12345",
#   "timestamp": 1710835200000,
#   "operation": "deploy"
# }

2. Force release if lock is old

# Delete lock file
aws s3 rm s3://${STATE_BUCKET}/cdkd/MyStack/us-east-1/lock.json

# Or use cdkd force-unlock command
cdkd force-unlock MyStack

3. Increase retry count

// Adjust in deploy-engine.ts
await lockManager.acquireLockWithRetry(
  stackName,
  owner,
  operation,
  5,      // maxRetries (default: 3)
  10000   // retryDelay (default: 5000ms)
);

Issue: Stale lock after a cancelled CI job

Symptoms

A CI job running cdkd deploy was cancelled (manually, or automatically by a newer run), and the next run fails with Failed to acquire lock even though no deploy is in progress.

A very common GitHub Actions setup for per-PR environments hits this:

concurrency:
  group: pr-env-${{ github.event.pull_request.number }}
  cancel-in-progress: true

Consecutive pushes to the same PR target the same stack, so the cancelled run's stale lock blocks the run that replaced it.

Causes

Cancellation is not a clean Ctrl-C. GitHub Actions escalates SIGINT → SIGTERM (~7.5 s later) → SIGKILL (~2.5 s after that); other CI systems (GitLab CI, docker stop, Kubernetes) typically send SIGTERM directly. cdkd's deploy / destroy / state destroy / rollback commands handle both SIGINT and SIGTERM gracefully (issue #1342): the first signal finishes in-flight operations, saves state, and releases the lock; a second signal force-quits with a best-effort lock release. But SIGKILL cannot be handled by any process — under GitHub Actions the whole escalation completes in ~10 seconds, so a job whose in-flight AWS operation takes longer than that is still killed before the lock-release cleanup finishes, stranding the lock. SIGTERM-only environments with a longer grace period (Kubernetes defaults to 30 s; docker stop to 10 s) give the graceful path a better chance to complete.

Solutions

1. Wait out the TTL — a stale lock is reclaimed automatically after the lock TTL (30 minutes by default). The next run after that succeeds without intervention.

2. Clear it immediately with:

cdkd force-unlock MyStack

3. Recommended CI pattern — when your workflow serializes runs per stack (as the concurrency group above does), it is safe to clear any stale lock at the start of the job, because no other run of the same group can be holding it legitimately:

- run: npm i -g @go-to-k/cdkd
- run: cdkd force-unlock MyStack || true  # only safe when runs are serialized per stack
- run: cdkd deploy MyStack --yes

Do not add an unconditional force-unlock to workflows where two jobs can legitimately operate on the same stack concurrently — it would break the lock that protects the running deploy.

Note on partially-applied deploys

A killed deploy is usually not a correctness problem beyond the lock: cdkd saves state incrementally after each completed resource, so a re-run resumes from the last saved state, and a rollback journal (when present) lets cdkd rollback revert the interrupted deploy instead. The remaining exposure is a resource whose create was in flight at the moment of the kill: it may have been created on AWS without reaching state, in which case the next run can surface an "already exists" conflict that needs manual reconciliation (delete the resource, or adopt it with cdkd import).

State Management Issues

Issue: "State was modified by another process"

Symptoms

StateError: State was modified by another process. Expected ETag: "abc123", but state has changed.

Causes

  • Two processes attempted to deploy simultaneously
  • Lock was acquired but conflict occurred when saving state

Solutions

1. Re-run deployment

Protected automatically by optimistic locking, so simply re-running should succeed:

node dist/cli.js deploy --app "..." --state-bucket ${STATE_BUCKET}

2. Adjust lock timeout

// lock-manager.ts
private readonly lockTTL = 30 * 60 * 1000;  // Extend to 30 minutes

Issue: State File is Corrupted

Symptoms

SyntaxError: Unexpected token in JSON at position 123

Causes

  • S3 upload was interrupted
  • JSON error during manual editing

Solutions

1. Restore from S3 versioning

# Get version list
aws s3api list-object-versions \
  --bucket ${STATE_BUCKET} \
  --prefix cdkd/MyStack/us-east-1/state.json

# Example output:
# {
#   "Versions": [
#     {
#       "Key": "cdkd/MyStack/us-east-1/state.json",
#       "VersionId": "abc123",
#       "LastModified": "2024-03-19T10:30:00.000Z"
#     },
#     {
#       "Key": "cdkd/MyStack/us-east-1/state.json",
#       "VersionId": "def456",
#       "LastModified": "2024-03-19T09:00:00.000Z"
#     }
#   ]
# }

# Restore old version
aws s3api get-object \
  --bucket ${STATE_BUCKET} \
  --key cdkd/MyStack/us-east-1/state.json \
  --version-id def456 \
  /tmp/state-backup.json

# Restore
aws s3 cp /tmp/state-backup.json \
  s3://${STATE_BUCKET}/cdkd/MyStack/us-east-1/state.json

2. Reset state and redeploy

# Delete state (resources remain)
aws s3 rm s3://${STATE_BUCKET}/cdkd/MyStack/us-east-1/state.json

# Redeploy (will error if existing resources exist)
node dist/cli.js deploy --app "..." --state-bucket ${STATE_BUCKET}

Issue: State and Resources Don't Match

Symptoms

  • Manually deleted/modified resources in AWS Console
  • cdkd tries to update non-existent resources

Causes

cdkd's state file and actual AWS resources have diverged.

Solutions

1. Reset state

# Delete state
aws s3 rm s3://${STATE_BUCKET}/cdkd/MyStack/us-east-1/state.json

# Redeploy (all resources treated as CREATE)
node dist/cli.js deploy --app "..." --state-bucket ${STATE_BUCKET}

2. Manually fix state (advanced)

# Download state
aws s3 cp s3://${STATE_BUCKET}/cdkd/MyStack/us-east-1/state.json /tmp/state.json

# Edit (remove entries for deleted resources)
vim /tmp/state.json

# Upload
aws s3 cp /tmp/state.json s3://${STATE_BUCKET}/cdkd/MyStack/us-east-1/state.json

3. Delete and recreate entire stack

# Delete all resources
node dist/cli.js destroy MyStack --force

# Redeploy
node dist/cli.js deploy --app "..." --state-bucket ${STATE_BUCKET}

Issue: "UnknownError" / cross-region state bucket

Symptoms

StateError: Failed to verify state bucket 'my-bucket': UnknownError
Caused by: UnknownError

…or similar AWS SDK v3 surface-level UnknownError on any S3 operation against the state bucket. The lock-path variant of the same root cause surfaced as a 301 PermanentRedirect instead:

LockError: Failed to acquire lock for stack 'MyStack' (ap-northeast-1):
The bucket you are attempting to access must be addressed using the
specified endpoint. Please send all future requests to this endpoint.

Cause

The state bucket lives in a region different from the one the AWS SDK client was constructed for. AWS SDK v3's region-redirect middleware does not handle the empty-body 301 HEAD response S3 returns in this case cleanly — the protocol parser falls through and produces a synthetic Unknown exception with the literal message UnknownError.

Solution

cdkd resolves this automatically: the state backend (PR #60, shipped v0.10.0), the lock manager (issue #803 — between PR #60 and that fix, state operations succeeded against a cross-region bucket but lock acquisition failed with the PermanentRedirect error above), and the custom-resource response path (issue #1195 — before that fix, deploying a stack with a Lambda-backed Custom Resource to a region different from the state bucket's region failed with the same 301 on the pre-signed ResponseURL) look up the bucket region via GetBucketLocation (a GET request, not a HEAD — avoids the SDK glitch) and rebuild their S3 clients to that region before any state, lock, or custom-resource response operation. If you still see either error, please file a bug with the full stack trace.

You no longer need to set the region to match the bucket region (the state-bucket client auto-detects it via GetBucketLocation). As of PR #63 (v0.12.0), --region is a first-class option only on cdkd bootstrap (where it picks the new bucket's region); on every other command it is deprecated (prefer AWS_REGION / your AWS profile) but still honored if passed. Use AWS_REGION or your AWS profile to control the SDK's default region for provisioning.

Deployment Errors

Issue: "Resource already exists" Error

Symptoms

ProvisioningError: Resource already exists: my-bucket-name
ResourceType: AWS::S3::Bucket

Causes

  • Resource with same name already exists
  • Previous deployment failed midway and state was not saved

Solutions

1. Change resource name

Make resource name unique in CDK code:

new s3.Bucket(this, 'MyBucket', {
  bucketName: `my-app-${cdk.Aws.ACCOUNT_ID}-${cdk.Aws.REGION}`,
});

2. Delete existing resource

# S3 bucket example
aws s3 rb s3://my-bucket-name --force

3. Import existing resource to state (planned for future implementation)

# cdkd import --stack MyStack --resource MyBucket=s3://my-bucket-name

Issue: "Provider not found" Error

Symptoms

Error: No provider registered for resource type: AWS::CustomService::Resource

Causes

  • Resource not supported by Cloud Control API
  • SDK Provider not implemented

Solutions

1. Check Cloud Control API support status

# Check AWS documentation
# https://docs.aws.amazon.com/cloudcontrolapi/latest/userguide/supported-resources.html

2. Implement SDK Provider

Refer to provider-development.md to implement a custom provider.

3. Temporarily use CloudFormation

For resources not supported by cdkd, use regular cdk deploy.

Issue: "Update requires replacement" Error

Symptoms

ProvisioningError: Cannot update property 'BucketName': Update requires replacement

Causes

  • Attempting to change property marked "Update requires: Replacement" in CloudFormation
  • Provider hasn't implemented replacement handling

Solutions

1. Implement replacement handling in provider

async update(...): Promise<ResourceUpdateResult> {
  const requiresReplacement = this.checkReplacementRequired(
    properties,
    previousProperties
  );

  if (requiresReplacement) {
    // Create new resource
    const createResult = await this.create(logicalId, resourceType, properties);

    // Delete old resource
    await this.delete(logicalId, physicalId, resourceType);

    return {
      physicalId: createResult.physicalId,
      wasReplaced: true,
      attributes: createResult.attributes,
    };
  }

  // Normal update process
  // ...
}

2. Manually delete and recreate resource

# Manually delete
aws s3 rb s3://old-bucket-name --force

# Redeploy
node dist/cli.js deploy --app "..." --state-bucket ${STATE_BUCKET}

Issue: "bucket is not empty" / "still contains images" on destroy

Symptoms:

Failed to delete S3 bucket MyBucket: bucket my-bucket is not empty. Matching
CloudFormation, cdkd does not delete a non-empty bucket unless it opted into
automatic emptying ...

Failed to delete ECR Repository MyRepo: repository my-repo still contains
images. Matching CloudFormation, cdkd does not force-delete an image-carrying
repository ...

Cause:

cdkd destroy matches CloudFormation's fail-and-protect behavior (issues #1340 / #1344): an S3 bucket (standard or S3 Express directory bucket) that still contains objects, or an ECR repository that still contains images, is NOT force-cleaned unless the resource opted in. Directory buckets have no template opt-in today (CDK has no autoDeleteObjects for them) — empty manually and destroy again.

Solutions:

  1. Opt in from the CDK app and redeploy, then destroy:

    • S3: autoDeleteObjects: true (with removalPolicy: DESTROY)
    • ECR: emptyOnDelete: true (or the legacy autoDeleteImages: true)

  2. Or empty the data manually and re-run the destroy:

    # Unversioned bucket
aws s3 rm s3://my-bucket --recursive
# Versioned bucket: also delete all object versions + delete markers
# ECR
aws ecr batch-delete-image --repository-name my-repo \
  --image-ids "$(aws ecr list-images --repository-name my-repo --query 'imageIds' --output json)"

See the "Destroy data guards" section in cli-reference.md for the full semantics.

Issue: "has DeletionPolicy: Snapshot, but ..." refusal on delete

Symptoms:

MyDb (AWS::RDS::DBInstance) has DeletionPolicy: Snapshot, but the resource is
managed via the Cloud Control API route (provisionedBy: cc-api), which has no
final-snapshot delete parameter ...

Cause:

CloudFormation creates a final snapshot before deleting a DeletionPolicy: Snapshot resource, and cdkd matches that (issues #1352 / #1353) for the FULL CFn-documented Snapshot-capable type list. The delete is refused only when cdkd cannot create the snapshot: the resource is an atomic-parameter type routed via Cloud Control (provisionedBy: cc-api, the #614 silent-drop routing — Cloud Control's DeleteResource has no final-snapshot parameter), or the template carries Snapshot on a type CloudFormation itself would refuse the attribute on.

Solutions:

  1. Snapshot the resource manually, then re-run with --skip-final-snapshot (the explicit data-loss opt-out), or
  2. Change the policy to Retain and delete the resource manually after snapshotting.

See the "DeletionPolicy: Snapshot" section in cli-reference.md for the per-type mechanics.

Asset Publishing Issues

Issue: "Asset publishing failed"

Symptoms

AssetPublisherError: Failed to publish asset: Access Denied

Causes

  • Asset storage doesn't exist for the target: in cdkd-assets mode the cdkd-assets-* bucket / cdkd-container-assets-* repo (someone deleted them after bootstrap), in legacy mode the CDK bootstrap bucket (cdk-hnb659fds-assets-*)
  • Insufficient IAM permissions

Solutions

1. Run cdkd bootstrap for the region

cdkd bootstrap creates the state bucket AND cdkd-owned asset storage for --region (asset bucket + container-asset ECR repo + opt-in marker), so no cdk bootstrap is needed:

cdkd bootstrap --region us-east-1

Normally this is automatic — the first cdkd deploy into a region auto-creates the storage (issue #1007), so this error usually means the auto-create was declined / opted out (--no-auto-asset-storage), failed (check the deploy output for the auto-create warning), or someone deleted the bucket/repo after opt-in. Deploys that stay in legacy mode publish to the CDK bootstrap bucket instead, which then must exist (npx cdk bootstrap aws://123456789012/us-east-1). See cli-reference.md.

Custom bootstrap: If you use a custom qualifier (e.g., --qualifier myqualifier), CDK synthesis will embed the custom bucket name in the asset manifest. cdkd reads destinations from the manifest (and, in cdkd-assets mode, redirects default-bootstrap-shaped destinations to cdkd-owned storage), so custom qualifiers are fully supported.

2. Skip asset publishing

# Skip during deployment
node dist/cli.js deploy --app "..." --skip-assets

3. Check IAM permissions

cdkd publishes assets with the caller's credentials directly (it never assumes CDK's cdk-hnb659fds-file-publishing-role-*). The caller needs S3 read/write on the asset bucket — cdkd-assets-* in cdkd-assets mode (adjust the ARN if the region was bootstrapped with a custom --asset-bucket name), cdk-hnb659fds-assets-* in legacy mode:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::cdkd-assets-*/*",
        "arn:aws:s3:::cdk-hnb659fds-assets-*/*"
      ]
    }
  ]
}

(Docker image assets additionally need ECR push permissions on the container-asset repo.)

Issue: Lambda Deployment Fails

Symptoms

ProvisioningError: Failed to create Lambda function: InvalidParameterValueException
The provided execution role does not have permissions to call CreateFunction.

Causes

  • Lambda asset (zip file) not published
  • IAM Role not created

Solutions

1. Verify asset publishing

# Check asset manifest
cat cdk.out/MyStack.assets.json

# Check asset bucket (cdkd-assets mode; use cdk-hnb659fds-assets-... in legacy mode)
aws s3 ls s3://cdkd-assets-${AWS_ACCOUNT_ID}-${AWS_REGION}/

2. Check IAM Role dependencies

Lambda functions depend on IAM Role, so verify proper ordering in DAG:

// Define Role first in CDK code
const role = new iam.Role(this, 'LambdaRole', { ... });

const func = new lambda.Function(this, 'MyFunction', {
  role: role,  // ← Dependency set
  // ...
});

Intrinsic Function Issues

Issue: "Unresolved intrinsic function" Error

Symptoms

Error: Cannot resolve intrinsic function: Fn::Select

Causes

CloudFormation intrinsic function not supported by cdkd is being used.

Support Status

Function Supported
Ref ✅
Fn::GetAtt ✅
Fn::Join ✅
Fn::Sub ✅
Fn::Select ✅
Fn::Split ✅
Fn::If ✅
Fn::Equals ✅
Fn::And ✅
Fn::Or ✅
Fn::Not ✅
Fn::ImportValue ✅
Fn::GetStackOutput ✅ (same-account; cross-account RoleArn not yet implemented)
Fn::FindInMap ✅
Fn::GetAZs ✅
Fn::Base64 ✅

Solutions

1. All intrinsic functions are now supported

All CloudFormation intrinsic functions are supported as of 2026-03-26, including Fn::GetAZs. If you encounter this error, ensure you are using the latest version of cdkd.

2. Extend intrinsic function implementation

If a new function needs support, add implementation to src/deployment/intrinsic-function-resolver.ts.

Example for Fn::Base64:

if ('Fn::Base64' in obj) {
  const value = await this.resolveValue(obj['Fn::Base64'], context);
  return Buffer.from(String(value)).toString('base64');
}

Issue: "AWS::AccountId not resolved"

Symptoms

Output value contains unresolved reference: ${AWS::AccountId}

Causes

Pseudo parameter not resolved.

Solutions

cdkd retrieves actual Account ID via STS GetCallerIdentity. Verify AWS credentials are properly configured:

# Check credentials
aws sts get-caller-identity

# Example output:
# {
#   "UserId": "AIDAI...",
#   "Account": "123456789012",
#   "Arn": "arn:aws:iam::123456789012:user/myuser"
# }

Permission Errors

Issue: "Access Denied" Error

Symptoms

ProvisioningError: Access Denied
User: arn:aws:iam::123456789012:user/myuser is not authorized to perform: s3:CreateBucket

Causes

IAM user/role lacks required permissions.

Solutions

1. Grant required permissions

Main permissions required by cdkd:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:*",
        "iam:*",
        "lambda:*",
        "dynamodb:*",
        "sqs:*",
        "cloudcontrol:*",
        "cloudformation:DescribeType",
        "sts:GetCallerIdentity"
      ],
      "Resource": "*"
    }
  ]
}

Note: In production, follow the principle of least privilege and grant only necessary permissions.

Note on cloudformation:DescribeType: cdkd uses it to resolve each resource type's writeOnlyProperties from the CloudFormation registry so that Cloud Control API updates re-include write-only properties in every patch document (Cloud Control's read-modify-write update would otherwise drop them — e.g. AWS::ECS::Service.VolumeConfigurations). If the permission is missing, cdkd logs a warning and gracefully falls back to a minimal patch (the pre-existing behavior), so deploys still work — but write-only properties may be dropped on update for affected resource types. cdkd export also uses cloudformation:DescribeType to resolve primary identifiers (with a hardcoded fallback table).

2. CloudFormation PassRole permission

When using IAM Role with Lambda, etc.:

{
  "Effect": "Allow",
  "Action": "iam:PassRole",
  "Resource": "arn:aws:iam::123456789012:role/MyLambdaRole"
}

Issue: "You are not authorized to perform sts:AssumeRole"

Symptoms

Error: You are not authorized to perform sts:AssumeRole on arn:aws:iam::...:role/cdk-*

Causes

Lack of AssumeRole permission for roles created by CDK Bootstrap.

Solutions

1. Check Bootstrap role trust policy

aws iam get-role --role-name cdk-hnb659fds-deploy-role-123456789012-us-east-1

2. Add your user/role to trust policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:user/myuser"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Performance Issues

Issue: Deployment is Slow

Symptoms

  • Takes 30+ seconds even for small stacks (5-10 resources)
  • Expected speedup not achieved

Causes

  • Long dependency chains in the DAG (the critical path caps how fast a deploy can finish, even with event-driven dispatch)
  • Cloud Control API rate limits
  • Asset publishing takes time

Solutions

1. Check dependencies

# Check execution plan with diff command
node dist/cli.js diff --app "..." --state-bucket ${STATE_BUCKET} --verbose

# Example output:
# Execution levels:
#   Level 0: [Bucket, Table] (2 resources, parallel)
#   Level 1: [Role] (1 resource)
#   Level 2: [Function] (1 resource)

2. Remove unnecessary dependencies

Reduce explicit dependencies in CDK code:

// Bad example
const bucket = new s3.Bucket(this, 'Bucket');
const role = new iam.Role(this, 'Role', { ... });
role.node.addDependency(bucket);  // ← Unnecessary dependency

// Good example
const bucket = new s3.Bucket(this, 'Bucket');
const role = new iam.Role(this, 'Role', { ... });
// Dependencies auto-detected from Ref/GetAtt

3. Parallelize asset publishing (planned for future implementation)

// Parallel execution in asset-publisher.ts
await Promise.all(
  assets.map(asset => publishAsset(asset))
);

Issue: Cloud Control API Rate Limit

Symptoms

Error: TooManyRequestsException: Rate exceeded

Causes

Cloud Control API has the following rate limits:

  • CreateResource: 5 TPS
  • UpdateResource: 5 TPS
  • DeleteResource: 5 TPS

Solutions

1. Retry logic with exponential backoff (built-in)

cdkd includes built-in retry logic for CREATE operations, with the backoff shape chosen per error class:

  • Throttling and other transient errors (rate limits, a resource still leaving Pending, an async delete releasing a dependency): exponential backoff 1s->2s->4s->8s->8s->8s->8s->8s, capped at 8s, up to 8 retries (47s of sleep). Hammering a throttled API is counter-productive, so this class deliberately backs off hard.
  • IAM propagation (Invalid IAM Instance Profile, cannot be assumed, not authorized to perform, Policy Error: PrincipalNotFound, ...): a denser 0.25s->0.5s->1s->2s->2s... schedule over 26 retries (47.75s of sleep). This class resolves in single-digit seconds — cdkd creates an IAM entity and consumes it ~1-3s later, faster than IAM propagates — so cdkd re-probes roughly every 2s instead of idling through a 4s or 8s step. The total window is at least as long as the generic one, so nothing that used to recover still recovers.

CC API polling uses its own 1s->2s->4s->8s->10s cap schedule. If rate limit errors persist, consider reducing parallelism or staggering deployments.

2. Use SDK Provider

Implement provider that uses SDK directly instead of Cloud Control API.

Orphaned Resources

Overview

Orphaned resources are AWS resources that exist in your account but are not tracked in cdkd's state file. This can happen when a deployment fails partway through — some resources may have been successfully created while others failed in flight.

How cdkd Prevents Orphans

cdkd uses a multi-layered approach to prevent orphaned resources:

  1. Per-resource in-memory state update: Each resource updates the in-memory state (newResources) immediately upon successful provisioning.

  2. Per-resource partial state save: After each successful resource provision, state is persisted to S3 (serialized via a save chain to avoid ETag conflicts). This prevents orphans if the process crashes mid-deploy.

  3. Pre-rollback state save: If any resource fails, cdkd saves the current in-memory state (including all successfully provisioned resources up to that point) to S3 before attempting rollback. This ensures that resources completed concurrently with the failed one are still tracked.

  4. Post-rollback state save: After rollback completes (or is skipped with --no-rollback), state is saved again to reflect the rolled-back resource state.

  5. Rollback journal: On a --no-rollback failure, a Ctrl+C interruption, or before an automatic rollback, cdkd writes a rollback-journal.json sibling of state.json recording exactly which operations completed (issue #1183). This is what lets the standalone cdkd rollback command revert the deploy later (see below). The journal is deleted on the next successful deploy and by cdkd destroy. After a clean automatic rollback it is settled to a failed-only segment instead of deleted (issue #1208): the completed ops are already reverted, but the failed resource's pre-op record is kept so cdkd rollback --revert-failed can still revert a possibly-half-applied resource; the next successful deploy clears it.

Reverting a failed --no-rollback / interrupted deploy: cdkd rollback

After a deploy fails with --no-rollback, is interrupted with Ctrl+C, or its automatic rollback dies partway, you have three options: fix forward (cdkd deploy again), revert (cdkd rollback), or clean up (cdkd destroy).

cdkd rollback MyStack        # revert to the pre-deploy state
cdkd rollback MyStack --force # skip the confirmation prompt
  • Exit 2 means the rollback was partial — one or more ops failed best-effort or were skipped with a warning (e.g. a resource whose physical id changed after a later fix-forward attempt, or an unrecoverable DELETE). The rollback journal is kept so you can re-run cdkd rollback — replay is idempotent (already-reverted resources are skipped).
  • Use --orphan <logicalId> (repeatable) to leave a specific resource alone during the revert (mirrors cdk rollback --orphan).
  • If cdkd rollback reports "nothing to roll back", the journal is already gone — the deploy either succeeded on a later attempt (journal deleted on success) or the process was killed before the journal was written (a SIGKILL before the PUT). In the latter case use cdkd deploy to resume or cdkd destroy to clean up.
  • See docs/cli-reference.md for the full flag reference and known limitations.

Detecting Orphaned Resources

If you suspect orphaned resources exist (e.g., due to a process crash before state could be saved), you can manually compare the state file against actual AWS resources:

# Download state file
aws s3 cp s3://${STATE_BUCKET}/cdkd/MyStack/us-east-1/state.json /tmp/state.json

# List resources tracked in state
cat /tmp/state.json | jq '.resources | keys[]'

# Compare against actual AWS resources using Cloud Control API
aws cloudcontrol list-resources --type-name AWS::S3::Bucket
aws cloudcontrol list-resources --type-name AWS::Lambda::Function

Future: cdkd orphans Command

A dedicated cdkd orphans (or cdkd check) command is planned to automate orphan detection. The approach:

  1. Read the state file for the target stack to get all tracked resources and their physical IDs.
  2. Read the synthesized template to get all expected resource types and logical IDs.
  3. Query AWS for each resource type in the template using Cloud Control API GetResource with the expected physical ID pattern, or by listing resources and matching tags/naming conventions.
  4. Compare: Resources that exist in AWS but are not in the state file are potential orphans. Resources in the state file but not in AWS indicate state drift.
  5. Report: Display a table of orphaned/drifted resources with recommended actions (import to state, delete from AWS, or remove from state).

Example planned interface:

# Check for orphaned resources
cdkd orphans MyStack

# Example output:
# Orphaned Resources (exist in AWS but not in state):
#   AWS::IAM::Role    my-stack-role-abc123    (likely from failed deploy on 2026-03-25)
#   AWS::S3::Bucket   my-stack-bucket-xyz     (likely from failed deploy on 2026-03-25)
#
# Recommended: Run 'cdkd deploy MyStack' to reconcile, or delete manually.

Recovering from Orphaned Resources

If state was saved (most cases):

Running cdkd deploy again will reconcile the state — existing resources will be detected as already created and handled as updates or no-ops.

If state was NOT saved (rare — process crash):

# Option 1: Delete state and redeploy (resources will error on CREATE if they exist)
aws s3 rm s3://${STATE_BUCKET}/cdkd/MyStack/us-east-1/state.json
cdkd deploy MyStack  # May need manual cleanup of duplicates

# Option 2: Manually reconstruct state
aws s3 cp s3://${STATE_BUCKET}/cdkd/MyStack/us-east-1/state.json /tmp/state.json
# Add entries for orphaned resources with their physical IDs
vim /tmp/state.json
aws s3 cp /tmp/state.json s3://${STATE_BUCKET}/cdkd/MyStack/us-east-1/state.json

# Option 3: Destroy everything and start fresh
# Manually delete orphaned resources first, then:
cdkd destroy MyStack --force
cdkd deploy MyStack

Known Leftover: FSx Final Backups

A successful cdkd destroy of an AWS::FSx::FileSystem can leave a chargeable final backup behind: cdkd keeps CloudFormation parity and calls DeleteFileSystem with API defaults, which take a final backup for Windows/ONTAP (observed on OpenZFS too). The backup is typically untagged, so find it via the backup's persisted FileSystem.FileSystemId rather than tags. See supported-resources.md, "FSx final backup on destroy" for the details and the aws fsx describe-backups / aws fsx delete-backup commands.

Debugging Methods

Adjust Log Level

# Enable verbose logging
node dist/cli.js deploy --app "..." --verbose

# Set log level with environment variable
export LOG_LEVEL=debug
node dist/cli.js deploy --app "..."

Check State File

# Download state file
aws s3 cp s3://${STATE_BUCKET}/cdkd/MyStack/us-east-1/state.json /tmp/state.json

# Format and display
cat /tmp/state.json | jq .

# Check specific resource
cat /tmp/state.json | jq '.resources.MyBucket'

Check API Calls with AWS CloudTrail

# Check recent events in CloudTrail
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateBucket \
  --max-results 10

Check Execution Plan with Dry Run

# Show plan only without actual execution
node dist/cli.js deploy --app "..." --state-bucket ${STATE_BUCKET} --dry-run

Previously Known Destroy Issues (All Resolved)

CloudFront OAI DELETE

Resolved via dedicated SDK Provider (cloudfront-oai-provider.ts).

Bedrock AgentCore Runtime IAM Propagation

Resolved via dedicated SDK Provider (agentcore-runtime-provider.ts).

Lambda Permission "No policy found"

Handled automatically by cdkd's idempotent delete logic (not-found errors treated as success).

Frequently Asked Questions (FAQ)

Q: Is a CloudFormation stack created?

A: No, cdkd does not use CloudFormation. Resources are provisioned directly via Cloud Control API and AWS SDK.

Q: Can I use CloudFormation and cdkd for the same stack?

A: No. Stacks deployed with CloudFormation should be managed with cdk deploy or aws cloudformation, and stacks deployed with cdkd should be managed with cdkd.

Q: What happens if I delete the state file?

A: On next deployment, all resources will be treated as CREATE. If existing resources exist, errors will occur, so manual deletion is required beforehand.

Q: Is there a rollback feature?

A: Yes. By default, cdkd rolls back on failure. Use --no-rollback to skip rollback and keep partial state (Terraform-style). On next execution, remaining changes are applied as diff. To revert a --no-rollback (or interrupted) deploy back to its pre-deploy state instead of fixing forward, run the standalone cdkd rollback <stack> command (issue #1183) — it replays a rollback journal cdkd persisted at failure time, with no synth needed.

Q: Are custom resources supported?

A: Yes, Lambda-backed custom resources (Custom::*) are supported.

Support

Issue Reporting

Report on GitHub Issues: https://github.com/YOUR_REPO/cdkd/issues

Questions

Ask questions on GitHub Discussions: https://github.com/YOUR_REPO/cdkd/discussions

Documentation