InvalidClientTokenId when provisioning the cold start baseline

[jochem725]

Getting started deploying our reference architecture today -

Whilst running atmos workflow init/tfstate -f baseline

This deploys the S3 bucket and Dynamo tables for the Terraform state perfectly fine, but I don't seem to be able to deploy aws_iam_role.default["xxx-core-gbl-root-tfstate"], which leads to the following error being thrown:

│ Error: creating IAM Role (xxx-core-gbl-root-tfstate): operation error IAM: CreateRole, https response error StatusCode: 403, RequestID: xxxx, api error InvalidClientTokenId: The security token included in the request is invalid

aws sts get-caller-identity gives me the expected role in Geodesic for the SuperAdmin role.

 √ . [xxx-identity] (HOST) infrastructure ⨠ aws sts get-caller-identity
{
    "UserId": "xxxxxxx",
    "Account": "xxxxxxx",
    "Arn": "arn:aws:iam::xxxxxxx:user/SuperAdmin"
}

Do you have any insights in what the possible cause for this can be? I've already checked MFA on both the root and SuperAdmin user as I believe that's a requirement to create IAM roles via the CLI.

Do you have any insights in what can cause this?

[milldr]

The InvalidClientTokenId error usually means Terraform is attempting to use credentials that AWS does not recognize, even though your CLI session with aws sts get-caller-identity looks correct. Please walk through some checks to make sure everything is set up properly.

1. Confirm SuperAdmin is configured correctly

• Make sure MFA is enabled and that your local session is authenticated with MFA.
• Please try signing out and signing back in as SuperAdmin to ensure you’re prompted for MFA.
• Reference: How to Create SuperAdmin User.

2. Verify you are using SuperAdmin locally

• From Geodesic, run aws sts get-caller-identity.
• You should see the SuperAdmin identity, as you’ve shown.

3. Check the tfstate-backend configuration

Terraform should be using your local role when bootstrapping the backend. In your stack configuration (infrastructure/stacks/catalog/tfstate-backend/defaults.yaml), confirm the following:

• backend.s3.role_arn and backend.s3.profile are set to null (so Terraform doesn’t try to assume another role).
• backend.s3.assume_role.role_arn should also be null or unset (often omitted entirely in this configuration).

4. Inspect the component directory

Sometimes a previous run generates backend configuration files that interfere with the cold start process.

• Look in components/terraform/tfstate-backend for any generated files.
• If backend.tf.json exists, delete it. On the initial apply, Terraform should use local state. On the second apply, Terraform will generate this file and migrate state to S3.
• Do not delete the .terraform directory, since that may contain your local state if the component was only partially applied.

5. Reapply with explicit commands

Run the first apply directly (replace use1 with your region/environment):

atmos terraform apply tfstate-backend -var=access_roles_enabled=false --stack core-use1-root --auto-generate-backend-file=false

If that succeeds, complete the S3 migration:

atmos terraform apply tfstate-backend -var=access_roles_enabled=false --stack core-use1-root --init-run-reconfigure=false

[jochem725]

Thank you - ultimately our issue was that a passkey was created as MFA, but it seems like it has to be a TOTP.