copilot hangs during copilot init --app

[ustrahlendorf]

Hi there,
I started to explore Copilot last week. Walk through the following workshop: I liked it and works well.
I cleaned up the deployments including deleting the app.
Want to start a new run but unfortunately the copilot init --app command hangs now for about 45minutes and I have no idea what went wrong. Here is the log from the command line:
uwes@MBP3-us work-folder % copilot init --app module1 --dockerfile svc-api-markdown/Dockerfile --name svc-api-markdown --type "Load Balanced Web Service"

Note: It's best to run this command in the root of your Git repository.
Welcome to the Copilot CLI! We're going to walk you through some questions
to help you get set up with a containerized application on AWS. An application is a collection of
containerized services that operate together.

Note: Looks like you're creating an application using credentials set by environment variables.
Copilot will store your application metadata in this account.
We recommend using credentials from named profiles. To learn more:
https://aws.github.io/copilot-cli/docs/credentials/

Ok great, we'll set up a Load Balanced Web Service named svc-api-markdown in application module1 listening on port 9090.

✔ Proposing infrastructure changes for stack module1-infrastructure-roles

• Creating the infrastructure for stack module1-infrastructure-roles [create complete] [38.3s]
  • A StackSet admin role assumed by CloudFormation to manage regional stacks [create complete] [13.7s]
  • An IAM role assumed by the admin role to create ECR repositories, KMS keys, and S3 buckets [create complete] [13.2s]
    ✔ The directory copilot will hold service manifests for application module1.

Note: Architecture type arm64 has been detected. We will set platform 'linux/x86_64' instead. If you'd rather build and run as architecture type arm64, please change the 'platform' field in your workload manifest to 'linux/arm64'.
✔ Wrote the manifest for service svc-api-markdown at copilot/svc-api-markdown/manifest.yml
Your manifest contains configurations like your container size and port (:9090).

• Update regional resources with stack set "module1-infrastructure" [failed] [72.2s]
  • Update resources in region "eu-west-1" [not started]

In my account I see the module1-infrastructure-roles stack as created complete but nothing else.

Any hints would be appreciated

[ustrahlendorf]

Meanwhile I could find the reason for this strange behavior, the AWS_PROFILE variable was not set correct.

This is something that is confusing. Some commands have the feature to provide a named profile, like copilot env init.
Any reason why not providing the --profile flag for other commands?

[efekarakus]

Hi @ustrahlendorf 👋

I'm sorry about the inconvenience 🙇 and I'm glad that you were able to find the root cause!

I can try to provide some context here around how Copilot works and how we can improve the user experience based on your feedback.

Context

AWS Copilot follows the "tooling account" pattern for organizing AWS account.

The CLI assumes that the default credential chain (in your example, the resolved AWS_PROFILE profile) always points to an "application" account where Copilot stores metadata about the application name, its environments and services as well as shared infrastructure such as CodePipeline pipelines across environments.
The env init command is the only command that takes a --profile flag because we want users to be able to initialize environments in different AWS accounts and regions. When an environment is initialized, Copilot creates an IAM role (EnvironmentManagerRole) that contains all the necessary permissions for the CLI to operate in the environment.

The other commands, such as copilot svc deploy --environment test, do not need the --profile flag because Copilot uses the default credential chain to find out if a "test" account exists and then assumes the EnvironmentManagerRole in the "test" account to deploy the service.

I hope that made the credential flow a bit clearer for Copilot.

Action items

Today, the issue is that it's not clear that Copilot always needs the default credential chain to point to the "application" account.
We should consider having a manifest.yaml file that enables users to specify credentials configuration for the application. For example,

# In copilot/manifest.yaml

type: Application
name: 'demo'

credentials: # Optional, otherwise we use the default credential chain.
  profile: 'some-profile'

[ustrahlendorf]

Hi Efe,
thank you for the clarification, it helps to understand the overall construct.
I think it would be very helpful to have some kind of, let say verification command , like a status where a user can check the current credentials that copilot sees for his commands.

The manifest.yaml is a good place for such "overall" config parts.

One other thing came to my mind. What about some config-settings for CloudFormation stacks like time-out period before a stack gets rollbacked or Rollback configuration options?

regards
Uwe

[ustrahlendorf]

smile ... one add on out of my practical work with assuming roles and credentials in different accounts (most from customers) ... I (same for my colleagues) use the tool awsume to switch into different roles and accounts. As I know awsume does not set AWS_PROFILE, it sets a variable called AWSUME_PROFILE.
(need to check now if it also could set AWS_PROFILE)
At the end that was the reason why I struggled, I just thought everything works because my credentials have been set as usual with awsume.
regards
Uwe

[efekarakus]

One other thing came to my mind. What about some config-settings for CloudFormation stacks like time-out period before a stack gets rollbacked or Rollback configuration options?

That's really interesting, we haven't thought of surfacing these options as configurable knobs for now because we hadn't run into a scenario in the past that required modification.

Today, some of the information is either hardcoded in the CLI or we surfaced it in the command (e.g. --no-rollback for the svc deploy command)

copilot-cli/internal/pkg/aws/cloudformation/cloudformation.go

Line 31 in 50e4927

request.WithWaiterMaxAttempts(1080), // Wait for at most 90 mins for any cfn action.

We'd love to learn more if there is a particular SDK setting is not surfaced in the CLI but is required for your workflows 🙏

[ustrahlendorf]

my observation is triggered when playing around with stacks/configurations during the "development phase". In this phase it is sometimes very annoying when there is a problem with the stack and you have to wait for the default time out (60minutes) until a rollback starts and you can start to fix the issue.

I dont know if this behavior is really an issue when working with copilot, but even if you do your best to prevent this, I have learned things go wrong, even if you think they can't.
Just my 2 ct.