request: ability to define tasks with manifest.yml files

[zackdotcomputer]

Current State

Currently, if a user wants to perform an action like a db migration where you:

• Require the roles, privileges, and secrets of the application's main service(s)
• Need to run code or scripts separate from your jobs or services - e.g. management code you don't want to ship in a production image or which needs to run once before deploying
• Want to clean up the resources used for the action after completion

Then the intended flows appears to be either

Using tasks:

1. Use copilot task run --generate-cmd ... to create a template command from their service's manifest (which may be thousands of characters long depending on the number of secrets - my project's output is nearly 7k characters)
2. Manually override the correct points of the generated command - most likely the dockerfile flag in this case, though possibly also the command flag.
3. Copy the generated-and-modified command back into your shell and execute.

Using jobs:

1. Define a job manifest with copilot job init
2. Set the job's schedule to none
3. Deploy the job with copilot job deploy
4. Run the job with copilot job run
5. Destroy the job resources with copilot job delete

Proposal

I propose that a more ideal flow would be a job-like manifest that can be executed in a one-off manner like a task. This could work in the following way:

1. Define a manifest.yml file (perhaps with copilot task init or copilot job init depending on which concept the team thinks this more closely aligns to). It should have the same variables and secrets as the service but a different type to indicate its one-off nature, such as "Transient Job" or "Task". (Relatedly - this could be further enhanced if [Feature Request] Ability to extend other manifests #3850 were implemented as the task manifest could extend the service manifest and inherit directly the secrets.)
2. Run with a variation of copilot task run -a application -e environment taskname (or maybe -t taskname to indicate that a task manifest is being used? I would suggest -n taskname but that flag is already taken by the task group name.)

This has the advantage of keeping the task definition living in a structured yml format rather than as a shell command that must either be regenerated live or preserved in a script file, while preserving the tidiness and control that you get from the task concept.

[Lou1415926]

Thank you @zackdotcomputer for the feature request! The proposal makes sense to me. It seems like task run has been used a lot for repeated executions, so I agree that introducing a way to preserve those configurations would make sense for task as well.

A quick question that is unrelated to your proposal and hence would not invalidate your proposal - On the first pain point of using task, I wonder if task run's --env-file flag would help you better?

[zackdotcomputer]

@Lou1415926 Glad to hear it's aligned with your thinking.

Our issue with using the --env-file is that we'd like this task to run in our CI and we aren't checking the .env in to our repo.

[iamhopaul123]

Hello @zackdotcomputer for --env-file you don't have to keep .env file into your repo. You would just need to make sure .env file exists when you run the copilot task run command, and under the hood we'll help you to upload the env file to a S3 bucket and refer to that in the task definition so that it can be used by ECS.

[bencehornak]

I've also thought about this feature request. I was going to create a Copilot task for db migrations (as a task is the recommended way to achieve that by this example), but since my container is relatively complex to initialize (lot of env vars, even side-cars to init the volume and an addon) it would be just too much for a single command line (and would not even be possible due to the add-on resources.

So the alternative solution I found was (ab)using a scheduled job with on schedule: none. This way I can declare a task with all the usual resources, and it won't get triggered automatically, only on demand with copilot job run.

This scheduled job approach gives me mostly what I want, given that all usual Copilot extensions (e.g. addons and side-cars) are available. The only minor drawback is that the resources and the CloudFormation stack are still there after the execution of the job, but I can leave with that limitation. And maybe the name 'scheduled job' could be renamed to simply job, because with schedule: none (which is a legit use case) the job is not scheduled (related confusing terminology: #5286).