This Terraform configuration deploys an AWS Deadline Cloud farm you can use to run jobs such as rendering images and reconstructing 3D scenes, or transforming your data in custom ways. It is the Terraform equivalent of the CloudFormation starter_farm template.
Sample jobs to submit are available in the deadline-cloud-samples on GitHub. Deadline Cloud provides many integrated submitter plugins for applications, and you can build your own jobs.
The deployed farm includes one or more service-managed fleets that you select during deployment. The production queue supports Conda virtual environments for the applications that jobs need, and the package build queue can be used to build more packages if needed.
It configures two Conda channels by default: a private channel on an S3 bucket you provide and the deadline-cloud channel. The deadline-cloud channel provides applications like Blender, Houdini, Maya, and Nuke. You can add the conda-forge channel to this list by setting the prod_conda_channels variable to "deadline-cloud conda-forge" to access packages created and maintained by the conda-forge community.
When supported applications need licenses to run, they will use Deadline Cloud's usage-based licensing. See Deadline Cloud pricing to learn which applications are supported and the associated costs.
Before deploying this Terraform configuration, check that you have the following resources created in your AWS Account. The AWS region should be the same as the one you use to deploy the Terraform configuration.
- Terraform >= 1.0 installed
- AWS credentials configured (via
aws configure, environment variables, or IAM role) - An Amazon S3 bucket to hold job attachments and your Conda channel. From the Amazon S3 management console, create an S3 bucket. You will need the bucket name to deploy the Terraform configuration.
- A Deadline Cloud monitor to view and manage the jobs you will submit to your queues. From the AWS Deadline Cloud management console, select the "Go to Monitor setup" option and follow the steps to enter a name for your monitor URL, enable IAM Identity Center, and then create a user login account to access the monitor. Your monitor URL will look similar to
https://..deadlinecloud.amazonaws.com/. You will need this URL to log in with the Deadline Cloud monitor desktop application.
This configuration creates the following resources:
| Resource | Description |
|---|---|
awscc_deadline_farm |
The Deadline Cloud farm |
awscc_deadline_queue (x2) |
Production queue and Package Build queue |
awscc_deadline_queue_environment |
Conda queue environment for the production queue |
awscc_deadline_fleet (up to 3) |
CPU Linux, CPU Windows, and/or CUDA Linux fleets |
awscc_deadline_queue_fleet_association (up to 6) |
Associations between queues and fleets |
aws_iam_role (x3) |
IAM roles for queues and fleet |
aws_iam_role_policy (x3) |
IAM policies for S3 access and CloudWatch Logs |
cd terraform/farm_templates/starter_farm
terraform initCreate a terraform.tfvars file or pass variables via command line:
# Required
job_attachments_bucket_name = "your-s3-bucket-name"
# Optional - customize as needed
aws_region = "us-west-2"
farm_name = "My Deadline Cloud Farm"
# Fleet configuration (set to empty string to skip)
cpu_linux_fleet_name = "CPU Linux Fleet"
cpu_windows_fleet_name = "" # Skip Windows fleet
cuda_linux_fleet_name = "" # Skip CUDA fleetterraform planterraform applyFrom the AWS Deadline Cloud management console, navigate to the farm that you created, and select the "Access management" tab. Select "Users", then "Add user", and then add the user you created for yourself from the prerequisites. Use the "Owner" access level to give yourself full access.
| Variable | Description | Default |
|---|---|---|
aws_region |
AWS region | us-west-2 |
job_attachments_bucket_name |
S3 bucket for job attachments | (required) |
farm_name |
Farm display name | Starter Deadline Cloud Farm |
prod_queue_name |
Production queue name | Production Job Queue |
package_build_queue_name |
Package build queue name | Package Build Queue |
prod_conda_channels |
Default Conda channels | deadline-cloud |
cpu_linux_fleet_name |
CPU Linux fleet name (empty to skip) | CPU Linux Fleet |
cpu_windows_fleet_name |
CPU Windows fleet name (empty to skip) | "" |
cuda_linux_fleet_name |
CUDA Linux fleet name (empty to skip) | "" |
max_cpu_linux_worker_count |
Max workers for CPU Linux fleet | 10 |
cpu_linux_instance_market_type |
spot or on-demand |
spot |
See main.tf for the complete list of configurable variables.
| Output | Description |
|---|---|
farm_id |
The Deadline Cloud farm ID |
farm_arn |
The Deadline Cloud farm ARN |
prod_queue_id |
The production queue ID |
package_build_queue_id |
The package build queue ID |
cpu_linux_fleet_id |
The CPU Linux fleet ID (if created) |
cpu_windows_fleet_id |
The CPU Windows fleet ID (if created) |
cuda_linux_fleet_id |
The CUDA Linux fleet ID (if created) |
- From the AWS Deadline Cloud management console, select the "Downloads" page on the left navigation area.
- Download and install the Deadline Cloud monitor desktop application. Use your monitor URL and the user account from the prerequisites to log in from the Deadline Cloud monitor desktop. This also provides AWS credentials to the Deadline Cloud CLI.
- Download and install the Deadline Cloud submitter installer for your platform, or install the
Deadline Cloud CLI into your existing Python installation from PyPI
using a command like
pip install "deadline[gui]". You can then use the commanddeadline handle-web-url --installto install the job attachments download handler on supported operating systems. - From the terminal, run the command
deadline config gui, and select the farm and production queue you deployed. Select OK to apply the settings.
Before submitting jobs, initialize the S3 Conda channel by publishing a package to it. See Publish packages to an Amazon S3 conda channel in the AWS Deadline Cloud Developer Guide for instructions.
This test job runs the imagemagick identify command on a directory of images to extract properties of the images and write them to a text file. Before proceeding with this test job, make sure the S3 Conda channel is initialized according to the instructions above. An uninitialized Conda channel will fail during the "Launch Conda" action.
- If you don't have a local copy of deadline-cloud-samples GitHub repository, you can make a git clone or download it as a ZIP.
- From the
job_bundlesdirectory ofdeadline-cloud-samples, run the following command:$ deadline bundle gui-submit cli_job - From the "Shared job settings" tab, give the job a name like "Starter farm test job", then enter "imagemagick" into the "Conda Packages" parameter and if it's not already included, add "conda-forge" to the "Conda Channels" parameter. These parameters are for the Conda queue environment that provides applications to the job.
- From the "Job-specific settings" tab, select the directory
turntable_with_maya_arnoldwithin the samples as the "Input/Output Data Directory". This directory has some .png files to process. - Replace the "Bash Script" text box contents with the following script:
find . -type f -iname "*.png" -exec magick identify {} \; | tee identified_images.txt - Select "Submit" and accept any prompts to submit the job to your queue.
- From Deadline Cloud monitor, navigate to the production queue to watch the job you submitted. When it is running, right click on the task and select "View logs". It may take a few minutes as Deadline Cloud starts an instance in your fleet to run the job. Within the log, you can find output that is similar to:
+ find . -type f -iname '*.png' -exec magick identify '{}' ';' + tee identified_images.txt ./screenshots/turntable_job_bundle_submitter_gui.png PNG 657x844 657x844+0+0 8-bit sRGB 59671B 0.000u 0:00.000 ./screenshots/windows_desktop_submitter_bat_file.png PNG 237x231 237x231+0+0 8-bit sRGB 29790B 0.000u 0:00.000 ./screenshots/turntable_job_output_video_screenshot.png PNG 962x693 962x693+0+0 8-bit sRGB 674715B 0.000u 0:00.000 - When it is complete, download the output of the job. The custom script you entered populates a text file with image metadata. The output is written to the provided input/output directory, so look in the
turntable_with_maya_arnolddirectory to find a fileidentified_images.txtwith contents matching the logged output from the job:./screenshots/turntable_job_bundle_submitter_gui.png PNG 657x844 657x844+0+0 8-bit sRGB 59671B 0.000u 0:00.000 ./screenshots/windows_desktop_submitter_bat_file.png PNG 237x231 237x231+0+0 8-bit sRGB 29790B 0.000u 0:00.000 ./screenshots/turntable_job_output_video_screenshot.png PNG 962x693 962x693+0+0 8-bit sRGB 674715B 0.000u 0:00.000
You can also submit the sample job with a single command from your terminal as follows:
$ deadline bundle submit cli_job \
--name "Starter farm test job" \
-p CondaPackages=imagemagick \
-p "CondaChannels=s3://your-s3-bucket-name/Conda/Default deadline-cloud conda-forge" \
-p DataDir=./turntable_with_maya_arnold \
-p 'BashScript=find . -type f -iname "*.png" -exec magick identify {} \; | tee identified_images.txt'
Use the AWS IAM Identity Center management console to create more users and groups, then give them permission to access the farm from the AWS Deadline Cloud management console.
See the Conda recipe samples to learn about the package building queue deployed by the template. If you write custom tools and plugins, you can write your own Conda package recipes to provide them to the farm.
Run jobs from the job bundle samples. Make copies of the code and build your own.
Run the submitter installer in the downloads section of the AWS Deadline Cloud management console, or start from the submitter source code on GitHub.
By deploying fleets with multiple different hardware configurations, you can create a farm that supports a wide variety of jobs. The starter farm Terraform configuration comes with these fleet configurations:
- a CPU Linux fleet
- a CPU Windows fleet
- a CUDA Linux fleet
Each fleet that you name will be deployed. If you set its name to be empty, it will be skipped.
When different steps of your jobs have different requirements, you can edit your job template to have hostRequirements for each step. These control the operating system and memory requirements, along with whether a GPU is available.
Each fleet has variables to control the maximum number of workers and the vCPUs and RAM of worker hosts, along with the choice of spot or on-demand instances. If you use spot instances, you generally want to include wider ranges of these properties when possible to increase the available instance types you can get.
The default Conda channels that come after the S3 Conda channel are controlled by the prod_conda_channels variable that defaults to "deadline-cloud". You can modify this to include conda-forge or channels such as bioconda.
The Terraform configuration includes a queue environment that creates Conda virtual environments for jobs to use. By default, the queue environment uses the template file conda_queue_env.yaml.tftpl. You can edit this file to customize the Conda environment behavior, such as changing the default channels and adjusting caching behavior or the environment creation logic.
See the queue environment samples for more ideas on how to configure queue environments.
If you want to organize the queues in your farm differently from this starter sample, or you need a different set of fleet configurations, you can copy this Terraform configuration and start editing it. See the CUDA farm CloudFormation template for an example where the starter farm has been simplified and specialized for jobs that use CUDA.
We recommend you follow Infrastructure as Code best practices, such as keeping your configurations in version control and strictly making changes by editing the configuration and deploying it instead of mixing Terraform together with manual infrastructure updates from the AWS console. See the AWS Well-Architected guidance on Infrastructure as Code to dive deeper into this topic.
All Terraform configurations have been validated with security scanning tools:
Run security scans on your modifications:
# Install tools
pip install checkov
brew install tflint # or see https://github.com/terraform-linters/tflint
# Run scans
checkov -d . --framework terraform
tflintThis template has been validated with:
- Checkov: 37 passed, 0 failed
- tflint: No issues
- terraform validate: Success
To destroy all resources:
terraform destroyThis Terraform configuration creates identical resources to the CloudFormation starter_farm template. See the parent README for a comparison table.