Skip to content

Commit fefa3cc

Browse files
crowecawcawcrowecawcaw
authored
feat: Add Blender turntable to Flow Production Tracking sample (aws-deadline#245)
Add a job bundle that renders an animated turntable in Blender, encodes a review-ready movie, extracts a poster-frame thumbnail, and publishes the result to Autodesk Flow Production Tracking (formerly ShotGrid) as a new Version on an Asset's review Task. The sample demonstrates two Deadline Cloud patterns: - post-render work modeled as discrete OpenJD steps with step dependencies (encode and thumbnail fan out in parallel, then publish), and - filling Flow job parameters from studio environment variables via a preSubmission hook, so the submitter does not enter them by hand. Flow credentials are read at runtime from AWS Secrets Manager using the worker's queue role. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
1 parent 2d04d2a commit fefa3cc

7 files changed

Lines changed: 1020 additions & 0 deletions

File tree

‎job_bundles/blender_turntable_to_flow/README.md‎

Lines changed: 294 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,294 @@
1+
# Blender Turntable to Autodesk Flow Production Tracking
2+
3+
## Introduction
4+
5+
This job bundle renders an animated turntable in Blender, encodes it into a
6+
review-ready movie, extracts a poster-frame thumbnail, and publishes the
7+
result to [Autodesk Flow Production Tracking](https://www.autodesk.com/products/flow-production-tracking/overview)
8+
(formerly ShotGrid) as a new `Version` on an `Asset`'s review `Task`.
9+
10+
![The published Version in Flow Production Tracking, showing the turntable movie, artist, status, and review Task.](screenshot.png)
11+
12+
It is a deliberately end-to-end, representative example of a render-and-publish
13+
pipeline on AWS Deadline Cloud, and it demonstrates two core patterns:
14+
15+
1. Post-render work modeled as job steps. Tasks like uploading to Flow,
16+
registering versions, and generating thumbnails are expressed as discrete
17+
steps in the job, not as side effects hidden inside the render.
18+
2. Chained, dependent, non-render tasks (movie encode, thumbnail, publish)
19+
that run after a render and fan out in parallel.
20+
21+
## Why post-render work belongs in a job step
22+
23+
The canonical way to do post-render work on Deadline Cloud — uploading to Flow,
24+
registering versions, generating thumbnails — is a discrete OpenJD step with a
25+
step dependency. Each unit of post-render work is a first-class farm task,
26+
which means it is:
27+
28+
- Observable — it shows up in the monitor with its own logs and status.
29+
- Independently retryable — a failed Flow publish can be retried without
30+
re-rendering a single frame.
31+
- Independently schedulable — the publish step can run on a cheap CPU fleet
32+
instead of the GPU render fleet (this sample keeps them on one fleet for
33+
simplicity, but the steps are separable).
34+
- Parallelizable — `GenerateMovie` and `GenerateThumbnail` run at the same
35+
time because neither depends on the other.
36+
37+
Modeling each piece of post-render work as its own step is what gives you all of
38+
the above, and it's the pattern to reach for when wiring a Flow publish into a
39+
render job.
40+
41+
## The step graph
42+
43+
```
44+
RenderTurntable (Blender; one task per frame, parameter-space over the frame range)
45+
├─ GenerateMovie (depends: RenderTurntable) ffmpeg frames → H.264 mp4
46+
├─ GenerateThumbnail (depends: RenderTurntable) mid frame → jpg
47+
└─ PublishToFlow (depends: GenerateMovie, GenerateThumbnail)
48+
create Version, upload movie + thumbnail, advance Task
49+
```
50+
51+
`GenerateMovie` and `GenerateThumbnail` both depend only on `RenderTurntable`, so
52+
they run in parallel — a clear visual demonstration of step fan-out in the
53+
monitor. `PublishToFlow` depends on both because it uploads both artifacts.
54+
55+
## How this maps to the Flow entity model
56+
57+
The sample follows idiomatic Flow conventions:
58+
59+
- The `Version` is the hero entity — the reviewable media record. The movie is
60+
uploaded to `sg_uploaded_movie` (what plays in the review player);
61+
`sg_path_to_movie` / `sg_path_to_frames` hold the filesystem paths.
62+
- The thumbnail belongs on the `Version` (uploaded via `upload_thumbnail`),
63+
which populates the Version's `image` field — that's what shows in the review
64+
grid. It is not attached to the Asset.
65+
- Status is advanced on the `Task`, not the Asset. In Flow, an Asset isn't
66+
marked "rendered"; instead the review Task (for example a "Model" or
67+
"Turntable" task) is advanced. Many studios use a `rev` ("pending review")
68+
status for this, but `rev` is not a stock status code — it varies per site.
69+
The `FlowTaskStatus` parameter defaults to `fin` because the demo site only
70+
defines the stock codes `wtg` / `ip` / `fin`; set it to whatever your site
71+
uses (`rev` is the conventional choice where it exists).
72+
- The work hangs off an Asset rather than a Shot. An "Asset turntable" — build
73+
an asset, render a turntable to review the model or look — is self-contained,
74+
with no sequence/shot hierarchy required, and maps directly to "render an
75+
object and review it."
76+
77+
## Prerequisites
78+
79+
The job needs Blender, FFmpeg, and Python. On Deadline Cloud
80+
service-managed fleets, use a [conda queue
81+
environment](https://docs.aws.amazon.com/deadline-cloud/latest/developerguide/provide-applications.html)
82+
with:
83+
84+
- `CondaPackages`: `blender ffmpeg python>=3.10 pip`
85+
- `CondaChannels`: `deadline-cloud conda-forge`
86+
87+
Blender is on the `deadline-cloud` channel; FFmpeg is on the `conda-forge`
88+
community channel.
89+
90+
> Note on `shotgun_api3`: the Flow Python client is published on PyPI
91+
> only (it is *not* on conda-forge). The `PublishToFlow` step therefore
92+
> `pip install`s `shotgun_api3` (and `boto3`) at runtime into the conda
93+
> environment. This mixed conda + pip approach is normal for Python client
94+
> libraries that have no conda package. A team that wants fully reproducible,
95+
> offline-capable environments can instead build an internal conda package or
96+
> bake the dependency into a custom worker image.
97+
98+
## Flow credentials: AWS Secrets Manager (do this once)
99+
100+
Flow script credentials are stored in AWS Secrets Manager. The `PublishToFlow`
101+
step reads them at runtime using the worker's queue role, so the credentials
102+
stay out of the job bundle and out of the job's parameters.
103+
104+
### 1. Create the secret
105+
106+
The secret value is JSON with three keys — your Flow site URL, the script
107+
name (from Flow → Admin → Scripts), and its API key:
108+
109+
```bash
110+
aws secretsmanager create-secret \
111+
--region us-west-2 \
112+
--name "deadline-cloud-samples/flow-production-tracking" \
113+
--description "Flow Production Tracking script credentials for the turntable demo job." \
114+
--secret-string '{
115+
"site_url": "https://your-site.shotgrid.autodesk.com",
116+
"script_name": "your-script-name",
117+
"api_key": "your-api-key"
118+
}'
119+
```
120+
121+
Note the returned `ARN` — you'll pass it as the `FlowSecretArn` job parameter.
122+
123+
To update the credentials later:
124+
125+
```bash
126+
aws secretsmanager put-secret-value \
127+
--region us-west-2 \
128+
--secret-id "deadline-cloud-samples/flow-production-tracking" \
129+
--secret-string '{ "site_url": "...", "script_name": "...", "api_key": "..." }'
130+
```
131+
132+
### 2. Grant the queue role permission to read the secret
133+
134+
The job runs under your queue role (the `roleArn` on your queue). That role
135+
must be allowed `secretsmanager:GetSecretValue` on the secret ARN. Find the role
136+
name:
137+
138+
```bash
139+
aws deadline get-queue \
140+
--region us-west-2 \
141+
--farm-id farm-XXXX --queue-id queue-XXXX \
142+
--query roleArn --output text
143+
# e.g. arn:aws:iam::<account>:role/service-role/AWSDeadlineCloudQueueRole-XXXXXXXX
144+
```
145+
146+
Attach an inline policy granting read access to just this secret (replace the
147+
role name and ARN):
148+
149+
```bash
150+
aws iam put-role-policy \
151+
--role-name AWSDeadlineCloudQueueRole-XXXXXXXX \
152+
--policy-name FlowProductionTrackingSecretRead \
153+
--policy-document '{
154+
"Version": "2012-10-17",
155+
"Statement": [
156+
{
157+
"Sid": "ReadFlowProductionTrackingSecret",
158+
"Effect": "Allow",
159+
"Action": "secretsmanager:GetSecretValue",
160+
"Resource": "arn:aws:secretsmanager:us-west-2:<account>:secret:deadline-cloud-samples/flow-production-tracking-XXXXXX"
161+
}
162+
]
163+
}'
164+
```
165+
166+
> Scope the `Resource` to the exact secret ARN (including the random suffix
167+
> Secrets Manager appends). Do not grant `secretsmanager:GetSecretValue` on `*`.
168+
169+
## Bundle layout
170+
171+
```
172+
blender_turntable_to_flow/
173+
├── template.yaml the OpenJD job template
174+
├── hooks.yaml submission hook configuration (preSubmission)
175+
└── scripts/
176+
├── build_turntable.py builds the turntable scene in Blender and renders a frame
177+
├── publish_to_flow.py creates the Version and uploads the movie + thumbnail
178+
└── flow_params_from_env.py preSubmission hook: fills the Flow parameters from the environment
179+
```
180+
181+
The two render/publish scripts are referenced by the template as `dataFlow: IN`
182+
`PATH` parameters (`BuildTurntableScript`, `PublishScript`). Deadline Cloud
183+
uploads them with the job's attachments and path-maps them onto the worker, so
184+
the steps run them as ordinary files rather than embedding the source inline.
185+
186+
## Filling Flow parameters from the environment (submission hook)
187+
188+
Studios usually already have environment or project-tracking tooling that sets
189+
environment variables when an artist opens a shell or launches an application
190+
(via Rez, a launcher, a "set project" script, and so on). This bundle uses a
191+
[Deadline Cloud submission hook](https://github.com/aws-deadline/deadline-cloud/blob/mainline/docs/submission-hooks.md)
192+
to read those variables and fill in the job's Flow parameters, so the
193+
submitting artist doesn't re-enter the project id, asset name, secret ARN, etc.
194+
by hand.
195+
196+
The Flow parameters in the template are `HIDDEN` and carry only placeholder
197+
defaults. `hooks.yaml` registers a `preSubmission` hook
198+
(`scripts/flow_params_from_env.py`) that runs as part of every submission — CLI
199+
or GUI — and rewrites those parameter defaults from the environment before the
200+
job is created.
201+
202+
| Environment variable | Job parameter | Required |
203+
|----------------------|---------------|----------|
204+
| `FLOW_PROJECT_ID` | `FlowProjectId` | yes |
205+
| `FLOW_ASSET_NAME` | `FlowAssetName` | yes |
206+
| `FLOW_SECRET_ARN` | `FlowSecretArn` | yes |
207+
| `FLOW_ASSET_TYPE` | `FlowAssetType` | no |
208+
| `FLOW_STEP_SHORT_NAME` | `FlowStepShortName` | no |
209+
| `FLOW_TASK_NAME` | `FlowTaskName` | no |
210+
| `FLOW_TASK_STATUS` | `FlowTaskStatus` | no |
211+
| `FLOW_PUBLISH` | `EnableFlowPublish` | no |
212+
213+
If a required variable is missing, the hook exits non-zero and the submission is
214+
aborted with a message telling you which variables to set. To submit without
215+
publishing (render → movie → thumbnail only), set `FLOW_PUBLISH=FALSE`; the
216+
required-variable check is then skipped.
217+
218+
Bundle hooks are disabled by default. Enable them once:
219+
220+
```bash
221+
deadline config set settings.allow_bundle_hooks true
222+
```
223+
224+
With the hook in place, a submission only needs the render parameters — the Flow
225+
parameters come from the environment:
226+
227+
```bash
228+
# A studio "set project" step would normally export these for you:
229+
export FLOW_PROJECT_ID=1234
230+
export FLOW_ASSET_NAME="Hero Vehicle"
231+
export FLOW_SECRET_ARN=arn:aws:secretsmanager:us-west-2:<account>:secret:deadline-cloud-samples/flow-production-tracking-XXXXXX
232+
233+
deadline bundle submit blender_turntable_to_flow/ -p ObjectShape=monkey -p FrameRange=1-48
234+
```
235+
236+
The hook reads and writes the template with PyYAML, so the Python that runs it
237+
(the Deadline Cloud CLI's Python) needs `pyyaml` available.
238+
239+
## Example submission
240+
241+
These assume the `FLOW_*` environment variables are set (see the previous
242+
section) and bundle hooks are enabled.
243+
244+
### GUI submission
245+
246+
```bash
247+
deadline bundle gui-submit blender_turntable_to_flow/
248+
```
249+
250+
### CLI submission
251+
252+
```bash
253+
deadline bundle submit blender_turntable_to_flow/ \
254+
-p ObjectShape=monkey \
255+
-p FrameRange=1-48
256+
```
257+
258+
### Render-only (no Flow)
259+
260+
To exercise just the render → movie → thumbnail pipeline without Flow
261+
credentials, set `FLOW_PUBLISH=FALSE` so the hook skips the required-variable
262+
check and the publish step is skipped:
263+
264+
```bash
265+
FLOW_PUBLISH=FALSE deadline bundle submit blender_turntable_to_flow/ \
266+
-p ObjectShape=torus
267+
```
268+
269+
## Parameters
270+
271+
| Parameter | Default | Description |
272+
|-----------|---------|-------------|
273+
| ObjectShape | `monkey` | Procedural subject: monkey, cube, torus, ico_sphere, cylinder, cone |
274+
| FrameRange | `1-48` | Frame range; a full 360° rotation is spread across it |
275+
| ResolutionX / ResolutionY | `960` / `540` | Render resolution |
276+
| Samples | `48` | Cycles render samples |
277+
| OutputDir | `output` | Working/output dir; frames in `OutputDir/frames`, movie+thumbnail in `OutputDir` |
278+
| FrameRate | `24` | Output movie fps |
279+
The Flow parameters below are `HIDDEN` and supplied by the preSubmission hook
280+
from the `FLOW_*` environment variables; the "default" column is only the
281+
template placeholder.
282+
283+
| Parameter | Default | Description |
284+
|-----------|---------|-------------|
285+
| EnableFlowPublish | `TRUE` | `FALSE` (via `FLOW_PUBLISH`) skips the Flow publish step |
286+
| FlowSecretArn | *(empty)* | Secrets Manager ARN holding `{site_url, script_name, api_key}` |
287+
| FlowProjectId | `0` | Numeric Flow Project id to publish into |
288+
| FlowAssetName | *(empty)* | Asset code (found-or-created) |
289+
| FlowAssetType | `Prop` | `sg_asset_type` used if the Asset is created |
290+
| FlowStepShortName | `MDL` | Pipeline Step short name for the review Task (if created) |
291+
| FlowTaskName | `Turntable` | Task content/name on the Asset (found-or-created) |
292+
| FlowTaskStatus | `fin` | Status set on the Task after publishing (`rev` where defined) |
293+
| BuildTurntableScript | `scripts/build_turntable.py` | Blender scene/render script shipped in the bundle (hidden) |
294+
| PublishScript | `scripts/publish_to_flow.py` | Flow publish script shipped in the bundle (hidden) |

‎job_bundles/blender_turntable_to_flow/hooks.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
version: "1.0"
2+
3+
# A preSubmission hook runs as part of every submission (CLI and GUI), just
4+
# before the job is hashed, uploaded, and created. This one reads the studio's
5+
# Flow environment variables and rewrites the job template's Flow parameter
6+
# defaults from them, then prints the modified template on stdout. The Deadline
7+
# Cloud client applies that template to the submission.
8+
#
9+
# The Flow parameters are HIDDEN in the template and have no usable default, so
10+
# this hook is the single source of their values. If a required environment
11+
# variable is missing, the hook exits non-zero and the submission is aborted.
12+
#
13+
# Bundle hooks are disabled by default. Enable them once with:
14+
# deadline config set settings.allow_bundle_hooks true
15+
preSubmission:
16+
- command: python3
17+
args: [scripts/flow_params_from_env.py]
18+
timeout: 10

‎job_bundles/blender_turntable_to_flow/screenshot.png‎

141 KB
Loading

0 commit comments

Comments
 (0)