Skip to content

Commit 7d793b7

Browse files
crowecawcawcrowecawcaw
authored
feat: add VFX shot-render-and-publish pipeline sample (aws-deadline#258)
Adds job_bundles/vfx_pipeline, a working studio-to-cloud VFX shot pipeline for AWS Deadline Cloud, for a studio that knows its own pipeline deeply but is new to Deadline Cloud. An artist submits a shot with a small launcher (studio_pipe), it renders per-frame on a service-managed fleet, and the frames, a preview movie, and a thumbnail come back to the studio drive, with a review Version published to Autodesk Flow Production Tracking. - studio_pipe/: a pip-installable launcher (resolve / launch / submit / autodownload) that resolves a shot's context from a layered config hierarchy and fills the job's parameters. One codebase for workstation and farm so the two environments can't drift. - job_bundle/blender_shot_render/: a static, parameterized OpenJD bundle: RenderShot (one task per frame) -> GenerateMovie / GenerateThumbnail (parallel) -> PublishToFlow. - Software delivery via Conda: Blender and an in-house Blender add-on (moonrise_scatter) are built from conda_recipes/ with rattler-build, published to an S3 Conda channel, and installed at run time by a Conda queue environment. - conda_channel/: a committed example channel so the layout is browsable. - Outputs return via deadline queue sync-output; submit sets OutputDir under studio/renders/<shot_id> so outputs restore straight onto the drive. - Flow publishing reads its credential from AWS Secrets Manager via the queue role, following the blender_turntable_to_flow pattern. Verified end to end on a Linux service-managed fleet: build and publish moonrise_scatter to an S3 Conda channel, submit, the queue environment installs blender plus the add-on, frames render with the add-on enabled, and the movie, thumbnail, and a Flow Version with attached media are produced and downloaded back to the studio drive. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
1 parent 8b6b6ea commit 7d793b7

43 files changed

Lines changed: 3291 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.images/vfx_pipeline_architecture.png‎

433 KB
Loading

‎job_bundles/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@ Nested collections provide their own complete indexes where applicable.
6060
| [Maya V-Ray tiled render](tile_render_with_maya_vray/) | V-Ray tile rendering followed by OpenImageIO assembly | You need tiled EXR output from Maya and V-Ray |
6161
| [V-Ray Linux region render](tile_render_with_vray_linux/) | Region rendering, asset discovery, path mapping, and image merge | You render `.vrscene` files in parallel regions on Linux |
6262
| [Maya Arnold turntable](turntable_with_maya_arnold/) | Building a scene around an OBJ, then rendering frames and encoding video | You need an easy-to-submit 3D asset review utility |
63+
| [Studio VFX shot pipeline](vfx_pipeline/) | A launcher, static render bundle, Conda software delivery, and Flow publishing mapped onto a studio's existing pipeline | You are porting a studio shot pipeline to Deadline Cloud |
6364
| [AutoDock Vina virtual screening](virtual_screening_vina/) | Parallel molecular docking and ranked result aggregation | You want to screen many ligands against a protein target |
6465
| [vLLM evaluation leaderboard](vllm_lm_eval_leaderboard/) | Parallel model evaluation and final CSV/Markdown aggregation | You need to compare multiple LLMs across benchmarks |
6566
| [V-Ray standalone render](vray_render/) | Rendering with a Conda-provided V-Ray executable | You need a basic standalone V-Ray bundle |

‎job_bundles/vfx_pipeline/.diagram/README.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Architecture diagram source
2+
3+
`vfx_pipeline_architecture.drawio` is the editable source for
4+
[`../../../.images/vfx_pipeline_architecture.png`](../../../.images/vfx_pipeline_architecture.png),
5+
shown at the top of the sample's [README](../README.md).
6+
7+
It is a [draw.io](https://www.drawio.com/) / diagrams.net document. Every shape
8+
and arrow is hand-placed at fixed coordinates (there is no auto-layout), so
9+
editing is deliberate: move a box by changing its `mxGeometry`, reroute an arrow
10+
by editing its waypoints.
11+
12+
## Edit
13+
14+
Open `vfx_pipeline_architecture.drawio` in the draw.io desktop app or at
15+
[app.diagrams.net](https://app.diagrams.net), or hand-edit the XML.
16+
17+
## Regenerate the PNG
18+
19+
With the draw.io desktop app installed (`brew install --cask drawio`), export
20+
headlessly from this directory:
21+
22+
```bash
23+
/Applications/draw.io.app/Contents/MacOS/draw.io \
24+
--export --format png --scale 2 --border 10 \
25+
-o ../../../.images/vfx_pipeline_architecture.png \
26+
vfx_pipeline_architecture.drawio
27+
```
28+
29+
`--scale 2` gives a crisp ~3300px-wide image. `--border 10` adds a small margin.
30+
31+
## Conventions
32+
33+
- Icons use the built-in AWS 2019 shape library (`mxgraph.aws4.*`): S3 for the
34+
bucket and shared storage, Deadline Cloud for the queue, EC2 for the worker,
35+
and the user shape for the artist workstation.
36+
- Edge colours are a legend without a legend box:
37+
blue = asset/file movement, orange = job submission, grey dashed = scheduling
38+
(control plane), purple = the on-worker step graph and publish, green = results
39+
coming home.
40+
- The main flow is one left-to-right band (workstation → S3 → queue → worker →
41+
step graph → Autodesk Flow); the two green return arrows ride a dedicated lower
42+
lane so they never cross the forward arrows.

‎job_bundles/vfx_pipeline/.diagram/vfx_pipeline_architecture.drawio‎

Lines changed: 199 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,199 @@
1+
<mxfile host="hand-authored" type="device">
2+
<diagram name="vfx-pipeline" id="vfx-pipeline">
3+
<mxGraphModel dx="1600" dy="900" grid="0" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1800" pageHeight="700" math="0" shadow="0">
4+
<root>
5+
<mxCell id="0"/>
6+
<mxCell id="1" parent="0"/>
7+
8+
<mxCell id="title" value="Sample VFX render pipeline on AWS Deadline Cloud"
9+
style="text;html=1;align=center;verticalAlign=middle;fontStyle=1;fontSize=22;fontColor=#232F3E;"
10+
vertex="1" parent="1">
11+
<mxGeometry x="450" y="16" width="900" height="40" as="geometry"/>
12+
</mxCell>
13+
14+
<mxCell id="studio" value="Studio (on-prem)"
15+
style="rounded=1;arcSize=6;whiteSpace=wrap;html=1;fillColor=#EEF6FB;strokeColor=#9CC3DB;verticalAlign=top;align=left;spacingLeft=12;spacingTop=8;fontStyle=1;fontSize=15;fontColor=#232F3E;"
16+
vertex="1" parent="1">
17+
<mxGeometry x="40" y="90" width="340" height="530" as="geometry"/>
18+
</mxCell>
19+
20+
<mxCell id="workstation" value="Artist workstation&#10;(studio-pipe)"
21+
style="sketch=0;outlineConnect=0;dashed=0;html=1;fontColor=#232F3E;fontSize=12;verticalLabelPosition=bottom;verticalAlign=top;align=center;shape=mxgraph.aws4.user;fillColor=#232F3E;"
22+
vertex="1" parent="1">
23+
<mxGeometry x="92" y="150" width="64" height="64" as="geometry"/>
24+
</mxCell>
25+
26+
<mxCell id="autodl" value="Auto-downloader&#10;(deadline queue sync-output)"
27+
style="rounded=1;arcSize=12;whiteSpace=wrap;html=1;fillColor=#FFFFFF;strokeColor=#57606A;fontColor=#232F3E;fontSize=12;strokeWidth=2;"
28+
vertex="1" parent="1">
29+
<mxGeometry x="80" y="330" width="230" height="60" as="geometry"/>
30+
</mxCell>
31+
32+
<mxCell id="storage" value="Shared storage&#10;(assets, renders)"
33+
style="sketch=0;outlineConnect=0;dashed=0;html=1;fontColor=#232F3E;fontSize=12;verticalLabelPosition=bottom;verticalAlign=top;align=center;shape=mxgraph.aws4.file_storage;fillColor=#7AA116;"
34+
vertex="1" parent="1">
35+
<mxGeometry x="163" y="490" width="64" height="64" as="geometry"/>
36+
</mxCell>
37+
38+
<mxCell id="aws" value="AWS"
39+
style="rounded=1;arcSize=4;whiteSpace=wrap;html=1;fillColor=#EEF3F8;strokeColor=#B4C4D4;verticalAlign=top;align=left;spacingLeft=12;spacingTop=8;fontStyle=1;fontSize=15;fontColor=#232F3E;"
40+
vertex="1" parent="1">
41+
<mxGeometry x="460" y="90" width="1100" height="530" as="geometry"/>
42+
</mxCell>
43+
44+
<mxCell id="bucket" value="Amazon S3&#10;(shot attachments +&#10;Conda channel)"
45+
style="sketch=0;html=1;verticalLabelPosition=top;labelBackgroundColor=none;verticalAlign=bottom;align=center;outlineConnect=0;fontColor=#232F3E;fontSize=11;shape=mxgraph.aws4.resourceIcon;resIcon=mxgraph.aws4.s3;fillColor=#7AA116;gradientColor=#4D852F;strokeColor=#ffffff;"
46+
vertex="1" parent="1">
47+
<mxGeometry x="506" y="272" width="68" height="68" as="geometry"/>
48+
</mxCell>
49+
50+
<mxCell id="farm" value="AWS Deadline Cloud farm"
51+
style="rounded=1;arcSize=4;whiteSpace=wrap;html=1;fillColor=#EAF3FB;strokeColor=#A9C7E0;verticalAlign=top;align=left;spacingLeft=12;spacingTop=8;fontStyle=1;fontSize=14;fontColor=#232F3E;"
52+
vertex="1" parent="1">
53+
<mxGeometry x="600" y="110" width="920" height="400" as="geometry"/>
54+
</mxCell>
55+
56+
<mxCell id="queue" value="Queue"
57+
style="sketch=0;html=1;verticalLabelPosition=bottom;labelBackgroundColor=none;verticalAlign=top;align=center;outlineConnect=0;fontColor=#232F3E;fontSize=12;shape=mxgraph.aws4.resourceIcon;resIcon=mxgraph.aws4.deadline_cloud;fillColor=#ED7100;gradientColor=#C8500A;strokeColor=#ffffff;"
58+
vertex="1" parent="1">
59+
<mxGeometry x="638" y="272" width="68" height="68" as="geometry"/>
60+
</mxCell>
61+
62+
<mxCell id="fleet" value="Fleet"
63+
style="rounded=1;arcSize=6;whiteSpace=wrap;html=1;fillColor=#F3F7FB;strokeColor=#C2D4E4;verticalAlign=top;align=left;spacingLeft=10;spacingTop=6;fontStyle=1;fontSize=13;fontColor=#232F3E;"
64+
vertex="1" parent="1">
65+
<mxGeometry x="768" y="150" width="730" height="320" as="geometry"/>
66+
</mxCell>
67+
68+
<mxCell id="worker" value="Worker"
69+
style="sketch=0;html=1;verticalLabelPosition=bottom;labelBackgroundColor=none;verticalAlign=top;align=center;outlineConnect=0;fontColor=#232F3E;fontSize=12;shape=mxgraph.aws4.resourceIcon;resIcon=mxgraph.aws4.ec2;fillColor=#ED7100;gradientColor=#C8500A;strokeColor=#ffffff;"
70+
vertex="1" parent="1">
71+
<mxGeometry x="800" y="272" width="68" height="68" as="geometry"/>
72+
</mxCell>
73+
74+
<mxCell id="steps" value="Worker session (Open Job Description)"
75+
style="rounded=1;arcSize=6;whiteSpace=wrap;html=1;fillColor=#F4EEFC;strokeColor=#C9B6EC;verticalAlign=top;align=left;spacingLeft=10;spacingTop=6;fontStyle=1;fontSize=13;fontColor=#232F3E;"
76+
vertex="1" parent="1">
77+
<mxGeometry x="920" y="200" width="560" height="200" as="geometry"/>
78+
</mxCell>
79+
80+
<mxCell id="stage" value="Conda queue env&#10;installs Blender + add-on&#10;(cached per host)"
81+
style="rounded=1;arcSize=14;whiteSpace=wrap;html=1;fillColor=#FFF3E8;strokeColor=#ED7100;fontColor=#232F3E;fontSize=10;strokeWidth=2;spacingLeft=2;spacingRight=2;"
82+
vertex="1" parent="1">
83+
<mxGeometry x="934" y="273" width="124" height="66" as="geometry"/>
84+
</mxCell>
85+
86+
<mxCell id="render" value="Render frames&#10;(one task / frame)"
87+
style="rounded=1;arcSize=14;whiteSpace=wrap;html=1;fillColor=#FFFFFF;strokeColor=#8C4FFF;fontColor=#232F3E;fontSize=12;strokeWidth=2;"
88+
vertex="1" parent="1">
89+
<mxGeometry x="1078" y="280" width="116" height="52" as="geometry"/>
90+
</mxCell>
91+
92+
<mxCell id="thumb" value="Make thumbnail"
93+
style="rounded=1;arcSize=14;whiteSpace=wrap;html=1;fillColor=#FFFFFF;strokeColor=#8C4FFF;fontColor=#232F3E;fontSize=12;strokeWidth=2;"
94+
vertex="1" parent="1">
95+
<mxGeometry x="1214" y="224" width="120" height="52" as="geometry"/>
96+
</mxCell>
97+
98+
<mxCell id="movie" value="Make preview movie"
99+
style="rounded=1;arcSize=14;whiteSpace=wrap;html=1;fillColor=#FFFFFF;strokeColor=#8C4FFF;fontColor=#232F3E;fontSize=12;strokeWidth=2;"
100+
vertex="1" parent="1">
101+
<mxGeometry x="1214" y="336" width="120" height="52" as="geometry"/>
102+
</mxCell>
103+
104+
<mxCell id="publish" value="Publish for review"
105+
style="rounded=1;arcSize=14;whiteSpace=wrap;html=1;fillColor=#FFFFFF;strokeColor=#8C4FFF;fontColor=#232F3E;fontSize=12;strokeWidth=2;"
106+
vertex="1" parent="1">
107+
<mxGeometry x="1356" y="280" width="110" height="52" as="geometry"/>
108+
</mxCell>
109+
110+
<mxCell id="flow" value="Autodesk Flow&#10;/ ShotGrid"
111+
style="rounded=1;arcSize=12;whiteSpace=wrap;html=1;fillColor=#FFFFFF;strokeColor=#57606A;fontColor=#232F3E;fontSize=12;strokeWidth=2;"
112+
vertex="1" parent="1">
113+
<mxGeometry x="1650" y="280" width="120" height="52" as="geometry"/>
114+
</mxCell>
115+
116+
<mxCell id="e_rw" value="reads / writes"
117+
style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;dashed=1;strokeColor=#7D8590;strokeWidth=2;startArrow=none;endArrow=none;fontSize=11;fontColor=#4D5358;labelBackgroundColor=#EEF6FB;exitX=0;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;"
118+
edge="1" parent="1" source="workstation" target="storage">
119+
<mxGeometry x="0.7" relative="1" as="geometry">
120+
<Array as="points"><mxPoint x="58" y="182"/><mxPoint x="58" y="522"/></Array>
121+
<mxPoint as="offset"/>
122+
</mxGeometry>
123+
</mxCell>
124+
125+
<mxCell id="e_write" value="write renders"
126+
style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#3F8624;strokeWidth=2;endArrow=block;fontSize=11;fontColor=#3F8624;labelBackgroundColor=#EEF6FB;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=0;entryDx=0;entryDy=0;"
127+
edge="1" parent="1" source="autodl" target="storage">
128+
<mxGeometry relative="1" as="geometry"/>
129+
</mxCell>
130+
131+
<mxCell id="e1" value="1. upload shot"
132+
style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#146EB4;strokeWidth=3;endArrow=block;fontSize=12;fontColor=#146EB4;labelBackgroundColor=#EEF3F8;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.35;entryDx=0;entryDy=0;"
133+
edge="1" parent="1" source="workstation" target="bucket">
134+
<mxGeometry relative="1" as="geometry">
135+
<Array as="points"><mxPoint x="470" y="182"/><mxPoint x="470" y="296"/></Array>
136+
</mxGeometry>
137+
</mxCell>
138+
139+
<mxCell id="e2" value="2. submit job"
140+
style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#C8500A;strokeWidth=3;endArrow=block;fontSize=12;fontColor=#C8500A;labelBackgroundColor=#EEF3F8;exitX=1;exitY=0.2;exitDx=0;exitDy=0;entryX=0.5;entryY=0;entryDx=0;entryDy=0;"
141+
edge="1" parent="1" source="workstation" target="queue">
142+
<mxGeometry relative="1" as="geometry">
143+
<Array as="points"><mxPoint x="672" y="163"/></Array>
144+
</mxGeometry>
145+
</mxCell>
146+
147+
<mxCell id="e3" value="3. schedule job"
148+
style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;dashed=1;strokeColor=#7D8590;strokeWidth=2;endArrow=block;fontSize=11;fontColor=#4D5358;labelBackgroundColor=#EAF3FB;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;"
149+
edge="1" parent="1" source="queue" target="worker">
150+
<mxGeometry relative="1" as="geometry"><mxPoint as="offset" y="-10"/></mxGeometry>
151+
</mxCell>
152+
153+
<mxCell id="e4" value="4. pull shot + Conda packages"
154+
style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#146EB4;strokeWidth=3;endArrow=block;fontSize=12;fontColor=#146EB4;labelBackgroundColor=#EAF3FB;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0.5;entryY=0;entryDx=0;entryDy=0;"
155+
edge="1" parent="1" source="bucket" target="worker">
156+
<mxGeometry x="0.55" relative="1" as="geometry">
157+
<Array as="points"><mxPoint x="614" y="306"/><mxPoint x="614" y="248"/><mxPoint x="834" y="248"/></Array>
158+
<mxPoint as="offset"/>
159+
</mxGeometry>
160+
</mxCell>
161+
162+
<mxCell id="e_runs" value="enters env,&#10;then runs steps"
163+
style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;dashed=1;strokeColor=#7D8590;strokeWidth=2;endArrow=block;fontSize=11;fontColor=#4D5358;labelBackgroundColor=#F3F7FB;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;"
164+
edge="1" parent="1" source="worker" target="stage">
165+
<mxGeometry x="0.3" relative="1" as="geometry"><mxPoint as="offset" y="-28"/></mxGeometry>
166+
</mxCell>
167+
168+
<mxCell id="s1" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#8C4FFF;strokeWidth=2;endArrow=block;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;" edge="1" parent="1" source="stage" target="render"><mxGeometry relative="1" as="geometry"/></mxCell>
169+
<mxCell id="s2" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#8C4FFF;strokeWidth=2;endArrow=block;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;" edge="1" parent="1" source="render" target="thumb"><mxGeometry relative="1" as="geometry"/></mxCell>
170+
<mxCell id="s3" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#8C4FFF;strokeWidth=2;endArrow=block;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;" edge="1" parent="1" source="render" target="movie"><mxGeometry relative="1" as="geometry"/></mxCell>
171+
<mxCell id="s4" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#8C4FFF;strokeWidth=2;endArrow=block;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0.5;entryY=0;entryDx=0;entryDy=0;" edge="1" parent="1" source="thumb" target="publish"><mxGeometry relative="1" as="geometry"/></mxCell>
172+
<mxCell id="s5" style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#8C4FFF;strokeWidth=2;endArrow=block;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0.5;entryY=1;entryDx=0;entryDy=0;" edge="1" parent="1" source="movie" target="publish"><mxGeometry relative="1" as="geometry"/></mxCell>
173+
174+
<mxCell id="e_pub" value="publish review"
175+
style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;dashed=1;strokeColor=#8C4FFF;strokeWidth=2;endArrow=block;fontSize=11;fontColor=#8C4FFF;labelBackgroundColor=#EEF3F8;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;"
176+
edge="1" parent="1" source="publish" target="flow">
177+
<mxGeometry x="-0.2" relative="1" as="geometry"><mxPoint as="offset"/></mxGeometry>
178+
</mxCell>
179+
180+
<mxCell id="e5" value="5. upload outputs"
181+
style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;dashed=1;strokeColor=#3F8624;strokeWidth=3;endArrow=block;fontSize=12;fontColor=#3F8624;labelBackgroundColor=#EAF3FB;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=1;entryDx=0;entryDy=0;"
182+
edge="1" parent="1" source="worker" target="bucket">
183+
<mxGeometry relative="1" as="geometry">
184+
<Array as="points"><mxPoint x="834" y="450"/><mxPoint x="540" y="450"/></Array>
185+
</mxGeometry>
186+
</mxCell>
187+
188+
<mxCell id="e6" value="6. download outputs"
189+
style="edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#3F8624;strokeWidth=3;endArrow=block;fontSize=12;fontColor=#3F8624;labelBackgroundColor=#EEF3F8;exitX=0;exitY=0.7;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;"
190+
edge="1" parent="1" source="bucket" target="autodl">
191+
<mxGeometry relative="1" as="geometry">
192+
<Array as="points"><mxPoint x="420" y="320"/><mxPoint x="420" y="360"/></Array>
193+
</mxGeometry>
194+
</mxCell>
195+
196+
</root>
197+
</mxGraphModel>
198+
</diagram>
199+
</mxfile>

0 commit comments

Comments
 (0)