Skip to content

Commit 0b0491b

Browse files
crowecawcawcrowecawcaw
authored
feat: add Blender wedge render job with CSV-to-task-parameters submission hook (aws-deadline#257)
* feat: add Blender wedge render job with CSV-to-task-parameters submission hook Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com> * fix: reject unsafe wedge names and tolerate extra CSV columns in expansion hook Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com> * docs: add hero image showing CSV-to-task-grid expansion Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com> --------- Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
1 parent 56e3462 commit 0b0491b

9 files changed

Lines changed: 663 additions & 0 deletions

File tree

‎job_bundles/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ Nested collections provide their own complete indexes where applicable.
2020
| [CARLA autonomous-driving simulation](autonomous_driving_carla/) | GPU container simulation, parameter sweeps, and multi-sensor capture | You want to distribute autonomous-driving scenarios |
2121
| [Blender render](blender_render/) | A compact frame-parallel DCC render job with application packages | You need a minimal Blender or general CLI-render starting point |
2222
| [Blender turntable to Flow](blender_turntable_to_flow/) | Rendering, encoding, thumbnail extraction, and publishing to Flow Production Tracking | You are building a render-to-review studio workflow |
23+
| [Blender wedge render from CSV](blender_wedge_from_csv/) | A pre-submission hook that expands CSV rows into task parameters | A spreadsheet defines your wedge variations, shot list, or parameter sweep |
2324
| [Bash CLI job](cli_job/) | Submitting a multi-line shell script with an attached data directory | You want to run an ad hoc command-line workload |
2425
| [Copy S3 prefix to job attachments](copy_s3_prefix_to_job_attachments/) | Distributed hashing and copying into content-addressable job attachment storage | Existing S3 datasets should be staged without workstation uploads |
2526
| [Custom submitters](custom_submitters/) | A collection of in-application submission interfaces, including Maya | Artist context and DCC state require a custom submission UI |

‎job_bundles/blender_wedge_from_csv/README.md‎

Lines changed: 241 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,241 @@
1+
# Blender wedge render from a CSV file
2+
3+
This job bundle renders a wedge — a set of look-development variations of the same Blender scene,
4+
one image per variation — where the variations are rows of a CSV file. Choose it when a spreadsheet
5+
defines your job's task list: wedge variations, shot lists, simulation parameter sweeps, per-asset
6+
QC checks, and similar structured data that does not fit a numeric frame range.
7+
8+
![The wedges.csv file expanded by the pre-submission hook into one render task per row, shown as a grid of the six output images.](screenshot.png)
9+
10+
## What this sample demonstrates
11+
12+
A [pre-submission hook](https://github.com/aws-deadline/deadline-cloud/blob/mainline/docs/submission-hooks.md)
13+
that lives inside the job bundle and expands the CSV into the job's task parameters at submission
14+
time, so each CSV row becomes one task on the farm. The CSV is the artist-facing interface, and the
15+
hook translates it into Open Job Description task parameters — no template editing per wedge. Unlike
16+
the workstation-wide [submission hook samples](../../submission_hooks/), the hook here is bundle-local:
17+
it ships with the job in `hooks.yaml` and applies only to this bundle's submissions.
18+
19+
## Prerequisites
20+
21+
- [Deadline Cloud CLI](https://github.com/aws-deadline/deadline-cloud) >= 0.58.0
22+
(submission hooks with template modification support), with PyYAML available
23+
to `python3` (`pip install pyyaml`)
24+
- A farm and queue with a Conda queue environment that provides the `blender`
25+
package (the default `CondaPackages` job parameter), such as the
26+
[Conda queue environment sample](../../queue_environments/)
27+
- Bundle hooks enabled once per workstation:
28+
29+
```console
30+
deadline config set settings.allow_bundle_hooks true
31+
```
32+
33+
The hook runs `python3`. On Windows, or if only `python` is on your PATH, edit
34+
the `command` in [`hooks.yaml`](hooks.yaml) accordingly.
35+
36+
## How it works
37+
38+
The wedge CSV has one row per variation:
39+
40+
```csv
41+
wedge,roughness,sun_rotation,samples
42+
mirror,0.05,20,64
43+
glossy,0.2,20,64
44+
satin,0.45,20,64
45+
matte,0.8,20,64
46+
backlit,0.2,160,64
47+
noisy_preview,0.2,20,8
48+
```
49+
50+
The job template's `RenderWedge` step declares one task parameter per CSV
51+
column, with placeholder single-value ranges, and zips them together with a
52+
[combination expression](https://github.com/OpenJobDescription/openjd-specifications/wiki/2023-09-Template-Schemas#34-parameterspacedefinition):
53+
54+
```yaml
55+
parameterSpace:
56+
combination: "(WedgeName, Roughness, SunRotation, Samples)"
57+
taskParameterDefinitions:
58+
- name: WedgeName
59+
type: STRING
60+
range: [placeholder]
61+
# ... Roughness, SunRotation, Samples ...
62+
```
63+
64+
Without the `combination` expression, OpenJD would build the cross product of
65+
all parameter values. The associative `(A, B, C, D)` form instead pairs the
66+
Nth value of every range together, which is exactly a CSV's row structure.
67+
68+
At submission time, the pre-submission hook
69+
([`scripts/expand_wedge_csv.py`](scripts/expand_wedge_csv.py), configured in
70+
[`hooks.yaml`](hooks.yaml)) receives the submission metadata as JSON on stdin,
71+
reads the CSV named by the `WedgeCsvFile` job parameter, validates it, and
72+
replaces each placeholder range with the corresponding CSV column:
73+
74+
```yaml
75+
- name: WedgeName
76+
type: STRING
77+
range: [mirror, glossy, satin, matte, backlit, noisy_preview]
78+
- name: Roughness
79+
type: FLOAT
80+
range: [0.05, 0.2, 0.45, 0.8, 0.2, 0.2]
81+
# ...
82+
```
83+
84+
The hook prints the modified template on stdout under the `template` key, and
85+
the Deadline Cloud client uses it for the CreateJob call. The bundle's
86+
`template.yaml` on disk is never modified, and the CSV itself is uploaded with
87+
the job (it is a `dataFlow: IN` path parameter) as a record of what was
88+
requested.
89+
90+
If the CSV is missing, empty, has malformed values, or duplicate wedge names,
91+
the hook exits non-zero and the submission is aborted before anything is
92+
uploaded.
93+
94+
Each task then builds the same procedural scene — a metallic Suzanne on a
95+
ground plane under a sun lamp — with that row's values applied
96+
([`scripts/render_wedge.py`](scripts/render_wedge.py)). Building the scene
97+
procedurally keeps every task fully independent and the sample self-contained;
98+
there is no `.blend` file to ship. In a production wedge the same task
99+
parameters would instead be applied to your scene file with a `--python-expr`
100+
override or a small driver script.
101+
102+
```text
103+
blender_wedge_from_csv/
104+
├── template.yaml # Job template with placeholder task parameter ranges
105+
├── hooks.yaml # Bundle hook configuration
106+
├── wedges.csv # The wedge definitions (one row per task)
107+
└── scripts/
108+
├── expand_wedge_csv.py # Pre-submission hook: CSV rows -> task parameters
109+
└── render_wedge.py # Blender script: builds the scene, renders one wedge
110+
```
111+
112+
## Run or submit
113+
114+
From this directory:
115+
116+
```console
117+
deadline bundle submit .
118+
```
119+
120+
The CLI asks for confirmation before running the bundle's hooks, then the hook
121+
reports what it expanded:
122+
123+
```text
124+
[pre-submission hook 1] Expanded 6 wedge row(s) from wedges.csv into 'RenderWedge'
125+
task parameters: mirror, glossy, satin, matte, backlit, noisy_preview
126+
```
127+
128+
To wedge your own values, edit `wedges.csv` — or keep several CSVs and pick one
129+
at submission:
130+
131+
```console
132+
deadline bundle submit . -p WedgeCsvFile=/path/to/my_wedges.csv
133+
```
134+
135+
GUI submission works too (`deadline bundle gui-submit .`); the CSV chosen in
136+
the file picker is the one the hook expands.
137+
138+
When the job finishes, download the images with:
139+
140+
```console
141+
deadline job download-output --job-id <job-id>
142+
```
143+
144+
### Run it locally
145+
146+
You can verify the full expansion + render flow without a farm, using the
147+
[Open Job Description CLI](https://github.com/OpenJobDescription/openjd-cli)
148+
and a local Blender install. Simulate what the submission hook does, writing
149+
the expanded template to a file:
150+
151+
```console
152+
echo '{"jobBundleDir": "'$PWD'"}' | python3 scripts/expand_wedge_csv.py \
153+
| python3 -c 'import json,sys; print(json.load(sys.stdin)["template"])' \
154+
> /tmp/expanded_template.yaml
155+
```
156+
157+
Then render one wedge from it:
158+
159+
```console
160+
openjd run /tmp/expanded_template.yaml --step RenderWedge \
161+
--tasks '[{"WedgeName": "noisy_preview", "Roughness": 0.2, "SunRotation": 20.0, "Samples": 8}]' \
162+
-p WedgeCsvFile=$PWD/wedges.csv \
163+
-p RenderWedgeScript=$PWD/scripts/render_wedge.py \
164+
-p OutputDir=/tmp/wedge_out
165+
```
166+
167+
Omit `--tasks` to render all six wedges sequentially.
168+
169+
## Parameters and outputs
170+
171+
| Parameter | Default | Description |
172+
|---|---|---|
173+
| `WedgeCsvFile` | `wedges.csv` | CSV with one wedge per row; required columns `wedge`, `roughness`, `sun_rotation`, `samples` (extra columns are ignored) |
174+
| `OutputDir` | `output` | Directory where the wedge images are written |
175+
| `ResolutionX` / `ResolutionY` | 960 / 540 | Render resolution in pixels |
176+
| `CondaPackages` | `blender` | Packages for a Conda queue environment to provide |
177+
178+
Each CSV row produces one image, `<OutputDir>/wedge_<name>.png`, applied as:
179+
180+
| CSV column | Applied as |
181+
|---|---|
182+
| `wedge` | Output image name, `wedge_<name>.png` (letters, digits, `_`, `.`, and `-` only) |
183+
| `roughness` | Principled BSDF roughness on the subject's material |
184+
| `sun_rotation` | Sun lamp rotation around the vertical axis, in degrees |
185+
| `samples` | Cycles sample count (denoising off, so sample wedges stay visible) |
186+
187+
## Security, cost, and cleanup
188+
189+
Bundle hooks execute local scripts from the job bundle at submission time, which is why they are
190+
disabled by default and gated behind the `settings.allow_bundle_hooks` setting plus a per-submission
191+
confirmation prompt. Review [`hooks.yaml`](hooks.yaml) and the hook script — as you should for any
192+
bundle — before enabling. The hook here reads only the wedge CSV and the bundle's own template, and
193+
modifies nothing on disk.
194+
195+
Submitting the job runs Blender render tasks on your farm's fleet and stores job attachments in your
196+
queue's S3 bucket; both are billable at your farm's normal rates. The default CSV renders six small
197+
images and completes in a few minutes on a single worker. There are no resources to clean up beyond
198+
normal job attachment lifecycle in your S3 bucket.
199+
200+
## Troubleshooting
201+
202+
| Symptom | Cause | Fix |
203+
|---|---|---|
204+
| Hooks confirmation never appears and the job has one task | Bundle hooks not enabled | `deadline config set settings.allow_bundle_hooks true` |
205+
| `expand_wedge_csv: CSV file ... missing required column(s)` | Header row does not match the expected schema | Match the column names in `wedges.csv`, or update `CSV_TO_TASK_PARAMETER` in the hook |
206+
| Submission aborts with a CSV error | Malformed value, duplicate wedge name, or empty CSV | The stderr message names the file, line, and column to fix |
207+
| Hook fails with `ModuleNotFoundError: yaml` | PyYAML not installed for `python3` | `pip install pyyaml` |
208+
| Tasks fail with `blender: command not found` | Queue has no Conda environment providing `blender` | Attach a [Conda queue environment](../../queue_environments/) or adjust `CondaPackages` |
209+
210+
Task logs are available per task in the Deadline Cloud monitor; the hook's own output appears in the
211+
submission console before upload begins.
212+
213+
## Adapting the pattern
214+
215+
To wedge different values, change all three layers together — they are coupled
216+
by name:
217+
218+
1. **CSV columns** — the artist-facing schema.
219+
2. **`CSV_TO_TASK_PARAMETER`** in `expand_wedge_csv.py` — maps each column to a
220+
task parameter name and type.
221+
3. **Task parameters** in `template.yaml` — one definition per column, all
222+
listed in the `combination` expression, plus wiring the value into the
223+
render command.
224+
225+
Notes and limits:
226+
227+
- Open Job Description allows at most 1024 values per task parameter range and
228+
16 task parameters per step; the hook enforces the former.
229+
- The hook trusts the CSV's header names, not column order, and ignores extra
230+
columns — so artists can annotate rows with notes columns freely.
231+
- Values a hook emits for `PATH`-typed *job* parameters on stdout must be
232+
absolute paths; this sample only rewrites *task* parameter ranges inside the
233+
template, which has no such restriction.
234+
235+
## Related resources
236+
237+
- [Submission hooks documentation](https://github.com/aws-deadline/deadline-cloud/blob/mainline/docs/submission-hooks.md)
238+
- [Workstation-wide submission hook samples](../../submission_hooks/) — the same mechanism deployed studio-wide via `DEADLINE_HOOKS_DIR`
239+
- [Blender turntable to Flow](../blender_turntable_to_flow/) — a bundle hook that fills job parameters from studio environment variables
240+
- [Blender render](../blender_render/) — the minimal frame-range Blender bundle this sample builds on
241+
- [OpenJD parameter space and combination expressions](https://github.com/OpenJobDescription/openjd-specifications/wiki/2023-09-Template-Schemas#34-parameterspacedefinition)

‎job_bundles/blender_wedge_from_csv/hooks.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
version: "1.0"
2+
3+
# A preSubmission hook runs as part of every submission (CLI and GUI), just
4+
# before the job is hashed, uploaded, and created. This one reads the wedge
5+
# CSV file (from the WedgeCsvFile job parameter) and expands its rows into the
6+
# RenderWedge step's task parameter ranges, then prints the modified template
7+
# on stdout. The Deadline Cloud client applies that template to the
8+
# submission, so each CSV row becomes one task on the farm.
9+
#
10+
# If the CSV is missing or malformed, the hook exits non-zero and the
11+
# submission is aborted before anything is uploaded.
12+
#
13+
# Bundle hooks are disabled by default. Enable them once with:
14+
# deadline config set settings.allow_bundle_hooks true
15+
preSubmission:
16+
- command: python3
17+
args: [scripts/expand_wedge_csv.py]
18+
timeout: 10

‎job_bundles/blender_wedge_from_csv/screenshot.png‎

293 KB
Loading

0 commit comments

Comments
 (0)