Immutability vs. dynamic adjustments

[jvanns]

I think one of the problems we face with a JDL is the notion of deferred or lazy evaluation of parts of the job not yet realised. I.e. a job that adjusts itself or expands at dispatch time to more jobs or tasks - something we see modern DCC tools and libraries doing. This makes it hard to describe the job fully upfront (and especially its resource requests, dependencies etc.). Personally I'm a fan of immutable jobs post-submission since it makes for reliable predictability and reproducibility (for scheduling decisions, debug & diagnosis etc.). But that's not the way (some) tools are evolving! Have you had any discussions prior to this public RFC around that?

[ddneilson]

Hi James!

Great question! The immutability thing was something that we talked about a lot. "since it makes for reliable predictability and reproducibility (for scheduling decisions, debug & diagnosis etc.)." -- heavily this, but in combination with an observation that immutable jobs seems to be pretty common in existing (public) render management systems, are what tipped the scales in favour of our proposal.

I'll admit to not being as up-to-date regarding modern DCC tools increasingly adding job mutability. Would you mind pointing me at some examples so that I can do some learning?

Thanks!

[mwiebe]

This is a fantastic topic to dig deeper on. One lens we looked at it with, is, assuming we want to support both approaches, which one should we start with? We concluded that starting with expressing the structure of the job ahead of time is where to start, because it's the more constrained approach. By trying to express a variety of jobs that don't quite fit in the schema, we can learn what specific dynamic job structures we might add, and use the specification's RFC process to add incremental features that satisfy the need.

Since submitting extensions to a job is not part of the spec, very dynamic self-expanding jobs aren't something it supports in a portable way that's independent of a specific render farm. You can still do this, by including the render farm's own job submission command inside the job. When people do this, those resulting jobs will be useful to build similar abstractions into the spec itself. I hope this approach leads us to a more broadly useful spec, than if we started by trying to include it.

[pkpriority]

My experience has mostly been around optimizing farm operations and managing teams to scale farm output without scaling costs or manual labor. From that perspective, having deferred evaluation of the full scope of the tasks in a job is risky; +1 to @jvanns notes. A job that keeps unpacking scope after submission: at a high enough level looks like a long running job. As a pattern, it kinda feels more like machine reservation (if the process unpacks locally) than batch processing. It is harder to scope (I feel) than long running immutable scope jobs, because at least the tasks count is finite, and the scheduler can evaluate the maximum number of machines entangled. Depending on how mutable task/parameter scopes unpack, and when in the evaluation (say a previous step) unpacks, the scheduler can get into states that are extremely hard to make any predictions about. (I'm basically saying what @jvanns is saying but worse 😄 and from an operations lens). Very difficult for infra admins to answer "when will it be done?", "is it supposed to take this long or is it sick?", "can this be optimized?" which are key questions to answer for production management.

Let's dive a little deeper; @jvanns do you have a list of a few particular DCCs you had in mind when you mentioned that this is seems to be where things are trending? I'd love to dive into their docs about this. Maybe there are suggested job management strategies that help us think about it.

At the end of the day, there's nothing stopping a job's instructions from being defined to call the render farm to submit more jobs. But why, I wonder?

+1 to @mwiebe and @ddneilson's replies too. I love this thread. Great topic, @jvanns !

[jvanns]

Sorry - I forgot all about these discussions! FWIW, one integration that comes to mind (proprietary software aside), I think, is recent versions of Houdini that have the PDG framework where TOPs can have dynamic work items? I could be wrong about this since its been a while since I've had to work with it but IIRC this can be generative or non-deterministic beyond submission time.

[pkpriority]

@jvanns I wanted to give an update; we're trying to find a good example of this internally but it's been harder than I'd like to track down. (I was a lighter, never an FX person, so Houdini is not my forte.) If you or a good FX crew friend have a good example that is generic enough to share here, that would be super helpful. Our team are eager to play around with a working example to try and describe it in OpenJD.

[rvinluan-sidefx]

@pkpriority Check out the Houdini PDG product page -- https://www.sidefx.com/products/houdini/pdg/

There are a couple of videos near the top of the page, "Intro to PDG" and "VFX TOP Workflow" that explain what is PDG, the motivation, and the basics of how it works. But more importantly, further down the page there is a "Crowd FX" video that provides a dynamic job example.

There are other examples on the product page but I believe they are static job examples.

I hope this helps!

Cheers,
Rob

[rvinluan-sidefx]

Hi All!

Just chiming in about dynamic jobs.

We had a great discussion with the OpenJD folks about a month ago regarding this topic and they pointed out that task specifications/templates can be standalone; they do not need to be embedded in a job template. I believe this is all that is needed by the OpenJD spec in order to support dynamic jobs.

Standalone task templates means that you can submit your job template first and then at a later time submit a task template to be attached to the already running job. For example, in pseudo-code:

# Create job template.
job_template = ....

# Submit job template to the job scheduling system and get back the job id.
job_id = jobSchedAPI.submitJob(job_template)

# Do work to determine what task needs to be added to the job.
# .....

# Create task template.
task_template = ....

# Submit task template to be attached to running job.
task_id = jobSchedAPI.submitTaskToJob(job_id, task_template)

So the onus would be on the job scheduling system to add support for adding OpenJD tasks to previously submitted OpenJD jobs (i.e. jobSchedAPI.submitTaskToJob() ) in order to support dynamic jobs. That would be enough to handle jobs submitted by Houdini's PDG framework that was brought up earlier in this discussion.

Cheers,
Rob

[ddneilson]

Thanks Rob. To add to this, there's nothing in the spec right now that really says that you can't have a render manager, or render manager integration, that accepts the StepTemplate fragment of a Job Template as a way to add to an existing Job.

However, I do think that we could be more explicit about supporting this sort of workflow in a future revision of the specification -- e.g. by making a Root Element for this use-case that's similar to what we've done for Environment Templates. Basically, just have a template that's the spec revision and a step template.

Similarly, the openjd packages could support this use-case better as well. e.g. A function like create_job that ensures that the given Step Templates to append to a Job are compatible with the Job Template that created the Job.

I think that this is an interesting thread to pull on to see where it goes, so thanks for chatting with us about how Houdini PDG works and thanks to @jvanns for starting the thread.