Task param space for files in a folder

[ddneilson]

Copied over from aws-deadline/deadline-cloud#447
Original submitter: https://github.com/Mathieson

---

I need my task space to be a subset of strings in a larger string array for my project. This is not currently supported, but I've found a solution that I am pretty happy with, and more official support to streamline it would be fantastic. Here is what I've done.

I've created a folder dedicated to hosting many text files. Each text file contains a slice of the list of strings I want processed. For example...

• my_job_bundle
  • my_task_values
    • 0.txt
    • 1.txt
    • 2.txt

In my template.yaml, I've defined a top-level param to specify the task_count, which will determine how many of these files I will process.

  - name: MyTaskCount
    type: INT

My parameter space defines a simple range...

    parameterSpace:
      taskParameterDefinitions:
        - name: TaskFileIndex
          type: INT
          range: "1-{{Param.MyTaskCount}}"

Then, in my script, I use Task.Param.TaskFileIndex to pull the appropriate file from that directory for the task, read its contents, and continue processing as normal.

I currently use Deadline Cloud with the Python submitter, so this workflow allows me to generate those task files dynamically in my submission script. This is extra awesome because it means I can enlist the help of more-itertools to create my ranges incredibly easily. As for local OpenJD workflow, it is also easy to generate and keep some of these files in my working area.

My hope is that this approach will become streamlined through official support, and we might be able to specify something such as...

    parameterSpace:
      taskParameterDefinitions:
        - name: TaskFile
          type: DirectoryContents
          range: "{{Param.MyTaskDirectory}}"

It would give you a path for each file found within the specified directory, or even...

    parameterSpace:
      taskParameterDefinitions:
        - name: Whatever
          type: TaskFileContents
        - name: Another
          type: TaskFileContents
      combination: "({Task.Param.Whatever},{Task.Param.Another})"

It could go so far as to give you the values found within said file and implicitly determine the file via a predictable folder structure matching up with the step name and taskParameter name. For example...

• my_job_bundle
  • task_files
    • step_name
      • Whatever
        • 0.txt
        • 1.txt
        • 2.txt
      • Another
        • 0.txt
        • 1.txt
        • 2.txt

Interested in hearing others' thoughts. Thanks!

[ddneilson]

This is a cool idea

It ties in with a "meta template" idea that I've been pondering. A constraint with the actual job template is that there can't be anything in it that defines job structure based on the contents of something local to the submitting workstation. Reason being that the job template is the thing that you submit to a render management system, and it's the backend server of that system that needs to process the template to determine job structure; that backend system doesn't have access to the submitting workstation's local environment.

So, the idea of a "meta template" is something that looks like a job template but can have directives in it to pull data from local sources. Then it needs to be "baked" into a job template to be submitted.

[sarahthompson-dev]

To piggy back on this, let's say you have a step that needs to be run multiple times, in parallel, but the only thing that changes across runs is a string. In other words, the step would consist of multiple child jobs all running in parallel on the farm.

An example might be that you have ["circle", "sqaure", "triangle"] and what the job does is to locate that object within a scene and output a render of it. All the jobs are doing the same thing but just with a different target.

Would it be feasible to add a taskParameterDefinition of type LIST where you can pass in the list of strings and it would just loop over those?

This would essentially do exactly what Mathieson is requesting, but with fewer extra steps.

Also to provide context, I am trying to replicate what Mathieson is doing, but programmatically from within an API. I'd prefer not to generate actual files so am attempting to use create_job(...) directly as shown in the examples, however i'm hitting a snag in replicating this task piece.

[ddneilson]

Hi @sarahthompson-dev . Just to clarify ...

Would it be feasible to add a taskParameterDefinition of type LIST where you can pass in the list of strings and it would just loop over those?

You can do this today with jobtemplate-2023-09:

parameterSpace: 
    taskParameterDefinitions:
    - name: StringVar
      type: STRING
      value:
      - circle
      - square
      - triangle

Are you looking for being able to provide the list ['circle', 'square', 'triangle'] as a job parameter that you could then reference as the value array of the string-typed task parameter?

[sarahthompson-dev]

Ah thank you for this info, this is great to know. I've just learned that we had explored this solution already but ran into issues due to the character limit on STRING params. I redact my request and appreciate your prompt follow-up :)

[sarahthompson-dev]

Is there any way you could add in some sort of support for chunking? In our case, we have a list of hundreds of items, but we want to be able to process them in chunks of 10. In our particular case, that means getting something like 'circle square triangle' within the data part, followed by 'pentagon hexagon heptagon' for another child job, etc

[mwiebe]

@sarahthompson-dev we've been working on chunking specifically scoped to integer task parameters in https://github.com/OpenJobDescription/openjd-specifications/blob/mainline/rfcs/0001-task-chunking.md, but I think generalizing that to STRING and PATH parameters makes good sense as a follow-up RFC. I think the representation of chunks would need to be something like JSON '["circle", "square", "triangle"]' because the strings themselves could have spaces, but I don't anticipate there being any difficulty forming an extension of that task chunking work.

[sarahthompson-dev]

Yes that makes sense. Appreciate the follow-up on this, looking forward to having it implemented :)