Your feedback on ESM-Tools workflow manager

[nwieters]

Dear ESM-Tools Users,

we are planning to improve the functionality and usability of the ESM-Tools workflow manager.

To understand the opportunities for improvement, we would appreciate your feedback on this topic.

In this discussion we would like to kindly asks you to provide us any kind of feedback you may have on the usage of the workflow manager with respect to the following aspects:

• Is the purpose of the workflow manager clearly defined? Is it clear to you what it is intended for?
• Usability and syntax: How do you rate the ease of use?
• Applicability: Is it suitable for your application or use-cases?
• Support, documentation, examples: Is the so far provided documentation, examples helpful?

Please describe what you have used the workflow manager for and where the main difficulties were, problems that could not be solved with workflow manager, and also things that you particularly like about it and that you think should be kept.

Your feedback is of great importance to us, as it helps us to understand the requirements you as a user have for this tool.

Thank you in advance for your cooperation.

The ESM-Tools Developer Team

[nwieters]

Dear @seb-wahl, @joakimkjellsson, @crodehacke, @a270105, @JanStreffing , @ackerlar,

we would be particularly interested in your feedback on the workflow manager.

Thank you very much

[JanStreffing]

I will be better able to provide feedback one I have used the workflow manager. So far I have stuck the the plugin manager, as I did not need to have setups more complex that a preprocessing script + a run.

I've done this simple workflow addition in multiple cases, always just by doing monkey see monkey do from Miguels first script call.

[nwieters]

I was able to run the exercise that was presented in the ESM-Tools workshop (https://github.com/esm-tools/workshops/blob/main/202207_AWI_ESM-Tools/exercises/7.5_workflow_manager.md).

I found this a good first try.

But I realized that some of the keywords that need to be set in this example were not described in the documentation (https://esm-tools.readthedocs.io/en/latest/workflow.html).

To improve the documentation you could

• explain all keywords
• distinguish between keywords for section workflow and section subjobs
• explain where to look for any info about the subjob (I found log files in the log-folder of the experiment)
• explain more clearly what kind of scripts (only bash? the example in the workshop used a Python example script?) can be used

[nwieters]

Some suggestions are also given in issue #960.

[nwieters]

Dear @esm-tools/all-users,

if you have any feedback on the use of the ESM-Tools workflow manager, please consider adding it to this discussion by the end of this week.

Thank you very much for your cooperation.

The ESM-Tools Developer Team

[nwieters]

Dear @esm-tools/all-users ,

I would like to kindly remind you to give us your feedback on the workflow manager.

If you have any requests for improvement, e.g. regarding data processing via the workflow manager or offline coupling, please feel free to add them to the discussion.
Your input will lead to an even better adaptation of this feature to the needs of the users.

Once again, thank you for your cooperation.

The ESM-Tools Developer Team

[ackerlar]

Dear @nwieters ,

So far, I used the workflow manager only for the iterative coupling between AWIESM and PISM. I am now very used to it. Additionally to the iterative coupling, I now needed a PISM postprocessing job that runs after every PISM run (additionally to the coupling job after every PISM chunk). With the help of the workshop material and the user manual, I managed to implement this into my run config very fast. Regarding your questions:

• I would say, I understand what the workflow manager is for
• the syntax is clear to me
• it is very suitable for my applications
• from my point: the iterative coupling needs examples and some additional information (e.g. that you need at least three run configs for one model setup, and you end up with three different date files). I can provide a minimal AWIESM-PISM example

Best wishes!

[nwieters]

Dear @ackerlar ,

Thank you for your feedback. We will take it into consideration.

To expand the number of examples, a minimal example for iterative coupling would be very helpful.
Thank you very much.

The ESM-Tools Developer Team

[ackerlar]

It's a bit late, but another point would be to make esm_tools somehow "remember" which jobs or subjobs have been run already. For example, in my coupled AWIESM-PISM setup I sometimes run into the situation where I have to resubmit the first year of an AWIESM chunk (because it broke for some reason), and then the couple_in job is rerun (because it is triggered by prepcompute). However, the couple_in alters certain restart files that already have been altered, leading to inconsistencies between them leading to a model crash.

So in my particular case it would be nice if I could resubmit an AWIESM year without resubmitting the couple_in subjob if it has already run.

I guess it is straightforward to include this into my coupling script without modifying the èsm_tools` but maybe similar issues apply also to other people.

[nwieters]

Hi @ackerlar,
thanks again for your feedback.

Can you please send me your AWIESM-PISM workflow? Then I can better understand your suggestions about the couple_in step and also add this into the documentation.

Thank you.

[ackerlar]

I created a PR that includes PISM coupling on albedo and example runscripts #1048

[seb-wahl]

To be honest: I implemented my post processing for NEMO and ECHAM6 once based on an example provided by @mandresm when the workflow manager was introduced and never touched it again.
From reading through the examples and the manual of the workflow manager I get the feeling that it supersedes the plugin-manager that we use to trigger the OIFS preprocessing. Since the plugin-manager is very confusing (installs additional python code just to execute a shell script) I would be grateful for an example on how to handle everything with the workflow manager. Would a preprocessing script be correctly triggered with the settings below:

# Pre processing
                        workflow:
                                next_run_triggered_by: tidy # what does this entry actually do?
                                subjobs:
                                        oifs_preprocessing:
                                                batch_or_shell: shell
                                                order_in_cluster: sequential
                                                run_on_queue: None
                                                run_after: copy_files_to_work
                                                script_dir: ${general.esm_function_dir}/setups/focioifs/
                                                submit_to_batch_system: False
                                                script: "
                                                   oifs_preprocessing.sh option1 option2 option3"
                                                nproc: 1

Another thing: both the prepcompute_recipe: (do you ever change that?) and the workflow manager control the workflow of a run including post processing. prepcompute_recipe: obviously allows fine-grained control on how a run is prepared, while the workflow manager is somehow embracing the prepcompute_recipe:, right. Isn't that in a way two approaches for the same thing?
Or is the

[nwieters]

Hi @seb-wahl,
thank you for your feedback.
The way I understand it so far is that the plugin manager and the workflow manager are not two approaches of the same thing. But they are both possible entry points to implement additional processing tasks by the user. One by adapting a certain recipe and the other by integrating an additional process in the workflow.
From your feedback I understand that you would prefer a solution within the workflow manager, if your preprocessing task (in the example) can be done with it.
Thank you for your example. I will have a look at it, and try to run it. Maybe we can also think about a different syntax from your example.

[seb-wahl]

The example will probably not work, because I didn't test it. I just put what I think should do the job.