Nexus: [Draft] Expanding dynamic workflows

[kayahans]

This is a draft PR that combines two different aspects of handling dynamic workflows. Unfortunately it combines a few different directions but it is not intended to be pushed to the code. Orthogonal smaller PRs will follow including testing.

1. Abstraction on dynamic workflows: Building on Jaron's initial code, this PR implements a chained dynamic workflow.
2. Pwscf error handling: This reuses the chain dynamic workflow and tries different ways to converge QE calculations using sequential strategies. Error handling is integrated into Pwscf class, open for discussion.

Two examples provided: 1. 03_chain reproduces Jaron's examples in 02. 2. 04_pwscf_recover reruns 03_example but using electron_maxstep=2 to simulate non-converging QE calculations for each ecut value. Running that job produces the following output in Nexus log in verbose mode:

Applying user settings
    Entering ./runs/ecut_50 0
      writing input files  0 scf
    Entering ./runs/ecut_50 0
      sending required files  0 scf
      submitting job  0 scf
    Entering ./runs/ecut_50 0
      Executing:
        export OMP_NUM_THREADS=1
        mpirun -np 4 pw.x -input scf.in
      recovering failed run  0 scf
      parsed products{'errors': ['electronic_convergence'], 'job_done': True, 'nbnd': 4}
      observed decision  next_params     = {'mixing_beta': 0.48999999999999994, 'electron_maxstep': 80, 'attempt': 1, 'restart_mode': 'from_scratch'}
        params          = {'mixing_beta': 0.7, 'electron_maxstep': 2, 'attempt': 0}
        products        = {'errors': ['electronic_convergence'], 'job_done': True, 'nbnd': 4}
        status          = continue

      saved attempt
      applied patch
      wrote input files  0 scf
      reset indicators
      submitting job  0 scf

params vs next_params indicate the changes in the input parameters. It indicates that the existing QE object is recycled while input parameters are modified.

Here is how the scf_50 directory looks like after these attempts:

>> pwd
examples/dynamic_workflows/04_pwscf_recover/runs/ecut_50
>> ls
C.BFD.upf      pwscf_output   scf_attempt1   scf.err        scf.in         scf.out        scf.struct.xsf scf.struct.xyz sim_scf
>> ls scf_attempt1/
scf.err scf.in  scf.out

After ecut_50 job is completed, the script advances to ecut_70 and so on.

What type(s) of changes does this code introduce?

• New feature
• Code style update (formatting, renaming)

Does this introduce a breaking change?

• No

What systems has this change been tested on?

Checklist

• • I have read the pull request guidance and develop docs
• • [ X] This PR is up to date with the current state of 'develop'
• • Code added or changed in the PR has been clang-formatted
• • This PR adds tests to cover any new code, or to catch a bug that is being fixed
• • Documentation has been added (if appropriate)

[jtkrogel]

Thanks Kayahan, this looks great! It will take me some time to dig through this.