reproman rerun

[yarikoptic]

How should we instruct people to rerun datalad run annotated analyses using their remote resources?

The only way now is to reproman login to the remote resource, get the dataset, and do rerun, logout, fetch update, logout.

We need to brain storm how we should do it reproman way

[kyleam]

It's worse than not having the ability to rerun from the local machine. reproman run records for concurrent jobs are not compatible with datalad rerun. See the run record bullet point in de60efa (NF: orchestrators: Support concurrent jobs, 2019-05-16) and

reproman/reproman/support/jobs/orchestrators.py

Lines 978 to 989 in 7c8800e

	cmds = self.job_spec["command_array"]
	if len(cmds) == 1:
	cmd = cmds[0]
	else:
	# FIXME: Can't use unexpanded command because of unknown
	# placeholders.
	cmd = self.jobid
	for res in run_command(
	# FIXME: How to represent inputs and outputs given that
	# they are formatted per subjob and then expanded by
	# glob?

[yarikoptic]

migrated from duplicate #525:

Inspired by the successful runs of the demo workflow of #438 , it seems to be time to consider implementing a rerun functionality. Since we are not creating datalad run records (although it could have been done with command being some reproman-run shim etc - but probably too ad-hoc and inflexible? or am I wrong?), I guess we are doomed to develop reproman rerun.

Unlike datalad run, I think it would make sense to allow for reproman rerun to change (replace) some of the recorded parameters (e.g. resource, submitter, orchestrator; batch params etc), so it could be very easy to fulfill use cases such as

• reproman rerun --bp "pl=newsubject" - "do the same by on new subject(s) data"
• `reproman rerun --jp "container=fmriprep-1.6.0" - do the same by using another registered versioned container
• reproman rerun -r cloud1 --bp "pl=10:20", reproman rerun -r cloud2 --bp "pl=20:30" to run new subjects spreading across two cloud resources

or of cause

• reproman rerun --since= for a similar to pure datalad rerun reproducible execution.