New feature
When using Google Batch, it'd be nice if spot could be enabled for individual processes. The reason being, in some pipelines I may have processes that run over 1 hour that cannot be paused and restarted, running the risk of them wasting hours of compute time, or possibly never finishing at all, if the spot resources are reclaimed. I haven't experimented to see if the maxSpotAttempts configuration would prevent the fail for Google Batch, but all the same it would be nice to be able to have on-demand be able to be set as the default.
Apologies if I have missed something in the documentation about this. I did try setting google.batch.spot = true within a withName config, but that seemed to still enable it for the entire workflow.
Usage scenario
As above.
Suggest implementation
Simply as an end-user, I do not know enough about the architecture of Google Batch or NextFlow's interaction with it to know how feasible this is. Currently the implementation is at the workflow level, but it would be nice to have it as a configuration for the process, the same way that machineType works and is implemented for Google Batch processes.
Thanks!
New feature
When using Google Batch, it'd be nice if spot could be enabled for individual processes. The reason being, in some pipelines I may have processes that run over 1 hour that cannot be paused and restarted, running the risk of them wasting hours of compute time, or possibly never finishing at all, if the spot resources are reclaimed. I haven't experimented to see if the
maxSpotAttemptsconfiguration would prevent the fail for Google Batch, but all the same it would be nice to be able to have on-demand be able to be set as the default.Apologies if I have missed something in the documentation about this. I did try setting
google.batch.spot = truewithin awithNameconfig, but that seemed to still enable it for the entire workflow.Usage scenario
As above.
Suggest implementation
Simply as an end-user, I do not know enough about the architecture of Google Batch or NextFlow's interaction with it to know how feasible this is. Currently the implementation is at the workflow level, but it would be nice to have it as a configuration for the process, the same way that
machineTypeworks and is implemented for Google Batch processes.Thanks!