Standard resources

[kdm9]

Hello all,

Forgive what is quite possibly a stupid question, but is there a good reason why this executor does not translate/respect snakemake standard job resources into task requests? Specifically, here I mean things like cores (threads), mem_mb, runtime, etc, as happens in other executor plugins for HPC clusters (e.g. the slurm one).

I have hacked together an extremely dodgy fork of this executor which does exactly this, and it seems to work at least for the three resources above. Before I make my fork more than just a 15 minute hack, I'd like to know if this is something you considered and decided was unwise, or if it was just never attempted? If this wasn't attempted and it sounds like a good idea, should I make this something a bit more robust and try to upstream it?

best,
K

[tadasbar]

Hello,

It appears that this is not implemented. I have personally been using executor-specific resources, such as googlebatch_machine_type and googlebatch_boot_disk_gb, but from your description it seems that you can provide the Batch API with the standard resources you mentioned, and an appropriate VM will be allocated for you?

If you already have a some kind of a solution implemented, it would be great to share via a PR. I could contribute to integrating it in the codebase, as I've been working on this plugin this week fixing some other issues.

[kdm9]

It appears that this is not implemented. I have personally been using executor-specific resources, such as googlebatch_machine_type and googlebatch_boot_disk_gb, but from your description it seems that you can provide the Batch API with the standard resources you mentioned, and an appropriate VM will be allocated for you?

Not quite, no. As a hack, I have a mapping between the number of threads a job wants (job.resources["_cores"]) and the machine size for a given family, pulling a list of machines by family for the given region via the API, and matching it to the requested number of CPUs

If you already have a some kind of a solution implemented, it would be great to share via a PR. I could contribute to integrating it in the codebase, as I've been working on this plugin this week fixing some other issues.

OK, I will make a WIP PR next week, just for the purpose of conversation. As I said, what I have currently is a bit of a fragile hack, but we can talk about how to make this robust. I have also deleted a lot of functionality from the original to simplify things, which I wanted to do until I got it operational. I would plan to revert that eventually, but initially the patch will be very large.

best,
Kevin