We want to limit the ability of sessions that aren't actively acquiring data to use cluster resources. To accomplish this, we would like to send dask workers that process older data to a preemptible partition so that they don't prevent sessions that are actively collecting micrographs from running. This would require some sort of logic that would change the partition used by the SLURMCluster if all images are above a certain cutoff period (i.e., images that were acquired more than 4 days ago).
We want to limit the ability of sessions that aren't actively acquiring data to use cluster resources. To accomplish this, we would like to send dask workers that process older data to a preemptible partition so that they don't prevent sessions that are actively collecting micrographs from running. This would require some sort of logic that would change the partition used by the
SLURMClusterif all images are above a certain cutoff period (i.e., images that were acquired more than 4 days ago).