Question about cluster.empty_infra

[Rivendile]

Hi, I'm using shortest first scheduler in your simulator but get an output "This cluster is not large enough to run the job", which seems unreasonable as shortest first scheduler is preemptive. By digging deeper, I find that the empty_infra in _Cluster class does not reset mem of each node. I'm a little confused about this. Is this specially designed or a bug? Looking forward to your reply!

[juncgu]

@Rivendile
Thanks for trying out Tiresias. The message should mean that the cluster can not satisfy the resource requirement (especially, GPU) of the top job in the queue. Would you please check if there is any extra large (but short) job in the job file?
We assume GPU machines are always with very large main memory space. Therefore, main memory is not considered as a constraint in resource allocation.

[Rivendile]

@juncgu
Hi, I use n32g4 cluster setup, check the 60_job trace provided in this repo, and find no extra large job.
Each job is set to use 5GB memory per GPU in your code. I think that If the memory is not reset when preemption happens, the memory will be used up sooner or later.
I reset the memory in _Cluster.empty_infra, and get reasonable results.