remove nr of jobs warning

[mihofer]

remove max nr of jobs warning and instead use the max_materialize option in the sub file.
if more then X jobs are submitted, the user is informed that those will materialize at a slower pace to not kill the scheduler

[JoschD]

I find it very hard to decide, WHEN to do that. Because the person might run things locally. Or has another limit than the one we assume. Or is doing only a dry-run anyway. Or would like to have a different max_materialize limit entirely.
So I would leave the warning and maybe update it with the hint, that for htcondor you can set a max_materialize limit via the htc_arguments.

[mihofer]

I am not sure I completely understand the when problem. If someone is running it locally, then this warning is not applicable anyway and shouldn't be triggered either way. Dry-run with htc submission should probably trigger a warning. I don't think many know of the max_materialize option anyway, so I wouldn't expect many wanting to modify it, and if so, an aptly named global variable in the code should do the trick. We can leave the warning but I'd still add the command by default to the .sub to avoid any potential problems due to a forgotten command. The warning could be rephrased to say something à la "more than X jobs submitted. note that in order not to overload HTC, only Y will materialize at a time, but all jobs are submitted".

[JoschD]

I'd still add the command by default

This is my problem with it. What if the person does not want it to be added? That's why I would just add the hint to that it exist to the warning and let the user decide themself

[mihofer]

Then a middle ground could be to add it as default to htc_arguments. This way if one wants to disable it, one needs to add only htc_arguments=" " to overwrite it. The warning could then be rephrased to "large number of jobs detected, use of max_materialize is recommended (set by default in htc arguments to X)".

[JoschD]

Idk. If it's not a default thing in HTCondor, why should we make it a default thing? Also, what if you have more htc_arguments. If you know that max_materialize has a default, you would expect htc_arguments= {'other': arg} to keep this default, but in your suggestion it would override it. It's the eoption, add all over again.