[8.0] feat: PoolCE takes care of RAM requirements

[fstagni]

BEGINRELEASENOTES

*WMS
NEW: InProcessComputingElement takes RAM requirements into consideration for jobs' matching
NEW: PoolComputingElement takes RAM requirements into consideration when subdividing the pool
NEW: SingularityComputingElement can enforce CG2 RAM limits

ENDRELEASENOTES

• This does not change anything at the matching level, where a mechanism using Tags was already present. It should anyway be checked --
  • this line seems wrong
• I have kept for the moment the definition of MaxRAM in GBs. Maybe MBs is better? The value has to be an integer
• This is a large PR for v8, and I am considering moving it to v9. What do you think?
• check for TS -> WMS propagation of requirements

[aldbr]

I might have overlooked something, do we specify a default RAM value for the jobs?

Because if I understand correctly, getRAMInUse() depends on the value returned by _getMemoryForJobs(), which would be 0 if the tag *GB or *GB_MAX is not specified (?).
So if we have n jobs with no "RAM" tag, we will always have getRAMInUse = 0.

[fstagni]

There's no obvious default value for RAM (for processors we specify 1 which is a slightly more obvious default value). So, yes, as it stands the default value for RAM is 0.

[atsareg]

RAM and NumberOfProcessors should be treated in exactly the same way. These are resources with integer values, having default values and summing up for jobs on the same node. So, may be common methods (or class) can be introduced for those, may be other tags can be added eventually, e.g. disk space.

[chrisburr]

Having RAM in GBs isn't fine enough for an integer value, having less than 1GB per core is sensible on some systems and jobs.

[chrisburr]

Could this be made MB instead?

[sfayer]

Hi,

I've done a quick review: Please let me know if you need anything else from me on this. (I didn't quite hear what you said about it in the most recent BiLD meeting: there was some local noise just as you were talking about it).

The other CEs (singularity/inprocess) should probably set the same memory setting based on the tags too if they've not already been set by PoolCE for cases where the PoolCE isn't in use.

Regards,
Simon

[fstagni]

This PR is replaced by #8366 which targets v9.0. As said, these are a bit too many changes for 8.0 branch.