[FEATURE]: Add MOAB scheduler integration

[shahzebsiddiqui]

Please describe your feature

Let's try adding moab scheduler integration with buildtest. This would require a couple of changes to codebase.

Suggest potential solution

1. We need a top-level executor declaration named moab in buildtest configuration file something like. This will define an executor name my_queue that is mapped to queue name dev.

executors:
  moab:
    my_queue:
       queue: dev

We need to update the json schema in https://github.com/buildtesters/buildtest/blob/devel/buildtest/schemas/settings.schema.json

2. We need to add keyboard for msub in the buildspec to do a simple job submission test such as sleep example

buildspecs:
  hostname_test:
    type: script
    executor: generic.moab.my_queue
    description: run sleep for 5 seconds 
    msub: ["-l nodes=32:ppn=2,pmem=1800mb,walltime=3600"]
    run: |
      sleep 5

This will require updates to the following

• Update definition.schema.json and update https://github.com/buildtesters/buildtest/blob/devel/buildtest/schemas/script.schema.json.
• Add implementation for generation of batch scheduler directives see

  buildtest/buildtest/builders/base.py

  Lines 772 to 805 in 8c4e188

  	def get_job_directives(self):
  	"""This method returns a list of lines containing the scheduler directives"""
  	lines = []
  	if self.sbatch:
  	for line in self.sbatch:
  	lines.append(f"#SBATCH {line}")
  	lines += [f"#SBATCH --job-name={self.name}"]
  	lines += [f"#SBATCH --output={self.name}.out"]
  	lines += [f"#SBATCH --error={self.name}.err"]
  	if self.bsub:
  	for line in self.bsub:
  	lines.append(f"#BSUB {line}")
  	lines += [f"#BSUB -J {self.name}"]
  	lines += [f"#BSUB -o {self.name}.out"]
  	lines += [f"#BSUB -e {self.name}.err"]
  	if self.pbs:
  	for line in self.pbs:
  	lines.append(f"#PBS {line}")
  	lines.append(f"#PBS -N {self.name}")
  	lines.append(f"#PBS -o {self.name}.o")
  	lines.append(f"#PBS -e {self.name}.e")
  	if self.cobalt:
  	for line in self.cobalt:
  	lines.append(f"#COBALT {line}")
  	lines.append(f"#COBALT --jobname={self.name}")
  	return lines

• Add core implementation for moab executor class https://github.com/buildtesters/buildtest/tree/devel/buildtest/executors and moab scheduler class https://github.com/buildtesters/buildtest/tree/devel/buildtest/scheduler. We can call it moab.py with class MoabExecutor and MoabJob.
• Also we need to add method for executor checks during configuration loadup see

  buildtest/buildtest/config.py

  Lines 168 to 180 in 8c4e188

  	def _executor_check(self):
  	"""Validate executors"""
  	if self.verbose:
  	console.print("Initiating executor check ...", style="bold blue")
  	self._validate_local_executors()
  	self._validate_slurm_executors()
  	self._validate_lsf_executors()
  	self._validate_cobalt_executors()
  	self._validate_pbs_executors()
  	self._validate_torque_executors()
  	self._validate_container_executors()

  . We need a new method to verify moab queues are up and running from ones specified in configuration file. For instance if queue dev is not enabled, we should make the executor invalid

Additional Information

No response

Post question in Slack

• I agree that I posted my question in slack before creating this issue

Is there an existing issue

• I confirm there is no existing issue for this issue