Allow custom scheduler to be specified in NIMService and NIMPipeline

[shengnuo]

This PR adds allows NIMs to be scheduled with a custom scheduler.

• custom scheduler can be specified in .spec.services[].spec.schedulerName for NIMPipeline; and in .spec.schedulerName for NIMServices.
• Default to default-scheduler if the .schedulerName is unspecified

Scheduled with Volcano

$ kubectl get nimpipeline llama3-1b-pipeline
NAME                 STATUS   AGE
llama3-1b-pipeline   Ready    31m
$ kubectl get nimpipeline llama3-1b-pipeline -o json | jq '.spec.services[].spec.schedulerName'
{
  "type": "volcano"
}
$ kubectl get pods meta-llama3
-1b-instruct-74cc4c5c9b-dtkdn -o json | jq '.spec.schedulerName'
"volcano"

Describing the created NIM pod, we can see that the pod was scheduled with Volcano

Events:
  Type     Reason     Age                 From     Message
  ----     ------     ----                ----     -------
  Normal   Scheduled  32m                 volcano  Successfully assigned nemo/meta-llama3-1b-instruct-74cc4c5c9b-dtkdn to nim-operator-9vg0z43

Scheduled with default-scheduler

If the scheduler is unspecified, NIM operator will choose the default-scheduler as default.

$ kubectl get nimservice meta-llama3-8b-instruct -o json | jq '.spec.scheduler'
null

Describing the NIM pod, we can see that the pod was scheduled with default-scheduler

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  12m                   default-scheduler  Successfully assigned nemo/meta-llama3-8b-instruct-6d95cd589b-qp96s to nim-operator-9vg0z43

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[visheshtanksale]

Is there any value around having some global config that will set the scheduler for all the pods created by NIM Operator?
Having scheduler on the CRDs is necessary. But it is not a value thats going to change across multiple objects created using NIM Operator, so having a global config makes it easy to use

[shivamerla]

LGTM