[Feature] Pin 5-Spot controller to control-plane nodes to prevent deletion deadlock

[atgreen]

Description of Problem

When 5-Spot manages ephemeral worker nodes, the controller pod can be scheduled onto one of those worker nodes. If 5-Spot then tries to delete that worker (e.g., schedule window closes), it deadlocks. The controller cannot complete the deletion because it's running on the node being deleted.

Potential Solutions

Maybe something like this?

        nodeSelector:
          node-role.kubernetes.io/control-plane: ""
        tolerations:
          - key: node-role.kubernetes.io/control-plane
            effect: NoSchedule
          - key: node-role.kubernetes.io/master
            effect: NoSchedule