Remove base/beefy `spack.io/node-pool` nodeSelectors

[mvandenburgh]

When I was debugging resource requests for #1219, I kept running into an issue where I had to flip the node selector between base and beefy depending on the amount of CPU and memory I was requesting. This is because of the spack.io/node-pool nodeSelector, which is used to specify what Karpenter NodePool a pod should be scheduled on. While this is useful for scheduling job pods based on microarchitecture, it just makes things confusing for other pods that don't have that requirement (i.e. pods in the base, beefy, and gitlab nodepools), and its usage for those pods is just an artifact from before we switched to Karpenter.

Not only will removing this nodeSelector make the infrastructure less complex, but it will also likely save some money on AWS. This is because, for non-gitlab/runner pods for example, Karpenter will have more flexibility for scheduling pods beyond just whether it needs a "base" or "beefy" node; it will be able to select an EC2 instance that minimally satisfies resource requirements without being limited to what the nodeSelector is set to.

Steps to do this -

• Ensure all affected pods in the cluster have resource requests
• Remove spack.io/node-pool nodeSelector from those pods
• Remove node.kubernetes.io/instance-type requirement from base NodePool
  • We want to allow Karpenter to select the instance type it believes is best without limiting it
• Remove beefy NodePool (and maybe rename base to default or something?)

[jjnesbitt]

I think this is a great idea, although I'd like to make sure any concerns we had originally aren't forgotten. Do you (and/or @zackgalbreath) know what reasons we had for separating them in the first place?

For example, the gitlab node pool only selects amd64arch, the beefy node pool only selects the r5.2xlarge instance type (very specific?), and the base node pool selects only the t3.small, t3.medium, and m4.xlarge instance types.

I don't quite recall our reasoning for doing so. Maybe it just got caught up in all the work we already had to do for the micro-architecture selection?

[mvandenburgh]

Last I knew, GitLab only supports amd64. So removing the gitlab nodepool would mean having to override every single pod managed by the GitLab helm chart to have a kubernetes.io/arch=amd64 label (which is maybe fine, as we currently override every pod's nodeSelector to be gitlab)

But for base and beefy, I believe the only reason we separated them was because cluster-autoscaler did not provide this nodepool capability. So, we implemented a custom version of it by using a combination of nodeSelector and hardcoded lists of specific instance types that satisfy certain resource requirements without being too expensive. Then, when we switched to Karpenter, we never removed them.

That's my understanding, at least. @zackgalbreath, please correct me if I'm wrong on any of those points.

[zackgalbreath]

That's my recollection as well. When we were using CA it was a fairly involved process to add support for additional instance types, so we always had a limited set. When we switched to Karpenter, I believe we simply copied the configuration for base and beefy from how they were defined in the CA days, rather than making them more general.