Ability to set node affinity and/or tolerations on RKE-deployed addon manifests

[Oats87]

RKE does not currently allow you to set tolerations or affinity rules on the deployment/daemonset manifests that it deploys, for components like CoreDNS, kube-dns, nginx-ingress, etc.

This is a critical feature required in order to allow for architecting robust highly available clusters when planning for minimizing noisy neighbor problems.

Related issues to this are:
#1066
#1365

[ekristen]

Big +1 to get this added ASAP. Thanks!

[olegTarassov]

+1 for us as well. We have master nodes that act as etcd + controlplane node but run our critical stuf like coredns and ingress on those 2 nodes. Easier to create Load balancing and troubleshooting.

Given this limitation, we spin cluster using RKE then modify the daemonset config to allow tolerations for etcd + controlepane and with the node_selection set, the pods lift on the master nodes. the downside, we never touch RKE after the deployments to not overwrite daemonset configs.

hope this makes it soon enough.

[deniseschannon]

With the ability to add taints to nodes adding in RKE v0.3.0 (#157), all system add-ons will now be set with a wildcard toleration for the worker nodes. The taints will be used to request that these add-ons are not scheduled onto these nodes.

By introducing/adding node selector (rancher/rancher#22447) for the remaining RKE add-ons, you can use node selectors to determine which nodes these system add-ons should be scheduled to.

With the combination of taints and node tolerations, you will be able to plan out which nodes your system add-ons should be deployed to.

I am closing this issue in favor of the other two issues listed above.

[Oats87]

@deniseschannon what issue did you keep open for this?

[Oats87]

I don't believe that rancher/rancher#22447 or #157 solve this issue, so I am reopening it.