Extend helm chart to improve controller availability

[MM53]

While installing the operator in our setup we did some modifications to the resources to improve the availability of the controller Pods. Since those changes might also help others, it would be great to include them in the Helm chart.

Following changes were applied:

• Add a value to set the priorityClassName of the Deployment and Jobs. This supports proper handling of scheduling and eviction in case of low resources
• Allow to configure topologySpreadConstraints of the controller Pods. This provides a more detailed control of Pod placement than affinity rules.
• Add an optional PodDisruptionBudget to avoid evicting all controller Pods at once in case there are multiple replicas.

Please let me know what you think about these changes.

[hashicorp-cla-app]

All committers have signed the CLA.

[hashicorp-cla-app]

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement

Learn more about why HashiCorp requires a CLA and what the CLA includes

_{Have you signed the CLA already but the status is still pending? Recheck it.}

[andrewseling]

@MM53 are you still interested in contributing this? I need the priorityClassName change and would like to help push this across the line if so.

[MM53]

Yes, this change would still be helpful for us.

[andrewseling]

@MM53 It seems like checks got stuck on this PR. Do you want to re-run them or open a new PR to kick them off again?

I'm told that once the checks (including CLA) all pass, this PR will be sent to the reviewing team for their approval.

You can also reach out to me at firstName.lastName @ (employer in my profile).com if you want to work through this more directly.

[MM53]

@andrewseling I updated the branch to resolve any conflicts, but I guess the missing check needs to be triggered by some reviewer since I don't have a possibility to run it.

[andrewseling]

@benashz Thanks for giving this a push! So I can set expectations with my customer, do you have any estimate for when this change might proceed to review + merge + release?