AI-1700 Adjust HPA & Memory Allocations

[dbarkowsky]

Summary

AI-1700

Changes

• Increased the memory allocations to the backend-services app and the temporal worker. This amount is intended to put the pods below the percent needed to trigger the scaling (which was also increased a bit). Used the current prod memory usage as a gauge, although it may rise further once given more memory to work with.
• Brought the max replicas down to 3. This was done with the concern that higher memory limits would have a greater impact on the namespace quota.

Testing

• Will see if I can test by inserting some of these configs directly. Otherwise we'll see how they operate in TEST.
  • After altering configs directly, there alerts appear resolved. No alerts related to this triggered this afternoon.

Other

I noticed there are also config yaml files at this directory: /deployments/openshift/kustomize/components/prod-resources.
They don't appear to be referenced anywhere. Are they actually picked up, or is this dead code?
I did not adjust these ones.

Checklist

By submitting this pull request, I acknowledge that I have attempted to meet the following:

• a self-review of my code
• commented code particularly in hard-to-understand areas
• corresponding changes to the documentation where required
• changes tested to the best of my ability
• no new errors or non-functional code