Feat: Add server readiness probe

[quangngotan95]

What was changed

Adds configuration for server pods readinessProbe

Why?

At the moment we only have livenessProbe and no readinessProbe for server pods.
On startup pods are getting requests right away before being ready.

Checklist

1. Closes [Bug] Missing readiness probes on web server #710

2. How was this tested:

• Helm template
• Helm test
• The changes had been running in our fork on production for few months.

3. Any docs updates needed?
   N/A

[robholland]

I think readiness probably only makes sense for the frontend(-internal) services. Other services don't receive incoming requests via a Kubernetes mechanism so readiness isn't really useful. For the frontend service there is a useful health check mechanism we can use over GRPC:

readinessProbe:
  grpc:
    port: <your-grpc-port>
    service: temporal.api.workflowservice.v1.WorkflowService

Note that we can't use a named port for GRPC probes for some reason, so it will need to be interpolated as the RPC port number is in the template.

[quangngotan95]

@robholland so you're saying for example history pods don't use k8s readinessProbe for anything and temporal control history nodes internally? We're setting 300s initial delay for history readinessProbe as a way to slow down history pods deployment 🤔

[robholland]

Yes history health is managed internally via the membership system (ringpop I think?). To slow down history deployments (which is a totally valid goal) I would use minReadySeconds on the deployment spec. Feel free to add that to this PR ;) Please do add the readinessprobe for frontend(-internal) though, I think that will be useful for people.

[quangngotan95]

Got it. We used minReadySeconds before and the while it works for the deployment.
The problem is that PodDisruptionBudget does not respect that. Hence there'll still be multiple history pods recycling when one of our node goes down; that's why we switched to use this readinessProbe and it works for both cases.