Reduce API errors during rolling deployments

[scotwells]

Problem

When the Milo API server is updated, clients occasionally receive brief 502 errors. This happens because the gateway continues sending requests to pods that are shutting down before it realizes they're no longer available.

The issue resolves itself within minutes, but it creates noisy alerts and a degraded experience for anyone making API calls during that window.

Expected Behavior

Deployments should be seamless — clients should never see errors when the API server is being updated.

Proposed Solution

Add an Envoy Gateway BackendTrafficPolicy to the existing gateway-api kustomize component alongside the HTTPRoute. This gives operators a safe operational baseline out of the box without needing to configure it themselves.

The policy should include:

• Retries on gateway errors so failed requests are transparently re-sent to a healthy pod
• Passive health checking to quickly eject unhealthy pods from the load balancer pool
• Active health checking so the gateway detects pod readiness faster than the default Kubernetes endpoint propagation

Additionally, add a PodDisruptionBudget to ensure a minimum number of API server pods remain available during voluntary disruptions (node drains, cluster upgrades).

Context

This is a recurring issue observed in staging. Most recently on 2026-04-02 where a single 502 was returned to a client calling the OpenAPI discovery endpoint during a routine image update rollout.

[scotwells]

Proposed Implementation

Add a BackendTrafficPolicy to the gateway-api kustomize component alongside the existing HTTPRoute:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: milo-apiserver
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: milo-apiserver
  retry:
    numRetries: 3
    retryOn:
      triggers:
        - gateway-error
        - connect-failure
        - reset
    perRetry:
      backOff:
        baseInterval: 100ms
        maxInterval: 1s
      timeout: 5s
  healthCheck:
    active:
      type: HTTPS
      https:
        path: /readyz
        hostname: milo-apiserver.datum-system.svc.cluster.local
      interval: 5s
      timeout: 3s
      unhealthyThreshold: 2
      healthyThreshold: 1
    passive:
      consecutive5XxErrors: 2
      consecutiveGatewayErrors: 1
      interval: 3s
      baseEjectionTime: 15s
      maxEjectionPercent: 33

Add a PodDisruptionBudget to the apiserver base:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: milo-apiserver
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: milo-apiserver

How these work together during a rollout

1. Old pod starts terminating → fails its next health check
2. Passive health check ejects it from the pool after 1 gateway error
3. Active health check confirms it's down within 5-10s, keeps it ejected
4. If a request slips through to the dying pod, retry sends it to a healthy pod — the client never sees an error
5. PDB ensures at least 2 pods remain available during voluntary disruptions

Notes

• The active health check hostname may need to be made configurable since operators patch the target namespace
• Operators can still override thresholds via kustomize patches in their infra repos