CI jobs occasionally hitting node not ready

[BenTheElder]

My suspicion is an issue with the approach to pod disruption budget on the GKE clusters. I'm less clear on the AWS clusters.

We've seen similar problems on both.

Filing this as an umbrella to note that we're aware this has happened and to collect examples.

/sig k8s-infra
/sig testing

[BenTheElder]

GKE can't perform maintenance because the Pod Disruption Budget allows for 0 Pod evictions. Update the Pod Disruption Budget.

"view details" shows:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"policy/v1","kind":"PodDisruptionBudget","metadata":{"annotations":{},"labels":{"infra.k8s.io/instance":"gke-prow-build"},"name":"prow-pods","namespace":"test-pods"},"spec":{"maxUnavailable":0,"selector":{"matchLabels":{"created-by-prow":"true"}}}}
  creationTimestamp: "2020-09-11T17:30:42Z"
  generation: 1
  labels:
    infra.k8s.io/instance: gke-prow-build
  name: prow-pods
  namespace: test-pods
  resourceVersion: "2676853668"
  uid: cc419911-f4a7-4ea6-b995-1a037b62fd6a
spec:
  maxUnavailable: 0
  selector:
    matchLabels:
      created-by-prow: "true"
status:
  conditions:
  - lastTransitionTime: "2024-05-21T11:28:52Z"
    message: ""
    observedGeneration: 1
    reason: InsufficientPods
    status: "False"
    type: DisruptionAllowed
  currentHealthy: 102
  desiredHealthy: 0
  disruptionsAllowed: 0
  expectedPods: 0
  observedGeneration: 1