raycluster: replace workers stuck terminating

[1fanwang]

Why are these changes needed?

When a worker pod is stuck Terminating, KubeRay still counts it toward the desired replica total. A worker group can remain below its requested active capacity until someone removes the terminating pod manually.

The reconciler now excludes pods with a deletion timestamp from worker health checks and replica accounting. Kubernetes continues deleting the old pod while KubeRay creates the missing active worker.

Related issue number

Fixes #4078

Labels

• No doc-updates-required label is needed because this restores existing reconciliation behavior.
• No breaking-change label is needed.

Checks

• make test WHAT=./controllers/ray passes.
• Testing Strategy
  • Unit tests
  • Manual tests on kind

Manual test instructions

With a one-worker RayCluster and the operator running outside the kind cluster:

NS=kuberay-4078-live
CLUSTER=terminating-worker-live
WORKER=$(kubectl get pod -n "$NS" \
  -l "ray.io/cluster=$CLUSTER,ray.io/node-type=worker" \
  -o jsonpath='{.items[0].metadata.name}')
kubectl patch pod -n "$NS" "$WORKER" --type=merge \
  -p '{"metadata":{"finalizers":["test.kuberay.io/hold"]}}'
kubectl delete pod -n "$NS" "$WORKER" --wait=false
kubectl get pods -n "$NS" -l "ray.io/cluster=$CLUSTER" \
  -o custom-columns='NAME:.metadata.name,TYPE:.metadata.labels.ray\.io/node-type,PHASE:.status.phase,DELETING:.metadata.deletionTimestamp' \
  --no-headers
kubectl get pods -n "$NS" \
  -l "ray.io/cluster=$CLUSTER,ray.io/node-type=worker" -o json |
  jq -r '"ACTIVE_WORKERS=\([.items[] | select(.metadata.deletionTimestamp == null)] | length)\nTOTAL_WORKERS=\(.items | length)"'

Raw logs

Before:

terminating-worker-live-head-69qtz             head     Running   <none>
terminating-worker-live-workers-worker-c5bpd   worker   Running   2026-09-03T15:31:58Z
ACTIVE_WORKERS=0
TOTAL_WORKERS=1

After:

terminating-worker-live-head-69qtz             head     Running   <none>
terminating-worker-live-workers-worker-2djd9   worker   Running   <none>
terminating-worker-live-workers-worker-c5bpd   worker   Failed    2026-09-03T15:32:00Z
ACTIVE_WORKERS=1
TOTAL_WORKERS=2

The original pod remains only because the test finalizer deliberately holds deletion.