SparkApplication framework: PodsReady ignores executor state, and Stop leaves orphan executor pods across requeue cycles #11675

What happened:
Two independent bugs in the v0.18 SparkApplication integration framework (pkg/controller/jobs/sparkapplication/) which together break Kueue's requeue cycle for Spark workloads.

Bug 1 — PodsReady returns true while executors are still pending

Current implementation:

// pkg/controller/jobs/sparkapplication/sparkapplication_controller.go
func (j *SparkApplication) PodsReady(ctx context.Context, _ client.Client) bool {
    return j.Status.AppState.State == sparkv1beta2.ApplicationStateRunning
}

AppState.State == Running transitions to Running as soon as the driver is up, regardless of executor state. When executors are unschedulable (resource shortage, missing node label, etc.), PodsReady is permanently true, so waitForPodsReady.timeout never fires. The Workload sits forever with the driver running and 0 executors ready.

Bug 2 — Stop only sets spec.suspend=true, leaving orphan executor pods across requeue cycles

The framework relies on the default stopJob flow which only patches spec.suspend=true on the SparkApplication and assumes Spark Operator cleans up. In practice Spark Operator's suspend handling only kills the driver; executor pods are owned by the SparkApplication CR via a non-controller OwnerReference and do not get GC'd when the driver dies.

Result — observed across two requeue cycles:

$ kubectl get pods -n default
NAME                               READY   STATUS    RESTARTS   AGE
pythonpi-7f25689e671b40fd-exec-1   0/1     Pending   0          4m58s   <- cycle 1
pythonpi-7f25689e671b40fd-exec-2   0/1     Pending   0          4m58s   <- cycle 1
pythonpi-db71229e671def82-exec-1   0/1     Pending   0          2m3s    <- cycle 2
pythonpi-db71229e671def82-exec-2   0/1     Pending   0          2m3s    <- cycle 2

Each evict -> re-admit cycle leaks N executor pods. With waitForPodsReady.requeuingStrategy.backoffLimitCount: 5 and executor.instances: 2, the cluster accumulates 10 zombie executor pods before the Workload is finally deactivated.
What you expected to happen:

• PodsReady returns true only when the driver and all requested executors are running (or completed), so waitForPodsReady.timeout actually fires when executors cannot be scheduled.
• After eviction, Stop tears down driver and all executor pods so the next requeue cycle starts clean. After 5 cycles, 0 pods remain (not 10).

How to reproduce it (as minimally and precisely as possible):

1. Cluster with Kueue v0.18 + Spark Operator >= v2.4.0, feature gate SparkApplicationIntegration=true.
2. ClusterQueue + LocalQueue using default-flavor, with quota sufficient for the driver. Bug 2 manifests on any eviction; Bug 1 needs the executor to be unschedulable so the Workload waits for the timeout that never fires.
3. Configure Kueue with waitForPodsReady.timeout: 2m and requeuingStrategy.backoffLimitCount: 5.
4. Apply the following SparkApplication. The fake nodeSelector deliberately makes executors unschedulable so Bug 1 is reachable:

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-bug-repro
  namespace: default
  labels:
    kueue.x-k8s.io/queue-name: spark-test
spec:
  type: Python
  mode: cluster
  image: "docker.io/apache/spark:3.5.3-python3"
  imagePullPolicy: IfNotPresent
  mainApplicationFile: "local:///opt/spark/examples/src/main/python/pi.py"
  sparkVersion: "3.5.3"
  driver:
    cores: 1
    memory: "512m"
  executor:
    cores: 1
    instances: 2
    memory: "512m"
    nodeSelector:
      non-existent-node-label: "true"   # forces executors to stay Pending
  restartPolicy:
    type: Never

Observed (Bug 1): The Workload never gets a PodsReadyTimeout eviction. Driver stays Running, executors stay Pending, indefinitely. With Bug 1 patched out (PodsReady also checks executor state), Evicted=True / reason=PodsReadyTimeout fires at the configured 2-minute mark.

Observed (Bug 2) — using a local patch to make Bug 1 trigger evictions, or by manually requeuing the workload: each cycle accumulates 2 zombie executor pods. After 5 cycles -> 10 zombie pods; cycle 1's executors are still Pending when cycle 5 starts.

Anything else we need to know?:

Why these two bugs travel together

Bug 1 prevents the requeue cycle from ever firing (timeout never triggers), so Bug 2 normally does not manifest in production. They were both discovered while testing the requeue cycle end-to-end with a local workaround for Bug 1 in place.

Stop ordering matters — two "obvious" implementations are incorrect

Correct ordering: (1) spec.suspend=true (Spark Operator gracefully tears down the driver, AppState stays transitionable to Running on resume); (2) bounded poll (<= 5s) until driver pod is gone; (3) delete all executor pods. This needs to live in a framework custom Stop since JobWithCustomStop bypasses the default Suspend() + RestorePodSetsInfo() flow in pkg/controller/jobframework/reconciler.go.
Environment:

• Kubernetes version: 1.32.0
• Kueue version: v0.18
• Spark Operator version: 2.5.0