Worker pods not cleaned up upon `MPIJobEvicted` event

[shaowei-su]

If the worker pod got evicted, the entire MPIJob will run into Failed state:

status:
  conditions:
  - lastTransitionTime: "2024-08-14T19:45:39Z"
    lastUpdateTime: "2024-08-14T19:45:39Z"
    message: MPIJob xxx is created.
    reason: MPIJobCreated
    status: "True"
    type: Created
  - lastTransitionTime: "2024-08-14T19:48:02Z"
    lastUpdateTime: "2024-08-14T19:48:02Z"
    message: MPIJob xxx is running.
    reason: MPIJobRunning
    status: "False"
    type: Running
  - lastTransitionTime: "2024-08-15T04:01:42Z"
    lastUpdateTime: "2024-08-15T04:01:42Z"
    message: 1/8 workers are evicted
    reason: MPIJobEvicted
    status: "True"
    type: Failed
  replicaStatuses:
    Launcher:
      failed: 1
    Worker:
      active: 7
      failed: 1
  startTime: "2024-08-14T19:45:39Z"

However, the run policy is not honored as a result and the worker pods are kept in running state.

  runPolicy:
    backoffLimit: 1
    cleanPodPolicy: Running
    ttlSecondsAfterFinished: 10800

[Sridhar1030]

/assign

[Sridhar1030]

Root cause:

When a worker pod is evicted, updateMPIJobStatus() correctly detects the eviction and sets the JobFailed condition with reason MPIJobEvicted. However, it does not set status.completionTime. The cleanup path in syncHandler requires both isFinished() and completionTime != nil to trigger cleanUpWorkerPods(). Since completionTime is never set for the eviction case, the cleanup is permanently skipped and the remaining running worker pods are never deleted — regardless of what cleanPodPolicy is configured.

Fix:

Set completionTime (and increment the failure counter) when worker eviction is detected, matching the existing pattern in updateMPIJobFailedStatus for launcher failures. This allows the existing cleanup logic to work correctly on the next reconciliation cycle.

Changes:

pkg/controller/mpi_job_controller.go — set CompletionTime in the eviction block of updateMPIJobStatus
pkg/controller/mpi_job_controller_test.go — added two tests: one verifying CompletionTime is set on eviction detection, and another verifying worker pods are deleted on the subsequent reconcile
Will open a PR shortly.

[Sridhar1030]

Does this look right @shaowei-su ?