What happened:
When a federated Job fails in a member cluster, the aggregated status written to the control plane is rejected by the apiserver and the Job never reaches a terminal state on the control plane.
helper.ParsingJobStatus (pkg/util/helper/job.go) builds a fresh JobStatus and appends JobFailed when any member reports failure:
if len(jobFailed) != 0 {
newStatus.Conditions = append(newStatus.Conditions, batchv1.JobCondition{
Type: batchv1.JobFailed,
Status: corev1.ConditionTrue,
...
})
}
It never copies temp.Conditions from the member status, so the FailureTarget condition the member cluster reported is discarded. Kubernetes' Job status validation requires FailureTarget to be present before Failed=True, so the update is refused:
Job.batch "<name>" is invalid: status.conditions: Invalid value: cannot set Failed=True
condition without the FailureTarget=true condition
The result is a Job that is finished in the member cluster but permanently non-terminal on the control plane. Two consequences follow, and the second is the expensive one:
ttlSecondsAfterFinished never fires for it, so failed Jobs accumulate on the control plane indefinitely.
- Anything that counts "active" Jobs from the control plane (our dispatcher, and equally kube-state-metrics or a KEDA-style scaler) counts a dead Job as alive forever.
Note the asymmetry with the success path, which is what makes this look like an oversight rather than a design choice: the same function does handle JobSuccessCriteriaMet for the Complete path, added alongside the completion condition, go if len(status) > 0 && successCriteriaMetClusters == len(status) { newStatus.Conditions = append(newStatus.Conditions, batchv1.JobCondition{ Type: batchv1.JobSuccessCriteriaMet, ... }) } , but there is no equivalent for FailureTarget on the failure path.
What you expected to happen:
A Job that failed in its member cluster should reach Failed=True on the control plane, with the FailureTarget condition present so apiserver validation accepts the update, mirroring how SuccessCriteriaMet is emitted for the completion path.
How to reproduce it (as minimally and precisely as possible):
- Register at least one member cluster.
- Propagate a
batch/v1 Job that is guaranteed to fail, with backoffLimit: 0, e.g. a container running sh -c 'exit 1'.
- Watch the member cluster: the Job ends with conditions
[FailureTarget, Failed], as expected.
- Watch the control plane:
kubectl --kubeconfig karmada.config get job <name> -o yaml shows the Job with no terminal condition, and karmada-controller-manager logs the validation error above on every reconcile.
Anything else we need to know?:
Related but distinct: #7135 / PR #7138 fixed the neighbouring status.startTime immutability problem in the same function. That fix is in and works; this one is the other half of the same write: the aggregated status is still rejected when the failure path is involved.
How we observed it, on Karmada v1.17.1 with a control-plane apiserver at Kubernetes v1.35.2: with our Lua workaround removed, we ran a Job that fails. The member cluster reported [FailureTarget, Failed], the ResourceBinding on the control plane received both conditions, and the Job on the control plane was left with only {startTime} while the controller retried the rejected update in a loop.
Why it matters beyond the stuck object: anything that counts "active" control-plane Jobs treats an unfinishable Job as alive forever. In our case a scheduler subtracts active Jobs from what it may launch, so each one permanently reduced the capacity available to that queue.
Our workaround is a ResourceInterpreterCustomization with a Lua statusAggregation that ports ParsingJobStatus and additionally carries FailureTarget through from the member status. Happy to send a Go patch upstream if maintainers agree on the shape: the minimal change is to track whether any member reported FailureTarget=true and emit it before Failed, symmetric to the existing successCriteriaMetClusters handling.
Environment:
- Karmada version: v1.17.1 (checked against
master while filing: ParsingJobStatus still does not carry FailureTarget through)
- kubectl-karmada or karmadactl version: n/a (reproduced through the Karmada API directly)
- Others: control plane apiserver at Kubernetes v1.35.2; member clusters are EKS and GKE
What happened:
When a federated Job fails in a member cluster, the aggregated status written to the control plane is rejected by the apiserver and the Job never reaches a terminal state on the control plane.
helper.ParsingJobStatus(pkg/util/helper/job.go) builds a freshJobStatusand appendsJobFailedwhen any member reports failure:It never copies
temp.Conditionsfrom the member status, so theFailureTargetcondition the member cluster reported is discarded. Kubernetes' Job status validation requiresFailureTargetto be present beforeFailed=True, so the update is refused:The result is a Job that is finished in the member cluster but permanently non-terminal on the control plane. Two consequences follow, and the second is the expensive one:
ttlSecondsAfterFinishednever fires for it, so failed Jobs accumulate on the control plane indefinitely.Note the asymmetry with the success path, which is what makes this look like an oversight rather than a design choice: the same function does handle
JobSuccessCriteriaMetfor theCompletepath, added alongside the completion condition,go if len(status) > 0 && successCriteriaMetClusters == len(status) { newStatus.Conditions = append(newStatus.Conditions, batchv1.JobCondition{ Type: batchv1.JobSuccessCriteriaMet, ... }) }, but there is no equivalent forFailureTargeton the failure path.What you expected to happen:
A Job that failed in its member cluster should reach
Failed=Trueon the control plane, with theFailureTargetcondition present so apiserver validation accepts the update, mirroring howSuccessCriteriaMetis emitted for the completion path.How to reproduce it (as minimally and precisely as possible):
batch/v1 Jobthat is guaranteed to fail, withbackoffLimit: 0, e.g. a container runningsh -c 'exit 1'.[FailureTarget, Failed], as expected.kubectl --kubeconfig karmada.config get job <name> -o yamlshows the Job with no terminal condition, andkarmada-controller-managerlogs the validation error above on every reconcile.Anything else we need to know?:
Related but distinct: #7135 / PR #7138 fixed the neighbouring
status.startTimeimmutability problem in the same function. That fix is in and works; this one is the other half of the same write: the aggregated status is still rejected when the failure path is involved.How we observed it, on Karmada v1.17.1 with a control-plane apiserver at Kubernetes v1.35.2: with our Lua workaround removed, we ran a Job that fails. The member cluster reported
[FailureTarget, Failed], theResourceBindingon the control plane received both conditions, and the Job on the control plane was left with only{startTime}while the controller retried the rejected update in a loop.Why it matters beyond the stuck object: anything that counts "active" control-plane Jobs treats an unfinishable Job as alive forever. In our case a scheduler subtracts active Jobs from what it may launch, so each one permanently reduced the capacity available to that queue.
Our workaround is a
ResourceInterpreterCustomizationwith a LuastatusAggregationthat portsParsingJobStatusand additionally carriesFailureTargetthrough from the member status. Happy to send a Go patch upstream if maintainers agree on the shape: the minimal change is to track whether any member reportedFailureTarget=trueand emit it beforeFailed, symmetric to the existingsuccessCriteriaMetClustershandling.Environment:
masterwhile filing:ParsingJobStatusstill does not carryFailureTargetthrough)