Created Git Jobs not inheriting node selectors from Fleet values.yaml

[charleshu-8]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

I've set the Helm chart values nodeSelector and tolerations to confine Fleet to a certain set of nodes. The deployments generated by Fleet all have these node selectors and tolerations (e.g., fleet-controller, gitjob) but whenever a new K8s job is created in order to sync a GitRepo, the job is assigned with the tolerations but not the selectors, causing it to get scheduled on untainted nodes.

Digging a little deeper, it appears that the job will inherit tolerations from the Fleet Controller:

// add tolerations from the fleet-controller deployment
jobSpec.Template.Spec.Tolerations = append(
	jobSpec.Template.Spec.Tolerations,
	fleetControllerDeployment.Spec.Template.Spec.Tolerations...,
)

(source)

but only inherits node selectors when Fleet is sharded:

nodeSelector := map[string]string{"kubernetes.io/os": "linux"}
if shardID != "" && len(strings.TrimSpace(r.JobNodeSelector)) > 0 {
	var shardNodeSelector map[string]string
	if err := json.Unmarshal([]byte(r.JobNodeSelector), &shardNodeSelector); err != nil {
		return nil, fmt.Errorf("could not decode shard node selector: %w", err)
	}

	maps.Copy(nodeSelector, shardNodeSelector)
}

(source)

I don't want to shard Fleet, so the jobs stay selector-less and aren't scheduled correctly for me.

Expected Behavior

The ability to assign node selectors to generated jobs without having the enable Fleet sharding.

Steps To Reproduce

No response

Environment

Logs

Anything else?

No response

[weyfonk]

Additional QA

Problem

Installing Fleet with a non-empty nodeSelector Helm value would constrain Fleet controller deployments, but that would not have any effect on git jobs, which pods could still be created on any node.

Solution

Git job creation now takes the fleet-controller deployment's node selector into account, and includes it into the job spec.

Testing

Engineering Testing

Manual Testing

1. Spinned up a k3d cluster
2. Patched one of its nodes with label "deploy-here": "true"
3. Installed Fleet with --set-json nodeSelector='{"deploy-here": "true"}'
4. Created a GitRepo
5. Observed that the git job pod was created on the node labeled in step 2.

Automated Testing

Unit tests covering propagation of the fleet-controller's node selector into a git job's spec.

QA Testing Considerations

When both a sharded node selector and the global nodeSelector value are set, when creating a sharded GitRepo, the created git job should inherit both node selectors (the sharded one and the global one).
However, Fleet does not check for possible incompatible or overlapping node selectors at this point.

Regressions Considerations

Node selectors set on shards should still work: when creating a sharded GitRepo, the created git job should inherit the node selector from the GitRepo's shard.