Cannot move Namespace into a Rancher Project if it were deployed via Fleet

[jamescooke-xyz]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

For any Namespaces created as part of a Fleet deployment, after being moved to a Rancher Project (manually or otherwise) the next time Fleet reconciles, it will revert this move.

In theory we could set the project label/annotation in the fleet deployment, but in practice this is not viable. We will be deploying to hundreds of downstream clusters which will have different ids for the 'target' project.

Expected Behavior

Either:

• labels and annotations not specified in fleet should not be deleted by fleet when it reconciles, or
• add a mechanism to ignore certain labels and annotations for fleet deployments to allow us control this, or
• provide a mechanism to move a namespace into a project using a displayName or predetermined label

Note:

• unlike spec changes, labels and annotation changes don't trigger drift detection so don't seem to be considered 'desired state' per say (but they are reverted when the next reconciliation occurs)
• certain labels and annotations remain, but these are hardcoded (e.g. Keep the kubernetes.io/metadata.name)
• this issue stretched beyond just Rancher Projects, as labels and annotations are metadata and may be used for other reasons that cannot always be predetermined

Steps To Reproduce

1. Deploy a namespace via fleet
2. Move the namespace into a Rancher Project
3. Force Reconcile of the GitRepo

Environment

- Architecture: amd64
- Fleet Version: 0.11.4 (default version with Rancher 2.10.3)
- Cluster:
  - Provider: EKS
  - Options: 2 nodes, basic configuration
  - Kubernetes Version: 1.30.14

Logs

Anything else?

No response

[kkaempf]

This is not (yet) expected to work. Added to Rancher 2.15 backlog.

[EdKingscote]

@jamescooke-xyz

I have adopted the pattern of creating projects with known IDs whereby they have the clustername embedded, which then allows targetCustomizations to set the matching annotations easily. (e.g. p-cluster1-myproject p-cluster2-myproject) - the displayName for the project is the same across all clusters (but it doesn't have to be)

There are a few issues I've worked around :

1. Fleet/Helm creating all the projects happens on the local cluster, and a good chunk of cleverness is needed if different projects are needed in different clusters/cluster sets to handle things - thankfully helm and some API calls has enabled me to manage what I need - although I'm at <10 clusters and <10 projects right now, I have a chart/values structure that works (albeit not as DRY as I'd like). I'm not in front of my work machine right now, but I'm pretty sure I pull a list of clusters and iterate on them and then match the config to clustername for alignment with the config. I also create the projectroletemplatebindings here too.

2. Bug: namespaceLabels/namespaceAnnotations targetCustomizations off-target #4210 and Feature Request: Support ClusterLabels for namespaceLabels in fleet.yaml #4140 - For third party charts my workaround right now is umbrella charts that handle the namespace creation and allow setting of the appropriate annotations as necessary. For stuff we are developing, it is simply taken care of in the charts for our apps. Even if 4210 was fixed, the values would still have to be set manually as a targetCustomization for every cluster, which doesn't scale very well, so the workaround I have still covers both bases. I know I force helm takeOwnership inside fleet in most places, and I think it is part of making this all work!

3. When using the internal Rancher cluster ID (${ get .ClusterLabels "management.cattle.io/cluster-name" }) in fleet to populate the annotation, role bindings are also synchronized as expected.

[tomitesh]

@kkaempf
We are having the same problem, but with finalizers. We are operating on a resource added by fleet and adding a finalizer, but fleet is deleting that finalizer. We cannot control all source objects (to add this finalizer onto all resources in git) so we need a fix for this, or workaround, or we will have to look for an alternate to using fleet for our GitOps deployments.