Fleet incorrectly recreates fleet-agent-local and bootstrap secret even with Local Cluster disabled

[ElGranLoky]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

I upgraded the Fleet installation in the cattle-fleet-system namespace to version fleet:108.0.0+up0.14.0 using the following values file:

affinity: {}
agent:
  leaderElection:
    leaseDuration: 60s
    renewDeadline: 20s
    retryPeriod: 10s
  reconciler:
    workers:
      bundledeployment: '1'
      drift: '1'
  replicas: 1
agentCheckinInterval: 30m
agentImage:
  imagePullPolicy: IfNotPresent
  repository: rancher/fleet-agent
  tag: v0.14.0
agentTLSMode: system-store
apiServerCA: |-
  -----BEGIN CERTIFICATE-----
  -----END CERTIFICATE-----
apiServerURL: https://server.k3
bootstrap:
  agentNamespace: cattle-fleet-local-system
  branch: ''
  enabled: false
  namespace: ''
  paths: ''
  repo: ''
  secret: ''
controller:
  reconciler:
    workers:
      bundle: '1'
      bundledeployment: '1'
      cluster: '1'
      clustergroup: '1'
      gitrepo: '1'
      imagescan: '1'
      schedule: '1'
  replicas: 1
  disableDefaultCluster: true
  disableLocalCluster: true
debug: ''
debugLevel: 0
disableSecurityContext: false
garbageCollectionInterval: 15m
gitClientTimeout: 120s
gitjob:
  replicas: 1
gitops:
  enabled: true
  syncPeriod: 2h
global:
  cattle:
    systemProjectId: p-bd46g
helmops:
  enabled: false
  replicas: 1
ignoreClusterRegistrationLabels: false
image:
  imagePullPolicy: IfNotPresent
  repository: rancher/fleet
  tag: v0.14.0
insecureSkipHostKeyChecks: false
leaderElection:
  leaseDuration: 60s
  renewDeadline: 20s
  retryPeriod: 10s
metrics:
  enabled: true
migrations:
  clusterRegistrationCleanup: true
  gitrepoJobsCleanup: true
noProxy: 127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local
nodeSelector: {}
priorityClassName: rancher-critical
propagateDebugSettingsToAgents: true
resources: {}
tolerations:
  - effect: NoSchedule
    key: node-role.kubernetes.io/control-plane
    operator: Equal
    value: 'true'
  - effect: NoExecute
    key: node-role.kubernetes.io/etcd
    operator: Equal

After applying this configuration, the Fleet agent deployment is being reinstalled every few minutes.
Additionally, Fleet keeps triggering the bootstrap installation even though bootstrap is explicitly disabled in the configuration.

Expected Behavior

The Fleet agent should remain stable and should not be redeployed every few minutes. Once the agent is deployed and connected, it should continue running without being recreated or restarted by the system.

Steps To Reproduce

No response

Environment

- Architecture: x86-64
- Fleet Version: v0.14.0
- Cluster:
  - Provider: K3S
  - Kubernetes Version: v1.32.10 +k3s1

Logs

Agent logs with the redeploy

I1211 13:05:25.515405       1 leaderelection.go:257] attempting to acquire leader lease cattle-fleet-local-system/fleet-agent...
{"level":"info","ts":"2025-12-11T13:05:25Z","logger":"setup","msg":"new leader","identity":"fleet-agent-5f9b4c7f4c-jp8m4-1"}
I1211 13:06:28.941902       1 leaderelection.go:271] successfully acquired lease cattle-fleet-local-system/fleet-agent
{"level":"info","ts":"2025-12-11T13:06:28Z","logger":"setup","msg":"renewed leader","identity":"fleet-agent-5f9b4c7f4c-w5l4j-1"}
time="2025-12-11T13:06:28Z" level=warning msg="Cannot find fleet-agent secret, running registration"
time="2025-12-11T13:06:28Z" level=info msg="Creating clusterregistration with id '89h9gdlqrzvtl7dmvdcdwgq2nvv9r4x8zpmsl8dshzdml2fs629rsm' for new token"
time="2025-12-11T13:06:31Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-e079af68e897baf118b4f5c7cb397f3723c831f118877af044059c9ddf5b3' on management cluster for request 'fleet-local/request-l9qll': secrets \"c-e079af68e897baf118b4f5c7cb397f3723c831f118877af044059c9ddf5b3\" not found"
{"level":"info","ts":"2025-12-11T13:06:33Z","logger":"setup","msg":"successfully registered with upstream cluster","namespace":"cluster-fleet-local-local-1a3d67d0a899"}
{"level":"info","ts":"2025-12-11T13:06:33Z","logger":"setup","msg":"listening for changes on upstream cluster","cluster":"local","namespace":"cluster-fleet-local-local-1a3d67d0a899"}
{"level":"info","ts":"2025-12-11T13:06:33Z","logger":"setup","msg":"Starting controller","metricsAddr":":8080","probeAddr":":8081","systemNamespace":"cattle-fleet-local-system"}
{"level":"info","ts":"2025-12-11T13:06:33Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2025-12-11T13:06:33Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2025-12-11T13:06:33Z","msg":"starting server","name":"health probe","addr":"[::]:8081"}
{"level":"info","ts":"2025-12-11T13:06:33Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8080","secure":false}
{"level":"info","ts":"2025-12-11T13:06:33Z","logger":"setup","msg":"Starting cluster status ticker","checkin interval":"30m0s","cluster namespace":"fleet-local","cluster name":"local"}
{"level":"info","ts":"2025-12-11T13:06:33Z","msg":"Starting EventSource","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","source":"kind source: *v1alpha1.BundleDeployment"}
{"level":"info","ts":"2025-12-11T13:06:33Z","msg":"Starting EventSource","controller":"drift-reconciler","source":"channel source: 0xc0002797a0"}
{"level":"info","ts":"2025-12-11T13:06:33Z","msg":"Starting Controller","controller":"drift-reconciler"}
{"level":"info","ts":"2025-12-11T13:06:33Z","msg":"Starting workers","controller":"drift-reconciler","worker count":1}
{"level":"info","ts":"2025-12-11T13:06:33Z","msg":"Starting Controller","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment"}
{"level":"info","ts":"2025-12-11T13:06:33Z","msg":"Starting workers","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","worker count":1}
time="2025-12-11T13:08:28Z" level=warning msg="signal received: \"terminated\", canceling context..."
{"level":"info","ts":"2025-12-11T13:08:28Z","msg":"Stopping and waiting for non leader election runnables"}
{"level":"info","ts":"2025-12-11T13:08:28Z","logger":"setup","msg":"termination requested, exiting"}
{"level":"info","ts":"2025-12-11T13:08:28Z","msg":"Stopping and waiting for leader election runnables"}

Anything else?

The cluster is active and all bundles are deployed correctly.

Suddenly, the kubelet redeploys the Fleet agent deployment:

After the agent reconnects, the cluster disconnects again. A few minutes or even seconds later, the cycle repeats:

kubectl get deployment -n cattle-fleet-local-system --watch -o wide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
fleet-agent 1/1 1 1 46s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 1/1 1 1 63s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 0 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 0 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 0 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 1 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 1/1 1 1 1s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 1/1 1 1 5m34s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 0 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 0 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 0 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 1 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 1 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 0 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 0 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 0 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 1 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 1 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 0 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 0 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 0 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 1 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 1 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 0 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 0 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 0 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 0/1 1 0 0s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent
fleet-agent 1/1 1 1 1s fleet-agent rancher/fleet-agent:v0.14.0 app=fleet-agent

Maybe same problem: https://slack-archive.rancher.com/t/32726079/hi-all-since-a-few-days-we-ve-noticed-that-our-fleet-agent-i

[LIV2]

We are seeing the same issue after upgrading from Rancher 2.11.2 to 2.13.1

As with the slack conversation above we constantly see

{"level":"info","ts":"2026-01-14T23:49:24Z","logger":"clustergroup-cluster-handler","msg":"Cluster changed, enqueue matching cluster groups","namespace":"fleet-local","name":"local"}

from the controller and continual redeployment of the fleet agent

[kkaempf]

after upgrading from Rancher 2.11.2 to 2.13.1

Did you upgrade 2.11 -> 2.13 in one step or via 2.12 ?
Did you give Fleet ample time to update all fleet-agents in downstream clusters ?

[hoo29]

We are seeing the same after an upgrade from 2.11.0 -> 2.12.3 on v1.33.7+rke2r1 but only if a node is down. Similar log of
{"level":"info","ts":"2026-01-17T18:08:24Z","logger":"clustergroup-cluster-handler","msg":"Cluster changed, enqueue matching cluster groups","namespace":"fleet-local","name":"local"}

It continues until all nodes are back up.

[LIV2]

Did you upgrade 2.11 -> 2.13 in one step or via 2.12 ?

We went straight from 2.11 to 2.13

Did you give Fleet ample time to update all fleet-agents in downstream clusters ?

Yes we did, it is still happening now with continual re-deployment of the fleet agent and the same log messages claiming the cluster has changed

When fleet-agent manages to run for a little bit, I get the following message:

{"level":"error","ts":"2026-01-18T20:53:04Z","msg":"Reconciler error","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"fleet-agent-local","namespace":"cluster-fleet-local-local-1a3d67d0a899"},"namespace":"cluster-fleet-local-local-1a3d67d0a899","name":"fleet-agent-local","reconcileID":"12046980-f254-413a-a2e1-7995acc54796","error":"failed deploying bundle: cannot re-use a name that is still in use","errorCauses":[{"error":"failed deploying bundle: cannot re-use a name that is still in use"}],"stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:296"}

Edit: I fixed the bundle deployment error by removing the sh.helm.release.v1.fleet-agent-local releases from secrets and that error is gone but we still have continuous redeployment of fleet-agent and the cluster changed messages

[kkaempf]

We went straight from 2.11 to 2.13

Ouch. That's uncharted territory. 🙈

but we still have continuous redeployment of fleet-agent and the cluster changed messages

Please attach a full log here.

[LIV2]

Here are the logs

fleet-agent.log
fleet-controller-6649c4b6db-rv95z_fleet-agentmanagement.log
fleet-controller-6649c4b6db-rv95z_fleet-controller.log

[LIV2]

We rolled back to 2.11 and then tried the upgrade stepping up incrementally to 2.12.3 and have encountered the same problem

Here is the latest fleet-controller log
fleet-controller-586bdb76fd-lvk55_fleet-controller.log