Fleet agent installation is stuck

[manno]

The fleet agent installation/update gets stuck. This happened when the server was under heavy load.

Workaround: Delete the agent's helm secrets and restart it or force redeploy it.

{"level":"info","ts":"2025-09-16T14:04:31Z","logger":"bundledeployment.helm-deployer.install","msg":"Installing helm release","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"fleet-agent-d0-k3k-downstream001-downstream0045","namespace":"cluster-fleet-default-d0-k3k-downstream001-downstream0045-f8390"},"namespace":"cluster-fleet-default-d0-k3k-downstream001-downstream0045-f8390","name":"fleet-agent-d0-k3k-downstream001-downstream0045","reconcileID":"fd2484ac-45c5-46fb-bc4b-0d76434786e8","commit":"","dryRun":false}                                                                                                   
{"level":"error","ts":"2025-09-16T14:04:31Z","msg":"Reconciler error","controller":"bundledeployment","controllerGroup":"fleet.cattle.io","controllerKind":"BundleDeployment","BundleDeployment":{"name":"fleet-agent-d0-k3k-downstream001-downstream0045","namespace":"cluster-fleet-default-d0-k3k-downstream001-downstream0045-f8390"},"namespace":"cluster-fleet-default-d0-k3k-downstream001-downstream0045-f8390","name":"fleet-agent-d0-k3k-downstream001-downstream0045","reconcileID":"fd2484ac-45c5-46fb-bc4b-0d76434786e8","error":"failed deploying bundle: cannot re-use a name that is still in use","errorCauses":[{"error":"failed deploying bundle: cannot re-use a name that is still in use"}],"stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202"}                                                                                                

I can see helm list is in pending-upgrade:

# helm list -n cattle-fleet-system --all
NAME                                           	NAMESPACE          	REVISION	UPDATED                                	STATUS         	CHART                                                                                                                 	APP VERSION
fleet-agent-d0-k3k-downstream001-downstream0045	cattle-fleet-system	2       	2025-09-16 13:23:38.061597057 +0000 UTC	pending-upgrade	fleet-agent-d0-k3k-downstream001-downstream0045-v0.0.0+s-c80a5ad64879318fcc81259f14f38c4df7ae3e9900d7e19b9633f67c24e6f

We fixed pending-install in https://github.com/rancher/fleet/pull/4065/files

[manno]

Testing with a version that has manno/fleet@e90365d

I see the fleet-agent is deployed (manager initiated registration) and can handle bundledeployments, i.e. it deployed 50 workload bundles.
However it can't deploy its own helm chart and adopt itself. Cluster is stuck at 50/51 bundles.

{
  "level": "info",
  "ts": "2025-09-24T15:28:05Z",
  "logger": "bundledeployment.helm-deployer.install",
  "msg": "Installing helm release",
  "controller": "bundledeployment",
  "controllerGroup": "fleet.cattle.io",
  "controllerKind": "BundleDeployment",
  "BundleDeployment": {
    "name": "fleet-agent-d0-k3k-downstream018-downstream1727",
    "namespace": "cluster-fleet-default-d0-k3k-downstream018-downstream1727-718b0"
  },
  "namespace": "cluster-fleet-default-d0-k3k-downstream018-downstream1727-718b0",
  "name": "fleet-agent-d0-k3k-downstream018-downstream1727",
  "reconcileID": "38a8012a-1f5c-457e-b62e-86b061eebe0f",
  "commit": "",
  "dryRun": false
}
{
  "level": "error",
  "ts": "2025-09-24T15:28:05Z",
  "msg": "Reconciler error",
  "controller": "bundledeployment",
  "controllerGroup": "fleet.cattle.io",
  "controllerKind": "BundleDeployment",
  "BundleDeployment": {
    "name": "fleet-agent-d0-k3k-downstream018-downstream1727",
    "namespace": "cluster-fleet-default-d0-k3k-downstream018-downstream1727-718b0"
  },
  "namespace": "cluster-fleet-default-d0-k3k-downstream018-downstream1727-718b0",
  "name": "fleet-agent-d0-k3k-downstream018-downstream1727",
  "reconcileID": "38a8012a-1f5c-457e-b62e-86b061eebe0f",
  "error": "failed deploying bundle: cannot re-use a name that is still in use",
  "errorCauses": [
    {
      "error": "failed deploying bundle: cannot re-use a name that is still in use"
    }
  ],
  "stacktrace": "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:296"
}

After increasing redeployAgentGeneration:

% helm status -n cattle-fleet-system  fleet-agent-d0-k3k-downstream018-downstream1727
NAME: fleet-agent-d0-k3k-downstream018-downstream1727
LAST DEPLOYED: Wed Sep 24 14:45:20 2025
NAMESPACE: cattle-fleet-system
STATUS: superseded
REVISION: 2
TEST SUITE: None

% helm history -n cattle-fleet-system  fleet-agent-d0-k3k-downstream018-downstream1727
REVISION	UPDATED                 	STATUS    	CHART                                                                                                                 	APP VERSION	DESCRIPTION
2       	Wed Sep 24 14:45:20 2025	superseded	fleet-agent-d0-k3k-downstream018-downstream1727-v0.0.0+s-a394e34ef708418c7ef06c155ff7862261616418d1c7e434c999e0080455f	           	Upgrade complete

Cluster status shows ErrApplied(1) [Bundle fleet-agent-d0-k3k-downstream018-downstream1727: cannot re-use a name that is still in use]'.

[0xavi0]

This also happened to me when upgrading rancher from 2.11.3 to 2.12.0

[mmartin24]

I tried to reproduce this using single node 1.31.6 +k3s1 cluster, deploying nginx gitrepo and later upgrading via Helmfrom 2.11.2 to 2.11.3 and later to 2.12.0 and fleet agent seemed to be ok all the time.

Screencast.from.2025-11-06.16-09-48.webm

Here you can see 2.11.2:

Later 2.11.3 (fleet-agent looks good)

And finally to 2.12.0 (fleet-agent looks good)

Not sure if perhaps I am missing something here.