[BUG] Some bundle deployments are not deployed / do not contain status release when running many clusters

[mmartin24]

Issue

Found when running scalability tests on fleet chart 0.15.0-alpha.4 and onwards.
Some bundle deployments are not deployed / do not contain status release when running many clusters
I tried to run the scalability tests with 30 downstream clusters attempting to deploy this gitrepo:

metadata:
  name: scale-50-single
  namespace: fleet-default
spec:
  repo: https://github.com/manno/fleet-experiments
  branch: main
  paths:
    - scale-50-bundles/ten
    - scale-50-bundles/twenty
    - scale-50-bundles/thirty
    - scale-50-bundles/fourty
    - scale-50-bundles/fifty
  targetNamespace: scale-50-single
  targets:
    - clusterSelector: {}

In essence the test will create several downstream clusters, will deploy the specified number of bundledeployments and secrets, and them delete them. In this case, the case could not finish because some bundledeployments were not completed.

I am adding here the logs of the yamls of an example where the cluster is ok vs one where is not

bundledeployment.ok.yaml

apiVersion: fleet.cattle.io/v1alpha1
kind: BundleDeployment
metadata:
creationTimestamp: "2026-02-04T10:08:41Z"
finalizers:
- fleet.cattle.io/bundle-deployment-finalizer
generation: 1
labels:
  fleet.cattle.io/bundle-name: scale-50-single-scale-50-bundles-fourty-thirty-three
  fleet.cattle.io/bundle-namespace: fleet-default
  fleet.cattle.io/cluster: cluster-7
  fleet.cattle.io/cluster-namespace: fleet-default
  fleet.cattle.io/commit: 116d6830e3d337fc8a6ff807376da8df04d8b7ac
  fleet.cattle.io/content-name: s-21d4568ce16f9b2b635e976f918b33eca5e2a7387a2871997f8e1c938197f
  fleet.cattle.io/managed: "true"
  fleet.cattle.io/repo-name: scale-50-single
name: scale-50-single-scale-50-bundles-fourty-thirty-three
namespace: cluster-fleet-default-cluster-7-8ee3d948c358
resourceVersion: "35283"
uid: 6d7c45fa-fb85-417c-b7f0-cc0840bf7afa
spec:
deploymentID: s-21d4568ce16f9b2b635e976f918b33eca5e2a7387a2871997f8e1c938197f:59a3323776341d4e1d2d7c2306463480d55713d0cf6580c5b31d784c62b06928
options:
  defaultNamespace: default
  helm:
    chart: config-chart
  namespace: scale-50-single
stagedDeploymentID: s-21d4568ce16f9b2b635e976f918b33eca5e2a7387a2871997f8e1c938197f:59a3323776341d4e1d2d7c2306463480d55713d0cf6580c5b31d784c62b06928
stagedOptions:
  defaultNamespace: default
  helm:
    chart: config-chart
  namespace: scale-50-single
valuesHash: 91db637c5c3ef20fadb44acf2677cbe1903d1d9808462fa349b5afb4468d5fcc
status:
appliedDeploymentID: s-21d4568ce16f9b2b635e976f918b33eca5e2a7387a2871997f8e1c938197f:59a3323776341d4e1d2d7c2306463480d55713d0cf6580c5b31d784c62b06928
conditions:
- lastUpdateTime: "2026-02-04T10:08:45Z"
  status: "True"
  type: Installed
- lastUpdateTime: "2026-02-04T10:08:45Z"
  status: "True"
  type: Deployed
- lastUpdateTime: "2026-02-04T10:09:03Z"
  status: "True"
  type: Ready
- lastUpdateTime: "2026-02-04T10:09:03Z"
  status: "True"
  type: Monitored
display:
  deployed: "True"
  monitored: "True"
  state: Ready
nonModified: true
ready: true
release: scale-50-single/scale-50-single-scale-50-bundles-fourty-thirty-three:1
resourceCounts:
  desiredReady: 1
  ready: 1
resources:
- apiVersion: v1
  createdAt: "2026-02-04T10:08:45Z"
  kind: ConfigMap
  name: test-config-thirty-three
  namespace: scale-50-single
syncGeneration: 0  

bundledeployment.no_status_release.yaml

apiVersion: fleet.cattle.io/v1alpha1
kind: BundleDeployment
metadata:
creationTimestamp: "2026-02-04T10:08:42Z"
finalizers:
- fleet.cattle.io/bundle-deployment-finalizer
generation: 1
labels:
  fleet.cattle.io/bundle-name: scale-50-single-scale-50-bundles-fifty-forty-six
  fleet.cattle.io/bundle-namespace: fleet-default
  fleet.cattle.io/cluster: cluster-8
  fleet.cattle.io/cluster-namespace: fleet-default
  fleet.cattle.io/commit: 116d6830e3d337fc8a6ff807376da8df04d8b7ac
  fleet.cattle.io/content-name: s-21d4568ce16f9b2b635e976f918b33eca5e2a7387a2871997f8e1c938197f
  fleet.cattle.io/managed: "true"
  fleet.cattle.io/repo-name: scale-50-single
name: scale-50-single-scale-50-bundles-fifty-forty-six
namespace: cluster-fleet-default-cluster-8-2a1fe93aeebd
resourceVersion: "34032"
uid: 1bf47df5-ad8a-426d-b3c7-c34b8a342c7b
spec:
deploymentID: s-21d4568ce16f9b2b635e976f918b33eca5e2a7387a2871997f8e1c938197f:f7c4f3cf3023c425489e0e3d29354a9a8ff64cf4aa38a1fe70a97fb041a4138c
options:
  defaultNamespace: default
  helm:
    chart: config-chart
  namespace: scale-50-single
stagedDeploymentID: s-21d4568ce16f9b2b635e976f918b33eca5e2a7387a2871997f8e1c938197f:f7c4f3cf3023c425489e0e3d29354a9a8ff64cf4aa38a1fe70a97fb041a4138c
stagedOptions:
  defaultNamespace: default
  helm:
    chart: config-chart
  namespace: scale-50-single
valuesHash: bd3b26c846cb88ae2c0908f7f53ba4f6e5b04b74da48d55939a636a3c3e4354d
status:
appliedDeploymentID: s-21d4568ce16f9b2b635e976f918b33eca5e2a7387a2871997f8e1c938197f:f7c4f3cf3023c425489e0e3d29354a9a8ff64cf4aa38a1fe70a97fb041a4138c
conditions:
- lastUpdateTime: "2026-02-04T10:08:48Z"
  message: 'not ready: unable to continue with install: ConfigMap "test-config-default-name"
    in namespace "scale-50-single" exists and cannot be imported into the current
    release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name"
    must equal "scale-50-single-scale-50-bundles-fifty-forty-six": current value
    is "scale-50-single-scale-50-bundles-twenty-sixteen"'
  reason: Error
  status: "False"
  type: Ready
- lastUpdateTime: "2026-02-04T10:08:48Z"
  message: 'not installed: unable to continue with install: ConfigMap "test-config-default-name"
    in namespace "scale-50-single" exists and cannot be imported into the current
    release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name"
    must equal "scale-50-single-scale-50-bundles-fifty-forty-six": current value
    is "scale-50-single-scale-50-bundles-twenty-sixteen"'
  reason: Error
  status: "False"
  type: Installed
- lastUpdateTime: "2026-02-04T10:08:48Z"
  status: "True"
  type: Deployed
display:
  deployed: "True"
  state: NotReady
nonModified: true

Also adding an example of error.json on the gitrepo pod (due to length constraint is linked to another place)

---

To reproduce:

• Use this guideline to configure the setup. Note: you can use 50 clusters as is preconfigured or adapt it to 30.
• Once you the script 40-run-tests.sh, some of the bundledeployments will fail. To determine in which exactly to look at I used this script:

#!/bin/bash

# Troubleshooting script to open tunnels to downstream clusters
# Loop through downstream clusters from 1 to 30
# If the configmap count is less than 52, stop the loop

for ds_n in {1..30}; do ## Adaot to 50 if you deploy 50 clusters
    echo "Checking downstream-0-${ds_n}..."
    
    bash default_config/open-tunnels-to-downstream-0-${ds_n}-server-0.sh && \
    export KUBECONFIG="$PWD/default_config/downstream-0-${ds_n}.yaml" && \
    kubectl config use-context downstream-0-${ds_n}
    
    configmap_count=$(k get -n scale-50-single configmaps | wc -l)
    echo "Configmap count for downstream-0-${ds_n}: ${configmap_count}"
    
    if [ "$configmap_count" -lt 52 ]; then
        echo "Count is less than 52, stopping at downstream-0-${ds_n}"
        break
    fi
done

• Once you have the one that does not contain 52 configmaps, just tunnel to that specific one to troubleshoot

[mmartin24]

I just rested after upgrade to 0.15.0-alpha.10 and this issue seems ok now. Pasting log of completed round with 50 clusters / 2500 bundledeployments:

Fleet Version: 0.15.0-alpha.10
----------------------------------------
Starting test at: Thu 12 Feb 2026 04:41:53 PM CET

Clusters: 50
Agents: 50
Bundles: 50d
Total BundleDeployments: 2550
Fleet Version: 0.15.0-alpha.10
Helm Version: version.BuildInfo{Version:"v4.0.0", GitCommit:"99cd1964357c793351be481d55abbe21c6b2f4ec", GitTreeState:"clean", GoVersion:"go1.25.3", KubeClientVersion:"v1.34"}

gitrepo.fleet.cattle.io/scale-50-single created

created gitrepo for 50 bundles: Thu 12 Feb 2026 04:41:54 PM CET
..
bundles exist: Thu 12 Feb 2026 04:41:58 PM CET
.
bundledeployments exist: Thu 12 Feb 2026 04:42:10 PM CET
.....
bundles ready: Thu 12 Feb 2026 04:42:39 PM CET
.
gitrepo ready: Thu 12 Feb 2026 04:42:45 PM CET

clusters ready: Thu 12 Feb 2026 04:42:46 PM CET

delete 
gitrepo.fleet.cattle.io "scale-50-single" deleted from fleet-default namespace

clusters 1/1: Thu 12 Feb 2026 04:42:51 PM CET
No resources found

secrets removed: Thu 12 Feb 2026 04:42:51 PM CET
Warning: Permanently added 'ec2-98-93-6-185.compute-1.amazonaws.com' (ED25519) to the list of known hosts.
Warning: Permanently added 'ip-172-16-100-172.ec2.internal' (ED25519) to the list of known hosts.
523M    /var/lib/rancher/rke2/server/db/

Test completed at: Thu 12 Feb 2026 04:42:54 PM CET
----------------------------------------

[mmartin24]

Closing issue since it seems fixed after #4624