BundleDeployment has not been updated 3 hours since the bundle is being updated.

[aDisplayName]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

We have observed, in our setup, that 3 hours after the content in the git repository has been updated, the change has not been deployed into our target cluster.

The fleet.yaml in our git repository is being updated fairly frequently by the CI/CD pipeline, a dozen times per hour,

In Rancher local server, we checked the bundle secret in fleet-default namespace, at it is always the latest:

But spot checked a few bundle deployment secrets, they are at least 3 hours late:

Which matches to the content deployed to the downstream clusters.

Expected Behavior

We would expect the corresponding bundledeployment for each target should always be up-to-date

Steps To Reproduce

No response

Environment

- Rancher: v2.12.1
  - Provider: GKE
  - Kubernetes Version: v1.32.9-gke.1108000
- Fleet Version: 0.13.1
- Downstream Cluster:
  - Provider: k3s
  - Options: 
  - Kubernetes Version: 1.28.1
- Rancher

Logs

Anything else?

The other Fleet GitRepos in the same rancher server doesn't seem to have the same deploy problem. Update of the change are always deployed to downstream clusters in a reasonable time line.

We are using gitjob to trigger the GitRepo update with source git repositories are hosted in azure devops.

Can you tell us where to get the required log?

[aDisplayName]

We Recreated the GitRepo with a different GitRepo object name, now the delay seems reduced to a few minutes. (targeting 100+ clusters).

The rancher server was upgraded from 2.10.1 to 2.12.2 directly, then we saw the fleet controller was not upgraded correctly (Still stuck in 0.11.x), so we downgraded rancher server to 2.11.x, and then upgraded again to 2.12.1.

[kkaempf]

The rancher server was upgraded from 2.10.1 to 2.12.2 directly, then we saw the fleet controller was not upgraded correctly (Still stuck in 0.11.x), so we downgraded rancher server to 2.11.x, and then upgraded again to 2.12.1.

Skipping minor versions on upgrades is not tested and not supported.

[weyfonk]

Closing this, as the issue seems resolved. If not, please provide more information.