Fleet clusters not reflecting disconnection under Continuous delivery

[mmartin24]

Setup

• Rancher version: 2.9.2 and onwards
• Rancher UI Extensions:
• Browser type & version: Brave v1.70.126

Describe the bug
A downstream cluster that is disconnected appears as Ready in Continuous Delivery Dashboard and Active in Continuous Deliver Clusters while it correctly shows status in Cluster Management

To Reproduce

• Deploy Rancher 2.9.2 or later
• Add a downstream cluster
• Deploy a gitrepo on downstream cluster
• Temporarirly stop the downstream cluster (ie if you used k3s downstream: sudo systemctl stop k3s.service )
• Once stopped, observe how in Cluster Management, it correctly shows that state
• Navigate to Continuous Delivery > Dashboard and observe how they are shown as Clusters Ready 1/1
• Navigate to Continuous Delivery > Clusters and observe how state is Active

Result
Clusters are shown as Ready in Continuous Delivery Dashboard and Active` in Continuous Deliver Clusters.

Expected Result
Clusters should be shown as Unavailable / Disconnected or similar like in Cluster management:

Screenshots

Additional context

[richard-cox]

The UI shows

• Cluster state via /v1/fleet.cattle.io.clusters metadata.state
• Dashboard home page repo state via /v1/fleet.cattle.io.bundles (that match gitrepo repoName + name, and namespace) that report a state of active

Moving to fleet team to triage further

[mmartin24]

Thanks @richard-cox for the input. I just moved it to our board.

[manno]

Since metadata.state is not computed by Fleet, but by Steve we need more information.

Two points:

• gitrepo.status.readyClusters does not indicate the cluster is alive and healthy, it shows that bundles all targeted at the cluster were deployed to the cluster.
• we are working on offline detection for clusters by using the agents hearbeat: Detect offline clusters fleet#2933

[manno]

We are treating this as a duplicate of rancher/fleet#594 for now.

[weyfonk]

@nwmac/@richard-cox We are wondering what kind of UX would be expected around this, as our implementation would depend on that. Do we want to:

• only impact statuses of GitRepos for offline clusters, and if so to what extent? Showing a regular error message in red, updating the colour of the badge for actual vs expected ready resources?
• update cluster status as well, at least in Continuous Delivery?
• take a possibly wider-ranging approach, greying things out to prevent some actions (eg. editing config) for offline clusters?
• add tooltips with warnings, eg. when a user tries to force update a GitRepo for which at least one target cluster is offline?

Happy to discuss.

[kkaempf]

ping @gaktive, /cc @torchiaf