[RFE] Build BRO Grafana dashboards

[jbiers]

The Backup&Restore Operator is now able to export metrics. Dashboards should be created to monitor these metrics in a meaningful way.

[jbiers]

For QA

Use Chart

Version: 69.8.2-rancher.10
From ClusterRepo:

apiVersion: catalog.cattle.io/v1
kind: ClusterRepo
metadata:
  name: bro-dashboards
spec:
  gitBranch: bro-dashboards
  gitRepo: https://github.com/jbiers/ob-team-charts

Testing notes

• Deploy rancher-monitoring from the branch indicated above. Make sure to set this to true:

rancherBackupMonitoring:

  ## Deploys the rancher-backup specific dashboards. Make sure to also have metrics enabled in the rancher-backup chart.
  dashboards:
    enabled: true

• Deploy BRO with metrics and the serviceMonitor enabled.
• Create backups and restores, make sure the new dashboards shows information that makes sense with your actions (number of backup, their type, etc.) Experiment with recurring backups.

[omkarkhatavkar]

@jbiers @susesamu I verified this with latest rancher versions,

Rancher	v2.12-9e47832a794d17cd66b559fe2bc5c7b20f31781c-head 
Dashboard	release-2.12 a53e85f 
Rancher CLI	v2.12.0 
Helm	dev 
Machine	v0.15.0-rancher131 
rancher-monitoring-crd:69.8.2-rancher.10

Steps performed:
Applied the following setting during the installation of the monitoring chart:

rancherBackupMonitoring:

  ## Deploys the rancher-backup specific dashboards. Make sure to also have metrics enabled in the rancher-backup chart.
  dashboards:
    enabled: true

Observed behavior:

When installing the monitoring chart with the above setting, the installation throws the following error:

Release "rancher-monitoring-crd" has been upgraded. Happy Helming!
NAME: rancher-monitoring-crd
LAST DEPLOYED: Mon Aug 11 10:35:41 2025
NAMESPACE: cattle-monitoring-system
STATUS: deployed
REVISION: 2
TEST SUITE: None
---------------------------------------------------------------------
SUCCESS: helm upgrade --install=true --labels=catalog.cattle.io/cluster-repo-name=bro-dashboards --namespace=cattle-monitoring-system --timeout=10m0s --values=/home/shell/helm/values-rancher-monitoring-crd-69.8.2-rancher.10.yaml --version=69.8.2-rancher.10 --wait=true rancher-monitoring-crd /home/shell/helm/rancher-monitoring-crd-69.8.2-rancher.10.tgz
---------------------------------------------------------------------
helm upgrade --install=true --labels=catalog.cattle.io/cluster-repo-name=bro-dashboards --namespace=cattle-monitoring-system --timeout=10m0s --values=/home/shell/helm/values-rancher-monitoring-69.8.2-rancher.10.yaml --version=69.8.2-rancher.10 --wait=true rancher-monitoring /home/shell/helm/rancher-monitoring-69.8.2-rancher.10.tgz
Release "rancher-monitoring" does not exist. Installing it now.
Error: template: rancher-monitoring/templates/rancher-monitoring/dashboards/rancher/backup-restore-dashboard.yaml:1:83: executing "rancher-monitoring/templates/rancher-monitoring/dashboards/rancher/backup-restore-dashboard.yaml" at <.Values.rancherBackupMonitoring.dashboards.enabled>: nil pointer evaluating interface {}.enabled```

While the chart is installed but not able to see the settings being applied.  

[jbiers]

Hey @omkarkhatavkar I just pushed a quick fix, you should now be able to test it properly. Just set these values in rancher-monitoring:

rancherBackupMonitoring:
  dashboards:
    enabled: true

[omkarkhatavkar]

Verified On

Rancher	v2.12-9e47832a794d17cd66b559fe2bc5c7b20f31781c-head 
Dashboard	release-2.12 a53e85f 
Rancher CLI	v2.12.0 
Helm	dev 
Machine	v0.15.0-rancher131 
rancher-monitoring-crd:69.8.2-rancher.10

Steps Executed

• Installed the Monitoring chart with mentioned setting

rancherBackupMonitoring:

  ## Deploys the rancher-backup specific dashboards. Make sure to also have metrics enabled in the rancher-backup chart.
  dashboards:
    enabled: true



• Deploy BRO with metrics and the serviceMonitor enabled.
• Able to fetch the backup-restore dashboard with backup-restore metrics



Observation

The Failed Recurring Backups over time chart on the Rancher monitoring dashboard appears to be non-functional. It seems that when a recurring backup fails, its status changes to pending and the backup does not retry until the issue is resolved. Consequently, the backup does not recur automatically after a failure. Given this behaviour, do you think this chart remains useful or necessary?

@jbiers let me know your thoughts here for the observation ^^

Result

Feature is working as expected apart from the observation as mentioned earlier - Pass

[mallardduck]

Great work @omkarkhatavkar - I think that we can look into improving that issue on the BRO side. So we should make a new issue to track that and work to solve that in a future release.

[jbiers]

The Failed Recurring Backups over time chart on the Rancher monitoring dashboard appears to be non-functional. It seems that when a recurring backup fails, its status changes to pending and the backup does not retry until the issue is resolved. Consequently, the backup does not recur automatically after a failure. Given this behaviour, do you think this chart remains useful or necessary?

Thanks for the observation @omkarkhatavkar. There is certainly improvements to be made here. I'd say the panel is still useful in cases where a recurring backup fails a few times but then recovers by itself, which is the data we were most interested in when building the panel. This is useful because a user can open up the dashboards and check if at any point in time their recurring backups failed in the past.

[omkarkhatavkar]

The Failed Recurring Backups over time chart on the Rancher monitoring dashboard appears to be non-functional. It seems that when a recurring backup fails, its status changes to pending and the backup does not retry until the issue is resolved. Consequently, the backup does not recur automatically after a failure. Given this behaviour, do you think this chart remains useful or necessary?

Thanks for the observation @omkarkhatavkar. There is certainly improvements to be made here. I'd say the panel is still useful in cases where a recurring backup fails a few times but then recovers by itself, which is the data we were most interested in when building the panel. This is useful because a user can open up the dashboards and check if at any point in time their recurring backups failed in the past.

If that is the expectation, I tried same many times updating the recurring backup failed->passed->failed ->passed this process many times for recurring backups, and not able to see anything in the chart. This needs to be improved in this case.