[RFE] Add Prometheus alerting rules

[jbiers]

The Backup&Restore Operator is now able to export metrics. Alerts should be created to monitor these metrics in a meaningful way.

[prachidamle]

SURE-3035

[omkarkhatavkar]

@jbiers Can you please add steps how to enable this feature and metrics list for QE.

[jbiers]

@omkarkhatavkar For sure!

This PR essentially creates a new alert that will fire when a backup operation fails.
You need to set .Values.monitoring.metrics.enabled: true, .Values.monitoring.serviceMonitor.enabled: true for the metrics to be exported and .Values.monitoring.prometheusRules.defaultAlert.enabled: true for the alert to be created. You should also be able to set custom alerts with the field .Values.monitoring.prometheusRules.customRules.rules.

The main thing to be tested here is if the alert is firing correctly on backup failure and if it stops firing once the backup succeeds.

[omkarkhatavkar]

Test Executed :

Rancher Version : 2.12.0-alpha12

1. Make sure the default rule gets created and alerts should be generated - PASSED

2. Alerts should be removed or deleted if the backups are successful - PASSED

3. Alerts should be generated for the backup and restore custom rule as well. - PASSED