[RFE] Provide 2 default ResourceSet configs out of the box based on use-case

[mallardduck]

🚀 Feature Request

Background

Historically Rancher Backups (BRO) has opted to exclude sensitive data from backups by default as it's seen as a security risk. This in turn can lead to some less than stellar experience for customers using BRO from Backups or Migrations. Of particular concern are areas related to Fleet, Provisioning, and cluster config related credentials.

In the more recent era of BRO we've faced far too many support issues to continue with the philosophy of excluding sensitive data from Backups. As that ultimately leads to support cases during some extremely tense situations. To balance these concerns we've gradually shifted focus to push users to utilize the encryption features.

With the recent change on this feature we fully support the same Encryption options k8s does. This opens the door for us to adjust how we handle Secrets in our default ResourceSet and to take our efforts to suggest encrypting backups even further.

Is your feature request related to a problem? Please describe.

Yes, there are a number of open issues this is related to.

Put simply, this RFE will help address many issues that are related to BRO capturing or skipping various sensitive data (Secrets, etc). In the past BRO opted to intentionally not capture sensitive data to avoid potential risk of those secrets being accessible via the Backup file. Recent changes to BRO allow more complex encryption configs to be used which provides the opportunity for us to include and encrypt sensitive data in the backups.

Describe the solution you'd like

There are two overlapping concepts that this issue relates to within BRO, that is:

• Ability to Encrypt BRO backups at rest, and
• the rules in the default ResourceSet that BRO provides.

Using the functionality provided by the new Encryption features we can ship 2 default ResourceSet versions; one matching our existing one, and one that's meant to capture all the necessary secrets for Rancher to work. In our documentation and in our examples, we will need to compare and contrast the pros/cons of each; these mainly boil down to:

• On the "Basic" ResourceSet, the backup files will not contain any sensitive or secrets data and not need encryption to be suggested/required to use. In this use-case the secrets necessary for the cluster which cannot/will-not be recreated should be backed up outside of BRO, or otherwise configured to deploy via CI/CD.
• On the "Full" ResourceSet, the backup files will include any necessary secrets to fully restore Rancher to a working state. Any examples and documentation for this mode must also show usage of an EncryptionConfiguation.
  • Note: In setup docs, we should provide users with an example showing what resources we suggest they encrypt; however, we should exclude the keys from the example and instead direct users to create their own. (hopefully to prevent users from using any default/example keys given in docs)

In order to implement this without backwards compatability breaks, we actually will need to create both the above mentioned ResourceSets as new items. This way we can deprecate the old one without modifying or removing it in this release.

Once the two ResourceSet resources are being prepared we will need to do:

1. Adjust how the current ResourceSet is composed,
2. Add new ResourceSet items with a new names,
3. On the "Basic" set, we can start from the current one with Rules for secrets and such removed,
4. On the "Full" resource set we will need to collaborate with other teams (such as Fleet, Harvester, etc) to verify we capture all correct secrets; if not we will use the respective teams expertise to create new rules.
5. Release notes, to inform users that this change has happened and the deprecation.

Then in 2.10.2 (or later) we can follow up on this and change the UI default from the deprecated ResourceSet to use the new "Basic" one instead by default. Ensuring that users will have to fully opt-in to having the backups contain sensitive data that must be secured.

Describe alternatives you've considered

We have a method to capture secrets based on labels/annotations that may be helpful here. However, this alternative is unlikely to be a complete solution if implemented alone. It will likely be something we consider to implement in addition to these changes.

Additional context

Here are a list of Related issues that this solution could help address:

Important

A PR on this list is here purely because it's related to BRO/Secrets as a concept. If a PR is here it is not a guaruntee that it will be fixed by the PR to address this. Simply that the topics overlap and if possible we will improve or fix these issues along the way.

• [BUG] Rancher doesn't properly label harvester secrets when migrated rancher#46276 / SURE-8118
• Restore renders downstream cluster unusable if this one resides in a non-fleet-default namespace #574
• [BUG] RKE2 provisioned registry secret not being backed up by Backup/Restore operator rancher#44033
• missing harvesterconfig secrets #282
• [v2.8] BRO not backing up namespaces created for FleetWorkspaces #482
• Secret not found after a restore #177
• [BUG] Secrets created via Fleet UI are not being backed up dashboard#11211
  • [BUG] Need script | Secrets created via Fleet UI are not being backed up by BRO rancher#45812
• Fleet related resources in general.

Note

Some items linked above are long merged/solved PRs that are relevant context behind existing rules.

[mallardduck]

@jbiers - once you have a PR cooked up for this one we will have some support from R.Apetz (find in slack) who has a customer affected by #574. So one we can produce an RC chart/image that includes your improvements they and their customer may be interested in testing it out.

[jbiers]

The first PR in resolving this issue is ready for review: #634. It introduces the two new resourceSets, basic and full, without removing the old standard one but treating it as deprecated. It will be removed completely in a later release.

Merging this PR will not close the issue. Here are the other necessary steps:

• Update this documentation page to include a ResourceSet subsection detailing when users should choose full or basic. It will make clear that opting for full means backups will contain sensitive information and, as a best practice, should be encrypted. Also explain that when using basic the user is fully responsible for backing up their secrets. Include note about default resourceset deprecation. Also, add an example to this page.
• Open issue to UI team to update the Backup creation screen (see current state below). Another section under the name of ResourceSets will be added, corresponding to the field spec.resourceSetName. This section will contain a dropdown menu with all available ResourceSets, and will currently default to rancher-resource-set. Along with the dropdown menu, we could add a warning text about using the rancher-resource-set-full with encryption.
  
• Initiate conversations with other teams to map out all secrets needed for rancher to work correctly.
• Open a follow-up issue with a milestone set to 2.12.0 for handling the complete removal of the old default resourceset.

@mallardduck and @alexandreLamarre once the holidays are over I'd appreciate some feedback on the initial PR as well as on the action plan detailed above 🤝

[jbiers]

Docs and UI issues have been created. UI will probably only get to this by 2.12, which I personally would not consider that big of an issue as users can still use the new resourcesets by editing the Backup CR via YAML (the way they do it now).