DryRun: Preview Changes to Resources in the Managed Clusters

[BonySmoke]

Is your feature request related to a problem? Please describe.
Before deploying a Helm chart to a managed cluster, we want to see what would be deployed (added/removed/modified).

The feature will be similar to the result of running helm template or helm diff with this plugin; however, the biggest advantage is that Sveltos can template chart values and merge them before running the mentioned commands.

The existing DryRun mode allows us to see what will change in resources in the management cluster only. This is great but it doesn't show what resources will be added/removed/modified in the target cluster after updating the Helm chart version/values.

Describe the solution you'd like
As we discussed in Slack, the biggest challenge is to find a place to store the result of helm template/diff.

I would like to propose a couple of options, but I'm not sure which one is best.

Config Maps

• When a Profile is deployed with the syncMode: DryRun, Addon Controller checks if the chart is new or updated, templates the resources per the operation type, and creates a ConfigMap with the template/diff per chart. The content of the config map key is compressed using gzip and encoded with base64. I templated the Kyverno chart using this approach, and the resulting file size was below 500K. If the size exceeds 1MB, we can either truncate it or tell the user that the chart size is too big to be shown.
• Addon Controller creates ClusterReport and references the names of the config maps in it
• When running the sveltosctl show dryrun command with the --raw-diff flag, the tool reads every config map, decompresses its content, and shows it to the user.

This flow looks cloud-native, but it may still hit the limit of the allowed config map size.
Additionally, we need a way to keep track of the created config maps and clean them up when they are not needed.

Volume

• When a Profile is deployed with the syncMode: DryRun, Addon Controller checks if the chart is new or updated, templates the resources and stores the result per chart in the /tmp folder of the pod.
  • If the chart is not yet installed, the controller templates all resources, taking user values into account
  • If the chart is upgraded, the controller checks the diff between the current and new releases, taking user values into account
• Addon Controller creates ClusterReport and references the names of the files in it
• When running the sveltosctl show dryrun command with the --raw-diff flag, the tool execs into the current leader pod of the addon controller and reads the content of the changed/added charts and displays it in the terminal.

A couple of considerations about this solution:

• Does addon-controller support leader election in HA setup or it will always run in replica 1? If we have 2 replicas, we need to find the pod with the right content
• If the pod restarts, temporary files will be lost, and there will be a discrepancy between the ClusterReport resource and the filesystem.

This solution looks less attractive to me but it addresses the issue with the size limitations of config maps.

CLI

• When a Profile is deployed with the syncMode: DryRun, Addon Controller templates the old and new final chart values and stores them either in the ClusterReport or ConfigMap
• • When running the sveltosctl show dryrun command with a new flag, e.g. --template, the tool finds chart values for the old and new chart version and runs the local helm template/diff command and shows the output in the user terminal

This approach looks fairly simple at first, but we also need a way to create a release with the old chart values so we can compare the new chart version to it. I'm sure it's possible but I haven't tested it 😅

Describe alternatives you've considered
Alternatively, we could provide a way to template the final values of the new Helm chart and show them to the user. Then, the user would need to run helm template/diff commands locally manually.

Additional context
Related Slack Thread