Make externally managed Fleet an opt-in feature

[salasberryfin]

Description

The import controller automatically applies the annotation provisioning.cattle.io/externally-managed on every Rancher cluster object (representing a CAPI cluster) that tells Rancher that Fleet is externally managed. This logic is not conditional and will be applied on each reconciliation loop which means there's no way around it.

Originally this was implemented when Turtles was a stand-alone project tightly coupled to CAAPF (as it was installed by default) and it delegated all Fleet responsibilities to the add-on provider. Without this annotation, there were duplicates of Fleet clusters when a CAPI cluster was imported into Rancher.

As Turtles became a core part of Rancher, CAAPF was no longer installed by default and this creates a void in some scenarios where, if you're not using CAAPF but want to use Fleet, CAPI workload clusters don't have a Fleet agent.

Use case

As a user I would like to be able to use Fleet on imported CAPI clusters even if I'm not interested in using CAAPF.

What needs to be implemented

The externally managed annotation should only be set when the user explicitly configures Turtles for using CAAPF. Otherwise it should be disabled and the regular Rancher management of Fleet should be maintained (as with other non-CAPI clusters). This could be implemented with a user configurable feature gate.

Outcome

Configuring Turtles for using CAAPF is an opt-in feature. Fleet agent installation should align with other provisioning mechanisms by default.

Documentation

There must be proper documentation that informs users of the mutually exclusive nature of these features:

• Default behavior does not support using CAAPF reliably.
• Opting to use CAAPF means that Fleet is no longer managed by Rancher but CAAPF.

[kkaempf]

enhancement 👉🏻 moving to 2.15.0