-
Notifications
You must be signed in to change notification settings - Fork 274
Allow cluster-scoped instance CRDs via schema scope #885
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 3 commits
138b7ee
47f1282
bcb1cc5
ca3260f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| # Use ResourceDescriptor for External Reference Scope Resolution | ||
|
|
||
| ## Problem statement | ||
|
|
||
| When reconciling instances, the controller needs to interact with external references (resources not managed by the instance but referenced by it). To do this, it needs to know the GroupVersionResource (GVR) and the scope (namespaced or cluster-scoped) of the external resource. | ||
|
|
||
| Previously, the controller relied on `restMapper.RESTMapping` at runtime to resolve the GVK to GVR and determine the scope. This introduces a runtime dependency on the discovery client and can lead to errors if the REST mapper is not up-to-date or if the discovery fails. It also adds unnecessary latency to the reconciliation loop. | ||
|
|
||
| ## Proposal | ||
|
|
||
| ### Overview | ||
|
|
||
| We propose to use the `ResourceDescriptor` interface, which is populated during the ResourceGraphDefinition (RGD) compilation phase, to determine the GVR and scope of external references. The `ResourceDescriptor` already contains this information, as it is resolved when the RGD is processed. | ||
|
|
||
| ### Design details | ||
|
|
||
| The `readExternalRef` function in `pkg/controller/instance/controller_reconcile.go` will be updated to: | ||
|
|
||
| 1. Retrieve the `ResourceDescriptor` for the given resource ID from the runtime. | ||
| 2. Use `descriptor.GetGroupVersionResource()` to get the GVR. | ||
| 3. Use `descriptor.IsNamespaced()` to determine if the resource is namespaced. | ||
| 4. Construct the dynamic client using this information. | ||
|
|
||
| ```go | ||
| func (igr *instanceGraphReconciler) readExternalRef(ctx context.Context, resourceID string, resource *unstructured.Unstructured) (*unstructured.Unstructured, error) { | ||
| descriptor := igr.runtime.ResourceDescriptor(resourceID) | ||
| gvr := descriptor.GetGroupVersionResource() | ||
|
|
||
| var dynResource dynamic.ResourceInterface | ||
| if descriptor.IsNamespaced() { | ||
| namespace := igr.getResourceNamespace(resourceID) | ||
| dynResource = igr.client.Resource(gvr).Namespace(namespace) | ||
| } else { | ||
| dynResource = igr.client.Resource(gvr) | ||
| } | ||
| // ... | ||
| } | ||
| ``` | ||
|
|
||
| ## Benefits | ||
|
|
||
| * **Performance**: Removes the need for a REST mapper lookup during every reconciliation of an external reference. | ||
| * **Reliability**: Relies on the static analysis performed during RGD compilation, which is consistent with how other resources are handled. | ||
| * **Consistency**: Aligns the handling of external references with managed resources, which already use `ResourceDescriptor`. | ||
|
|
||
| ## Tradeoffs | ||
|
|
||
| * **Static Definition**: This assumes that the scope of a resource does not change between RGD compilation and runtime. In Kubernetes, changing the scope of a CRD is a breaking change and requires re-creation, so this assumption holds true for practical purposes. | ||
|
|
||
| ## Scoping | ||
|
|
||
| ### What is in scope for this proposal? | ||
|
|
||
| * Modifying `pkg/controller/instance/controller_reconcile.go`. | ||
| * Updating `readExternalRef` implementation. | ||
|
|
||
| ### What is not in scope? | ||
|
|
||
| * Changes to RGD compilation logic (the information is already there). | ||
| * Changes to other parts of the controller. | ||
|
|
||
| ## Testing strategy | ||
|
|
||
| ### Test plan | ||
|
|
||
| * **Unit Tests**: Existing unit tests for the instance controller should pass. | ||
| * **Manual Verification**: Verify that external references are correctly resolved and read by the controller. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -516,10 +516,14 @@ func (igr *instanceGraphReconciler) setManaged( | |
|
|
||
| igr.instanceLabeler.ApplyLabels(instancePatch) | ||
|
|
||
| updated, err := igr.client.Resource(igr.gvr). | ||
| Namespace(obj.GetNamespace()). | ||
| Apply(ctx, instancePatch.GetName(), instancePatch, | ||
| metav1.ApplyOptions{FieldManager: FieldManagerForLabeler, Force: true}) | ||
| instanceClient := igr.client.Resource(igr.gvr) | ||
| var namespacedClient dynamic.ResourceInterface = instanceClient | ||
| if ns := obj.GetNamespace(); ns != "" { | ||
| namespacedClient = instanceClient.Namespace(ns) | ||
| } | ||
|
|
||
| updated, err := namespacedClient.Apply(ctx, instancePatch.GetName(), instancePatch, | ||
| metav1.ApplyOptions{FieldManager: FieldManagerForLabeler, Force: true}) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("failed to update managed state: %w", err) | ||
| } | ||
|
|
@@ -552,10 +556,14 @@ func (igr *instanceGraphReconciler) setUnmanaged( | |
| return nil, fmt.Errorf("failed to remove finalizer: %w", err) | ||
| } | ||
|
|
||
| updated, err := igr.client.Resource(igr.gvr). | ||
| Namespace(obj.GetNamespace()). | ||
| Apply(ctx, instancePatch.GetName(), instancePatch, | ||
| metav1.ApplyOptions{FieldManager: FieldManagerForLabeler, Force: true}) | ||
| instanceClient := igr.client.Resource(igr.gvr) | ||
| var namespacedClient dynamic.ResourceInterface = instanceClient | ||
| if ns := obj.GetNamespace(); ns != "" { | ||
| namespacedClient = instanceClient.Namespace(ns) | ||
| } | ||
|
||
|
|
||
| updated, err := namespacedClient.Apply(ctx, instancePatch.GetName(), instancePatch, | ||
| metav1.ApplyOptions{FieldManager: FieldManagerForLabeler, Force: true}) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("failed to update unmanaged state: %w", err) | ||
| } | ||
|
|
@@ -571,26 +579,23 @@ func (igr *instanceGraphReconciler) delayedRequeue(err error) error { | |
| // readExternalRef fetches an external reference from the cluster. | ||
| // External references are resources that exist outside of this instance's control. | ||
| func (igr *instanceGraphReconciler) readExternalRef(ctx context.Context, resourceID string, resource *unstructured.Unstructured) (*unstructured.Unstructured, error) { | ||
| gvk := resource.GroupVersionKind() | ||
| restMapping, err := igr.restMapper.RESTMapping(gvk.GroupKind(), gvk.Version) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("failed to get REST mapping for %v: %w", gvk, err) | ||
| } | ||
| descriptor := igr.runtime.ResourceDescriptor(resourceID) | ||
| gvr := descriptor.GetGroupVersionResource() | ||
|
|
||
| var dynResource dynamic.ResourceInterface | ||
| if restMapping.Scope.Name() == meta.RESTScopeNameNamespace { | ||
| if descriptor.IsNamespaced() { | ||
| namespace := igr.getResourceNamespace(resourceID) | ||
| dynResource = igr.client.Resource(restMapping.Resource).Namespace(namespace) | ||
| dynResource = igr.client.Resource(gvr).Namespace(namespace) | ||
| } else { | ||
| dynResource = igr.client.Resource(restMapping.Resource) | ||
| dynResource = igr.client.Resource(gvr) | ||
|
||
| } | ||
|
|
||
| clusterObj, err := dynResource.Get(ctx, resource.GetName(), metav1.GetOptions{}) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("failed to get external ref %s/%s: %w", resource.GetNamespace(), resource.GetName(), err) | ||
| } | ||
|
|
||
| igr.log.V(2).Info("read external ref", "gvk", gvk, "namespace", resource.GetNamespace(), "name", resource.GetName()) | ||
| igr.log.V(2).Info("read external ref", "gvr", gvr, "namespace", resource.GetNamespace(), "name", resource.GetName()) | ||
| return clusterObj, nil | ||
| } | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
namespacedClientname is misleading IMO It could be cluster-scoped or namespaced client depending on the resourceI would suggest defining a
getGVRClientused here, in thesetUnmanagedand in thegetResourceClientfunction to ensure namespacing is well-managed everywhere