add cross-workspace implementation for ValidatingAdmissionPolicy

Signed-off-by: olalekan odukoya <[email protected]>

Author: olamilekan000

docs/content/concepts/apis/admission-webhooks.md

@@ -1,13 +1,18 @@
+---
+description: >
+  How admission webhooks and validating admission policies work across workspaces in kcp.
+---
+
 # Admission Webhooks
 
-kcp extends the vanilla [admission plugins](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/) for webhooks, and makes them cluster-aware.
+kcp extends the vanilla [admission plugins](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/) for webhooks and validating admission policies, and makes them cluster-aware.
 
 ```mermaid
 flowchart TD
     subgraph ws1["API Provider Workspace ws1"]
         export["Widgets APIExport"]
         schema["Widgets APIResourceSchema<br/>(widgets.v1.example.org)"]
-        webhook["Mutating/ValidatingWebhookConfiguration<br/>for widgets.v1.example.org<br/><br/>Handle a from ws2 (APIResourceSchema)<br/>Handle b from ws3 (APIResourceSchema)<br/>Handle a from ws1 (CRD)"]
+        webhook["Mutating/ValidatingWebhookConfiguration<br/>ValidatingAdmissionPolicy<br/>for widgets.v1.example.org<br/><br/>Handle a from ws2 (APIResourceSchema)<br/>Handle b from ws3 (APIResourceSchema)<br/>Handle a from ws1 (CRD)"]
         crd["Widgets CustomResourceDefinition<br/>(widgets.v1.example.org)"]
         
         export --> schema
@@ -37,9 +42,39 @@ flowchart TD
     class export,schema,webhook,crd,binding1,binding2 resource;
 ```
 
-When an object is to be mutated or validated, the webhook admission plugin ([`apis.kcp.io/MutatingWebhook`](https://github.com/kcp-dev/kcp/tree/main/pkg/admission/mutatingwebhook) and [`apis.kcp.io/ValidatingWebhook`](https://github.com/kcp-dev/kcp/tree/main/pkg/admission/validatingwebhook) respectively) looks for the owner of the resource schema. Once found, it then dispatches the handling for that object in the owner's workspace. There are two such cases in the diagram above:
+When an object is to be mutated or validated, the admission plugins ([`apis.kcp.io/MutatingWebhook`](https://github.com/kcp-dev/kcp/tree/main/pkg/admission/mutatingwebhook), [`apis.kcp.io/ValidatingWebhook`](https://github.com/kcp-dev/kcp/tree/main/pkg/admission/validatingwebhook), and [`ValidatingAdmissionPolicy`](https://github.com/kcp-dev/kcp/tree/main/pkg/admission/validatingadmissionpolicy) respectively) look for the owner of the resource schema. Once found, they then dispatch the handling for that object in the owner's workspace. There are two such cases in the diagram above:
+
+* **Admitting bound resources.** During the request handling, Widget objects inside the consumer workspaces `ws2` and `ws3` are picked up by the respective admission plugin. The plugin sees the resource's schema comes from an APIBinding, and so it sets up an instance of the admission plugin to be working with its APIExport's workspace, in `ws1`. Afterwards, normal admission flow continues: the request is dispatched to all eligible webhook configurations or validating admission policies inside `ws1` and the object in request is mutated or validated.
+* **Admitting local resources.** The second case is when the webhook configuration or validating admission policy exists in the same workspace as the object it's handling. The admission plugin sees the resource is not sourced via an `APIBinding`, and so it looks for eligible webhook configurations or policies locally, and dispatches the request accordingly. The same would of course be true if `APIExport` and its `APIBinding` lived in the same workspace: the `APIExport` would resolve to the same cluster.
+
+## ValidatingAdmissionPolicy Support
+
+kcp supports cross-workspace `ValidatingAdmissionPolicy` and `ValidatingAdmissionPolicyBinding` resources, similar to how it supports cross-workspace webhooks. When a resource is created in a consumer workspace that is bound via an `APIBinding`, the `ValidatingAdmissionPolicy` plugin will:
+
+1. Check the `APIBinding` to find the source workspace (`APIExport` workspace)
+2. Look for `ValidatingAdmissionPolicy` and `ValidatingAdmissionPolicyBinding` resources in the source workspace
+3. Apply those policies to validate the resource in the consumer workspace
+
+This means that policies defined in an `APIExport` workspace will automatically apply to all resources created in consuming workspaces, providing a consistent validation experience across all consumers of an API.
+
+### Example
+
+Consider a scenario where:
+- **Provider workspace** (`root:provider`) has:
+  - An `APIExport` for `cowboys.wildwest.dev`
+  - A `ValidatingAdmissionPolicy` that rejects cowboys with `intent: "bad"`
+  - A `ValidatingAdmissionPolicyBinding` that binds the policy
+  
+- **Consumer workspace** (`root:consumer`) has:
+  - An `APIBinding` that binds to the provider's `APIExport`
+  - A user trying to create a cowboy with `intent: "bad"`
+
+When the user creates the cowboy in the consumer workspace, the `ValidatingAdmissionPolicy` plugin will:
+1. Detect that the cowboy resource comes from an `APIBinding`
+2. Look up the source workspace (provider workspace)
+3. Find and apply the policy from the provider workspace
+4. Reject the cowboy creation because it violates the policy
 
-* **Admitting bound resources.** During the request handling, Widget objects inside the consumer workspaces `ws2` and `ws3` are picked up by the respective webhook admission plugin. The plugin sees the resource's schema comes from an APIBinding, and so it sets up an instance of `{Mutating,Validating}Webhook` to be working with its APIExport's workspace, in `ws1`. Afterwards, normal webhook admission flow continues: the request is dispatched to all eligible webhook configurations inside `ws1` and the object in request is mutated or validated.
-* **Admitting local resources.** The second case is when the webhook configuration exists in the same workspace as the object it's handling. The admission plugin sees the resource is not sourced via an APIBinding, and so it looks for eligible webhook configurations locally, and dispatches the request to the webhooks there. The same would of course be true if APIExport and its APIBinding lived in the same workspace: the APIExport would resolve to the same cluster.
+This ensures that API providers can enforce consistent validation rules across all consumers of their APIs.
 
-Lastly, objects in admission review are annotated with the name of the workspace that owns that object. For example, when Widget `b` from `ws3` is being validated, its caught by `ValidatingWebhookConfiguration` in `ws1`, but the webhook will see `kcp.io/cluster: ws3` annotation on the reviewed object.
+Lastly, objects in admission review are annotated with the name of the workspace that owns that object. For example, when Widget `b` from `ws3` is being validated, its caught by `ValidatingWebhookConfiguration` or `ValidatingAdmissionPolicy` in `ws1`, but the webhook or policy evaluator will see `kcp.io/cluster: ws3` annotation on the reviewed object.

pkg/admission/validatingadmissionpolicy/validating_admission_policy.go

@@ -18,10 +18,13 @@ package validatingadmissionpolicy
 
 import (
 	"context"
+	"fmt"
 	"io"
 	"sync"
 
 	"k8s.io/apimachinery/pkg/api/meta"
+	"k8s.io/apimachinery/pkg/labels"
+	"k8s.io/apimachinery/pkg/runtime/schema"
 	"k8s.io/apiserver/pkg/admission"
 	"k8s.io/apiserver/pkg/admission/initializer"
 	"k8s.io/apiserver/pkg/admission/plugin/policy/generic"
@@ -33,12 +36,15 @@ import (
 	"k8s.io/client-go/informers"
 	"k8s.io/client-go/kubernetes"
 	"k8s.io/client-go/restmapper"
+	"k8s.io/client-go/tools/cache"
 	"k8s.io/klog/v2"
 
 	kcpdynamic "github.com/kcp-dev/client-go/dynamic"
 	kcpkubernetesinformers "github.com/kcp-dev/client-go/informers"
 	kcpkubernetesclientset "github.com/kcp-dev/client-go/kubernetes"
 	"github.com/kcp-dev/logicalcluster/v3"
+	apisv1alpha2 "github.com/kcp-dev/sdk/apis/apis/v1alpha2"
+	corev1alpha1 "github.com/kcp-dev/sdk/apis/core/v1alpha1"
 	kcpinformers "github.com/kcp-dev/sdk/client/informers/externalversions"
 	corev1alpha1informers "github.com/kcp-dev/sdk/client/informers/externalversions/core/v1alpha1"
 
@@ -59,7 +65,7 @@ func Register(plugins *admission.Plugins) {
 func NewKubeValidatingAdmissionPolicy() *KubeValidatingAdmissionPolicy {
 	return &KubeValidatingAdmissionPolicy{
 		Handler:   admission.NewHandler(admission.Connect, admission.Create, admission.Delete, admission.Update),
-		delegates: make(map[logicalcluster.Name]*stoppableValidatingAdmissionPolicy),
+		delegates: make(map[string]*stoppableValidatingAdmissionPolicy),
 	}
 }
 
@@ -75,15 +81,18 @@ type KubeValidatingAdmissionPolicy struct {
 	serverDone                      <-chan struct{}
 	authorizer                      authorizer.Authorizer
 
+	getAPIBindings func(clusterName logicalcluster.Name) ([]*apisv1alpha2.APIBinding, error)
+
 	lock      sync.RWMutex
-	delegates map[logicalcluster.Name]*stoppableValidatingAdmissionPolicy
+	delegates map[string]*stoppableValidatingAdmissionPolicy
 
 	logicalClusterDeletionMonitorStarter sync.Once
 }
 
 var _ admission.ValidationInterface = &KubeValidatingAdmissionPolicy{}
 var _ = initializers.WantsKubeClusterClient(&KubeValidatingAdmissionPolicy{})
 var _ = initializers.WantsKubeInformers(&KubeValidatingAdmissionPolicy{})
+var _ = initializers.WantsKcpInformers(&KubeValidatingAdmissionPolicy{})
 var _ = initializers.WantsServerShutdownChannel(&KubeValidatingAdmissionPolicy{})
 var _ = initializers.WantsDynamicClusterClient(&KubeValidatingAdmissionPolicy{})
 var _ = initializer.WantsAuthorizer(&KubeValidatingAdmissionPolicy{})
@@ -95,6 +104,33 @@ func (k *KubeValidatingAdmissionPolicy) SetKubeClusterClient(kubeClusterClient k
 
 func (k *KubeValidatingAdmissionPolicy) SetKcpInformers(local, global kcpinformers.SharedInformerFactory) {
 	k.logicalClusterInformer = local.Core().V1alpha1().LogicalClusters()
+	k.getAPIBindings = func(clusterName logicalcluster.Name) ([]*apisv1alpha2.APIBinding, error) {
+		return local.Apis().V1alpha2().APIBindings().Lister().Cluster(clusterName).List(labels.Everything())
+	}
+
+	_, _ = local.Core().V1alpha1().LogicalClusters().Informer().AddEventHandler(
+		cache.ResourceEventHandlerFuncs{
+			DeleteFunc: func(obj interface{}) {
+				cl, ok := obj.(*corev1alpha1.LogicalCluster)
+				if !ok {
+					return
+				}
+
+				clName := logicalcluster.Name(cl.Annotations[logicalcluster.AnnotationKey])
+
+				k.lock.Lock()
+				defer k.lock.Unlock()
+
+				prefix := clName.String() + ":"
+				for key, delegate := range k.delegates {
+					if len(key) > len(prefix) && key[:len(prefix)] == prefix {
+						delete(k.delegates, key)
+						delegate.stop()
+					}
+				}
+			},
+		},
+	)
 }
 
 func (k *KubeValidatingAdmissionPolicy) SetKubeInformers(local, global kcpkubernetesinformers.SharedInformerFactory) {
@@ -129,18 +165,42 @@ func (k *KubeValidatingAdmissionPolicy) Validate(ctx context.Context, a admissio
 		return err
 	}
 
-	delegate, err := k.getOrCreateDelegate(cluster.Name)
+	sourceCluster, err := k.getSourceClusterForGroupResource(cluster.Name, a.GetResource().GroupResource())
+	if err != nil {
+		return err
+	}
+
+	delegate, err := k.getOrCreateDelegate(cluster.Name, sourceCluster)
 	if err != nil {
 		return err
 	}
 
 	return delegate.Validate(ctx, a, o)
 }
 
+func (k *KubeValidatingAdmissionPolicy) getSourceClusterForGroupResource(clusterName logicalcluster.Name, groupResource schema.GroupResource) (logicalcluster.Name, error) {
+	objs, err := k.getAPIBindings(clusterName)
+	if err != nil {
+		return "", err
+	}
+
+	for _, apiBinding := range objs {
+		for _, br := range apiBinding.Status.BoundResources {
+			if br.Group == groupResource.Group && br.Resource == groupResource.Resource {
+				return logicalcluster.Name(apiBinding.Status.APIExportClusterName), nil
+			}
+		}
+	}
+
+	return clusterName, nil
+}
+
 // getOrCreateDelegate creates an actual plugin for clusterName.
-func (k *KubeValidatingAdmissionPolicy) getOrCreateDelegate(clusterName logicalcluster.Name) (*stoppableValidatingAdmissionPolicy, error) {
+func (k *KubeValidatingAdmissionPolicy) getOrCreateDelegate(clusterName logicalcluster.Name, sourceCluster logicalcluster.Name) (*stoppableValidatingAdmissionPolicy, error) {
+	delegateKey := fmt.Sprintf("%s:%s", clusterName.String(), sourceCluster.String())
+
 	k.lock.RLock()
-	delegate := k.delegates[clusterName]
+	delegate := k.delegates[delegateKey]
 	k.lock.RUnlock()
 
 	if delegate != nil {
@@ -150,7 +210,7 @@ func (k *KubeValidatingAdmissionPolicy) getOrCreateDelegate(clusterName logicalc
 	k.lock.Lock()
 	defer k.lock.Unlock()
 
-	delegate = k.delegates[clusterName]
+	delegate = k.delegates[delegateKey]
 	if delegate != nil {
 		return delegate, nil
 	}
@@ -185,17 +245,17 @@ func (k *KubeValidatingAdmissionPolicy) getOrCreateDelegate(clusterName logicalc
 	plugin.SetDrainedNotification(ctx.Done())
 	plugin.SetAuthorizer(k.authorizer)
 	plugin.SetClusterName(clusterName)
-	plugin.SetSourceFactory(func(_ informers.SharedInformerFactory, client kubernetes.Interface, dynamicClient dynamic.Interface, restMapper meta.RESTMapper, clusterName logicalcluster.Name) generic.Source[validating.PolicyHook] {
+	plugin.SetSourceFactory(func(_ informers.SharedInformerFactory, client kubernetes.Interface, dynamicClient dynamic.Interface, restMapper meta.RESTMapper, _ logicalcluster.Name) generic.Source[validating.PolicyHook] {
 		return generic.NewPolicySource(
-			k.globalKubeSharedInformerFactory.Admissionregistration().V1().ValidatingAdmissionPolicies().Informer().Cluster(clusterName),
-			k.globalKubeSharedInformerFactory.Admissionregistration().V1().ValidatingAdmissionPolicyBindings().Informer().Cluster(clusterName),
+			k.globalKubeSharedInformerFactory.Admissionregistration().V1().ValidatingAdmissionPolicies().Informer().Cluster(sourceCluster),
+			k.globalKubeSharedInformerFactory.Admissionregistration().V1().ValidatingAdmissionPolicyBindings().Informer().Cluster(sourceCluster),
 			validating.NewValidatingAdmissionPolicyAccessor,
 			validating.NewValidatingAdmissionPolicyBindingAccessor,
 			validating.CompilePolicy,
 			nil,
 			dynamicClient,
 			restMapper,
-			clusterName,
+			sourceCluster,
 		)
 	})
 
@@ -204,7 +264,7 @@ func (k *KubeValidatingAdmissionPolicy) getOrCreateDelegate(clusterName logicalc
 		return nil, err
 	}
 
-	k.delegates[clusterName] = delegate
+	k.delegates[delegateKey] = delegate
 
 	return delegate, nil
 }
@@ -213,19 +273,24 @@ func (k *KubeValidatingAdmissionPolicy) logicalClusterDeleted(clusterName logica
 	k.lock.Lock()
 	defer k.lock.Unlock()
 
-	delegate := k.delegates[clusterName]
-
 	logger := klog.Background().WithValues("clusterName", clusterName)
 
-	if delegate == nil {
+	prefix := clusterName.String() + ":"
+	found := false
+	for key, delegate := range k.delegates {
+		if len(key) > len(prefix) && key[:len(prefix)] == prefix {
+			delete(k.delegates, key)
+			delegate.stop()
+			found = true
+		}
+	}
+
+	if !found {
 		logger.V(3).Info("received event to stop validating admission policy for logical cluster, but it wasn't in the map")
 		return
 	}
 
 	logger.V(2).Info("stopping validating admission policy for logical cluster")
-
-	delete(k.delegates, clusterName)
-	delegate.stop()
 }
 
 type stoppableValidatingAdmissionPolicy struct {