Merge pull request #3743 from olamilekan000/implement-crossworskapce-validating-admission-policy

add cross-workspace implementation for ValidatingAdmissionPolicy

Author: kcp-ci-bot

docs/content/concepts/apis/admission-webhooks.md

@@ -1,13 +1,18 @@
+---
+description: >
+  How admission webhooks and validating admission policies work across workspaces in kcp.
+---
+
 # Admission Webhooks
 
-kcp extends the vanilla [admission plugins](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/) for webhooks, and makes them cluster-aware.
+kcp extends the vanilla [admission plugins](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/) for webhooks and validating admission policies, and makes them cluster-aware.
 
 ```mermaid
 flowchart TD
     subgraph ws1["API Provider Workspace ws1"]
         export["Widgets APIExport"]
         schema["Widgets APIResourceSchema<br/>(widgets.v1.example.org)"]
-        webhook["Mutating/ValidatingWebhookConfiguration<br/>for widgets.v1.example.org<br/><br/>Handle a from ws2 (APIResourceSchema)<br/>Handle b from ws3 (APIResourceSchema)<br/>Handle a from ws1 (CRD)"]
+        webhook["Mutating/ValidatingWebhookConfiguration<br/>ValidatingAdmissionPolicy<br/>for widgets.v1.example.org<br/><br/>Handle a from ws2 (APIResourceSchema)<br/>Handle b from ws3 (APIResourceSchema)<br/>Handle a from ws1 (CRD)"]
         crd["Widgets CustomResourceDefinition<br/>(widgets.v1.example.org)"]
         
         export --> schema
@@ -37,9 +42,39 @@ flowchart TD
     class export,schema,webhook,crd,binding1,binding2 resource;
 ```
 
-When an object is to be mutated or validated, the webhook admission plugin ([`apis.kcp.io/MutatingWebhook`](https://github.com/kcp-dev/kcp/tree/main/pkg/admission/mutatingwebhook) and [`apis.kcp.io/ValidatingWebhook`](https://github.com/kcp-dev/kcp/tree/main/pkg/admission/validatingwebhook) respectively) looks for the owner of the resource schema. Once found, it then dispatches the handling for that object in the owner's workspace. There are two such cases in the diagram above:
+When an object is to be mutated or validated, the admission plugins ([`apis.kcp.io/MutatingWebhook`](https://github.com/kcp-dev/kcp/tree/main/pkg/admission/mutatingwebhook), [`apis.kcp.io/ValidatingWebhook`](https://github.com/kcp-dev/kcp/tree/main/pkg/admission/validatingwebhook), and [`ValidatingAdmissionPolicy`](https://github.com/kcp-dev/kcp/tree/main/pkg/admission/validatingadmissionpolicy) respectively) look for the owner of the resource schema. Once found, they then dispatch the handling for that object in the owner's workspace. There are two such cases in the diagram above:
+
+* **Admitting bound resources.** During the request handling, Widget objects inside the consumer workspaces `ws2` and `ws3` are picked up by the respective admission plugin. The plugin sees the resource's schema comes from an APIBinding, and so it sets up an instance of the admission plugin to be working with its APIExport's workspace, in `ws1`. Afterwards, normal admission flow continues: the request is dispatched to all eligible webhook configurations or validating admission policies inside `ws1` and the object in request is mutated or validated.
+* **Admitting local resources.** The second case is when the webhook configuration or validating admission policy exists in the same workspace as the object it's handling. The admission plugin sees the resource is not sourced via an `APIBinding`, and so it looks for eligible webhook configurations or policies locally, and dispatches the request accordingly. The same would of course be true if `APIExport` and its `APIBinding` lived in the same workspace: the `APIExport` would resolve to the same cluster.
+
+## ValidatingAdmissionPolicy Support
+
+kcp supports cross-workspace `ValidatingAdmissionPolicy` and `ValidatingAdmissionPolicyBinding` resources, similar to how it supports cross-workspace webhooks. When a resource is created in a consumer workspace that is bound via an `APIBinding`, the `ValidatingAdmissionPolicy` plugin will:
+
+1. Check the `APIBinding` to find the source workspace (`APIExport` workspace)
+2. Look for `ValidatingAdmissionPolicy` and `ValidatingAdmissionPolicyBinding` resources in the source workspace
+3. Apply those policies to validate the resource in the consumer workspace
+
+This means that policies defined in an `APIExport` workspace will automatically apply to all resources created in consuming workspaces, providing a consistent validation experience across all consumers of an API.
+
+### Example
+
+Consider a scenario where:
+- **Provider workspace** (`root:provider`) has:
+  - An `APIExport` for `cowboys.wildwest.dev`
+  - A `ValidatingAdmissionPolicy` that rejects cowboys with `intent: "bad"`
+  - A `ValidatingAdmissionPolicyBinding` that binds the policy
+  
+- **Consumer workspace** (`root:consumer`) has:
+  - An `APIBinding` that binds to the provider's `APIExport`
+  - A user trying to create a cowboy with `intent: "bad"`
+
+When the user creates the cowboy in the consumer workspace, the `ValidatingAdmissionPolicy` plugin will:
+1. Detect that the cowboy resource comes from an `APIBinding`
+2. Look up the source workspace (provider workspace)
+3. Find and apply the policy from the provider workspace
+4. Reject the cowboy creation because it violates the policy
 
-* **Admitting bound resources.** During the request handling, Widget objects inside the consumer workspaces `ws2` and `ws3` are picked up by the respective webhook admission plugin. The plugin sees the resource's schema comes from an APIBinding, and so it sets up an instance of `{Mutating,Validating}Webhook` to be working with its APIExport's workspace, in `ws1`. Afterwards, normal webhook admission flow continues: the request is dispatched to all eligible webhook configurations inside `ws1` and the object in request is mutated or validated.
-* **Admitting local resources.** The second case is when the webhook configuration exists in the same workspace as the object it's handling. The admission plugin sees the resource is not sourced via an APIBinding, and so it looks for eligible webhook configurations locally, and dispatches the request to the webhooks there. The same would of course be true if APIExport and its APIBinding lived in the same workspace: the APIExport would resolve to the same cluster.
+This ensures that API providers can enforce consistent validation rules across all consumers of their APIs.
 
-Lastly, objects in admission review are annotated with the name of the workspace that owns that object. For example, when Widget `b` from `ws3` is being validated, its caught by `ValidatingWebhookConfiguration` in `ws1`, but the webhook will see `kcp.io/cluster: ws3` annotation on the reviewed object.
+Lastly, objects in admission review are annotated with the name of the workspace that owns that object. For example, when Widget `b` from `ws3` is being validated, its caught by `ValidatingWebhookConfiguration` or `ValidatingAdmissionPolicy` in `ws1`, but the webhook or policy evaluator will see `kcp.io/cluster: ws3` annotation on the reviewed object.

pkg/admission/validatingadmissionpolicy/validating_admission_policy.go

@@ -22,6 +22,8 @@ import (
 	"sync"
 
 	"k8s.io/apimachinery/pkg/api/meta"
+	"k8s.io/apimachinery/pkg/labels"
+	"k8s.io/apimachinery/pkg/runtime/schema"
 	"k8s.io/apiserver/pkg/admission"
 	"k8s.io/apiserver/pkg/admission/initializer"
 	"k8s.io/apiserver/pkg/admission/plugin/policy/generic"
@@ -33,12 +35,15 @@ import (
 	"k8s.io/client-go/informers"
 	"k8s.io/client-go/kubernetes"
 	"k8s.io/client-go/restmapper"
+	"k8s.io/client-go/tools/cache"
 	"k8s.io/klog/v2"
 
 	kcpdynamic "github.com/kcp-dev/client-go/dynamic"
 	kcpkubernetesinformers "github.com/kcp-dev/client-go/informers"
 	kcpkubernetesclientset "github.com/kcp-dev/client-go/kubernetes"
 	"github.com/kcp-dev/logicalcluster/v3"
+	apisv1alpha2 "github.com/kcp-dev/sdk/apis/apis/v1alpha2"
+	corev1alpha1 "github.com/kcp-dev/sdk/apis/core/v1alpha1"
 	kcpinformers "github.com/kcp-dev/sdk/client/informers/externalversions"
 	corev1alpha1informers "github.com/kcp-dev/sdk/client/informers/externalversions/core/v1alpha1"
 
@@ -75,15 +80,18 @@ type KubeValidatingAdmissionPolicy struct {
 	serverDone                      <-chan struct{}
 	authorizer                      authorizer.Authorizer
 
-	lock      sync.RWMutex
-	delegates map[logicalcluster.Name]*stoppableValidatingAdmissionPolicy
+	getAPIBindings func(clusterName logicalcluster.Name) ([]*apisv1alpha2.APIBinding, error)
+
+	delegatesLock sync.RWMutex
+	delegates     map[logicalcluster.Name]*stoppableValidatingAdmissionPolicy
 
 	logicalClusterDeletionMonitorStarter sync.Once
 }
 
 var _ admission.ValidationInterface = &KubeValidatingAdmissionPolicy{}
 var _ = initializers.WantsKubeClusterClient(&KubeValidatingAdmissionPolicy{})
 var _ = initializers.WantsKubeInformers(&KubeValidatingAdmissionPolicy{})
+var _ = initializers.WantsKcpInformers(&KubeValidatingAdmissionPolicy{})
 var _ = initializers.WantsServerShutdownChannel(&KubeValidatingAdmissionPolicy{})
 var _ = initializers.WantsDynamicClusterClient(&KubeValidatingAdmissionPolicy{})
 var _ = initializer.WantsAuthorizer(&KubeValidatingAdmissionPolicy{})
@@ -95,6 +103,32 @@ func (k *KubeValidatingAdmissionPolicy) SetKubeClusterClient(kubeClusterClient k
 
 func (k *KubeValidatingAdmissionPolicy) SetKcpInformers(local, global kcpinformers.SharedInformerFactory) {
 	k.logicalClusterInformer = local.Core().V1alpha1().LogicalClusters()
+	k.getAPIBindings = func(clusterName logicalcluster.Name) ([]*apisv1alpha2.APIBinding, error) {
+		return local.Apis().V1alpha2().APIBindings().Lister().Cluster(clusterName).List(labels.Everything())
+	}
+
+	_, _ = local.Core().V1alpha1().LogicalClusters().Informer().AddEventHandler(
+		cache.ResourceEventHandlerFuncs{
+			DeleteFunc: func(obj interface{}) {
+				cl, ok := obj.(*corev1alpha1.LogicalCluster)
+				if !ok {
+					return
+				}
+
+				clName := logicalcluster.Name(cl.Annotations[logicalcluster.AnnotationKey])
+
+				k.delegatesLock.Lock()
+				defer k.delegatesLock.Unlock()
+
+				for key, delegate := range k.delegates {
+					if key == clName {
+						delete(k.delegates, key)
+						delegate.stop()
+					}
+				}
+			},
+		},
+	)
 }
 
 func (k *KubeValidatingAdmissionPolicy) SetKubeInformers(local, global kcpkubernetesinformers.SharedInformerFactory) {
@@ -129,26 +163,48 @@ func (k *KubeValidatingAdmissionPolicy) Validate(ctx context.Context, a admissio
 		return err
 	}
 
-	delegate, err := k.getOrCreateDelegate(cluster.Name)
+	sourceCluster, err := k.getSourceClusterForGroupResource(cluster.Name, a.GetResource().GroupResource())
+	if err != nil {
+		return err
+	}
+
+	delegate, err := k.getOrCreateDelegate(sourceCluster)
 	if err != nil {
 		return err
 	}
 
 	return delegate.Validate(ctx, a, o)
 }
 
+func (k *KubeValidatingAdmissionPolicy) getSourceClusterForGroupResource(clusterName logicalcluster.Name, groupResource schema.GroupResource) (logicalcluster.Name, error) {
+	objs, err := k.getAPIBindings(clusterName)
+	if err != nil {
+		return "", err
+	}
+
+	for _, apiBinding := range objs {
+		for _, br := range apiBinding.Status.BoundResources {
+			if br.Group == groupResource.Group && br.Resource == groupResource.Resource {
+				return logicalcluster.Name(apiBinding.Status.APIExportClusterName), nil
+			}
+		}
+	}
+
+	return clusterName, nil
+}
+
 // getOrCreateDelegate creates an actual plugin for clusterName.
 func (k *KubeValidatingAdmissionPolicy) getOrCreateDelegate(clusterName logicalcluster.Name) (*stoppableValidatingAdmissionPolicy, error) {
-	k.lock.RLock()
+	k.delegatesLock.RLock()
 	delegate := k.delegates[clusterName]
-	k.lock.RUnlock()
+	k.delegatesLock.RUnlock()
 
 	if delegate != nil {
 		return delegate, nil
 	}
 
-	k.lock.Lock()
-	defer k.lock.Unlock()
+	k.delegatesLock.Lock()
+	defer k.delegatesLock.Unlock()
 
 	delegate = k.delegates[clusterName]
 	if delegate != nil {
@@ -210,8 +266,8 @@ func (k *KubeValidatingAdmissionPolicy) getOrCreateDelegate(clusterName logicalc
 }
 
 func (k *KubeValidatingAdmissionPolicy) logicalClusterDeleted(clusterName logicalcluster.Name) {
-	k.lock.Lock()
-	defer k.lock.Unlock()
+	k.delegatesLock.Lock()
+	defer k.delegatesLock.Unlock()
 
 	delegate := k.delegates[clusterName]