Cannot create resources in non-default namespace of projects

[scotwells]

Summary

I can't create a resource in a non-default directory of a project because it complains the namespace can't be found even though the namespace exists.

Expected Behavior

Be able to create resources in non-default namespaces.

Notes

Output from running kubectl apply on this kustomization program.

$ kubectl --kubeconfig .tmp/config apply -k config/gateway --server-side -v 6
I1105 10:15:06.495899   75437 loader.go:373] Config loaded from file:  .tmp/config
I1105 10:15:07.023297   75437 round_trippers.go:553] GET https://api.datum.net/apis/resourcemanager.miloapis.com/v1alpha1/projects/datum-cloud/control-plane/openapi/v2?timeout=32s 200 OK in 523 milliseconds
I1105 10:15:07.158139   75437 round_trippers.go:553] GET https://api.datum.net/apis/resourcemanager.miloapis.com/v1alpha1/projects/datum-cloud/control-plane/openapi/v3?timeout=32s 200 OK in 44 milliseconds
I1105 10:15:07.160484   75437 round_trippers.go:553] GET https://api.datum.net/openapi/v3/api/v1?hash=72C5066E2BFF35261236CEB8E92E38F20190DAC513654EC303B7C36862E44E8681C25446D931945CDE027302361687EB59EE59176B0292943FE02992D17CDE4C&timeout=32s 200 OK in 1 milliseconds
I1105 10:15:07.172202   75437 round_trippers.go:553] GET https://api.datum.net/openapi/v3/apis/discovery.k8s.io/v1?hash=77D8692D7E58CF63427EE8D4C4DDFF61F2AAFDB5834DFF3B0FB814B800EDD4DC0E5EAA341CBFAB6884B53628585D62283D2640641E9A2F641068319EE741BF56&timeout=32s 200 OK in 0 milliseconds
I1105 10:15:07.175556   75437 round_trippers.go:553] GET https://api.datum.net/openapi/v3/apis/gateway.networking.k8s.io/v1?hash=CFB5776FAA314BC75524731AFEF91682161AB1080E93A7A787F452B2223A9C83B0546AADFE3CCF680CA33C1E3E9F29AC19DCA43FFEB60C5F525E0C19972015F7&timeout=32s 200 OK in 1 milliseconds
I1105 10:15:07.363690   75437 round_trippers.go:553] PATCH https://api.datum.net/apis/resourcemanager.miloapis.com/v1alpha1/projects/datum-cloud/control-plane/api/v1/namespaces/datum-net?fieldManager=kubectl&fieldValidation=Strict&force=false 200 OK in 173 milliseconds
namespace/datum-net serverside-applied
I1105 10:15:07.476057   75437 round_trippers.go:553] PATCH https://api.datum.net/apis/resourcemanager.miloapis.com/v1alpha1/projects/datum-cloud/control-plane/apis/discovery.k8s.io/v1/namespaces/datum-net/endpointslices/datum-net?fieldManager=kubectl&fieldValidation=Strict&force=false 404 Not Found in 110 milliseconds
I1105 10:15:07.685150   75437 round_trippers.go:553] PATCH https://api.datum.net/apis/resourcemanager.miloapis.com/v1alpha1/projects/datum-cloud/control-plane/apis/gateway.networking.k8s.io/v1/namespaces/datum-net/gateways/datum-net-gateway?fieldManager=kubectl&fieldValidation=Strict&force=false 404 Not Found in 208 milliseconds
I1105 10:15:07.778681   75437 round_trippers.go:553] PATCH https://api.datum.net/apis/resourcemanager.miloapis.com/v1alpha1/projects/datum-cloud/control-plane/apis/gateway.networking.k8s.io/v1/namespaces/datum-net/httproutes/datum-net-http-route?fieldManager=kubectl&fieldValidation=Strict&force=false 404 Not Found in 93 milliseconds
I1105 10:15:07.865484   75437 round_trippers.go:553] PATCH https://api.datum.net/apis/resourcemanager.miloapis.com/v1alpha1/projects/datum-cloud/control-plane/apis/gateway.networking.k8s.io/v1/namespaces/datum-net/httproutes/datum-net-redirect?fieldManager=kubectl&fieldValidation=Strict&force=false 409 Conflict in 86 milliseconds
I1105 10:15:07.866540   75437 helpers.go:246] server response object: [{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "namespace \"datum-net\" not found",
  "reason": "NotFound",
  "details": {
    "name": "datum-net",
    "kind": "namespace"
  },
  "code": 404
}]
I1105 10:15:07.866586   75437 helpers.go:246] server response object: [{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "namespace \"datum-net\" not found",
  "reason": "NotFound",
  "details": {
    "name": "datum-net",
    "kind": "namespace"
  },
  "code": 404
}]
I1105 10:15:07.866596   75437 helpers.go:246] server response object: [{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "namespace \"datum-net\" not found",
  "reason": "NotFound",
  "details": {
    "name": "datum-net",
    "kind": "namespace"
  },
  "code": 404
}]

Confirmed the namespace exists in the project.

$ kubectl --kubeconfig .tmp/config get namespaces                            
NAME          STATUS   AGE
datum-net     Active   56d
default       Active   56d
milo-os-com   Active   56d
milo-system   Active   13d

Confirmed I can retrieve existing resources in the namespace.

$ kubectl --kubeconfig .tmp/config get gateways -n datum-net -v 6
I1105 10:20:23.394181   81631 loader.go:373] Config loaded from file:  .tmp/config
I1105 10:20:23.831796   81631 round_trippers.go:553] GET https://api.datum.net/apis/resourcemanager.miloapis.com/v1alpha1/projects/datum-cloud/control-plane/apis/gateway.networking.k8s.io/v1/namespaces/datum-net/gateways?limit=500 200 OK in 429 milliseconds
NAME                CLASS                         ADDRESS                                                           PROGRAMMED   AGE
datum-net-gateway   datum-external-global-proxy   87a38568-b47a-4368-80af-ee787b1de9cd.prism.global.datum-dns.net   True         56d

FYI @zachsmith1 - I expect this is related to the virtualization we did with project control planes.

[scotwells]

@zachsmith1 good hint on this potentially being part of the admission plugin. I was able to validate that this is being caused by the deny-unapproved-user ValidatingAdmissionPolicy we have in place for users that are still on the waitlist.

$ kubectl --kubeconfig datumcfg create configmap -n test-configmap-creation test-configmap                                                    
error: failed to create configmap: namespace "test-configmap-creation" not found

$ kubectl -n datum-system exec deployments/kubectl -it -- kubectl --kubeconfig /etc/kubernetes/auth/kubeconfig delete validatingadmissionpolicies 
deny-unapproved-user
validatingadmissionpolicy.admissionregistration.k8s.io "deny-unapproved-user" deleted

$ kubectl --kubeconfig datumcfg create configmap -n test-configmap-creation test-configmap
configmap/test-configmap created

[zachsmith1]

@scotwells ya validating and mutating webhooks all have similar logic here to getNamespace. Looking at how we are easily wrap or override this currently: https://github.com/kubernetes/apiserver/blob/master/pkg/admission/plugin/policy/validating/dispatcher.go#L181