kube-mgmt resource replication requires cluster wide access

[shrinandj]

I tried to run the kube-mgmt sidecar to opa on my Kubernetes cluster and granted it limited privileges. The role that the deployment was running with only had access to it's own namespace and nothing else.

I only had --replicate=v1/pods as the argument to kube-mgmt. I saw the following errors:

E0412 05:31:24.735947       1 reflector.go:201] github.com/open-policy-agent/kube-mgmt/pkg/policies/configmap.go:100: Failed to list *v1.ConfigMap: unknown (get configmaps)

If I understand the code right, it seems that kube-mgmt currently watches for resources across all namespaces.

File: pkg/policies/configmap.go

        source := cache.NewListWatchFromClient(
                client,
                "configmaps",
                v1.NamespaceAll, <<<---------------
                fields.Everything())

File: ./pkg/data/generic.go

        source := cache.NewListWatchFromClient(
                client,
                s.ns.Resource,
                api.NamespaceAll,  <<<--------------
                fields.Everything())

As a result, kube-mgmt can only run if it is given a role that has cluster wide access to these resources.

I changed the cluster-binding to cluster-admin (basically, ran opa and kube-mgmt as root) and things worked fine.

It'll be good if, kube-mgmt can watch resources in the namespace that the user provides (and maybe default to all).

[tsandall]

That seems like a reasonable configuration setting.

The reason that kube-mgmt defaults to watching all namespaces is that it's intended to support OPA as an external admission controller for the cluster.

This means that OPA is responsible for enforcing policy decisions that affect the entire cluster.

To do this, OPA may need to have Kubernetes resources from any namespace.

Minimally we could include some more information in the README about the permissions that OPA has to have.

[alkar]

Since --policies is passing in the list of namespaces that policies should be taken from, couldn't we change the v1. NamespaceAll flag so that we can achieve least privileged role here?

I understand that it might need to look across all namespaces but the current default behaviour is to constrain it to three namespaces and I'm assuming most users would want to restrict to a subset of namespaces.

[tsandall]

@alkar yes that makes sense. If you'd like to submit a PR, please go ahead. EDIT: The only bit I'm unsure about is whether client-go lets you specifying multiple namespaces to read from. Maintaining a set of N readers might be a bit more work.

[alkar]

@tsandall very good point. I'll have a look and see if it's possible.

[alkar]

Looking at client-go it seems informers is either one namespace or all as suspected. Unfortunately, it seems the only option would be to create a pool of informers watching individual namespaces and then aggregate their results. At this point, I'm not sure it's worth the effort.

[rosskusler]

Some of us are using kube-mgmt for things other than an admission controller and we don't always have the ability to access resources in every namespace. For these use cases it would be really useful if we could specify a list of namespaces on a per resource type basis. Something along the lines of:

--replicate=v1/pods:default,ns1,ns2
--replicate=v1/secrets:ns3

This would only allow us to replicate pods from the namespaces named default, ns1, and ns2 while secrets would only be replicated from ns3.

[tsandall]

@rosskusler that seems reasonable. PRs are welcome.