How to define Kubernetes NetworkPolicies?

[bygui86]

Hi guys

this tool is amazing but I'm struggling to let it work when k8s NetworkPolicies are in place.

With ingress NetPol like following no problem:

# deny all ingress connections
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: ingress-default-deny-all
spec:
  podSelector: {}
  policyTypes:
    - Ingress

Unfortunately with egress NetPol like following Reloader does not work anymore:

# deny all egress connection excepts to DNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: egress-default-deny-all
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: TCP
          port: 53
        - protocol: UDP
          port: 53

---

# allow egress connection to k8s api-server
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: egress-from-reloader
spec:
  podSelector:
    matchLabels:
      app: reloader
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: MASTER_NODE_IP/32
      ports:
        - protocol: TCP
          port: 443

There are COMPLETELY NO LOGS so I don't really understand why.

Which connection does Reloader require? Who is Reloader talking to?

Thanks in advance for any help!

[stefan-marc]

@bygui86

Can you the following:
k cluster-info
Get the IP from "Kubernetes control plane is running at <IP_addr>"

Put that in the cidr section. Should work :).
Let me know how it goes.

P.S.
It would be nice to have this integrated in the helm chart together with the monitor allowed namespaces as apposed to ignore namespaces. :))

[bygui86]

@smarcu92 thanks for the suggestion, here below the NetworkPolicies working for me:

# ingress - deny all (whole namespace)
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: ingress-default-deny-all
spec:
  podSelector: {}
  policyTypes:
    - Ingress
---
# egress - deny all but DNS (whole namespace)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: egress-default-deny-all
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: TCP
          port: 53
        - protocol: UDP
          port: 53
---
# egress - allow k8s api-server only from reloader pods
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: egress-from-reloader
spec:
  podSelector:
    matchLabels:
      app: reloader-reloader
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: CONTROL_LANE_IP_ADDRESS/32
      ports:
        - protocol: TCP
          port: 443

And here the Prometheus operator ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: reloader
  namespace: reloader
  labels:
    app: reloader
spec:
  selector:
    matchLabels:
      app: reloader
  namespaceSelector:
    matchNames:
      - reloader
  endpoints:
    - port: http
      interval: 30s
      scrapeTimeout: 29s

I suggest to add them not only in the Helm chart, but also in the plain manifests.

[stefan-marc]

Hi @bygui86,

Awesome job! 👏 thanks for the manifests as well!
Just a suggestion. I'm not sure if you need the DNS setting for egress though🤔 . I would test the configuration without and see if it's working. The stricter the policy the better.

Thanks!

[bygui86]

@smarcu92 I haven't tested Reloader itself, but some other operators (Strimzi for Kafka, Prometheus, Instaclustr for Cassandra, etc) don't work without DNS-allow egress.

I noticed that, in order to communicate with the k8s api-server (and getting back the control plane IP address), an operator passes through the DNS asking to resolve a k8s Service named kubernetes in the default Namespace.
There is no need to allow the operator to communicate with default namespace or kubernetes service itself, but without the DNS the operator won't reach the control plane.

[faizanahmad055]

@bygui86 do you still face this issue with the latest version?

[bygui86]

Yes, still there :(

[faizanahmad055]

Can you please share you reloader specs as well as the logs?

[bygui86]

@faizanahmad055 everything seems to work properly now with following NetworkPolicies:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: egress-default-deny-all
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: TCP
          port: 53
        - protocol: UDP
          port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: egress-from-reloader
spec:
  podSelector:
    matchLabels:
      app: reloader
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: CONTROL_PLANE_IP_ADDRESS/28
      ports:
        - protocol: TCP
          port: 443
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: ingress-default-deny-all
spec:
  podSelector: {}
  policyTypes:
    - Ingress

Thanks for your support, I think we can close this issue :)

[devopstales]

Can this be added to the helm chart?