Remove kube-rbac-proxy in favor of WithAuthenticationAndAuthorization

[synhershko]

Fix #1094 #1093

[bananasplit393]

The PR looks like it removes the kube-rbac-proxy flag in its entirety.
I'm a little bit in doubt about how to use the WithAuthenticationAndAuthorization flag. Can someone send/share some documentation on how to implement it?

I don't see the WithAuthenticationAndAuthorization flag in the values file, which is why i am asking for help.

Should i just do something like this:

WithAuthenticationAndAuthorization:
enable: true

Do i need to set any other values?

[josedev-union]

@synhershko It looks good to me.
One thing from my mind is we can rename RBAC resources to *-metrics-auth-role and *-metrics-auth-rolebinding instead of *-proxy-role and *-proxy-rolebinding in both config dir and helm templates.
https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/charts/opensearch-operator/templates/opensearch-operator-proxy-role-cr.yaml
https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/charts/opensearch-operator/templates/opensearch-operator-proxy-rolebinding.yaml
https://github.com/opensearch-project/opensearch-k8s-operator/tree/main/opensearch-operator/config/rbac

[josedev-union]

The PR looks like it removes the kube-rbac-proxy flag in its entirety. I'm a little bit in doubt about how to use the WithAuthenticationAndAuthorization flag. Can someone send/share some documentation on how to implement it?

I don't see the WithAuthenticationAndAuthorization flag in the values file, which is why i am asking for help.

Should i just do something like this:

WithAuthenticationAndAuthorization: enable: true

Do i need to set any other values?

There is no need for helm chart value change. This change is just to replace kube-rbac-proxy (deprecated) with WithAuthenticationAndAuthorization.
controller-runtime supports auth and authz via TokenReviews and SubjsctAccessReviews out of the box now. So we can rely on this builtin option.
This is not optional. We enable as mandatory.
ref:
https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/metrics/filters#WithAuthenticationAndAuthorization

sample code snippet https://github.com/etcd-io/etcd-operator/blob/main/cmd/main.go#L107-L123

[josedev-union]

secured metrics port tested. (10.43.27.11 is operator svc IP.)

$ cat <<EOF|kubectl apply -f-
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: opensearch-operator-default-metrics-reader-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: opensearch-operator-default-metrics-reader
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
EOF
$ kubectl run -it curl --image=radial/busyboxplus:curl /bin/sh
$ TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
# without token, it returns `Unauthorized`
$ curl  -k -H "Authorization: Bearer $TOKEN" https://10.43.27.11:8443/metrics