Optionally serve metrics over TLS

[npapapietro]

Is your feature request related to a problem? Please describe.
Running the operator in environments with very strict TLS or mTLS rules (enforced by Istio for example), there are some edge cases that aren't configurable for TLS. The CRDs are pretty well covered to setup connections to Grafana and/or Datasources that respect mTLS.

There are two endpoints on the controller that need to expose some way to configure. The health probe and the metrics endpoint

(If applicable)If your feature request solves a bug please provide a link to the community issue

Describe the solution you'd like
Expose some more cli flags for pass in certs and boolean flag on TLS mode for health and metrics endpoints. Add these to the helm chart (enable ServiceMonitor tlsconfig)

Describe alternatives you've considered
Currently I just flag off metrics and remove probes, but in large environments or fast moving environments no metrics on a service is a blind spot.

Additional context
#2549

[weisdd]

@npapapietro thanks for opening the issue. The same question remains as in the linked PR:

[...] Aside from other details, it has to include an explanation on why TLS setup is not transparent to the end workloads like in more or less any mTLS setup.

So, could you, please, explain in details why you expect a workload to serve TLS instead of relying on transparent mTLS setup? - "there are some edge cases that aren't configurable for TLS" is not specific enough.
Also, please, include links to documentation where the limitation you're facing is mentioned.

[npapapietro]

The health probe TLS doesn't seem to be configurable by controller-runtime, but this is likely okay, because that's not client based but rather kubernetes.

The metrics endpoint on the other hand is being scraped by another service and is subject to TLS rules.

It is configured by this line

grafana-operator/main.go

Line 228 in 95d3b53

Metrics: metricsserver.Options{BindAddress: operatorConfig.MetricsAddr},

I would like to have some sort of cli flag to pass in these options https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/metrics/server. Most can be auto configured via the comment

	// CertDir is the directory that contains the server key and certificate. Defaults to
	// <temp-dir>/k8s-metrics-server/serving-certs.
	//
	// Note: This option is only used when TLSOpts does not set GetCertificate.
	// Note: If certificate or key doesn't exist a self-signed certificate will be used.

but the SecureServing flag would need to be exposed.

I believe the golang code modification can be achieved by this flag, and the rest of tls can be handled by the helm chart mounting a tls secret volume on that path

[weisdd]

Well, it still doesn't answer the question :)

I'm not an Istio expert (haven't seen it for ages), but, from what I gather, mTLS is transparent to the end apps, because traffic gets proxied either through Envoy sidecars or through ztunnel. Even if you have strict mTLS setup, everything is handled by Istio data plane, so the apps are not even aware they're within the service mesh.
So, I'm not sure why you'd like to make the operator serve something with TLS. That's why I'm asking you to explain everything in details.

[npapapietro]

Istio perhaps was a bad example then. I've not used it some time either, but I do remember their strict mode.

I'm running your operator in specific environments where E2E encryption at the application level is required. This is common where FIPS compliance is required. Basically even at the pod level, no clear text (including container to container communication within pod). I'm not always given the tool that certain environments use to enforce this, but it will be flagged nonetheless. This is my strongest motivation for this request

Another reason as well a service mesh type solution might not work is again back to certain environments, I can't control security auditors attempting to use a scraper that lives outside the cluster (not ideal, but sometimes that's not my call).

[weisdd]

Alright, now it makes sense, thanks for clarifying :)

[Baarsgaard]

Spent some time looking into this.
If someone has an explicit requirement of mTLS, no CNI can provide that.
Which Istio backs up
Cilium can provide MutualAuthentication, although similar, is not the same as mTLS as it's built on WireGuard.

Long story short, users have two options.

1. Use Linkerd, Istio, or similar style service mesh and enable FIPS compliance.
2. Manually configure applications in every pod/container with client certificates.
   This may be an even bigger challenge if the requirements include "Must be ECC certificates" due to patchy support.

But if the requirements specify inter-process mTLS, application level client certificates are required and that rules out option 1.