ComputeDomain: secure-by-default IMEX daemon network communication (via mTLS)

[jgehrcke]

That implies setting IMEX_ENABLE_AUTH_ENCRYPTION to 1 and

1. Either having Kubernetes cert manager automatically generate certificate and key material (that could be one option) or
2. Allowing for users to inject their own key material via Kubernetes secrets.

or both.

[jgehrcke]

We may also think about locking this down by default, by generating a standalone root CA and a shared end-entity cert per daemon ensemble.

The primary authentication mechanism would then be the very fact of being derived from a certain private key at all -- ignoring any CN/SAN-based authentication. This provides the desired secrecy, integrity, authenticity without being able to distinguish individual daemons.

In that case, we will have to be able to effectively disable hostname verification in the IMEX daemon's TLS library (when enabling (m)TLS: in 'proper TLS', a complication would otherwise be hostname verification -- however, we don't want to be in a situation where we have to dynamically regenerate all certificates when just one of the pod IPs in the daemon ensemble changes. :))

I had another look, to see if disabling hostname verification is already possible in the daemon's config. It seems to be, through a non-canonical config setting called IMEX_SECURITY_TARGET_OVERRIDE:
https://github.com/NVIDIA/k8s-dra-driver-gpu/blob/82a050c5776168461d08b10acd04b43703675534/templates/compute-domain-daemon-config.tmpl.cfg#L154

[jgehrcke]

CC @robertdavidsmith who asked about locking down the communication between IMEX daemons here: #354 (comment)

[jgehrcke]

I adjusted the title. Possible path forward, after thinking about this some more:

1. milestone 1: secure-by-default, optional opt-out for debugging -- unique root CA cert per CD. This yields actual security, and isolates CDs from each other. This might not check every compliance box, because the certs do not derive from an org's CA.
2. milestone 2: allow for users to inject their own root CA cert + (shared end-entity cert) -- this can be great for compliance, but allows for cross-connectivity between CDs, i.e. sacrifices security.
3. milestone 3 (punt on this for as long as we absolutely can): dynamic signing of end entity certs based on an external CA; this is quite a bit of effort and introduces many failure paths.

[robertdavidsmith]

Thanks for moving quickly on this, I'm discussing this with our security team now.

[robertdavidsmith]

Hi,

We’ve discussed this further internally at GR, including with our security team

For our point of view, either or both of these solutions would be a very acceptable fix

• Changing the nvidia-imex pod to be hostNetwork: false (slightly preferred)
• Using SSL with a unique root CA cert per CD (option 1 above)

At GR we don't have a requirement for doing both of these, though if you wish to that’s also fine by us.

Kind regards,

Rob

[klueska]

As of 25.3.0-rc.3, hostNetworking is already set to false and we instead use the pod IPs for IMEX daemon communication.

[robertdavidsmith]

Noted, thanks. So it already meets our requirements at GR. Your call what to do with this ticket.

[jgehrcke]

I think for as long as we can get away with it we should always defer this to the virtual network layer, i.e. hopefully this capability never needs to be built. Closing for now.