Add support for using DNSNames instead of raw IPs for IMEX daemons

[klueska]

This patch lays the foundation for eventually resolving the following issues:

• ComputeDomain: add support for node failure (follow workload as it is re-scheduled) #348
• ComputeDomain: add support for elastic workloads #349
• ComputeDomain: do not require a priori node count (numNodes) for creation #364
• ComputeDomain: support more than one domain per node, with subsets of GPUs #353

It does this by leveraging the new "hostnames" featured introduced in newer versions of the IMEX daemon. This new "hostnames" feature allows one to list a set of DNS resolvable names in the nodes_config.cfg file read by an IMEX daemon. In doing so, we are able to provide a level-of-indirection between each individual IMEX daemon and how it connects to other IMEX daemons in the same IMEX domain.

So long as all of the IMEX daemons agree on the same set of DNS names in their nodes_config.cfg files, we can dynamically change which IMEX daemon each of these names resolves to, thus providing natural support for things like (1) failing over worker pods to other nodes, (2) replacing nodes on node failures, and (2) expanding workloads to include more or fewer workers over time.

The only feature directly enabled by this PR is #348. The others are not directly addressed, but this PR does lay the foundation for resolving each of them in the future.

Fixes: #348

Test ran to verify its working as expected:

#!/bin/bash

MAX_ITER=100
TIMEOUT=300  # 5 minutes in seconds
WORKER_DELETE_DELAY=15 # seconds
DAEMON_DELETE_DELAY=30 # seconds

for ((i=1; i<=MAX_ITER; i++)); do
    echo "Starting iteration $i"

    kubectl apply -f nvbandwidth-test-job.yaml

    # Wait up to TIMEOUT seconds for the pod to complete
    SECONDS=0
    WORKER_DELETED=false
    DAEMON_DELETED=false
    while true; do
        STATUS=$(kubectl get pod -l job-name=nvbandwidth-test-launcher -o jsonpath="{.items[0].status.phase}" 2>/dev/null)

        # Pod succeeded
        if [ "$STATUS" == "Succeeded" ]; then
            echo "Pod completed successfully."
            break

        # Pod failed
        elif [ "$STATUS" == "Failed" ]; then
            echo "Pod failed."
            break

        # Delete worker pod after $WORKER_DELETE_DELAY seconds
        elif [ "$SECONDS" -ge $WORKER_DELETE_DELAY ] && [ "$WORKER_DELETED" = "false" ]; then
                kubectl delete pod nvbandwidth-test-worker-0
                WORKER_DELETED=true

        # Force delete (i.e. with SIGKILL) IMEX daemon pods after $DAEMON_DELETE_DELAY seconds
        elif [ "$SECONDS" -ge $DAEMON_DELETE_DELAY ] && [ "$DAEMON_DELETED" = "false" ]; then
                kubectl delete pod -n nvidia-dra-driver-gpu -l resource.nvidia.com/computeDomain --grace-period=0 --force
                DAEMON_DELETED=true

        # Timeout reached
        elif [ "$SECONDS" -ge $TIMEOUT ]; then
            echo "Timeout reached ($TIMEOUT seconds). Exiting loop."
            exit 1
        fi

        sleep 5
    done

    kubectl delete -f nvbandwidth-test-job.yaml
done

echo "Finished $MAX_ITER iterations successfully."

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[jgehrcke]

What I write next is probably premature before we have talked synchronously. But I already want to leave a thought / questions here that I have on my mind right now.

Does the daemon watch for node config file changes?

So far, I thought that here (after an update) we have to

1. send a signal to the child process to either make it "re-initiate all TCP connections" or "re-read the config" (and have it decide itself if it wants to (re-)initiate some TCP connections (I choose this wording because as the first step of a TCP connect() there's DNS (re)resolution (hopefully bypassing any cache (I've never nested parentheses in normal text so deeply so I thought I add another layer just for fun))))
2. or (as before) bluntly restart the child.

And zooming out a little further, in your thinking you are probably carefully distinguishing between these cases:

• 'growing' (new DNS name in pool)
• 'shrinking' (DNS name removal from pool)
• 'changing' (old DNS name pointing to a different IP address)
• 'not changing' (nothing changed)

And of course these are not all equally difficult to support. I assume your experiment / poccing focused on just one of them?

[jgehrcke]

About:

I thought that here (after an update) we have to send a signal to the child process to either make it "re-initiate all TCP connections"

Answered and discussed here: https://github.com/NVIDIA/k8s-dra-driver-gpu/pull/433/files#r2318598969

I assume your experiment / poccing focused on just one of them?

Given the test shared by @klueska I believe it's fair to say we are confirming here mainly

• 'changing' (old DNS name pointing to a different IP address)

and maybe implicitly also

• 'not changing' (nothing changed)'

[klueska]

In its current form, this only works if the rescheduled pods reland on the same node...

[klueska]

In its current form, this only works if the rescheduled pods reland on the same node...

The latest change fixes this issue.

It does this by ensuring a consistent dnsname --> IMEXDaemonIP mapping on all nodes in the IMEX daemon ensemble. Without this, the IMEX daemons were getting confused if the same DNS name was used by different nodes to point to different IMEX daemons in the ensemble.

And it make sense -- normally DNS names do provide such a global mapping — I was just abusing things by only maintaining a local mapping without any regard for how that mapping was done on other nodes.

[jgehrcke]

Exciting! After lots of discussion and iteration: let's get this in 🚀