Support Read-Only, systemd-less Systems

[Arc676]

Motivation

Informally, this PR adds (partial) support for Talos. Closes #356.

More formally, this PR adds support for systems with read-only filesystems and systems that do not run systemd.

Description

The MIG manager assumes that it will be able to copy the mig-parted binary to the host and use systemd to restart host-side GPU services. Neither of these is true for Talos, which is an immutable OS that doesn't run systemd. Proper Talos support would introduce a dependency on the Talos API, but that is beyond the scope of this PR and likely falls beyond the scope of what this tool should support.

This PR adds support for systems like Talos by introducing two new flags (both of which are required for Talos):

1. Identifying the host as read-only: prevent the manager from attempting to copy data to the host
2. Flagging the absence of systemd: tell the manager to skip all systemd operations that would otherwise cause the program to hang, since there would be no response on DBus

This PR includes nil-checks for the systemd manager that were not present before. In the original code, these checks are effectively unnecessary because this member is always initialized and the entire program blocks on this initialization if systemd is not present.

Improvements

This is the simplest possible solution to the problem described in the linked issue. All the MIG- and GPU-related operations work fine¹ on Talos. We simply need to skip over the parts that can't work on Talos. The obvious alternatives or improvements over this PR are:

1. Specifically catching the "read-only FS" error when attempting to copy the binary instead of requiring a flag to skip the operation entirely
2. Detecting the presence or absence of systemd, either by inspecting the running processes or by introducing a timeout on the DBus connection, and adjusting accordingly, instead of requiring a flag

Caveats

This PR exists more for discussion than with the goal of being merged. These changes were made based on a very cursory reading and superficial understanding of the MIG manager. There is likely a cleaner and more elegant way to achieve this. However, I'll submit the patch as a proof-of-concept: by disabling the host-copies and all systemd features, the MIG manager works properly on Talos. This is, at least for us, an important starting point.

Footnotes

1. CUDA validation yields ERROR: init 250 result=11s. I haven't yet figured out what this means, but so far it hasn't impacted the use of the GPU. The GPU workloads still run fine, as does the CUDA validation pod. ↩

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[linkages]

Just want to add my feedback to this. I just tested this with the following setup:

Environment:

• OS: Talos Linux v1.13.0
• Kubernetes: v1.34.0
• GPUs: NVIDIA B300 NVL and RTX 6000 Pro Server
• NVIDIA driver/toolkit: provided by Talos system extensions (580.159.03)
• Kernel Version: 6.18.29-talos
• GPU Operator: Helm chart v26.3.1

I had to build a new k8s-mig-manager based on @Arc676 repo. I then pushed it to docker.io/linkages/k8s-mig-manager:v0.14.1.

Then when I deploy the gpu-operator, I set the values for the helm chart using this:

driver:
  enabled: false

toolkit:
  enabled: false

hostPaths:
  driverInstallDir: /usr/local

mig:
  strategy: mixed

migManager:
  enabled: true
  repository: docker.io/linkages
  version: v0.14.1
  env:
    - name: READONLY_ROOTFS
      value: "true"
    - name: SYSTEMD_UNAVAILABLE
      value: "true"

operator:
  cleanupCRD: true

I then set the nvidia.com/mig.config label on all nodes to all-balanced and the mig-manager did the right thing in waiting for all the operator components to stop and then it adjusted the MiG settings and restarted everything back up. Shortly after the gpu-feature-discovery controller set the correct labels on the nodes.

This was tested on 2 different types of nodes in the same cluster:

2 x Lenovo nodes with 8 x NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs
and
1 x DGX B300 with 8 x NVIDIA B300 SXM6 AC

Thank you @Arc676 for this patch. I hope it or a more elegant version of this gets pulled upstream. For now this solves my problem.

[rajathagasthya]

Thanks @Arc676 for tackling this!

My preference is to replace the flag-based approach in favor of something with no new public surface area. The reason being WITH_SHUTDOWN_HOST_GPU_CLIENTS=false already encodes the intent "don't touch the host". The current bug is that reconfigure.New() unconditionally connects to systemd even when WITH_SHUTDOWN_HOST_GPU_CLIENTS=false.

We can document that on platforms without systemd WITH_SHUTDOWN_HOST_GPU_CLIENTS=false is required. To fully close the loop on getting this to work on Talos, I would also suggest creating an issue to decouple WITH_SHUTDOWN_HOST_GPU_CLIENTS from IS_HOST_DRIVER in the GPU Operator helm chart.

In the meantime, we should remove the unconditional systemd.NewManager() call from reconfigure.New(), and replace it with lazy initialization and a context timeout. This way, when systemd isn't available, the MIG manager fails fast with a clear error instead of hanging silently.

[Arc676]

Thanks for the feedback! I agree with your point regarding the public surface; adding flags was the easiest approach but both properties can be inferred from the system's behavior. Instead of requiring the user to set these flags, nvidia-mig-manager can set them automatically.

In the meantime, we should remove the unconditional systemd.NewManager() call from reconfigure.New(), and replace it with lazy initialization and a context timeout. This way, when systemd isn't available, the MIG manager fails fast with a clear error instead of hanging silently.

Support for systems without systemd requires the option to disable the connection to systemd entirely. I'm not sure I understand what exactly you have in mind here.

What would be the advantage of a lazy init for the systemd manager? I suppose the startup would be slightly faster by skipping a step if the desired MIG configuration is already applied, but I think that determining the availability of systemd straight away makes more sense. In strictly managed environments, startup occurs at known times when the GPU operator is updated. Deferring the systemd check to when it's needed would mean that the latency would be incurred when the user attempts to change the MIG configuration. The first repartitioning operation after startup would be slower than the others.

We could keep the unconditional systemd.NewManager but introduce the timeout as you mentioned; if the connection times out, then we would set a flag to indicate that systemd is unavailable. In particular, the MIG manager should not fail in this case, but perhaps output a warning to ensure that the user is aware. Or did you mean that WITH_SHUTDOWN_HOST_GPU_CLIENTS=false should be equivalent to "systemd unavailable"?

Unless you want to separate the features, I'd implement all these changes in this PR such that Talos support is covered.

I've created a new issue to track the change to the Helm chart per your suggestion.

[Arc676]

I've adapted the implementation based on the above comments:

• Assume the Helm chart will be adapted such that WITH_SHUTDOWN_HOST_GPU_CLIENTS correctly reflects the intent "do not touch the host"; I've removed the flag for readonly hosts
• Since the Reconfigure object is recreated each time, attempts to persist that fail due to a read-only FS issue a warning but don't return an error. Caching this finding would have to occur at the program's top-level; I suppose this could be added, but with the above point there are no attempts to write to the host at this level.
• The DBus connection uses the same Context throughout; I didn't find a way to set a timeout on just the initial connection. DBus closes the network connection when the context times out and issues a corresponding warning that is not wrapped with context.DeadlineExceeded. It's not particularly elegant but as a workaround I changed the initialization function to try twice: once with a timeout, after which systemd is flagged as unavailable (at least for the current reconfiguration attempt), and a second time with the parent context. As before, we'd need to query systemd outside the Reconfigure object to be able to cache the result. I've left in a flag to change the timeout, primarily to avoid having a fixed constant in the code. However, this does mean that every reconfiguration will have to wait for this timeout.

[Arc676]

Any update here? Not sure what's blocking the automated checks; I don't really use GitHub actions.

[rajathagasthya]

/ok-to-test 125d1db

[copy-pr-bot]

/ok-to-test 125d1db

@rajathagasthya, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

[rajathagasthya]

@Arc676 This is on my list. I'm going to take a closer look this week. Thanks for your patience!

[rajathagasthya]

@Arc676 Thanks for your patience and for reworking this! I expanded on some of my earlier suggestions in this review.

As for NVIDIA/gpu-operator#2501, I think the decoupling can be done in a non-breaking way in the entrypoint script like below:

WITH_SHUTDOWN_HOST_GPU_CLIENTS=${WITH_SHUTDOWN_HOST_GPU_CLIENTS:-$IS_HOST_DRIVER}

[rajathagasthya]

To expand on my earlier suggestion of lazy init of systemd manager:

1. We could remove the unconditional init in this New() method, move it to a cached getter method which all the callers that use systemd (shutdownHostGPUClients, hostStartSystemdServices and persistConfig) invoke. Each reconfiguration calls New() anyway, so latency doesn't change.
2. On Talos, shutdownHostGPUClients and hostStartSystemdServices are never invoked when WithShutdownHostGPUClients = true. persistConfig only runs if HOST_MIG_MANAGER_STATE_FILE already exists on the host (the default value is /etc/systemd/system/nvidia-mig-manager.service.d/override.conf which is not applicable to Talos).
3. Right now, you're treating connection timeout as "systemd not present" and ignoring it. On systems that have systemd, connection timeouts should be hard failures and shouldn't be equated to systemd not being present.

I think the code could look something like this:

// Cached getter
func (r *Reconfigure) getSystemdManager() (*systemd.Manager, error) {
      if r.systemdManager != nil {
              return r.systemdManager, nil
      }
      m, err := systemd.NewManager(r.ctx) // connect timeout lives inside NewManager
      if err != nil {
              return nil, fmt.Errorf("failed to connect to systemd (if the host does not run systemd, set WITH_SHUTDOWN_HOST_GPU_CLIENTS=false): %w", err)
      }
      r.systemdManager = m
      return m, nil
}

// Caller
func (r *Reconfigure) shutdownHostGPUClients() error {
      m, err := r.getSystemdManager()
      if err != nil {
              return err
      }
      stoppedServices, err := m.StopSystemdServices(services)
      ...
}

[Arc676]

Regarding point 3: this is true. Using the timeout was the simplest way to check for systemd without introducing much additional logic. However, it occurs to me that the systemd dependency is actually tied to features behind the WithShutdownHostGPUClients flag. This means that mig-parted doesn't need a systemd detection mechanism if we instead rely on the flag.

I've implemented the lazy-init as you suggested and removed the soft-fail for timeouts. I've kept the timeout check itself since this prevents the program from hanging in the event of an actual DBus connection issue. The new implementation thus assumes that WithShutdownHostGPUClients will be set only when systemd is running. Otherwise, the reconfiguration will always fail due to connection timeout. With the flags set properly, none of the systemd-dependent calls can be reached with the flag is disabled, so the initializer is never called.

As a side note for anyone using this patch on their own setup: you have to patch the operator manually to use this feature branch now.

Patch Details

The new changes make this build of mig-parted dependent on the decoupling of the flags in the chart. Until the chart issue is resolved, mig-parted can only be used in this form if the entrypoint script is patched in the operator image (which can be done using a volume mount and a custom ConfigMap).

Custom entrypoint:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nvidia-mig-manager-custom-entrypoint
data:
  configmap.yaml: |
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: nvidia-mig-manager-entrypoint
      namespace: "FILLED BY THE OPERATOR"
      labels:
        app: nvidia-mig-manager
    data:
      entrypoint.sh: |
        #!/bin/sh

        until [ -f /run/nvidia/validations/driver-ready ]
        do
          echo "waiting for the driver validations to be ready..."
          sleep 5
        done

        set -o allexport
        cat /run/nvidia/validations/driver-ready
        . /run/nvidia/validations/driver-ready
          
        # manually export additional envs required by mig-manager
        export WITH_SHUTDOWN_HOST_GPU_CLIENTS=${WITH_SHUTDOWN_HOST_GPU_CLIENTS:-$IS_HOST_DRIVER}
        echo "WITH_SHUTDOWN_HOST_GPU_CLIENTS=$WITH_SHUTDOWN_HOST_GPU_CLIENTS"

        echo "Starting nvidia-mig-manager"
        exec nvidia-mig-manager

Mounting into the operator (just showing the altered bits):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpu-operator
spec:
  template:
    spec:
      volumes:
        - name: mig-entrypoint
          configMap:
            name: nvidia-mig-manager-custom-entrypoint
      containers:
        - name: gpu-operator
          volumeMounts:
            - name: mig-entrypoint
              mountPath: /opt/gpu-operator/state-mig-manager/0420_configmap.yaml
              subPath: configmap.yaml

[lexfrei]

I opened #431 for the same bug before seeing this thread, and was pointed here — makes sense to land the fix in this PR rather than compete, since it's older and already hardware-tested. Happy to fold my work in (co-author, or just take the pieces), or step back if you'd rather carry it yourself.

I did end up at exactly the refactor requested in the 07-08 review, so it might save a round-trip. In #431:

• reconfigure.New() no longer dials D-Bus; the connection moved to a lazy cached getter (matches the sketch in the review), so on a systemd-less host with WITH_SHUTDOWN_HOST_GPU_CLIENTS=false it's never dialed and the reconfigure completes.
• The connect timeout lives inside NewManager and returns a hard error rather than nil, so on a real systemd host an unresponsive socket surfaces as an error instead of being silently treated as "no systemd". It's a single dial (avoiding the current double-dial, which leaks the first connection and then re-dials without a timeout), and the timeout is an internal constant — no new flag or public surface.
• Unit tests cover the timeout path, the missing-socket path, lazy construction, and nil-safe cleanup.

The diff is on #431 as a reference. Glad to push these here as commits with you as co-author, or hand them over — whatever's easiest. Just say which you prefer.

[Arc676]

@lexfrei thanks for reaching out! I actually prefer the timer-based approach (I hadn't thought of that); it's a lot cleaner than opening two connections. I also hadn't implemented any unit tests, since I tested the build in the cluster directly, so that's a useful addition for the pipeline here on GitHub. I cherry-picked your commits into my branch and updated the trailers in case the code-signing checks care about authorship. Conveniently, we used similar identifiers so there were almost no conflicts.

@rajathagasthya I went with the assumption that having unit tests is better than not having them. However, the commits are still distinct, so if you would prefer to review the tests separately, I can easily reset the branch to its HEAD~2 and re-separate the two PRs. Let me know what you think makes the most sense. I think at this point, by line count, this PR mostly contains the content of the other PR. They do the same thing so I don't really mind which one you'd prefer to follow going forward.

[Arc676]

Regarding point 3: this is true. Using the timeout was the simplest way to check for systemd without introducing much additional logic. However, it occurs to me that the systemd dependency is actually tied to features behind the WithShutdownHostGPUClients flag. This means that mig-parted doesn't need a systemd detection mechanism if we instead rely on the flag.

I've implemented the lazy-init as you suggested and removed the soft-fail for timeouts. I've kept the timeout check itself since this prevents the program from hanging in the event of an actual DBus connection issue. The new implementation thus assumes that WithShutdownHostGPUClients will be set only when systemd is running. Otherwise, the reconfiguration will always fail due to connection timeout. With the flags set properly, none of the systemd-dependent calls can be reached with the flag is disabled, so the initializer is never called.

As a side note for anyone using this patch on their own setup: you have to patch the operator manually to use this feature branch now.

Patch Details

The new changes make this build of mig-parted dependent on the decoupling of the flags in the chart. Until the chart issue is resolved, mig-parted can only be used in this form if the entrypoint script is patched in the operator image (which can be done using a volume mount and a custom ConfigMap).

Custom entrypoint:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nvidia-mig-manager-custom-entrypoint
data:
  configmap.yaml: |
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: nvidia-mig-manager-entrypoint
      namespace: "FILLED BY THE OPERATOR"
      labels:
        app: nvidia-mig-manager
    data:
      entrypoint.sh: |
        #!/bin/sh

        until [ -f /run/nvidia/validations/driver-ready ]
        do
          echo "waiting for the driver validations to be ready..."
          sleep 5
        done

        set -o allexport
        cat /run/nvidia/validations/driver-ready
        . /run/nvidia/validations/driver-ready
          
        # manually export additional envs required by mig-manager
        export WITH_SHUTDOWN_HOST_GPU_CLIENTS=${WITH_SHUTDOWN_HOST_GPU_CLIENTS:-$IS_HOST_DRIVER}
        echo "WITH_SHUTDOWN_HOST_GPU_CLIENTS=$WITH_SHUTDOWN_HOST_GPU_CLIENTS"

        echo "Starting nvidia-mig-manager"
        exec nvidia-mig-manager

Mounting into the operator (just showing the altered bits):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpu-operator
spec:
  template:
    spec:
      volumes:
        - name: mig-entrypoint
          configMap:
            name: nvidia-mig-manager-custom-entrypoint
      containers:
        - name: gpu-operator
          volumeMounts:
            - name: mig-entrypoint
              mountPath: /opt/gpu-operator/state-mig-manager/0420_configmap.yaml
              subPath: configmap.yaml

[lexfrei]

@Arc676 thanks for picking up my commits from #431 — the lazy-init shape is right.

One regression to flag, in cleanup():

func (r *Reconfigure) cleanup() {
	if mgr, _ := r.getSystemdManager(); mgr != nil {
		mgr.Close()
	}
}

getSystemdManager() only returns from cache when a connection already exists; otherwise it dials. So on a systemd-less host — where nothing ever established a manager — cleanup() now dials the D-Bus socket, waits out the timeout, gets an error, and closes nothing. That's exactly the path the lazy-init exists to remove, and it's back on precisely the hosts this PR targets. The original guard was load-bearing:

if r.systemdManager != nil {
	r.systemdManager.Close()
}

Reading r.systemdManager directly keeps cleanup side-effect free. TestNewDoesNotConnectSystemd won't catch this — it only pins New() — so a test calling cleanup() on a nil manager would be worth adding alongside the fix.

Separately, on DefaultSystemdTimeout = 1.0: that's a lot tighter than the 10s in the original. A busy host can lose a 1s D-Bus connect race legitimately, and anything that previously succeeded slowly would now fail instead. It's overridable, but 1s as the default looks likely to produce spurious failures.

[Arc676]

@lexfrei thanks for catching the cleanup regression. I've pushed the correction.

The default timeout of one second is actually outdated; I adopted your implementation of the manager setup, which does not propagate the user's command line options. Hence, the timeout is actually fixed at 10 seconds. I've removed the obsolete constant. I can put back the command line parameter if needed.