Devices are not re-prepared on node reboot causing workloads to run with inconsistent device state

[shivamerla]

The DRA driver currently does not distinguish between a plugin pod restart and a node reboot. During PrepareResourceClaims, if a claim entry exists in the checkpoint and is marked as PrepareCompleted, the driver skips device preparation.

This behavior works for plugin restarts, but is incorrect after a node reboot. Device configurations (e.g., MIG setup or VFIO bindings) are not guaranteed to persist across reboots. As a result, the driver may incorrectly assume devices are already prepared and skip necessary setup.

This leads to workload pods failing due to missing or inconsistent device state. The driver should detect node reboots or otherwise validate device state, and re-run device preparation when the underlying device configuration is no longer consistent with the checkpoint.

Root Cause

After a node reboot, device configurations are not guaranteed to persist. This includes cases such as:

• MIG (Multi-Instance GPU) configurations
• Devices bound to VFIO or other drivers
• MPS control daemon is not ready before the workloads could run.
• IMEX daemon is not ready before the workloads could run.
• Timeslicing settings

Despite this, the driver assumes the devices are still in the expected state based on the checkpoint and skips reconfiguration. This results in:

• Devices not being properly prepared
• Workload pods failing due to missing or inconsistent device state

Proposed Solutions

To address this issue, we need a reliable way to detect node reboots or validate device state before skipping preparation. Possible approaches include:

Persist Node Boot ID in Checkpoint

• Store the node’s boot ID (/proc/sys/kernel/random/boot_id) alongside the checkpointed state.
• On NodePrepare, compare the current boot ID with the stored one. If they differ, treat it as a node reboot and re-run device preparation.

Validate Actual Device State

• Instead of depending on the checkpoint state, verify that the current GPU/device configuration matches the expected prepared state. If there is a mismatch (e.g., missing MIG partitions or incorrect driver bindings), trigger re-configuration.

[guptaNswati]

@shivamerla Boot ID is giving 404

[shivamerla]

@shivamerla Boot ID is giving 404

updated with the sysfs entry

[guptaNswati]

This issue correctly identifies the node reboot case. We may need to address the GPU reset or GPU lost case also as it will become a valid use case when we expand the monitor and will suffer from stale PrepareCompleted checkpoint entries. Currently, when GPU lost / GPU reset happens, the health monitor catches it, blocking new allocations. But there is no checkpoint invalidation, no unprepare of existing claims.

Scenario	Hardware intact?	Checkpoint valid?	Handled?
Driver restart	Yes	Yes	Yes
Node reboot (across different device tyeps)	No	No	No
Per (GPU reset/lost)	No	No	No

[jgehrcke]

Great discussion!

is incorrect after a node reboot

I can see that this is currently true for

• a fast reboot (where the pod object survives -- i.e., the same claim object is re-used)
• for a subset of device types (that we should narrow down explicitly)

Let's maybe discuss a specific repro / example; addressing this correctly for all affected device types at once may be quite a bit of work.

Slow vs. fast reboot -- why that matters: the claim is prepared for a pod. When the node takes a bit of time to come back, the pod will be deleted (after a configurable time period, the node controller evicts the pod by setting DeletionTimestamp on it). In that case, the old claim will eventually be unprepared and any replacement pod will have a new claim associated with it. If the node reboot is fast enough, however, the original pod object survives in the API server. It is scheduled again on the rebooted node. The kubelet on the node restarts the containers.

Related:

• dynamic-resource-allocation: PrepareResource won't be called on node reboot kubernetes/kubernetes#119850
• DRA: call plugins for claims even if exist in cache kubernetes/kubernetes#120550

I think after kubernetes/kubernetes#120550 we get a guaranteed additional NodePrepareResources() call from the kubelet.

That call leads to a noop prepare in our business logic as you point out:

if a claim entry exists in the checkpoint and is marked as PrepareCompleted, the driver skips device preparation.

I think this can stay as-is. We should probably "just" introduce new "recreate-device-state-if-necessary-upon-plugin-startup" logic. This simply isn't there so far (not in the CD plugin, not in the GPU plugin).

That is, I would say that instead of an explicit "detect node reboot" we should generally upon every plugin startup walk all claims in PrepareCompleted state and inspect the actual device state and perform any state reconciliation necessary.

Kevin's opinion from before:

when the plugin tries to come online (for example after a node reboot or a simple restart), then it needs to do what's necessary to bring the hardware to that state before it marks itself as ready

[jgehrcke]

Re-prepare devices on node reboot to avoid stale checkpoint state

Strong opinion: a bug ticket title should describe a problem, not a solution.

A corresponding specific bug title could be

"MIG device not re-created after fast node reboot"

and a corresponding task title could be

"Reconcile PrepareCompleted claims with device state upon plugin startup"

(these wordings may not be perfect -- I just want to point out the decisive difference between problem space and solution space :-))

[varunrsekar]

That is, I would say that instead of an explicit "detect node reboot" we should generally upon every plugin startup walk all claims in PrepareCompleted state and inspect the actual device state and perform any state reconciliation necessary.

I'm not sure what you mean by state reconciliation here. Is it putting the device in the state expected by the checkpoint? Or just ensuring its in-memory state of allocatable/prepared devices?

I dont think it should be the responsibility of plugin startup to reconfigure PrepareCompleted devices without knowing if a workload needs it or not. If we do it, it means:

1. we dont ever trust the checkpoint
2. we're increasing the plugin startup time

Our SoT for device usage is always kubelet and its anyway guaranteed to call NodePrepare for such devices so why not do the preparation only as part of that?

[shivamerla]

+1, It would be very expensive to do this for every prepared claim during plugin startup while blocking prepare/unprepare calls. Instead, we should defer this work and trigger it on each subsequent NodePrepareResources call.

@jgehrcke this issue shows up in the fairly common case where a node reboots and comes back before Kubernetes completes its eviction flow. By default, the node controller waits ~40s (node-monitor-grace-period) before marking a node NotReady, and pod eviction is delayed by an additional ~5 minutes (pod-eviction-timeout). If the node returns within that ~5–6 minute window, pods are never evicted and the checkpoint state becomes stale, which is exactly the scenario we will hit as discussed here.

[varunrsekar]

We have the guarantee that after a node reboot, the GPUs will be reset. So as long as we can invalidate the checkpoints if we understand the node has rebooted, the GPUs will follow proper lifecycle through NodePrepare/NodeUnprepare after the plugin becomes ready.

Determining whether a node has rebooted is also fairly straightforward:

# k8s:
$ k get node -oyaml | yq -r '.items[] | .status.nodeInfo.bootID'
ccf7a670-9fa3-4dd0-a5a3-9d7823397f04

# linux:
$ cat /proc/sys/kernel/random/boot_id
ccf7a670-9fa3-4dd0-a5a3-9d7823397f04

[jgehrcke]

I think everything written makes sense, good discussion.

We're comparing two different approaches, right? I think I haven't decided yet which variant I prefer. Here are the two different strategies that I think we have thrown into the game so far (there may be different sub-variants within each of the two high-level strategies!):

1. In the NodePrepareResources handler, change the PrepareCompleted noop fast path to something more involved: do a deep check of the device state: is it still in the desired state? if not, make changes to the device state. This raises interesting questions: Are mutations done here always safe? Is this costly? Now, a variation: only do the deep check if the boot ID has changed since the last claim preparation. But how do I reliably know that the boot ID changed since the last claim preparation? By storing the boot ID in the checkpoint. Interesting. I am sure I haven't covered all relevant questions and remarks here. One decisive property of this strategy: the deeper logic may happen at any time during the lifetime of the plugin process (I don't say that this is good or bad; just an observation).

2. Upon plugin startup, iterate through all PrepareCompleted claims from the checkpoint. Most considerations from (1) are true, here too (one could also short-cut this depending on boot ID). However, the deeper logic would not happen during the later lifetime of the plugin process anymore (that's the big difference to (1)).

To me it seems that (1) and (2) are pretty much the same if we actually consider the boot ID.

I dont think it should be the responsibility of plugin [...] without knowing if a workload needs it or not.

That's what we mean when we say that the checkpoint is the source of truth for questions like that -- the plugin knows, by virtue of the checkpoint.