Skip to content

Create nvproxy device files on restore regardless of remap metadata. - #14784

Open
copybara-service[bot] wants to merge 1 commit into
masterfrom
test/cl982330762
Open

copybara-service[bot] wants to merge 1 commit into
masterfrom
test/cl982330762

Conversation

@copybara-service

Copy link
Copy Markdown

Create nvproxy device files on restore regardless of remap metadata.

Device remap metadata is only saved when at least one frontend device
file is open during save. When application CUDA state is checkpointed
with cuda-checkpoint before save, no frontend FDs remain open, so no
metadata is saved. Restoring such a snapshot on a machine with a
different set of GPUs left restored /dev filesystems without device
files for the new GPUs, so applications failed to open them.

This regressed with commit 6c8a221 ("Add support for
nvidia-imex."). Before it, the sandbox did not provide
/proc/driver/nvidia/params; in its absence, the CUDA userspace driver
defaults to dynamic device file management and creates missing
/dev/nvidia# files itself, so applications checkpointed with
cuda-checkpoint restored successfully on a different set of GPUs
despite the missing metadata. That commit began providing
/proc/driver/nvidia/params with ModifyDeviceFiles forced to 0
(consistent with libnvidia-container), so the driver no longer creates
missing device files and such restores began failing.

The set of GPUs available to each container after restore is known from
its spec and dev gofer, independent of any saved metadata. So always
create missing /dev/nvidia# files for that set in each container's
mount namespace (found via its root process). This subsumes the previous
remapping-gated creation, and additionally creates device files for new
GPUs beyond those mapped from saved devices. Device remap metadata is
still used to remap open FDs and driver objects, which is only needed
when frontend FDs were open during save.

FUTURE_COPYBARA_INTEGRATE_REVIEW=#14525 from ayushr2:nvproxy-restore-devfiles a241b10

Device remap metadata is only saved when at least one frontend device
file is open during save. When application CUDA state is checkpointed
with cuda-checkpoint before save, no frontend FDs remain open, so no
metadata is saved. Restoring such a snapshot on a machine with a
different set of GPUs left restored /dev filesystems without device
files for the new GPUs, so applications failed to open them.

This regressed with commit 6c8a221 ("Add support for
nvidia-imex."). Before it, the sandbox did not provide
/proc/driver/nvidia/params; in its absence, the CUDA userspace driver
defaults to dynamic device file management and creates missing
/dev/nvidia# files itself, so applications checkpointed with
cuda-checkpoint restored successfully on a different set of GPUs
despite the missing metadata. That commit began providing
/proc/driver/nvidia/params with ModifyDeviceFiles forced to 0
(consistent with libnvidia-container), so the driver no longer creates
missing device files and such restores began failing.

The set of GPUs available to each container after restore is known from
its spec and dev gofer, independent of any saved metadata. So always
create missing /dev/nvidia# files for that set in each container's
mount namespace (found via its root process). This subsumes the previous
remapping-gated creation, and additionally creates device files for new
GPUs beyond those mapped from saved devices. Device remap metadata is
still used to remap open FDs and driver objects, which is only needed
when frontend FDs were open during save.

FUTURE_COPYBARA_INTEGRATE_REVIEW=#14525 from ayushr2:nvproxy-restore-devfiles a241b10
PiperOrigin-RevId: 982330762
@copybara-service copybara-service Bot added the exported Issue was exported automatically label Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

exported Issue was exported automatically

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant