Create nvproxy device files on restore regardless of remap metadata. - #14784
Open
copybara-service[bot] wants to merge 1 commit into
Open
copybara-service[bot] wants to merge 1 commit into
copybara-service[bot] wants to merge 1 commit into
Conversation
Device remap metadata is only saved when at least one frontend device file is open during save. When application CUDA state is checkpointed with cuda-checkpoint before save, no frontend FDs remain open, so no metadata is saved. Restoring such a snapshot on a machine with a different set of GPUs left restored /dev filesystems without device files for the new GPUs, so applications failed to open them. This regressed with commit 6c8a221 ("Add support for nvidia-imex."). Before it, the sandbox did not provide /proc/driver/nvidia/params; in its absence, the CUDA userspace driver defaults to dynamic device file management and creates missing /dev/nvidia# files itself, so applications checkpointed with cuda-checkpoint restored successfully on a different set of GPUs despite the missing metadata. That commit began providing /proc/driver/nvidia/params with ModifyDeviceFiles forced to 0 (consistent with libnvidia-container), so the driver no longer creates missing device files and such restores began failing. The set of GPUs available to each container after restore is known from its spec and dev gofer, independent of any saved metadata. So always create missing /dev/nvidia# files for that set in each container's mount namespace (found via its root process). This subsumes the previous remapping-gated creation, and additionally creates device files for new GPUs beyond those mapped from saved devices. Device remap metadata is still used to remap open FDs and driver objects, which is only needed when frontend FDs were open during save. FUTURE_COPYBARA_INTEGRATE_REVIEW=#14525 from ayushr2:nvproxy-restore-devfiles a241b10 PiperOrigin-RevId: 982330762
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Create nvproxy device files on restore regardless of remap metadata.
Device remap metadata is only saved when at least one frontend device
file is open during save. When application CUDA state is checkpointed
with cuda-checkpoint before save, no frontend FDs remain open, so no
metadata is saved. Restoring such a snapshot on a machine with a
different set of GPUs left restored /dev filesystems without device
files for the new GPUs, so applications failed to open them.
This regressed with commit 6c8a221 ("Add support for
nvidia-imex."). Before it, the sandbox did not provide
/proc/driver/nvidia/params; in its absence, the CUDA userspace driver
defaults to dynamic device file management and creates missing
/dev/nvidia# files itself, so applications checkpointed with
cuda-checkpoint restored successfully on a different set of GPUs
despite the missing metadata. That commit began providing
/proc/driver/nvidia/params with ModifyDeviceFiles forced to 0
(consistent with libnvidia-container), so the driver no longer creates
missing device files and such restores began failing.
The set of GPUs available to each container after restore is known from
its spec and dev gofer, independent of any saved metadata. So always
create missing /dev/nvidia# files for that set in each container's
mount namespace (found via its root process). This subsumes the previous
remapping-gated creation, and additionally creates device files for new
GPUs beyond those mapped from saved devices. Device remap metadata is
still used to remap open FDs and driver objects, which is only needed
when frontend FDs were open during save.
FUTURE_COPYBARA_INTEGRATE_REVIEW=#14525 from ayushr2:nvproxy-restore-devfiles a241b10