I was trying to checkpoint and restart the example GPU workload mentioned https://github.com/NVIDIA/cuda-checkpoint to validate cuda-checkpointing functionality. Here are the commands I ran on a non-root account
Checkpoint
PID=221325
cuda-checkpoint --toggle --pid $PID # <-- No SUDO
criu dump -t $PID --shell-job --unprivileged # <-- No SUDO
Restore
criu restore --shell-job --restore-detached --images-dir dump --unprivileged # <-- No SUDO
cuda-checkpoint --toggle --pid $PID # <-- No SUDO
I managed to get both checkpoint and restart working without super user access
Then I tried CRIU GPU approach to do the same without sudo
criu dump -t $PID --shell-job --lib /usr/local/lib/criu --unprivileged
Warn (criu/kerndat.c:1769): ioctl(PAGEMAP_SCAN) doesn't support PAGE_IS_GUARD flag. CRIU dump will fail if dumped processes use madvise(MADV_GUARD_INSTALL). Please, consider updating your kernel.
Error (cuda_plugin.c:297): cuda_plugin: Failed to set ptrace options on interrupt for restore tid 221572: Operation not permitted
Error (cuda_plugin.c:297): cuda_plugin: Failed to set ptrace options on interrupt for restore tid 221572: Operation not permitted
Dumping worked when I use sudo to above command.
sudo criu dump -t $PID --shell-job --lib /usr/local/lib/criu --unprivileged
Warn (criu/kerndat.c:1195): $XDG_RUNTIME_DIR not set. Cannot find location for kerndat file
Warn (criu/kerndat.c:1195): $XDG_RUNTIME_DIR not set. Cannot find location for kerndat file
Warn (criu/kerndat.c:1769): ioctl(PAGEMAP_SCAN) doesn't support PAGE_IS_GUARD flag. CRIU dump will fail if dumped processes use madvise(MADV_GUARD_INSTALL). Please, consider updating your kernel.
Warn (compel/arch/x86/src/lib/infect.c:419): Will restore 221572 with interrupted system call
Warn (compel/arch/x86/src/lib/infect.c:419): Will restore 221580 with interrupted system call
Is there a way I can use direct CRIU GPU checkpointing approach without being the super user?
I was trying to checkpoint and restart the example GPU workload mentioned https://github.com/NVIDIA/cuda-checkpoint to validate cuda-checkpointing functionality. Here are the commands I ran on a non-root account
Checkpoint
Restore
I managed to get both checkpoint and restart working without super user access
Then I tried CRIU GPU approach to do the same without sudo
Dumping worked when I use sudo to above command.
Is there a way I can use direct CRIU GPU checkpointing approach without being the super user?