Pytorch version

[keysongkang]

Hello,

I am currently going through the initial steps of DeepH-pack and have encountered the following error message below during the training phase after loading the processed files (Command: deeph-train --config train.ini). I suspect this error may be related to compatibility issues between our NVIDIA A100 and PyTorch versions.

I am currently using PyTorch 1.9.1 with Cuda 11.2, but upon checking the recommended requirements, it appears that PyTorch 1.9.1 is strictly specified.

I would greatly appreciate any helpful comments or suggestions regarding this matter. Is it advisable to upgrade the PyTorch version? Your input would be of great help.

Thank you!

---

/u/kkang/.local/lib/python3.9/site-packages/torch/cuda/init.py:106: UserWarning:
NVIDIA A100-SXM4-40GB with CUDA capability sm_80 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA A100-SXM4-40GB GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
Traceback (most recent call last):
File "/u/kkang/.local/bin/deeph-train", line 8, in
sys.exit(main())
File "/u/kkang/.local/lib/python3.9/site-packages/deeph/scripts/train.py", line 20, in main
kernel.train(train_loader, val_loader, test_loader)
File "/u/kkang/.local/lib/python3.9/site-packages/deeph/kernel.py", line 513, in train
train_losses = self.kernel_fn(train_loader, 'TRAIN')
File "/u/kkang/.local/lib/python3.9/site-packages/deeph/kernel.py", line 658, in kernel_fn
output = self.model(
File "/u/kkang/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/u/kkang/.local/lib/python3.9/site-packages/deeph/model.py", line 620, in forward
batch_edge = batch[edge_idx[0]]
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

[mzjb]

It seems like there might be an issue with your PyTorch installation. I suggest running the following commands to check the version of PyTorch and the CUDA version:
python -c "import torch; print(torch.version.cuda)"
python -c "import torch; print(torch.__version__)"

If you find any compatibility issues, it may be advisable to upgrade your PyTorch version or reinstall it to ensure a proper installation.

[keysongkang]

Thank you for your comments!
After executing each command, I receive the following.

python -c "import torch; print(torch.version.cuda)" -> None
python -c "import torch; print(torch.__version__)" -> 1.9.1

Does this imply that CUDA was not detected?

[mzjb]

This indicates that you have installed the CPU version of PyTorch. I suggest removing your current PyTorch and reinstalling PyTorch using the following command: conda install pytorch=1.9.1 cudatoolkit=11.1 -c pytorch -c conda-forge. You may also need to reinstall PyTorch Geometric to ensure compatibility with PyTorch.

[keysongkang]

Thank you for your comments! Sorry for bothering you. On my machine, Cuda is only available for cuda/11.2, cuda/11.4, cuda/11.6 and cuda/12.1. Is it possible to combine one of them with pytorch=1.9.1?

[mzjb]

Based on my usage experience, the CUDA version of PyTorch does not need to be exactly the same as the CUDA version on your machine. For example, I have a computer with CUDA 11.4 installed, and I have installed PyTorch with CUDA version 11.1, and it works fine.

The reason I suggest you install PyTorch with CUDA 11.1 is that I checked the pre-compiled binary files provided by PyTorch for version 1.9.1, and they only have CUDA version 11.1 available. You can see this by searching for torch-1.9.1 on https://download.pytorch.org/whl/torch_stable.html.

That's also why you ended up installing the CPU version of PyTorch when you tried to install CUDA version 11.2. If you want to install PyTorch 1.9.1 with a different CUDA version, you'll need to compile and install it from the source code.

[keysongkang]

Thank you so much! I have come to realize that I can use PyTorch with a different CUDA version than the one installed on my machine. I successfully set up PyTorch with CUDA version 11.1 on a machine that has CUDA 11.4 installed.

Now,
python -c "import torch; print(torch.version.cuda)" -> 11.1
python -c "import torch; print(torch.__version__)" -> 1.9.1

I guess that the command deeph-train --config train.ini prints the following outputs from the training process.
Epoch #102 | Learning rate: 2.00e-04 | Epoch time: 35.28 | Train loss: 0.02211995 | Val loss: 0.02213686 | Best val loss: 0.02104361.

How long does the training set commonly take? I am testing a 2x2x2 Si conventional cell (64 atoms). It seems one step takes 40 secs. Is it possible to resume the calculation?

Thank you!

[mzjb]

Regarding this type of dataset, DeepH may take around six days to train a model, while DeepH-E3 can reduce the training time to three days.

For "resume", please refer to the documentation at https://deeph-pack.deepmodeling.com/en/latest/keyword/train.html and search for resume.