Summary
This issue reports a compatibility bug between DPGEN and the newly added cusolver KS solver in ABACUS LTSv3.10.0. Manual addition of the cusolver option in the DPGEN source code enables normal DFT task submission, but subsequent result parsing by dpdata fails with a NumPy reshaping error due to unrecognized output files generated by the cuSOLVER solver. It is expected to add native support for the cusolver ks_solver in DPGEN and dpdata to ensure complete workflow compatibility for GPU-accelerated ABACUS calculations.
Detailed Description
My software versions are as follows:
DPGEN Version: 0.13.3
ABACUS Version: LTSv3.10.0
DPData Version: 1.0.2
I am currently conducting sampling for the SiC system using DPGEN, yet several issues have arisen during DFT calculations with the ABACUS software. ABACUS has newly introduced the cusolver KS equation solver for GPU acceleration, which delivers remarkable computational speedup. However, when I adopt this newly added solver, DPGEN throws the following error:
INFO:dpgen:system 000 accurate_ratio: 0.8479 thresholds: 0.9980 and 0.9990 eff. task min and max -1 20 number of fp tasks: 20
Traceback (most recent call last):
File "/home/young/anaconda3/envs/dpgen1/bin/dpgen", line 6, in
sys.exit(main())
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/main.py", line 255, in main
args.func(args)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/generator/run.py", line 5712, in gen_run
run_iter(args.PARAM, args.MACHINE)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/generator/run.py", line 5061, in run_iter
make_fp(ii, jdata, mdata)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/generator/run.py", line 4011, in make_fp
make_fp_calculation(iter_index, jdata, mdata)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/generator/run.py", line 4032, in make_fp_calculation
make_fp_abacus_scf(iter_index, jdata)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/generator/run.py", line 3622, in make_fp_abacus_scf
ret_input = make_abacus_scf_input(fp_params, extra_file_path=pporb_path)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/generator/lib/abacus_scf.py", line 122, in make_abacus_scf_input
assert fp_params["ks_solver"] in [
AssertionError: 'ks_sover' should in 'cgx', 'dav', 'lapack', 'genelpa', 'hpseps', 'scalapack_gvx'.
I then modified the abacus_scf.py file by adding the cusolver option, which allows DFT calculation tasks to be normally submitted. Nevertheless, upon completion of the calculations, another error occurs when dpdata attempts to parse and collect ABACUS calculation results:
/home/young/.local/lib/python3.10/site-packages/paramiko/pkey.py:100: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from cryptography.hazmat.primitives.ciphers.algorithms in 48.0.0.
"cipher": algorithms.TripleDES,
/home/young/.local/lib/python3.10/site-packages/paramiko/transport.py:258: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from cryptography.hazmat.primitives.ciphers.algorithms in 48.0.0.
"class": algorithms.TripleDES,
INFO:dpgen:start running
INFO:dpgen:continue from iter 000 task 05
INFO:dpgen:=============================iter.000000==============================
INFO:dpgen:-------------------------iter.000000 task 06--------------------------
INFO:dpgen:system 000 candidate : 676 in 4444 15.21 %
INFO:dpgen:system 000 failed : 0 in 4444 0.00 %
INFO:dpgen:system 000 accurate : 3768 in 4444 84.79 %
INFO:dpgen:system 000 accurate_ratio: 0.8479 thresholds: 0.9980 and 0.9990 eff. task min and max -1 20 number of fp tasks: 20
INFO:dpgen:-------------------------iter.000000 task 07--------------------------
INFO:dpgen:-------------------------iter.000000 task 08--------------------------
Traceback (most recent call last):
File "/home/young/anaconda3/envs/dpgen1/bin/dpgen", line 6, in
sys.exit(main())
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/main.py", line 255, in main
args.func(args)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/generator/run.py", line 5712, in gen_run
run_iter(args.PARAM, args.MACHINE)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/generator/run.py", line 5067, in run_iter
post_fp(ii, jdata)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/generator/run.py", line 4940, in post_fp
post_fp_check_fail(iter_index, jdata)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/generator/run.py", line 4401, in post_fp_check_fail
sys = dpdata.LabeledSystem(os.path.join(single_sys), fmt="deepmd/npy")
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpdata/system.py", line 198, in init
self.from_fmt(
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpdata/system.py", line 235, in from_fmt
return self.from_fmt_obj(load_format(fmt), file_name, **kwargs)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpdata/system.py", line 1225, in from_fmt_obj
data = fmtobj.from_labeled_system(file_name, **kwargs)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpdata/plugins/deepmd.py", line 101, in from_labeled_system
return dpdata.formats.deepmd.comp.to_system_data(
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpdata/formats/deepmd/comp.py", line 80, in to_system_data
all_data.append(np.reshape(tmp, [tmp.shape[0], *shape]))
File "<array_function internals>", line 180, in reshape
File "/home/young/.local/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 298, in reshape
return _wrapfunc(a, 'reshape', newshape, order=order)
File "/home/young/.local/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 57, in _wrapfunc
return bound(*args, **kwds)
ValueError: cannot reshape array of size 0 into shape (20,64,3)
We would highly appreciate it if the maintainers could resolve this problem at your earliest convenience.
Further Information, Files, and Links
No response
Summary
This issue reports a compatibility bug between DPGEN and the newly added cusolver KS solver in ABACUS LTSv3.10.0. Manual addition of the cusolver option in the DPGEN source code enables normal DFT task submission, but subsequent result parsing by dpdata fails with a NumPy reshaping error due to unrecognized output files generated by the cuSOLVER solver. It is expected to add native support for the cusolver ks_solver in DPGEN and dpdata to ensure complete workflow compatibility for GPU-accelerated ABACUS calculations.
Detailed Description
My software versions are as follows:
DPGEN Version: 0.13.3
ABACUS Version: LTSv3.10.0
DPData Version: 1.0.2
I am currently conducting sampling for the SiC system using DPGEN, yet several issues have arisen during DFT calculations with the ABACUS software. ABACUS has newly introduced the cusolver KS equation solver for GPU acceleration, which delivers remarkable computational speedup. However, when I adopt this newly added solver, DPGEN throws the following error:
INFO:dpgen:system 000 accurate_ratio: 0.8479 thresholds: 0.9980 and 0.9990 eff. task min and max -1 20 number of fp tasks: 20
Traceback (most recent call last):
File "/home/young/anaconda3/envs/dpgen1/bin/dpgen", line 6, in
sys.exit(main())
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/main.py", line 255, in main
args.func(args)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/generator/run.py", line 5712, in gen_run
run_iter(args.PARAM, args.MACHINE)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/generator/run.py", line 5061, in run_iter
make_fp(ii, jdata, mdata)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/generator/run.py", line 4011, in make_fp
make_fp_calculation(iter_index, jdata, mdata)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/generator/run.py", line 4032, in make_fp_calculation
make_fp_abacus_scf(iter_index, jdata)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/generator/run.py", line 3622, in make_fp_abacus_scf
ret_input = make_abacus_scf_input(fp_params, extra_file_path=pporb_path)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/generator/lib/abacus_scf.py", line 122, in make_abacus_scf_input
assert fp_params["ks_solver"] in [
AssertionError: 'ks_sover' should in 'cgx', 'dav', 'lapack', 'genelpa', 'hpseps', 'scalapack_gvx'.
I then modified the abacus_scf.py file by adding the cusolver option, which allows DFT calculation tasks to be normally submitted. Nevertheless, upon completion of the calculations, another error occurs when dpdata attempts to parse and collect ABACUS calculation results:
/home/young/.local/lib/python3.10/site-packages/paramiko/pkey.py:100: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from cryptography.hazmat.primitives.ciphers.algorithms in 48.0.0.
"cipher": algorithms.TripleDES,
/home/young/.local/lib/python3.10/site-packages/paramiko/transport.py:258: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from cryptography.hazmat.primitives.ciphers.algorithms in 48.0.0.
"class": algorithms.TripleDES,
INFO:dpgen:start running
INFO:dpgen:continue from iter 000 task 05
INFO:dpgen:=============================iter.000000==============================
INFO:dpgen:-------------------------iter.000000 task 06--------------------------
INFO:dpgen:system 000 candidate : 676 in 4444 15.21 %
INFO:dpgen:system 000 failed : 0 in 4444 0.00 %
INFO:dpgen:system 000 accurate : 3768 in 4444 84.79 %
INFO:dpgen:system 000 accurate_ratio: 0.8479 thresholds: 0.9980 and 0.9990 eff. task min and max -1 20 number of fp tasks: 20
INFO:dpgen:-------------------------iter.000000 task 07--------------------------
INFO:dpgen:-------------------------iter.000000 task 08--------------------------
Traceback (most recent call last):
File "/home/young/anaconda3/envs/dpgen1/bin/dpgen", line 6, in
sys.exit(main())
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/main.py", line 255, in main
args.func(args)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/generator/run.py", line 5712, in gen_run
run_iter(args.PARAM, args.MACHINE)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/generator/run.py", line 5067, in run_iter
post_fp(ii, jdata)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/generator/run.py", line 4940, in post_fp
post_fp_check_fail(iter_index, jdata)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpgen/generator/run.py", line 4401, in post_fp_check_fail
sys = dpdata.LabeledSystem(os.path.join(single_sys), fmt="deepmd/npy")
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpdata/system.py", line 198, in init
self.from_fmt(
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpdata/system.py", line 235, in from_fmt
return self.from_fmt_obj(load_format(fmt), file_name, **kwargs)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpdata/system.py", line 1225, in from_fmt_obj
data = fmtobj.from_labeled_system(file_name, **kwargs)
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpdata/plugins/deepmd.py", line 101, in from_labeled_system
return dpdata.formats.deepmd.comp.to_system_data(
File "/home/young/anaconda3/envs/dpgen1/lib/python3.10/site-packages/dpdata/formats/deepmd/comp.py", line 80, in to_system_data
all_data.append(np.reshape(tmp, [tmp.shape[0], *shape]))
File "<array_function internals>", line 180, in reshape
File "/home/young/.local/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 298, in reshape
return _wrapfunc(a, 'reshape', newshape, order=order)
File "/home/young/.local/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 57, in _wrapfunc
return bound(*args, **kwds)
ValueError: cannot reshape array of size 0 into shape (20,64,3)
We would highly appreciate it if the maintainers could resolve this problem at your earliest convenience.
Further Information, Files, and Links
No response