Unstable results of daily intel-dav test

[pxlxingliang]

Describe the bug

I have run several times of the daily test example 003_12Pt111 with dav method compiled by intel, and the results are not exactly same.

More tests have been listed in #4022

Expected behavior

No response

To Reproduce

No response

Environment

No response

Additional Context

No response

Task list for Issue attackers (only for developers)

• Verify the issue is not a duplicate.
• Describe the bug.
• Steps to reproduce.
• Expected behavior.
• Error message.
• Environment details.
• Additional context.
• Assign a priority level (low, medium, high, urgent).
• Assign the issue to a team member.
• Label the issue with relevant tags.
• Identify possible related issues.
• Create a unit test or automated test to reproduce the bug (if applicable).
• Fix the bug.
• Test the fix.
• Update documentation (if necessary).
• Close the issue and inform the reporter (if applicable).

[pxlxingliang]

003.zip
Some test results:

Set symmetry = 0 and run 10 times, the results of last SCF step:

9.558E-08	9.558E-08	9.558E-08	9.558E-08	9.558E-08	9.558E-08	9.558E-08	9.558E-08	9.558E-08	9.558E-08

Set symmetry = 1 and run 10 times, the results of last SCF step:

2.765E-08	2.765E-08	2.780E-08	2.780E-08	2.765E-08	2.780E-08	2.765E-08	2.780E-08	2.780E-08	2.780E-08

[WHUweiqingzhou]

@maki49 could you have a look?

[maki49]

003.zip Some test results:

The link of the zip file seems not found?

[maki49]

I haven't seen the full logs of all the 10-time runs, but it seems that symmetry_prec=1e-6 as the default is not suitable for this case: some symmetry operations are blocked due to the structure precision.
symmetry_prec is commonly used in charge density symmetrization (such as determining whether to average the charge density of two grids), so it may be the subtle value that causes the small difference.
@pxlxingliang could you please try the same test with symmetry_prec=1e-5?

[pxlxingliang]

003 (3).zip
Please check the logs.

I have tested symmetry_prec=1e-5, and the drho of 10 times is same.

[pxlxingliang]

I noticed that there is some warning when set symmetry_prec=1e-6, and it tells that abacus will deal this structure as a primitive cell, but the K points are 13 which are less than the k point 25 when setting symmetry=0.

  WARNING: PRICELL: NCELL != NTRANS !
 NCELL=2, NTRANS=3
 Suggest solution: Use a larger `symmetry_prec`.
 Now regard the structure as a primitive cell.
 DONE(0.116236   SEC) : SYMMETRY
 DONE(0.248344   SEC) : INIT K-POINTS
 ---------------------------------------------------------
 Self-consistent calculations for electrons
 ---------------------------------------------------------
 SPIN    KPOINTS         PROCESSORS
 1       13              16
 ---------------------------------------------------------

[maki49]

I noticed that there is some warning when set symmetry_prec=1e-6, and it tells that abacus will deal this structure as a primitive cell, but the K points are 13 which are less than the k point 25 when setting symmetry=0.

Yes. It means that the space group symmetry is partly used while the primitive-cell translation symmetry is not used. They are 2 different types of symmetry sharing the same symmetry_prec to filter the operations.

[WHUweiqingzhou]

@maki49,
Why symmetry_prec = 1e-6 leads to different symmetry with same INPUT. Is it a bug?

[maki49]

@maki49, Why symmetry_prec = 1e-6 leads to different symmetry with same INPUT. Is it a bug?

No. symmetry_prec is the key parameter determining the precision to analyze symmetry. If symmetry_prec=0, no symmetry will be found.
But I have no idea why the final density is randomly different with the same symmetry analyzed by symmetry_prec=1e-6.

[pxlxingliang]

The image after 0418 may be the problem.
I have tested on a bohrium image generated at 0418, and run 003 example 10 times, and the results show that the dE of the last SCF step are different, which indicates the calculation is unstable. I also checkout to other commits (before 0418 and after 0418), they all give the same conclusions.
The last SCF step of 10 times run:

While I use a bohrium image generated at 0415, and run 10 times with different commits, and all of them give the same results.

[pxlxingliang]

At 20240428, one more example 008 failed the accuracy test.
The stress of 008 has a maximum absolute deviation (MAD) of 5e-4 kbar to the reference value.
At the same time, the MAD of stress for 006 is 0.012 kbar.

https://app.bohrium.dp.tech/abacustest/?request=GET%3A%2Fapplications%2Fabacustest%2Fjobs%2Fsched-abacustest-intel-dav-dbab2f

[pxlxingliang]

I have checked the difference between image at 0416 and 0419:

PACKAGE                                 0419-IMAGE1 (ghcr.io/deepmodeling/abacus-intel@sha256:91513122d597f47d64bf142ae9c6ef116e8cefe8928e2fa26d212bd8359a9f3e)        0416-IMAGE2 (ghcr.io/deepmodeling/abacus-intel@sha256:8ca79dc7cc8882a7cf950ca54a5ca8b63503060781626dc06791364609a18344)
-apt                                    2.4.12, 4M                                                                                                                2.4.11, 4M
-bash                                   5.1-6ubuntu1.1, 1.8M                                                                                                      5.1-6ubuntu1, 1.8M
-bsdutils                               1:2.37.2-4ubuntu3.4, 334K                                                                                                 1:2.37.2-4ubuntu3, 334K
-coreutils                              8.32-4.1ubuntu1.2, 6.9M                                                                                                   8.32-4.1ubuntu1.1, 6.9M
-dpkg                                   1.21.1ubuntu2.3, 6.6M                                                                                                     1.21.1ubuntu2.2, 6.6M
-intel-oneapi-common-vars               2024.1.0-579, 56K                                                                                                         2024.0.0-49406, 56K
-intel-oneapi-compiler-dpcpp-cpp        2024.1.0-963, 0                                                                                                           2024.0.2-49895, 0
-intel-oneapi-compiler-fortran          2024.1.0-963, 0                                                                                                           2024.0.2-49895, 0
-intel-oneapi-mkl-devel                 2024.1.0-691, 0                                                                                                           2024.0.0-49656, 0
-intel-oneapi-mpi-devel                 2021.12.0-534, 0                                                                                                          2021.11.0-49493, 0
-intel-oneapi-tcm-1.0                   1.0.1-175, 10.7M                                                                                                          1.0.0-435, 11.7M
-intel-oneapi-vtune                     2024.1.0-515, 1.5G                                                                                                        2024.0.1-11, 1.3G
-libapt-pkg6.0                          2.4.12, 3.1M                                                                                                              2.4.11, 3.1M
-libblkid1                              2.37.2-4ubuntu3.4, 323K                                                                                                   2.37.2-4ubuntu3, 323K
-libcurl3-gnutls                        7.81.0-1ubuntu1.16, 771K                                                                                                  7.81.0-1ubuntu1.15, 771K
-libcurl4                               7.81.0-1ubuntu1.16, 787K                                                                                                  7.81.0-1ubuntu1.15, 787K
-libmount1                              2.37.2-4ubuntu3.4, 382K                                                                                                   2.37.2-4ubuntu3, 382K
-libsmartcols1                          2.37.2-4ubuntu3.4, 209K                                                                                                   2.37.2-4ubuntu3, 209K
-libuuid1                               2.37.2-4ubuntu3.4, 134K                                                                                                   2.37.2-4ubuntu3, 134K
-linux-libc-dev                         5.15.0-102.112, 6.7M                                                                                                      5.15.0-100.110, 6.7M
-mount                                  2.37.2-4ubuntu3.4, 389K                                                                                                   2.37.2-4ubuntu3, 389K
-util-linux                             2.37.2-4ubuntu3.4, 3.3M                                                                                                   2.37.2-4ubuntu3, 3.3M
-vim                                    2:8.2.3995-1ubuntu2.16, 3.8M                                                                                              2:8.2.3995-1ubuntu2.15, 3.8M
-vim-common                             2:8.2.3995-1ubuntu2.16, 382K                                                                                              2:8.2.3995-1ubuntu2.15, 382K
-vim-runtime                            2:8.2.3995-1ubuntu2.16, 32M                                                                                               2:8.2.3995-1ubuntu2.15, 32M
-xxd                                    2:8.2.3995-1ubuntu2.16, 280K                                                                                              2:8.2.3995-1ubuntu2.15, 280K

And the mainly difference is the version of Inter oneapi. At the 0419 image, I re-install intel-oneapi-mpi-devel with version 2021.11.0-49493, and this time, the results are stable.

[caic99]

Hi @WHUweiqingzhou ,
I'm afraid that this issue still requires more investigation.

[caic99]

@dyzheng Do we have a UT for diagonalization?

[WHUweiqingzhou]

@haozhihan could you have a look?