MILC code (master branch) with QUDA version 1.1.0 ---> HISQ fattening unitarization error

[lcosmai]

I successfully compiled the MILC code (master branch) with QUDA version 1.1.0 using CUDA v11.8.

The QUDA compilation passed all the tests.

I compiled the su3_rhmc_hisq target for ks_imp_rhmc.

I then launched a test job on 4 nodes, each with 4 Nvidia A100 GPUs.

The job aborted with the following error:

“
ERROR: Error in unitarization component of the hisq fattening: 1048576 failures (/leonardo/pub/userexternal/lcosmai0/AREA_COMPILAZIONE_QUDA/quda-1.1.0/lib/interface_quda.cpp:4154 in computeKSLinkQuda())
“

Could you please provide any suggestions on how to resolve this issue?

Best regards,
Leonardo

[james-simone]

Hi Leonardo, I expect you should be using the 'develop' branch of milc_qcd rather than 'master'.

…

--jim -----Original Message----- From: Leonardo Cosmai ***@***.*** ***@***.***>> Reply-To: milc-qcd/milc_qcd ***@***.*** ***@***.***>> Date: Friday, November 3, 2023 at 12:07 PM To: milc-qcd/milc_qcd ***@***.*** ***@***.***>> Cc: Subscribed ***@***.*** ***@***.***>> Subject: [milc-qcd/milc_qcd] MILC code (master branch) with QUDA version 1.1.0 ---> HISQ fattening unitarization error (Issue #62) I successfully compiled the MILC code (master branch) with QUDA version 1.1.0 using CUDA v11.8. The QUDA compilation passed all the tests. I compiled the su3_rhmc_hisq target for ks_imp_rhmc. I then launched a test job on 4 nodes, each with 4 Nvidia A100 GPUs. The job aborted with the following error: “ ERROR: Error in unitarization component of the hisq fattening: 1048576 failures (/leonardo/pub/userexternal/lcosmai0/AREA_COMPILAZIONE_QUDA/quda-1.1.0/lib/interface_quda.cpp:4154 in computeKSLinkQuda()) “ Could you please provide any suggestions on how to resolve this issue? Best regards, Leonardo — Reply to this email directly, view it on GitHub <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_milc-2Dqcd_milc-5Fqcd_issues_62&amp;d=DwMFaQ&amp;c=gRgGjJ3BkIsb5y6s49QqsA&amp;r=9pGwH941nYC6vS4VlvwK0Q&amp;m=bCRcFK-TZO1RS9DZEjEVlFiKn5ptU3pkChGkE7N0D_NKgSMCuhlEXis4VT_wcDdz&amp;s=NIv_t3TGac50ORmaBEMLYzPlmBmiLjogVmN1sYRCmDk&amp;e=> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_milc-2Dqcd_milc-5Fqcd_issues_62&amp;amp;d=DwMFaQ&amp;amp;c=gRgGjJ3BkIsb5y6s49QqsA&amp;amp;r=9pGwH941nYC6vS4VlvwK0Q&amp;amp;m=bCRcFK-TZO1RS9DZEjEVlFiKn5ptU3pkChGkE7N0D_NKgSMCuhlEXis4VT_wcDdz&amp;amp;s=NIv_t3TGac50ORmaBEMLYzPlmBmiLjogVmN1sYRCmDk&amp;amp;e=&gt;>, or unsubscribe <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABRABTSNLQSR43KAEPMO73LYCUQGBAVCNFSM6AAAAAA64XS4O6VHI2DSMVQWIX3LMV43ASLTON2WKOZRHE3TMNRRGE3DEMQ&amp;d=DwMFaQ&amp;c=gRgGjJ3BkIsb5y6s49QqsA&amp;r=9pGwH941nYC6vS4VlvwK0Q&amp;m=bCRcFK-TZO1RS9DZEjEVlFiKn5ptU3pkChGkE7N0D_NKgSMCuhlEXis4VT_wcDdz&amp;s=PxEgQK042V3qw2aKnKOoOrkp125ZigfNRVhnCNuCq3k&amp;e=> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABRABTSNLQSR43KAEPMO73LYCUQGBAVCNFSM6AAAAAA64XS4O6VHI2DSMVQWIX3LMV43ASLTON2WKOZRHE3TMNRRGE3DEMQ&amp;amp;d=DwMFaQ&amp;amp;c=gRgGjJ3BkIsb5y6s49QqsA&amp;amp;r=9pGwH941nYC6vS4VlvwK0Q&amp;amp;m=bCRcFK-TZO1RS9DZEjEVlFiKn5ptU3pkChGkE7N0D_NKgSMCuhlEXis4VT_wcDdz&amp;amp;s=PxEgQK042V3qw2aKnKOoOrkp125ZigfNRVhnCNuCq3k&amp;amp;e=&gt;>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** ***@***.***>>

[lcosmai]

Hi Jim,

Thanks for your suggestion.

Unfortunately, when I tried to compile milc_qcd-develop, I received the following error messages:

../generic_ks/gauss_smear_ks_QUDA.c: In function 'gauss_smear_delete_2link_QUDA':
../generic_ks/gauss_smear_ks_QUDA.c:51:3: warning: implicit declaration of function 'qudaFreeTwoLink' [-Wimplicit-function-declaration]
51 | qudaFreeTwoLink();
| ^~~~~~~~~~~~~~~
../generic_ks/gauss_smear_ks_QUDA.c: In function 'gauss_smear_v_field_QUDA':
../generic_ks/gauss_smear_ks_QUDA.c:106:3: error: unknown type name 'QudaTwoLinkQuarkSmearArgs_t'
106 | QudaTwoLinkQuarkSmearArgs_t qsmear_args;
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~
../generic_ks/gauss_smear_ks_QUDA.c:107:14: error: request for member 'n_steps' in something not a structure or union
107 | qsmear_args.n_steps = iters;
| ^
../generic_ks/gauss_smear_ks_QUDA.c:108:14: error: request for member 'width' in something not a structure or union
108 | qsmear_args.width = width;
| ^
../generic_ks/gauss_smear_ks_QUDA.c:109:14: error: request for member 'compute_2link' in something not a structure or union
109 | qsmear_args.compute_2link = compute_2link_temp;
| ^
../generic_ks/gauss_smear_ks_QUDA.c:110:14: error: request for member 'delete_2link' in something not a structure or union
110 | qsmear_args.delete_2link = 0;
| ^
../generic_ks/gauss_smear_ks_QUDA.c:111:14: error: request for member 't0' in something not a structure or union
111 | qsmear_args.t0 = t0;
| ^
../generic_ks/gauss_smear_ks_QUDA.c:112:14: error: request for member 'laplaceDim' in something not a structure or union
112 | qsmear_args.laplaceDim = laplaceDim;
| ^
../generic_ks/gauss_smear_ks_QUDA.c:115:3: warning: implicit declaration of function 'qudaTwoLinkGaussianSmear' [-Wimplicit-function-declaration]
115 | qudaTwoLinkGaussianSmear( MILC_PRECISION, MILC_PRECISION, (void*) t_links, (void*) src, qsmear_args );
| ^~~~~~~~~~~~~~~~~~~~~~~~
make[1]: *** [../generic_ks/Make_template:384: gauss_smear_ks_QUDA.o] Error 1
make[1]: Leaving directory '/leonardo/pub/userexternal/lcosmai0/AREA_COMPILAZIONE_MILC/milc_qcd-develop/ks_imp_rhmc'
make: *** [Make_template:223: su3_rhmc_hisq] Error 2

[james-simone]

Hi, Double check that you have -DQUDA_SMEAR_GAUSS_TWOLINK=ON in the quad cmake step. -----Original Message----- From: Leonardo Cosmai ***@***.*** ***@***.***>> Reply-To: milc-qcd/milc_qcd ***@***.*** ***@***.***>> Date: Friday, November 3, 2023 at 1:53 PM To: milc-qcd/milc_qcd ***@***.*** ***@***.***>> Cc: James N Simone ***@***.*** ***@***.***>>, Comment ***@***.*** ***@***.***>> Subject: Re: [milc-qcd/milc_qcd] MILC code (master branch) with QUDA version 1.1.0 ---> HISQ fattening unitarization error (Issue #62) Hi Jim, Thanks for your suggestion. Unfortunately, when I tried to compile milc_qcd-develop, I received the following error messages: ../generic_ks/gauss_smear_ks_QUDA.c: In function 'gauss_smear_delete_2link_QUDA': ../generic_ks/gauss_smear_ks_QUDA.c:51:3: warning: implicit declaration of function 'qudaFreeTwoLink' [-Wimplicit-function-declaration] 51 | qudaFreeTwoLink(); | ^~~~~~~~~~~~~~~ ../generic_ks/gauss_smear_ks_QUDA.c: In function 'gauss_smear_v_field_QUDA': ../generic_ks/gauss_smear_ks_QUDA.c:106:3: error: unknown type name 'QudaTwoLinkQuarkSmearArgs_t' 106 | QudaTwoLinkQuarkSmearArgs_t qsmear_args; | ^~~~~~~~~~~~~~~~~~~~~~~~~~~ ../generic_ks/gauss_smear_ks_QUDA.c:107:14: error: request for member 'n_steps' in something not a structure or union 107 | qsmear_args.n_steps = iters; | ^ ../generic_ks/gauss_smear_ks_QUDA.c:108:14: error: request for member 'width' in something not a structure or union 108 | qsmear_args.width = width; | ^ ../generic_ks/gauss_smear_ks_QUDA.c:109:14: error: request for member 'compute_2link' in something not a structure or union 109 | qsmear_args.compute_2link = compute_2link_temp; | ^ ../generic_ks/gauss_smear_ks_QUDA.c:110:14: error: request for member 'delete_2link' in something not a structure or union 110 | qsmear_args.delete_2link = 0; | ^ ../generic_ks/gauss_smear_ks_QUDA.c:111:14: error: request for member 't0' in something not a structure or union 111 | qsmear_args.t0 = t0; | ^ ../generic_ks/gauss_smear_ks_QUDA.c:112:14: error: request for member 'laplaceDim' in something not a structure or union 112 | qsmear_args.laplaceDim = laplaceDim; | ^ ../generic_ks/gauss_smear_ks_QUDA.c:115:3: warning: implicit declaration of function 'qudaTwoLinkGaussianSmear' [-Wimplicit-function-declaration] 115 | qudaTwoLinkGaussianSmear( MILC_PRECISION, MILC_PRECISION, (void*) t_links, (void*) src, qsmear_args ); | ^~~~~~~~~~~~~~~~~~~~~~~~ make[1]: *** [../generic_ks/Make_template:384: gauss_smear_ks_QUDA.o] Error 1 make[1]: Leaving directory '/leonardo/pub/userexternal/lcosmai0/AREA_COMPILAZIONE_MILC/milc_qcd-develop/ks_imp_rhmc' make: *** [Make_template:223: su3_rhmc_hisq] Error 2 — Reply to this email directly, view it on GitHub <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_milc-2Dqcd_milc-5Fqcd_issues_62-23issuecomment-2D1792962103&amp;d=DwMCaQ&amp;c=gRgGjJ3BkIsb5y6s49QqsA&amp;r=9pGwH941nYC6vS4VlvwK0Q&amp;m=wMrA4-B_BFAGqMNTTvTbSIJaYNaYzXQrB7Q1ymOIVgjpudcJvUFLFVAf70Q4CMnw&amp;s=9uGOZL6cMU2bVe_5oUCtqT1MVgRbaPTgR5Xjry3QXHs&amp;e=> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_milc-2Dqcd_milc-5Fqcd_issues_62-23issuecomment-2D1792962103&amp;amp;d=DwMCaQ&amp;amp;c=gRgGjJ3BkIsb5y6s49QqsA&amp;amp;r=9pGwH941nYC6vS4VlvwK0Q&amp;amp;m=wMrA4-B_BFAGqMNTTvTbSIJaYNaYzXQrB7Q1ymOIVgjpudcJvUFLFVAf70Q4CMnw&amp;amp;s=9uGOZL6cMU2bVe_5oUCtqT1MVgRbaPTgR5Xjry3QXHs&amp;amp;e=&gt;>, or unsubscribe <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABRABTX2YYFBZRFEOPT3FELYCU4QRAVCNFSM6AAAAAA64XS4O6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJSHE3DEMJQGM&amp;d=DwMCaQ&amp;c=gRgGjJ3BkIsb5y6s49QqsA&amp;r=9pGwH941nYC6vS4VlvwK0Q&amp;m=wMrA4-B_BFAGqMNTTvTbSIJaYNaYzXQrB7Q1ymOIVgjpudcJvUFLFVAf70Q4CMnw&amp;s=t8M4vl5Af6_C27ckjPzJGQML9gzU3R7MVyHnx5F7cbQ&amp;e=> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABRABTX2YYFBZRFEOPT3FELYCU4QRAVCNFSM6AAAAAA64XS4O6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJSHE3DEMJQGM&amp;amp;d=DwMCaQ&amp;amp;c=gRgGjJ3BkIsb5y6s49QqsA&amp;amp;r=9pGwH941nYC6vS4VlvwK0Q&amp;amp;m=wMrA4-B_BFAGqMNTTvTbSIJaYNaYzXQrB7Q1ymOIVgjpudcJvUFLFVAf70Q4CMnw&amp;amp;s=t8M4vl5Af6_C27ckjPzJGQML9gzU3R7MVyHnx5F7cbQ&amp;amp;e=&gt;>. You are receiving this because you commented.Message ID: ***@***.*** ***@***.***>>

[maddyscientist]

@lcosmai you also need to use the develop version of QUDA from GitHub. We haven’t made a release tag since the two link smearing support was merged in. Thx

[lcosmai]

Following your suggestions, I successfully compiled the develop branch of QUDA with OPENMPI (-DQUDA_MPI=ON) and the develop branch of the MILC code (-DQUDA_SMEAR_GAUSS_TWOLINK=ON).

I also tested the MILC code on a GPU cluster equipped with 4 NVIDIA Ampere GPUs, 64GB HBM2, and 32 Intel Ice Lake cores per node.

I appreciate your kind support.