Failing tests on Aurora (SYCL - Intel GPU)

[mjschmidt271]

As referenced in PR #445

Failing tests:

The following tests FAILED:
         19 - mode_averages (Failed)
         64 - validate_stand_modal_aero_calcsize_sub_update_ptend (Failed)
         66 - validate_stand_calcsize_aero_model_wetdep_ts_379 (Failed)

Output from tests failing on Aurora

Start testing: May 22 00:28 UTC
----------------------------------------------------------
19/646 Testing: mode_averages
19/646 Test: mode_averages
Command: "/usr/bin/sh" "-c" "/home/mjschm/mam4xx/build/bin/test-launcher -- ./mode_averages --use-colour no"
Directory: /home/mjschm/mam4xx/build/src/tests
"mode_averages" start time: May 22 00:28 UTC
Output:
----------------------------------------------------------
Calling initialize_kokkos
 ExecSpace name: SYCL
 ExecSpace initialized: yes
 active avx set: 
 compiler id: IntelLLVM
 FPE support is enabled, current FPE mask: 0 (NONE)
 #host threads: 1


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
mode_averages is a Catch v2.13.8 host application.
Run with -? for options

-------------------------------------------------------------------------------
modal_averages
-------------------------------------------------------------------------------
/home/mjschm/mam4xx/src/tests/mode_averages_unit_tests.cpp:27
...............................................................................

/home/mjschm/mam4xx/src/tests/mode_averages_unit_tests.cpp:27: FAILED:
due to unexpected exception with message:
  The program was built for 1 devices
  Build program log for 'Intel(R) Data Center GPU Max 1550':
  Module <0x3d8fd10>:  Unresolved Symbol <nan>
  Module <0x3d8fd10>:  Unresolved Symbol <nan>

===============================================================================
test cases: 1 | 1 failed
assertions: 1 | 1 failed

EKAT is not managing resources.
RUN: OMP_PROC_BIND=spread OMP_PLACES=threads ./mode_averages --use-colour no
FROM: /home/mjschm/mam4xx/build/src/tests
<end of output>
Test time =   1.50 sec
----------------------------------------------------------
Test Failed.
"mode_averages" end time: May 22 00:28 UTC
"mode_averages" time elapsed: 00:00:01
----------------------------------------------------------

64/646 Testing: validate_stand_modal_aero_calcsize_sub_update_ptend
64/646 Test: validate_stand_modal_aero_calcsize_sub_update_ptend
Command: "/opt/aurora/24.347.0/spack/unified/0.9.2/install/linux-sles15-x86_64/gcc-13.3.0/python-venv-1.0-a4pusmc/bin/python3" "compare_mam4xx_mam4.py" "mam4xx_stand_modal_aero_calcsize_sub_update_ptend.py" "mam_stand_modal_aero_calcsize_sub_update_ptend.py" "True" "3e-5"
Directory: /home/mjschm/mam4xx/build/src/validation/calcsize
"validate_stand_modal_aero_calcsize_sub_update_ptend" start time: May 22 00:28 UTC
Output:
----------------------------------------------------------
dgnumdry_m
L1 4.758090000438264e-12
L2 6.120652745099863e-13
Linf 1.8500000014169075e-13
ptend_q
L1 0.0016484514219205918
L2 0.0010593508120839688
Linf 0.0007477462949000001
L1 rel_error 0.0003181953256389337
L2 rel_error 0.000204483111928284
Linf rel_error 0.00014433508481784857
qqcw
L1 0.0
L2 0.0
Linf 0.0
final pass array = [ True False  True]
Traceback (most recent call last):
  File "/home/mjschm/mam4xx/build/src/validation/calcsize/compare_mam4xx_mam4.py", line 136, in <module>
    assert(np.all(pass_all_tests))
AssertionError
<end of output>
Test time =   0.13 sec
----------------------------------------------------------
Test Failed.
"validate_stand_modal_aero_calcsize_sub_update_ptend" end time: May 22 00:28 UTC
"validate_stand_modal_aero_calcsize_sub_update_ptend" time elapsed: 00:00:00
----------------------------------------------------------

66/646 Testing: validate_stand_calcsize_aero_model_wetdep_ts_379
66/646 Test: validate_stand_calcsize_aero_model_wetdep_ts_379
Command: "/opt/aurora/24.347.0/spack/unified/0.9.2/install/linux-sles15-x86_64/gcc-13.3.0/python-venv-1.0-a4pusmc/bin/python3" "compare_mam4xx_mam4.py" "mam4xx_stand_calcsize_aero_model_wetdep_ts_379.py" "mam_stand_calcsize_aero_model_wetdep_ts_379.py" "True" "1.5e-3"
Directory: /home/mjschm/mam4xx/build/src/validation/calcsize
"validate_stand_calcsize_aero_model_wetdep_ts_379" start time: May 22 00:28 UTC
Output:
----------------------------------------------------------
dgnumdry_m
L1 4.383719999403568e-12
L2 5.946998775934814e-13
Linf 1.860000004629556e-13
ptend_q
L1 14.419584504071418
L2 10.195808596672368
Linf 7.209525398
L1 rel_error 0.19365119759473054
L2 rel_error 0.1369270067826688
Linf rel_error 0.09682201501840247
qqcw
L1 91388.94299998647
L2 25167.88959375477
Linf 9074.523000000045
L1 rel_error 0.0010769036541811079
L2 rel_error 0.0002965718978886232
Linf rel_error 0.0001069318306772845
final pass array = [ True False  True]
Traceback (most recent call last):
  File "/home/mjschm/mam4xx/build/src/validation/calcsize/compare_mam4xx_mam4.py", line 136, in <module>
    assert(np.all(pass_all_tests))
AssertionError
<end of output>
Test time =   0.11 sec
----------------------------------------------------------
Test Failed.
"validate_stand_calcsize_aero_model_wetdep_ts_379" end time: May 22 00:28 UTC
"validate_stand_calcsize_aero_model_wetdep_ts_379" time elapsed: 00:00:00
----------------------------------------------------------

End testing: May 22 00:28 UTC

Originally posted by @mjschmidt271 in #445 (comment)