Fix CTest support template and script

[grnydawn]

• Added build_only = False when SLURM_JOB_ID is set in os.environ within omega_ctest.py
• Updated paths to the mesh NetCDF files in the build/test script template

This PR fixes issues found from running Omega ctest on Chrysalis.

changing permissions on downloaded files
100% |#################################################################################| Time:  0:00:00
  done.
Traceback (most recent call last):
  File "/lcrc/group/e3sm/ac.kimy/omega/polaris/utils/omega/ctest/./omega_ctest.py", line 273, in <module>
    main()
    ~~~~^^
  File "/lcrc/group/e3sm/ac.kimy/omega/polaris/utils/omega/ctest/./omega_ctest.py", line 234, in main
    build_only=build_only,
               ^^^^^^^^^^
UnboundLocalError: cannot access local variable 'build_only' where it is not associated with a value

and

[100%] Linking CXX executable testField.exe
icpc: warning #10237: -lcilkrts linked in dynamically, static library not available
[100%] Built target testField.exe
+ cd test
+ ln -sfn /lcrc/group/e3sm/public_html/polaris/ocean/omega_ctest/ocean.QU.240km.151209.nc OmegaMesh.nc
+ ln -sfn /lcrc/group/e3sm/public_html/polaris/ocean/omega_ctest/PlanarPeriodic48x48.nc OmegaPlanarMesh.nc
+ ln -sfn /lcrc/group/e3sm/public_html/polaris/ocean/omega_ctest/cosine_bell_icos480_initial_state.230220.nc OmegaSphereMesh.nc
+ ./omega_ctest.sh
/lcrc/group/e3sm/ac.kimy/omega/polaris/build_omega/build_and_ctest_omega_chrysalis_intel.sh: line 51: ./omega_ctest.sh: No such file or directory

[xylar]

Thanks @grnydawn! I'll take a look tomorrow.

[xylar]

Thanks @grnydawn. It took me a bit to realize why I hadn't seen these issues myself. I always run the utility from a login node and let it launch a job, whereas you are running on a compute node from an interactive job. Both modes should work so your fixes are very much appreciated!

I am able to reproduce both the original problem and a successful run with these fixes on an interactive job. (However, tests seem to be taking much longer on Chrysalis in an interactive job than they do in a batch job.)

[xylar]

(However, tests seem to be taking much longer on Chrysalis in an interactive job than they do in a batch job.)

This is happening because each test runs as a separate batch job and therefore takes some queue time. This is a good reason not to try to run CTests interactively on Chrysalis.

[grnydawn]

Thanks @grnydawn. It took me a bit to realize why I hadn't seen these issues myself. I always run the utility from a login node and let it launch a job, whereas you are running on a compute node from an interactive job. Both modes should work so your fixes are very much appreciated!

I am able to reproduce both the original problem and a successful run with these fixes on an interactive job. (However, tests seem to be taking much longer on Chrysalis in an interactive job than they do in a batch job.)

@xylar . Yes, I ran the test on an interactive computing node, which I usually do when frequent changes in the code and configuration need to be tested quickly. Thanks