What are people's dev/test workflows for CIME now?

[billsacks]

I'm returning to some CIME development after not doing much for the last few years. I know there have been some valuable additions to the CIME dev/test workflows in the last few years, such as the addition of pre-commit configuration, GitHub actions, and maybe more. Given these changes, I'm curious what people's dev/test workflows tend to be now.

Two specific questions:

1. How do you run pre-commit - e.g., manually, or via git hooks, or just relying on the action after pushing? And if manually, do you run pre-commit run -a as is documented in the wiki, or some other way? I'm noticing that that takes a little while to run... I was thinking of generally running it manually before committing, but wondered if there's an easy way to restrict that to just the files that have been changed.
2. For a small/simple PR, what amount of testing do people generally do by hand at this point vs. just relying on the GitHub actions?

If you have other tips for your dev/test workflow with CIME, I'd also be happy to hear anything else.

@jedwards4b @jgfouca @jasonb5 - especially interested in your input.

[jgfouca]

@billsacks , glad to have back on CIME!

For 1, I have pre-commit in my PATH and I just run pre-commit run -a like you said once I feel my branch is ready to be made into a PR. I always have things black needs to fix up.

For 2, the github CI is pretty good at this point. On my machine, the scripts-regression-tests do not run successfully, so I just rely on the CI mostly. I do usually at least try to do a bit of manual testing on whatever I did. Also, for new features, it's always a good idea to add tests. The testing is a bit different than it was before; it relies much more on mock objects rather than setting up real CIME cases.

Jason has made a lot of progress on modernizing the CIME dev process, but CIME development is not fundamentally that different than it was back when you were an active CIME dev.

[jedwards4b]

I use git-hooks to run pre-commit, that only runs on files you've changed so you can end up with errors in github, but not often.
I try to run scripts_regression_tests locally, if I rely on github to do it I almost always need to ask jason for help if it fails...

[jgfouca]

@jedwards4b , that's a good point. Containers add complexity, but one good thing is that they should, in theory, allow you do duplicate CI errors on almost any machine. If you load a container image, it should get you an environment identical to the CI env. @jasonb5 , are there instructions for doing this somewhere?

[billsacks]

Thanks a lot, @jgfouca and @jedwards4b - it's very helpful to hear both of your workflows here!

[billsacks]

@jedwards4b where do you run scripts_regression_tests? I'm having issues everywhere I try it (my Mac, derecho, izumi). The first failure is #4874 but I hit others as well. Maybe user error?

[billsacks]

Partly just to document this for myself, and partly in case it's useful for others and/or anyone has any insight / suggestions: Here are the failures I'm seeing in scripts_regression_tests currently, when running in a CESM context (from grep -e '^ERROR' -e '^FAIL' in the output):

Failures on derecho:

ERROR: test_b_cime_f90_unit_tests (test_sys_unittest.TestUnitTest.test_b_cime_f90_unit_tests)
ERROR: test_generate_env_mach_specific (test_unit_configure.TestConfigure.test_generate_env_mach_specific)
ERROR: test_generate_baseline (test_unit_system_tests.TestUnitSystemTests.test_generate_baseline)
ERROR: test_filter_children_by_compiler (test_unit_xml_machines.TestUnitXMLMachines.test_filter_children_by_compiler)
ERROR: test_get_resolved_environment_variable (test_unit_xml_machines.TestUnitXMLMachines.test_get_resolved_environment_variable)
ERROR: test_has_batch_system (test_unit_xml_machines.TestUnitXMLMachines.test_has_batch_system)
ERROR: test_is_valid_MPIlib (test_unit_xml_machines.TestUnitXMLMachines.test_is_valid_MPIlib)
ERROR: test_is_valid_compiler (test_unit_xml_machines.TestUnitXMLMachines.test_is_valid_compiler)
ERROR: test_print_values (test_unit_xml_machines.TestUnitXMLMachines.test_print_values)
ERROR: test_print_values_details (test_unit_xml_machines.TestUnitXMLMachines.test_print_values_details)
FAIL: test_bless_test_results (test_sys_bless_tests_results.TestBlessTestResults.test_bless_test_results)
FAIL: test_rebless_namelist (test_sys_bless_tests_results.TestBlessTestResults.test_rebless_namelist)
FAIL: test_full_system (test_sys_full_system.TestFullSystem.test_full_system)
FAIL: test_run_restart_too_many_fails (test_sys_run_restart.TestRunRestart.test_run_restart_too_many_fails)
FAIL: test_c_use_existing (test_sys_test_scheduler.TestTestScheduler.test_c_use_existing)
FAIL TESTRUNFAIL_P1.f19_g16.A.derecho_intel (phase RUN)
FAIL: test_d_retry (test_sys_test_scheduler.TestTestScheduler.test_d_retry)
FAIL TESTRUNFAILRESET_P1.f19_g16.A.derecho_intel (phase RUN)
FAIL TESTRUNFAILRESET_P1.f19_g16.A.derecho_intel (phase RUN)
FAIL: test_user_concurrent_mods (test_sys_user_concurrent_mods.TestUserConcurrentMods.test_user_concurrent_mods)
FAILED (failures=7, errors=10, skipped=28)

Failures on my Mac:

ERROR: test_read_baseline_file_content (test_unit_baselines_performance.TestUnitBaselinesPerformance.test_read_baseline_file_content)
ERROR: test_generate_env_mach_specific (test_unit_configure.TestConfigure.test_generate_env_mach_specific)
ERROR: test_generate_baseline (test_unit_system_tests.TestUnitSystemTests.test_generate_baseline)
ERROR: test_filter_children_by_compiler (test_unit_xml_machines.TestUnitXMLMachines.test_filter_children_by_compiler)
ERROR: test_get_resolved_environment_variable (test_unit_xml_machines.TestUnitXMLMachines.test_get_resolved_environment_variable)
ERROR: test_has_batch_system (test_unit_xml_machines.TestUnitXMLMachines.test_has_batch_system)
ERROR: test_is_valid_MPIlib (test_unit_xml_machines.TestUnitXMLMachines.test_is_valid_MPIlib)
ERROR: test_is_valid_compiler (test_unit_xml_machines.TestUnitXMLMachines.test_is_valid_compiler)
ERROR: test_print_values (test_unit_xml_machines.TestUnitXMLMachines.test_print_values)
ERROR: test_print_values_details (test_unit_xml_machines.TestUnitXMLMachines.test_print_values_details)
FAIL: test_bless_test_results (test_sys_bless_tests_results.TestBlessTestResults.test_bless_test_results)
FAIL: test_rebless_namelist (test_sys_bless_tests_results.TestBlessTestResults.test_rebless_namelist)
FAIL: test_self_build_cprnc (test_sys_cime_case.TestCimeCase.test_self_build_cprnc)
ERROR: /Users/sacks/cesm/cesm2/cime/CIME/build_scripts/buildlib.cprnc FAILED, cat /Users/sacks/projects/scratch/scripts_regression_test.20251010_115558/ERS_Ln7.f19_g16.A.green_gnu.fake_testing_only_20251010_115820/bld/cprnc.bldlog.251010-115821
FAIL: test_full_system (test_sys_full_system.TestFullSystem.test_full_system)
FAIL ERIO_Ln11.f09_g16.X.green_gnu (phase RUN)
FAIL ERP_Ln9.f45_g37.A.green_gnu (phase SHAREDLIB_BUILD)
FAIL PET_Ln9_P4.f19_f19.A.green_gnu (phase SHAREDLIB_BUILD)
FAIL SMS_D_Ln9_Mmpi-serial.f19_g16.A.green_gnu (phase RUN)
FAIL: test_user_concurrent_mods (test_sys_user_concurrent_mods.TestUserConcurrentMods.test_user_concurrent_mods)
FAIL TESTRUNUSERXMLCHANGE_Mmpi-serial.f19_g16.X.green_gnu (phase RUN)
FAIL: test_wait_for_test_cdash_kill (test_sys_wait_for_tests.TestWaitForTests.test_wait_for_test_cdash_kill)
FAIL: test_wait_for_test_wait_kill (test_sys_wait_for_tests.TestWaitForTests.test_wait_for_test_wait_kill)
FAIL: test_read_baseline_file_multi_line (test_unit_baselines_performance.TestUnitBaselinesPerformance.test_read_baseline_file_multi_line)
FAILED (failures=8, errors=10, skipped=36)

My ideal goal is to get a clean run on my Mac so that I can reliably do testing there. To that end, I have looked into some of the failures from my Mac that are passing on derecho:

• cprnc build: see Can't get recent versions of cprnc to build on my Mac due to problems finding netcdf cprnc#29
• failures in test_unit_baselines_performance: I have a PR incoming that fixes this