Issues with run_neon with the `--experiment` flag starting in ctsm5.1.dev172

[wwieder]

Brief summary of bug

It seems like in the NEON refactor for PLUMBER2 we lost come capabilities using the --experiment flag? Specifically creating an AD case that includes the experiment flag in the case name.

General bug information

CTSM version you are using: dev175

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected: running NEON cases with --experiment flag.

Details of bug

I created two cases, one with dev171 & another with dev175, that were intended to evaluate impacts of dead arctic veg on NEON simulations in Alaska. The following command with dev175 creates an AD case, as expected, but without the experiment flag in the case name.

./run_neon --neon-sites BARR --run-type ad --experiment dev175 --output-root /glade/derecho/scratch/wwieder/neon_AK

the same command using dev 171 produces case names like this: BARR.dev171.ad

cases are all in my scratch directory:
/glade/u/home/wwieder/scratch/neon_AK

[ekluzek]

Thanks for pointing this out @wwieder. This must be in #2363, but because it's a refactoring step it's hard to point to the exact line.

This is where adding more testing for all the important configurations run_neon is used in would be critical to prevent this type of thing. We test a few -- but obviously not enough.

So @TeaganKing and @wwieder it would be good to sit down together and map out the list of important configuration of options you use for run_neon. It looks like right now our testing is really minimal, and we need to also test interactions of command line options with each other (like in this case).

Having this kind of comprehensive testing enables us to refactor and improve the design without losing important functionality. Rather than adding comprehensive testing up front we've been incrementally adding testing as we go, which in this case led to this problem. In Agile SDM it's considered important to have the comprehensive testing up front so you know you can refactor without regressing functionality. So we might want to discuss what kind of balance we want to have between those two things as we move forward.

[TeaganKing]

Thanks for catching this bug and making this issue. A few other notes I wanted to add are below.

• Without the run type specified, it looks like the actual build is not completing, and requires an extra step of running ./case.build separately from the main case directory (not the .transient case directory). I think this may be a more significant problem.
• I've made a fix to how run_neon.py calls run_case to specify the optional arguments which might resolve this issue; I'm testing a few examples now. I likely won't have actual results before Monday morning, but wanted to share a status update.

[wwieder]

A few more errors after trying to start postad runs

---- cloning the base case in /glade/derecho/scratch/wwieder/neon_AK/BARR.postad
Traceback (most recent call last):
  File "./run_neon", line 48, in <module>
    main(__doc__)
  File "/glade/u/home/wwieder/CTSM/tools/site_and_regional/../../python/ctsm/site_and_regional/run_neon.py", line 241, in main
    experiment,
  File "/glade/u/home/wwieder/CTSM/tools/site_and_regional/../../python/ctsm/site_and_regional/neon_site.py", line 103, in run_case
    base_case_root, run_type, prism, run_length, user_version, tower_type, user_mods_dirs
  File "/glade/u/home/wwieder/CTSM/tools/site_and_regional/../../python/ctsm/site_and_regional/tower_site.py", line 394, in run_case
    self.set_ref_case(case)
  File "/glade/u/home/wwieder/CTSM/tools/site_and_regional/../../python/ctsm/site_and_regional/tower_site.py", line 230, in set_ref_case
    symlink_force(reffile, os.path.join(rundir, os.path.basename(reffile)))
  File "/glade/u/home/wwieder/CTSM/cime/CIME/utils.py", line 1473, in symlink_force
    raise e
  File "/glade/u/home/wwieder/CTSM/cime/CIME/utils.py", line 1467, in symlink_force
    os.symlink(target, link_name)
FileNotFoundError: [Errno 2] No such file or directory: '/glade/derecho/scratch/wwieder/neon_AK/BARR.ad/run/BARR.ad.clm2.r.1018-01-01-00000.nc' -> '/glade/derecho/scratch/wwieder/neon_AK/BARR.postad/run/BARR.ad.clm2.r.1018-01-01-00000.nc'

[wwieder]

it seems like the run directory isn't being created in the new .postad case, the initial conditions can't be copied, and the jobs fail. Manually copying files into the run directory still doesn't pickup the right restart files (not sure what else isn't happening correctly...)

[TeaganKing]

Thanks for these extra details, @wwieder !