Test estimation software on other test cases

[ian-bertolacci]

For #6
My current implementation of the estimation tool has only been tested on my sinusoidal domain.
I need to try it on other TCL scripts to show that it works on other scripts as well, and doesn't "cause problems"
My main concern is that test scripts often orchestrate their own data distribution, which could generally be annoying at best, and cause issues with later runs at worst.

Done when:

• I've tested against the following cases:
  • all tests in parflow/tests/ that has a pfrun command
  • our benchmark tests clayl, ru-conus, tfg-conus
  • one or more test cases from hydroframe.org modeling v1.0 subset domain

Deliverables:

• Report
  • Listing any tests cases that fail and why.
  • Comment on any complications that arise (important for deciding future changes to the tool and to the ReadMe).

[ian-bertolacci]

Running on parflow/tests, observed the following types of failures:
Key failures:

1. Model producing invalid predictions (< 0)
2. non-zero exit codes
   • Some causes:
     • output file blah.pfb not created
     • Failed to open silo file blah.silo (pfload)
     • invalid expressions probably caused by not providing parameters
3. Bad json output
   • Probably caused by not providing parameters
   • in some cases, multipl json outputs, probably from multiple runs or tests?
4. Some actually seem to be running something?
   • ./richards_FBy.tcl (tclsh)
   • ./van-genuchten-file.tcl (tclsh)
   • ./richards_FBx.tcl (tclsh)
   • ./overland_flux.tcl (tclsh)
   • ./LW_var_dz_spinup.tcl (tclsh)
   • ./LW_var_dz.tcl (tclsh)
   • ./timing/clm-reuse/clm-reuse.tcl (tclsh)

Thoughts:

1. Handle multiple outputs. Report a list of one or more reports.
2. intercept and no-op the following pfcommands; OR ALTERNATIVE ***
   • pftestFile -> just output true
   • pfload
   • pfsave...
   • pfmdiff
3. *** Alternatively, could comment out anything not a pfset or set command...
4. Capture runname and put in configuration
5. add flag to set whether or not I care about exit code (--ignore-script-exit-code)

[ian-bertolacci]

added #22 based on thoughts from this issue