[WIP] JSON based parameters

[rgknox]

Description:

This set of changes migrates FATES away from using a netcdf based parameter file. The file format is changed to JSON (or XML depending on discussion). A fortran JSON parsing utility is provided (not external module), which utilizes a character-by-character file scan and indexing functions. Python utilities are planned to be converted from netcdf to json based as well.

A new utility is available, that converts pre-existing netcdf parameter datasets to the json (or xml) format.

Synchronized with: CTSM 3570

Collaborators:

@glemieux

To Do:

• Write python conversion utility that transforms pre-existing cdl files to json/xml
• Generate new default file
• Write Fortran JSON/XML parser (JSON parser feature complete)
• Write code that transfers new parameter data-structure into FATES primitive parameter memory
• Refactor: tools/modify_fates_paramfile.py
  • Include new commands appended to history
  • Include a change by name feature
• Refactor: tools/FatesPFTIndexSwapper.py
  • Include a change by name feature
• Refactor: tools/ncvarsort.py
  • Change name to sort_parameters.py
• Remove temporary file datatypes.txt
• Refactor: tools/BatchPatchParams.py
• Refactor: tools/UpdateParamAPI.py
• HLM PR:
  • Update default file path to be fates/parameters/fates_parameter_default.* instead of the binary
  • Update the HLM fortran code that calls FATES parameter reads
  • Update scripting in cime_config/testdefs/testmods_dirs/clm
  • Update at least 1 test configuration to create/use a modified version of the new file type in CASEDIR or scratch

Expectation of Answer Changes:

This should not have any meaningful changes, but reading in the parameters differently could have very small, inconsequential changes to answers (barely outside of roundoff diffs).

Checklist

If this is your first time contributing, please read the CONTRIBUTING document.

All checklist items must be checked to enable merging this pull request:

Contributor

• The in-code documentation has been updated with descriptive comments
• The documentation has been assessed to determine if updates are necessary
• Python manipulation tools completed

Integrator

• FATES PASS/FAIL regression tests were run
• Evaluation of test results for answer changes was performed and results provided
• FATES-CLM6 Code Freeze: satellite phenology regression tests are b4b

If satellite phenology regressions are not b4b, please hold merge and notify the FATES development team.

Documentation

• Technical Note update:
• User's Guide update:

Test Results:

CTSM (or) E3SM (specify which) test hash-tag:

CTSM (or) E3SM (specify which) baseline hash-tag:

FATES baseline hash-tag:

Test Output:

[rgknox]

Notes from the FATES model meeting:

Presentation

Poll results on JSON vs XML preference

Feature requests:

• Allow "comment" tags (this should automatically work).
• Add a "datatype" (ie string, integer or float) to the variable descriptors.

Provide additional support to migrate external workflows that interface with the parameter file, such as

• The PPE experiment (@adrifoster)
• OLMT (@dmricciuto)

Most people seemed to favor JSON, but we played the devils advocate and discussed arguments for going with XML. Familiarity XML and that it is used in various other workflows like CIME and GCAM seemed to be a point for it.

People also expressed hope that the new method would help mitigate complications with keeping personal parameter files up-to-date with format changes (like adding modifying or deleting parameters).

[rgknox]

I went through each parameter and jotted down their datatypes in this temporary text file:

tools/datatypes.txt

1 : float
2 : integer
3 : string

I will use this to generate the the "datatype" attribute in the first json file.