Y. GEOS‐JEDI
The intent of this Wiki is to provide users with a basic understanding of how JEDI has been brought into the GEOS ADAS Workflow and to allow for testing JEDI in the context of GEOS ADAS. The presentation here implements Phase I in the GEOS ADAS transition from GSI to JEDI. In this initial phase, the enhanced GEOS ADAS Workflow intentionally runs both GSI and JEDI in the same cycle to allow and facilitate a scientifically robust comparison of the two variational analysis systems.
In GEOS ADAS Workflow, a simple driver script now controls how JEDI is invoked and how its results can be used to replace GSI results and drive the underlying GEOS model integration. A depiction of the driver script controlling only the JEDI analysis within GEOS ADAS is shown in the figure below.
What follows provides a set of instructions for running JEDI within the GEOSadas. Users should be aware that the instructions here will be changing frequently, as minor inconveniences are expected to be uncovered the more people exercise the GEOS-JEDI capability.
It is also important to remark from the outset that the ultimate goal is to have SWELL drive the GEOS ADAS cycle. Although this is a future Phase in the path of transitioning from GSI to JEDI. Changes made to any JEDI-related configurations in GEOS-JEDI must be brought into SWELL. Users are expected to be responsible for carrying out integrations in SWELL or providing all needed information for that to take place.
Users should be aware of the following nuances in this initial version of GEOS-JEDI.
-
This version is provided simply for testing purposes; this is a not a final configuration suitable to near-real-time applications such as GEOS-FP, GEOS-IT and reanalysis. However, the configuration provided here is expected to evolve rapidly and eventually support applications such as GEOS-FP and GEOS-IT to run --- no reanalysis configuration is planned for GEOS-JEDI.
-
A second release (see table below) adds on to the initial JEDI settings for 3D-VAR and 3D-FGAT (default), including hybrid 3D and 4D configurations (see below).
-
An experiment input file, j3dfgat.input (handled by runjob), sets up a rather simplified configuration of GSI to have as much consistency as possible between the two analyses set to run concurrently. Specifically, the following is revised in the GSI configuration:
- Single outer-loop
- NetCDF-diags on
- No TLNMC
- No Dry Mass Constraint
- No NST analysis
- No flow dependence in B
- Q option 1 for moisture variable
- 100 inner iterations; 100 orthogonalization steps
- write out true value of cost to log file
- Off obs-systems: ATMS-N21, OMPS-NL (N21 & NPP), TCVitals
- No Ta2Tb conversion
The Perl program **setup_aanajedi.pl ** provides the mechanism to setup JEDI within GEOS. This program lives in the bin directory of a GEOS ADAS build. To get things going follow the steps below:
-
To setup a GEOS-JEDI experiment define the two env variables below:
setenv GEOSJEDI 1
-
Then use runjob -l j3dfgat.input to setup the ADAS experiment.
The difference in the settings of this ADAS experiment is that now, the setup creates a subdirectory in FVHOME/run named jedi, which has some controls and configurations associated with running JEDI in GEOS. The presence of this jedi directory in the experiment home, triggers JEDI --- if you move this directory out of the way, the ADAS will run normally, without knowing of the existence of JEDI.
Note that GEOS-JEDI runs in batch segments that cover only 6-hours of DA (just GEOS's hybrid 4DEnVar runs).
Note also that by construction, for now, JEDI does not run in the very first cycle of the ADAS experiment; JEDI only runs from the second cycle onwards.
When running within the GEOS ADAS Workflow, the configuration of JEDI is controlled in the file JEDIanaConfig.csh found under FVHOME/run/jedi. In this, the following are flags to know about; you should not change them unless you know better.
There are only a few env variables the typical user should be concerned with when running GEOS-JEDI:
OFFLINE_IODA_DIR
This sets the path to convert_ncdiags, SWELL-geneated, IODA files.
present default: /discover/nobackup/projects/gmao/dadev/rtodling/archive/541/Milan/x0050/ioda/
The default covers a short period of x0050, from roughly start of October 2023 to the middle of the month. See below for generating data for other periods.
JEDI_FEEDBACK_VARBC
0 - does not cycle var BC files (default; they are picked up from a previous run or the swell ncdiag converter
1 - cycle varBC; output generated by JEDI from a given cycle serves as input to the following cycle - JEDI cannot yet handle writing out updated variances for aircraft bias correction; for safety for now, the default is not to cycle varBC.
Note: a BOOTSTRAP mechanism is used in the first couple of cycles of a run in handling varBC.
JEDI_IAU_OVERWRITE
0 - generates an agcm_import)_rst file but does not use it in the cycle (the GCM does not see this file; default)
1 - moves the GSI-based agcm_import_rst out of the way and make the JEDI-generated file (above) visible to the GCM
JEDI_OBS_OPT
1 - point to x-exp-like set (data in tar-balls; data from an existing experiment; see jedi_ioda.acq & jedi_vbc.acq under FVHOME/run/jedi/Config)
2 - point to existing set of ncdiag-ioda-converted set generated by SWELL, in a location of the form: PATH/swell/.../DATE/geos_atmosphere; default) - see also env OFFLINE_IODA_DIR --- notice that, in the default configuration at present, one of the env variables set before running runjob already establishes this location.
3- generate IODA files on the fly from underlying GSI nc-diag files (To-be-implemented)
OFFLINE_IODA_DIR
Path of IODA SWELL-generated IODA files (see during running setup_aanajedi.pl)
JEDI_ROOT
This is a pointer to the JEDI build (version) the release of GEOS-JEDI has been tested against. Typically, there is not need to change this. However, users working to implement and test a new feature in JEDI will have their own build and should use this variable to point to that instead.
What else is found in FVHOME/run/jedi? Aside from the configuration file just discussed, the next most important thing found in this is the actual yaml file that drives the variational procedure, namely, FVHOME/run/jedi/Config/geosvar.yaml. This file is basically a conveniently templated form of the yaml file produced buy SWELL when setting up a particular variational configuration. Users might find it necessary, when testing a modification or new feature, to edit this file and changed it as needed. But please remember, after your testing is completed, it is expected that you work these changes not only in the GEOSadas fixture (its original location in the source code), but also to bring the corresponding change into SWELL.
Everything else in FVHOME/run/jedi can typically be ignored by users. They are mainly job (batch) control files responsible for the acquisition of various components needed to run JEDI, e.g., background files, ioda files, varBC files, and a couple other related to the setting of an ensemble-base ADAS, but not yet ported to FVHOME (these will come later when hyb4denvar becomes available -- presently under testing).
In all likelihood you will be interested in running GEOS-JEDI for a period we have no IODA observations readily available. The instructions here provide brief guidance for how you can go about getting the data you need. For this you will need to start from the output of GSI for an existing GEOS ADAS (x-)experiment). Only experiments ran with v5.42.0 onwards will have all necessary entries in the ncdiag files and corresponding varBC files -- Presently, a small set of IODA files is provided covering part of October 2023 (see default for OFFLINE_IODA_DIR env var above). The most recent experiment at the time of this release, has ncdiags being stored from mid Dec 2023 onwards, this will be converted into IODA some time soon. If you need other periods you will want to do the following to generate suitable IODA files:
-
Setup a regular (3D-Var) experiment with GEOS ADAS (no JEDI); use GEOSadas v5.42.1 or higher.
-
Use the standalone procedure to run GSI without needing to cycle; set gsi.rc.tmpl to write out ncdiag files (no need to have geovals written out). Run the standalone for as many cycles as desired.
-
Set up a SWELL convert_ncdiags suite. Edit the corresponding experiment.yaml as to point to the location of your ncdiag and varBC files. Run it for as many cycles as you have ncdiags for.
-
Use the OFFLINE_IODA_DIR env var in GEOS-JEDI to point to your database of IODA files.
An upcoming follow up release will have the option to get IODA files generated directly from the GSI ncdiag output; in store is also getting settings for JEDI to run directly from the bufr files used as inputs to GSI (though this will take a little more doing).
Let's consider the situation when a user submitted the GEOS ADAS job associated with the x51jedi experiment in consideration here. Let's assume further that the crash happens when GEOS is trying to run the JEDI component. As typical, a morgue directory will result from the crash. GEOS provides the ability for a user to run the JEDI component from the crashed pieces in the morgue. For that, the first thing to do is to move the morgue directory back to its original fvwork.#### name (cat FVHOME/.FVWORK to find out the original FVWORK name).
Assuming the user has something else to try to correct for the failure - and has made the correction where needed - the JEDI part can be re-submitted directly from where the crash landed by running:
A. Get an interactive batch queue large enough to run the JEDI configuration case you have
B. then execute ut_jedi.j at command line, i.e., type ./ut_jedi.j
Notice that each main step of the JEDI run drops a hidden file, .DONE_jedi_something in the fvwork directory. Depending where your fix is you might need to wipe out a .DONE associated with the step in question (in case it exists, or in case you want to job to go over that step again). For example, if the error was something related to the set of the configuration files (or input data files), you need to make sure the hidden file associated with the set step is not in the way, that is, the file of name .DONE_jedi_set.csh.yyyymmddhh needs to be removed.
Here is a snapshot of the output available at the end of a JEDI analysis:
List of Output files created by running JEDI within GEOS
| Output file | Description |
|---|---|
| EXPID.ana_jo.log.20231006_00z.txt | Summary of GSI Jo-related quantities |
| EXPID.jedi_var.log.20231005_21z.txt | Log out of JEDI variational run |
| EXPID.jedi_hofx.20231005_21z.tar | Collection of all hofx-ioda files from var run |
| EXPID.jedi_vbc.20231005_21z.tar | Collection of all varBC files out of var run |
| EXPID.jedi_set.log.20231005_21z.txt | Log file from the setup set of JEDI |
| EXPID.jedi_run.log.20231005_21z.txt | Log file from all procedures around JEDI var run |
| EXPID.jedi_agcmrst.20231005_21z.tar | All agcm_import files derived from JEDI analysis (one file in 3d case; 7 in 4d) |
| EXPID.jedi_ana.20231005_21z.tar | All analysis files output by JEDI var and possibly converted to lat-lon |
| EXPID.jedi_inc.20231005_21z.tar | All increment files output by JEDI var and possibly converted to lat-lon |
| EXPID.jedi_mkiau.log.20231005_21z.txt | Log from procedures controlling generation of agcm_import file(s) |
| EXPID.jedi_drv.log.20231005_21z.txt | Log from driver script controlling all procedure associated with running JEDI in GEOS |
The JEDI output is archived in the location where the GEOS ADAS experiment is archived in a subdirectory named jedi, that is, in addition to the typical, ana, chem, diag, etc, obs, and rs directories, there is now a directory named jedi with similarly named subdirectories of its own named: ana, etc, obs, rs; whose description should be self-evident.
By default the cycles provide the model with IAU input files (agcm_import) generated from the GSI analyses; similarly, forecasts are run by pulling the same IAU files to complete the forecast runs (controlled in the FVHOME/fcst).
As we have seem above, the env variable JEDI_IAU_OVERWRITE allows the cycle to use the IAU files created from the JEDI analysis, bypassing the GSI ones; in this case, for consistency, the user should want to run forecasts from the same JEDI-related IAU files. This, however, must be set by hand - for now.
The bringing of IAU files into the forecast experiment is controlled by the file rst4fcst.acq found under FVHOME/fcst. Specifically, a line such as the one below controls the fetching of IAU into the forecast env:
/discover/nobackup/projects/gmao/dadev/rtodling/archive/JEDI/x51//j3dfgat/rs/Y%y4/M%m2/j3dfgat.agcm_import_rst.%y4%m2%d2_%h2%n2z.nc4
In case of wanting to forecast from the JEDI-related IAU files you need to:
-
Unravel the tar balls jedi_agcmrst stored in the archive under EXPID/jedi/rs/Yyyyy/Mmm - so the agcm_import files will sit lose in this archive directory.
-
And edit the rst4fcst.acq file and insert the jedi name within the path in the line for agcm_import_rst, such as:
/discover/nobackup/projects/gmao/dadev/rtodling/archive/JEDI/x51//j3dfgat/jedi/rs/Y%y4/M%m2/j3dfgat.agcm_import_rst.%y4%m2%d2_%h2%n2z.nc4
- You can then follow the usual instructions to launch forecasts in GEOS ADAS.
Tip: the ultimate consistency check (to be done by the user) is to verify that the first prog.eta file created for any forecast is identical to the asm.eta file available from the cycle for that same date and time.
Note: the steps above are not necessary for running forecasts when GSI is not running (see Bypassing GSI below).
The second release of GEOS-JEDI makes available a number of flavors of variational analysis:
-
3D-Var (3dvar): traditional 3D scheme requiring only a single snapshot of the background, and producing an analysis in the center time of the variational window.
-
3D-FGAT (3dfgat): more commonly employer variational settings requiring a high-frequency background to run the 3D analysis; as in a 3D scheme, results are written out in the middle of the variational window; the default settings require 3-hourly backgrounds just like in the GEOS-GSI traditional 3D-Var settings.
-
Hybrid 4DEn-Var (hyb4denvar): Similar to the current GEOS default settings: hourly (lat-lon) ensemble makes up part of the hybrid background error covariance in an ensemble-based 4D scheme that produces updates (increments) hourly within the variational window. The settings in this option use GSIBEC to handle the whole hybrid background error covariance, thus being "identical" to how GSI works in GEOS.
-
Hybrid cubed-based 3DEn-Var (hyb3dcenvar): This is a hybrid ensemble-variational procedure that relies on GSIBEC for the climatological term and on BUMP for the ensemble term. As such, the ensemble is required to be on the cubed grid. As in a 3D procedure, results are written out only in the middle of the window.
-
hybrid cubed-based 4DEn-Var (hyb4dcenvar): This is similar to the hyb3dcenvar, but a 4D procedures with hourly background, ensemble members, and corresponding results.
As pointed out above, for the time being, the default GEOS-JEDI sets up a 3D-FGAT procedure. Advanced users can try some of the other procedures above by simply setting the following env variable before setting up the experiment, e.g.: sentenv JEDI_ATMVAR_SCHEME hyb4denvar
Warnings:
A. Users should be aware that these schemes are all under intense testing; be particularly cautious about the BUMP-based schemes since not true attempt has been made to come up with adequate localization scales, and weighting between climatological and ensemble terms.
B. Although the yaml files are available for these various options, the setup configurations are not available for all resolutions (particularly the BUMP-based schemes which are only available for c90 for now); the setup will halt if you choose something that is not available.
Description coming soon ...
Description coming soon ...
It might be desirable to run GEOS-JEDI without having to run GSI. Although users are largely discouraged at this stage to run this way since GEOS-JEDI is not yet fully vetted, this option is available for it is forward looking and along the lines of ridding GEOS of its dependence on GSI sometime soon.
The only thing to be done to bypass GSI is to set an env variable before running runjob over the input file, as such:
setenv SKIPGSI 1
Note: When this flag is set to one, the env variable JEDI_IAU_OVERWRITE is also automatically set to one, i.e., the JEDI analysis is used to create the IAU terms to force the model.
WARNING: In practice, GSI is only turn off after a very first cycle with the DAS. An initial cycle with GSI is still need for generating JEDI background input data - users with much intimacy with how GEOS-JEDI Workflow works should be able to go around this set if really needed.
This is a summary of a couple of things that should be adjusted by hand before the user starts cycling with GEOS-JEDI. These changes will make the run consistent in various ways.
A. Edit the AGCM.rc.tmpl (and corresponding BOOTSTRAP files) in FVHOME/run and FVHOME/fcst and set ANALYZE_TS: 0. This is to be consistent with the turning off of the NST analysis in GSI (and the fact that variational JEDI does not yet have an NST analysis component. Notice however, this really truly is not a necessary change: when NST not on, the increment of skin temperature over the ocean is zero, and the model state never changes as a consequence, regardless where the model uses the delta-Tskin coming from the analysis or not.
B. If cycling the ADAS with JEDI analyses, remember to edit the file FVHOME/fcst/rst4fcst.acq and adjust the location of the agcm_import tar ball to point to where the JEDI ones are stored.
C. Remember, before setting up your experiment using runjob you should at a minimum do:
setenv GEOSJEDI 1
setenv ARCHIVE /some_location_of_your_own/
if providing your own IODA location you can also set:
setenv OFFLINE_IODA_DIR /your_own_ioda_location/
if bypassing GSI
setenv SKIPGSI 1
-
Remember you are using a not-yet validated version of GEOS-JEDI ADAS.
-
Many people have contributed to making GEOS-JEDI a potentially viable alternative to GEOS-GSI; this is relevant when it comes to properly crediting those who worked to make it so.
-
Consult with senior GMAO staff if you are considering submitting to a peer reviewed publication work you have done using the GEOS-JEDI framework.
List of JEDI-enabled GEOS ADAS Versions
| Dev Phase | Release | JEDI Pin | Brief Summary | Known Issues |
|---|---|---|---|---|
| I | v5.42.3 | 02062025 | Initial release; requires SWELL-generated IODA obs done prior to ADAS run. | (i) JEDI analysis is found to have super convergence at times (likely an adjoint failure issue, to be investigated); (ii) JEDI does not yet cycle aircraft varBC variances; (iii) running without GSI (a basis for comparison) is highly discouraged at this stage. |
| I | v5.42.4 | 02062025 | Introduces hybrid 4dEnVar and a couple other hybrid options. | Users must use caution when employing alternative schemes. An oversight: mkiau.rc.env is wire to 360; please adjust by hand in your run/jedi/Config directory if running model at different resolution- will be fixed in upcoming upgrade. |
| I | v5.42.11 | 06182025 | JEDI I/O var name handling; 3d-pressure now an analysis variable (fixes use of RO); dual-resolution hybrid B error. |