Preparing for new version upgrade #13
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ing yml file. The perturbation follows the original HYCOM approach.
…perturb function;
…ction needs attention
… time units instead of giving it explicitly.
Adding random perturbation to forcing - an initial implementation
…g job arrays for the forecast phase
…ll perturbation variance for cycled testing;updating model configuration files for each cycle; put generated restart file to next cycle;
saved the old copy of config.py moved perturb/randoim_perturb.py to ./perturb.py temporarily, will try to merge with utils/random_perturb.py later output_dir is no longer needed, makedir function using os.makedirs but catching FileExistsError from race conditions, adding nens to the job_opts for batch mode generate_init_ensemble.py is removed and the function is merged into preprocess.py as time==time_start the restart_dir is where the initial ensemble will be obtained run_exp.py renamed into run_expt.py and made more clean
replacing the job_submit_cmd approach in the config file # Changes not staged for commit:
implemented OAR/SLURM submitters updated nextsim.dg.model.run to use the submitter
previously I have the perturb settings under the model_def section, defined for each model; but now the perturb section is made stand alone. The reason doing so is that I can now run perturbation scheme involving multiple model components. I'm keeping the model-specific perturb settings so that the code is still compatible. Note that model.preprocess method will run the model-specific perturb settings while the scripts/perturb.py will run the standalone perturb settings. Moving the perturb.py inside nextsim.dg module as well prepare_forcing and prepare_restart are included in the preprocess step
combined prepare_restart and prepare_forcing into preprocess since nextsim.dg config_file is holding the files and perturb options, they are initialized directly in model class, instead of passing through kwargs everytime.
-initialize grid inside __init__, can we define model grid parameters in default.yml? -include restart forcing modules read_var and write_var in model class -in preprocess, there is a bit issue with copying previous cycle restart files to the current cycle. In config file files:restart:format, there seems to be no ensemble member index?
since only ensemble forecast gets to use this job array functionality, moving it out of job_submit section, so that other scripts like assimilate.py and perturb.py will not get accidently assigned use_job_array=True. Note; if not specified, use_job_array is defaulted to False
__getattr__ is not safe if config_dict failed to initialize; runtime reassign of attr will not work, unless a corresponding __setattr__ is written to change values in config_dict
don't raise attribute error if __getattr__ doesn't find key, becasue it will resort to __getattribute__
use dateutil.parser to allow flexible time string defined in yaml file or through argparse added test to ensure time parsing works
run a single step is a functionality only for offline_filter, so moving the code into offline_filter __call__ function to make NEDAS.__main__ generally just call scheme(c)
fix some initialization steps that are machine-specific and would fail the test. Moving them into runtime funcs
refactor the submodules in assim_tools to be extensible by registry and get_* function approach; not fully tested, expect a lot of bugs
add cs2smos and osisaf.ice_conc synthetic obs type fix datetime tzinfo in forcing add model grid land mask
rm etkf eakf, use ETKF EAKF convension
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Develop branch has collected several major upgrades
With further tests, the new version 1.1 will be released soon