consolidate gfs data to gfs_aero_c12/bkg for 20201214.21z #107

HamidehGMAO · 2025-02-07T20:09:20Z

Description

In order to run coupled oasim with aerosol optical thickness derived from gfs, we need to use gfs_aerosol bkg data, but we need bkg data for surface variables like surface wind speed and surface pressure as well. so we need a set of bkg data that include both aerosol data and other gfs bkg data. so here I just consolidated the gfs bkg data to the gfs_aero_c12/bkg to be able to run coupled oasim using gfs_aero_c12/bkg data

Issue(s) addressed

Resolves #105

Impact

this needs to be merged before
https://github.com/JCSDA-internal/coupling/pull/53

Checklist

[ *] I have performed a self-review of my own code
I have made corresponding changes to the documentation
[ *] I have run the unit tests before creating the PR

fmahebert · 2025-02-19T05:01:21Z

This looks reasonable to me, but I hope @cmgas or @mer-a-o or @travissluka can also take a look!

cmgas

This is good with me!

jeromebarre · 2025-02-21T16:01:48Z

I understand the problem but this is still a duplication of the data. I think we all agree that there is no reason to have a separate folder for the atmosphere with or without atmospheric composition. I advocate having a consolidated folder with only one set of atmosphere backgrounds for GFS tests. With those new backgrounds we will have to update all GFS tracers related tests... a lot of unrewarding work... would EMC willing to help on this? @CoryMartin-NOAA @danholdaway Thanks

danholdaway · 2025-02-21T16:15:23Z

I'm not sure I understand what you need help with?

jeromebarre · 2025-02-21T16:52:42Z

Another solution could be to add the missing variables needed for OASIM to the already-existing GFS aero files in the GFS_aero dir, this way this would limit the amount of files being duplicated and not require updating all the aerosol-related tests.

travissluka · 2025-02-21T17:02:21Z

~~@HamidehGMAO could you do a symlink on the files instead of copying them?~~
That way we're not duplicating data and we can move forward with the PR, I realize its not the cleanest way, but there's no harm in a symlink and everyone else can figure out the "clean way" to do it at a later date.

Nevermind, ignore me

HamidehGMAO · 2025-02-21T17:17:04Z

just to clarify the tracer files in gfs and gfs_aero have different variables, gfs_c12/bkg> ncdump -h 20201214.230000.fv_tracer.res.tile3.nc
netcdf \20201214.230000.fv_tracer.res.tile3 {
dimensions:
xaxis_1 = 12 ;
yaxis_1 = 12 ;
zaxis_1 = 127 ;
Time = UNLIMITED ; // (1 currently)
variables:
double xaxis_1(xaxis_1) ;
xaxis_1:long_name = "xaxis_1" ;
xaxis_1:units = "none" ;
xaxis_1:cartesian_axis = "X" ;
double yaxis_1(yaxis_1) ;
yaxis_1:long_name = "yaxis_1" ;
yaxis_1:units = "none" ;
yaxis_1:cartesian_axis = "Y" ;
double zaxis_1(zaxis_1) ;
zaxis_1:long_name = "zaxis_1" ;
zaxis_1:units = "none" ;
zaxis_1:cartesian_axis = "Z" ;
double Time(Time) ;
Time:long_name = "Time" ;
Time:units = "time level" ;
Time:cartesian_axis = "T" ;
double sphum(Time, zaxis_1, yaxis_1, xaxis_1) ;
sphum:long_name = "specific_humidity" ;
sphum:units = "kgkg-1" ;
sphum:checksum = "D24A7D0E2FE7C942" ;
double ice_wat(Time, zaxis_1, yaxis_1, xaxis_1) ;
ice_wat:long_name = "cloud_liquid_ice" ;
ice_wat:units = "kgkg-1" ;
ice_wat:checksum = "AF3CD602DCA7B362" ;
double liq_wat(Time, zaxis_1, yaxis_1, xaxis_1) ;
liq_wat:long_name = "cloud_liquid_water" ;
liq_wat:units = "kgkg-1" ;
liq_wat:checksum = "D571C09BE0413390" ;
double o3mr(Time, zaxis_1, yaxis_1, xaxis_1) ;
o3mr:long_name = "ozone_mass_mixing_ratio" ;
o3mr:units = "kgkg-1" ;
o3mr:checksum = " A83291AF758B077" ;

this is aerosol file

gfs_aero_c12/bkg> ncdump -h 20201214.210000.fv_tracer.res.tile1.nc
netcdf \20201214.210000.fv_tracer.res.tile1 {
dimensions:
xaxis_1 = 12 ;
yaxis_1 = 12 ;
zaxis_1 = 127 ;
Time = UNLIMITED ; // (1 currently)
variables:
double xaxis_1(xaxis_1) ;
xaxis_1:long_name = "xaxis_1" ;
xaxis_1:units = "none" ;
xaxis_1:cartesian_axis = "X" ;
double yaxis_1(yaxis_1) ;
yaxis_1:long_name = "yaxis_1" ;
yaxis_1:units = "none" ;
yaxis_1:cartesian_axis = "Y" ;
double zaxis_1(zaxis_1) ;
zaxis_1:long_name = "zaxis_1" ;
zaxis_1:units = "none" ;
zaxis_1:cartesian_axis = "Z" ;
double Time(Time) ;
Time:long_name = "Time" ;
Time:units = "time level" ;
Time:cartesian_axis = "T" ;
double sphum(Time, zaxis_1, yaxis_1, xaxis_1) ;
sphum:long_name = "specific_humidity" ;
sphum:units = "kgkg-1" ;
sphum:checksum = "F0D44616EEA751E3" ;
double so4(Time, zaxis_1, yaxis_1, xaxis_1) ;
so4:long_name = "mass_fraction_of_sulfate_in_air" ;
so4:units = "ugkg-1" ;
so4:checksum = "83C65993578E73C8" ;
double bc1(Time, zaxis_1, yaxis_1, xaxis_1) ;
bc1:long_name = "mass_fraction_of_hydrophobic_black_carbon_in_air" ;
bc1:units = "ugkg-1" ;
bc1:checksum = "2F98E71EC3760A1F" ;
double bc2(Time, zaxis_1, yaxis_1, xaxis_1) ;
bc2:long_name = "mass_fraction_of_hydrophilic_black_carbon_in_air" ;
bc2:units = "ugkg-1" ;
bc2:checksum = "32B3B5286FF1C84E" ;
double oc1(Time, zaxis_1, yaxis_1, xaxis_1) ;
oc1:long_name = "mass_fraction_of_hydrophobic_organic_carbon_in_air" ;
oc1:units = "ugkg-1" ;
oc1:checksum = " 3D4E143E9D32BFE" ;
double oc2(Time, zaxis_1, yaxis_1, xaxis_1) ;
oc2:long_name = "mass_fraction_of_hydrophilic_organic_carbon_in_air" ;
oc2:units = "ugkg-1" ;
oc2:checksum = "1B7F487A30ED5839" ;
double dust1(Time, zaxis_1, yaxis_1, xaxis_1) ;
dust1:long_name = "mass_fraction_of_dust001_in_air" ;
dust1:units = "ugkg-1" ;
dust1:checksum = "12B0A4352DF4EE1D" ;
double dust2(Time, zaxis_1, yaxis_1, xaxis_1) ;
dust2:long_name = "mass_fraction_of_dust002_in_air" ;
dust2:units = "ugkg-1" ;
dust2:checksum = " 4C6F363837E6B67" ;
double dust3(Time, zaxis_1, yaxis_1, xaxis_1) ;
dust3:long_name = "mass_fraction_of_dust003_in_air" ;
dust3:units = "ugkg-1" ;
dust3:checksum = "C42D9E6CC01B341E" ;
double dust4(Time, zaxis_1, yaxis_1, xaxis_1) ;
dust4:long_name = "mass_fraction_of_dust004_in_air" ;
dust4:units = "ugkg-1" ;
dust4:checksum = "5DDC57DF718AC432" ;
double dust5(Time, zaxis_1, yaxis_1, xaxis_1) ;
dust5:long_name = "mass_fraction_of_dust005_in_air" ;
dust5:units = "ugkg-1" ;
dust5:checksum = "C002B2A3E75741A4" ;
double seas1(Time, zaxis_1, yaxis_1, xaxis_1) ;
seas1:long_name = "mass_fraction_of_sea_salt001_in_air" ;
seas1:units = "ugkg-1" ;
seas1:checksum = "AF337258E3F330AA" ;
double seas2(Time, zaxis_1, yaxis_1, xaxis_1) ;
seas2:long_name = "mass_fraction_of_sea_salt002_in_air" ;
seas2:units = "ugkg-1" ;
seas2:checksum = "BD6C3B11232E1BA7" ;
double seas3(Time, zaxis_1, yaxis_1, xaxis_1) ;
seas3:long_name = "mass_fraction_of_sea_salt003_in_air" ;
seas3:units = "ugkg-1" ;
seas3:checksum = "85A1E462FC3943C0" ;
double seas4(Time, zaxis_1, yaxis_1, xaxis_1) ;
seas4:long_name = "mass_fraction_of_sea_salt004_in_air" ;
seas4:units = "ugkg-1" ;
seas4:checksum = "E05CF1A2E7C0C3C7" ;
double seas5(Time, zaxis_1, yaxis_1, xaxis_1) ;
seas5:long_name = "mass_fraction_of_sea_salt005_in_air" ;
seas5:units = "ugkg-1" ;
seas5:checksum = "B307F64F0A1A21B4" ;
double no3an1(Time, zaxis_1, yaxis_1, xaxis_1) ;
no3an1:long_name = "mass_fraction_of_nitrate001_in_air" ;
no3an1:units = "ugkg-1" ;
no3an1:checksum = "5E1948C96427198C" ;
double no3an2(Time, zaxis_1, yaxis_1, xaxis_1) ;
no3an2:long_name = "mass_fraction_of_nitrate002_in_air" ;
no3an2:units = "ugkg-1" ;
no3an2:checksum = "50C11E9CBCDE134E" ;
double no3an3(Time, zaxis_1, yaxis_1, xaxis_1) ;
no3an3:long_name = "mass_fraction_of_nitrate003_in_air" ;
no3an3:units = "ugkg-1" ;
no3an3:checksum = "CD7E8071764F8B11" ;

That's why they need to be consolidated.

HamidehGMAO · 2025-02-21T17:27:59Z

Another solution could be to add the missing variables needed for OASIM to the already-existing GFS aero files in the GFS_aero dir, this way this would limit the amount of files being duplicated and not require updating all the aerosol-related tests.

@jeromebarre no, this won't limit the duplication, as anyway both tracer files needs to be consolidated and also we would need to add srf_wnd.res. and sfc_data data to aerosol bkg data which they do not belong there. So, I think this works best for now, but later on there could be a PR to consolidate aerosol data to gfs bkg data and remove the aerosol bkg data and update the corresponding tests.

mer-a-o · 2025-02-21T18:41:47Z

just to clarify the tracer files in gfs and gfs_aero have different variables, gfs_c12/bkg> ncdump -h 20201214.230000.fv_tracer.res.tile3.nc netcdf \20201214.230000.fv_tracer.res.tile3 { dimensions: xaxis_1 = 12 ; yaxis_1 = 12 ; zaxis_1 = 127 ; Time = UNLIMITED ; // (1 currently) variables: double xaxis_1(xaxis_1) ; xaxis_1:long_name = "xaxis_1" ; xaxis_1:units = "none" ; xaxis_1:cartesian_axis = "X" ; double yaxis_1(yaxis_1) ; yaxis_1:long_name = "yaxis_1" ; yaxis_1:units = "none" ; yaxis_1:cartesian_axis = "Y" ; double zaxis_1(zaxis_1) ; zaxis_1:long_name = "zaxis_1" ; zaxis_1:units = "none" ; zaxis_1:cartesian_axis = "Z" ; double Time(Time) ; Time:long_name = "Time" ; Time:units = "time level" ; Time:cartesian_axis = "T" ; double sphum(Time, zaxis_1, yaxis_1, xaxis_1) ; sphum:long_name = "specific_humidity" ; sphum:units = "kgkg-1" ; sphum:checksum = "D24A7D0E2FE7C942" ; double ice_wat(Time, zaxis_1, yaxis_1, xaxis_1) ; ice_wat:long_name = "cloud_liquid_ice" ; ice_wat:units = "kgkg-1" ; ice_wat:checksum = "AF3CD602DCA7B362" ; double liq_wat(Time, zaxis_1, yaxis_1, xaxis_1) ; liq_wat:long_name = "cloud_liquid_water" ; liq_wat:units = "kgkg-1" ; liq_wat:checksum = "D571C09BE0413390" ; double o3mr(Time, zaxis_1, yaxis_1, xaxis_1) ; o3mr:long_name = "ozone_mass_mixing_ratio" ; o3mr:units = "kgkg-1" ; o3mr:checksum = " A83291AF758B077" ;

this is aerosol file

gfs_aero_c12/bkg> ncdump -h 20201214.210000.fv_tracer.res.tile1.nc netcdf \20201214.210000.fv_tracer.res.tile1 { dimensions: xaxis_1 = 12 ; yaxis_1 = 12 ; zaxis_1 = 127 ; Time = UNLIMITED ; // (1 currently) variables: double xaxis_1(xaxis_1) ; xaxis_1:long_name = "xaxis_1" ; xaxis_1:units = "none" ; xaxis_1:cartesian_axis = "X" ; double yaxis_1(yaxis_1) ; yaxis_1:long_name = "yaxis_1" ; yaxis_1:units = "none" ; yaxis_1:cartesian_axis = "Y" ; double zaxis_1(zaxis_1) ; zaxis_1:long_name = "zaxis_1" ; zaxis_1:units = "none" ; zaxis_1:cartesian_axis = "Z" ; double Time(Time) ; Time:long_name = "Time" ; Time:units = "time level" ; Time:cartesian_axis = "T" ; double sphum(Time, zaxis_1, yaxis_1, xaxis_1) ; sphum:long_name = "specific_humidity" ; sphum:units = "kgkg-1" ; sphum:checksum = "F0D44616EEA751E3" ; double so4(Time, zaxis_1, yaxis_1, xaxis_1) ; so4:long_name = "mass_fraction_of_sulfate_in_air" ; so4:units = "ugkg-1" ; so4:checksum = "83C65993578E73C8" ; double bc1(Time, zaxis_1, yaxis_1, xaxis_1) ; bc1:long_name = "mass_fraction_of_hydrophobic_black_carbon_in_air" ; bc1:units = "ugkg-1" ; bc1:checksum = "2F98E71EC3760A1F" ; double bc2(Time, zaxis_1, yaxis_1, xaxis_1) ; bc2:long_name = "mass_fraction_of_hydrophilic_black_carbon_in_air" ; bc2:units = "ugkg-1" ; bc2:checksum = "32B3B5286FF1C84E" ; double oc1(Time, zaxis_1, yaxis_1, xaxis_1) ; oc1:long_name = "mass_fraction_of_hydrophobic_organic_carbon_in_air" ; oc1:units = "ugkg-1" ; oc1:checksum = " 3D4E143E9D32BFE" ; double oc2(Time, zaxis_1, yaxis_1, xaxis_1) ; oc2:long_name = "mass_fraction_of_hydrophilic_organic_carbon_in_air" ; oc2:units = "ugkg-1" ; oc2:checksum = "1B7F487A30ED5839" ; double dust1(Time, zaxis_1, yaxis_1, xaxis_1) ; dust1:long_name = "mass_fraction_of_dust001_in_air" ; dust1:units = "ugkg-1" ; dust1:checksum = "12B0A4352DF4EE1D" ; double dust2(Time, zaxis_1, yaxis_1, xaxis_1) ; dust2:long_name = "mass_fraction_of_dust002_in_air" ; dust2:units = "ugkg-1" ; dust2:checksum = " 4C6F363837E6B67" ; double dust3(Time, zaxis_1, yaxis_1, xaxis_1) ; dust3:long_name = "mass_fraction_of_dust003_in_air" ; dust3:units = "ugkg-1" ; dust3:checksum = "C42D9E6CC01B341E" ; double dust4(Time, zaxis_1, yaxis_1, xaxis_1) ; dust4:long_name = "mass_fraction_of_dust004_in_air" ; dust4:units = "ugkg-1" ; dust4:checksum = "5DDC57DF718AC432" ; double dust5(Time, zaxis_1, yaxis_1, xaxis_1) ; dust5:long_name = "mass_fraction_of_dust005_in_air" ; dust5:units = "ugkg-1" ; dust5:checksum = "C002B2A3E75741A4" ; double seas1(Time, zaxis_1, yaxis_1, xaxis_1) ; seas1:long_name = "mass_fraction_of_sea_salt001_in_air" ; seas1:units = "ugkg-1" ; seas1:checksum = "AF337258E3F330AA" ; double seas2(Time, zaxis_1, yaxis_1, xaxis_1) ; seas2:long_name = "mass_fraction_of_sea_salt002_in_air" ; seas2:units = "ugkg-1" ; seas2:checksum = "BD6C3B11232E1BA7" ; double seas3(Time, zaxis_1, yaxis_1, xaxis_1) ; seas3:long_name = "mass_fraction_of_sea_salt003_in_air" ; seas3:units = "ugkg-1" ; seas3:checksum = "85A1E462FC3943C0" ; double seas4(Time, zaxis_1, yaxis_1, xaxis_1) ; seas4:long_name = "mass_fraction_of_sea_salt004_in_air" ; seas4:units = "ugkg-1" ; seas4:checksum = "E05CF1A2E7C0C3C7" ; double seas5(Time, zaxis_1, yaxis_1, xaxis_1) ; seas5:long_name = "mass_fraction_of_sea_salt005_in_air" ; seas5:units = "ugkg-1" ; seas5:checksum = "B307F64F0A1A21B4" ; double no3an1(Time, zaxis_1, yaxis_1, xaxis_1) ; no3an1:long_name = "mass_fraction_of_nitrate001_in_air" ; no3an1:units = "ugkg-1" ; no3an1:checksum = "5E1948C96427198C" ; double no3an2(Time, zaxis_1, yaxis_1, xaxis_1) ; no3an2:long_name = "mass_fraction_of_nitrate002_in_air" ; no3an2:units = "ugkg-1" ; no3an2:checksum = "50C11E9CBCDE134E" ; double no3an3(Time, zaxis_1, yaxis_1, xaxis_1) ; no3an3:long_name = "mass_fraction_of_nitrate003_in_air" ; no3an3:units = "ugkg-1" ; no3an3:checksum = "CD7E8071764F8B11" ;

That's why they need to be consolidated.

@HamidehGMAO Is the difference between tracer files in gfs and gfs_aero only the missing aerosol variables? Or Am I missing something else?

Sorry, it's difficult to compare the ncdump output in the format you put in the comments. You can use ``` to make the code blocks more readable.

HamidehGMAO · 2025-02-21T18:47:05Z

@mer-a-o I see! gfs doesn't have the aerosol variables and also gfs-aero dosn't have ice_wat, "cloud_liquid_ice" ; liq_wat "cloud_liquid_water" ; o3mr "ozone_mass_mixing_ratio"

mer-a-o · 2025-02-21T19:11:04Z

Thanks.
I'm comparing /gfs_c12/bkg/20201214.210000.fv_tracer.res.tile1.nc with gfs_aero_c12/bkg/20210801.060000.fv_tracer.res.tile1.nc. A few questions:

the new tracer files you added to gfs_c12 has 127 levels compared to the gfs_aero tracer files. Is this intentional?
In gfs_aero_c12 there inputs for different dates. In 20210801.060000, "cloud_liquid_ice", "cloud_liquid_water" and "ozone_mass_mixing_ratio" are available, but it only includes dust aerosols and is missing other aerosols. For the other dates, cloud variables are missing but aerosol variables are available. @jeromebarre, we need to clean this up at some point.

I suggest, for now, to add your files to gfs_aero directly, simply because they are files that have aerosols in them. Later we can spend a few hours cleaning up the directly.

Are you also missing surface values like surface wind (you mentioned that in the description)? I don't see surface variables in the files you added in this PR.

HamidehGMAO · 2025-02-21T19:15:33Z

@mer-a-o We are not adding any surface variables here to fv_tracer, we are reading the surface variables from gfs/fv_srf_wnd and gfs/sfc_data data, which gfs_aero doesn't have them. I explained the issue here #105

HamidehGMAO · 2025-03-14T19:56:27Z

@mer-a-o @jeromebarre Could you please take a look, and see if you are happy with the new changes? Thanks.

jeromebarre

great thx!

fmahebert · 2025-04-30T18:37:40Z

@mer-a-o do you feel your comments above were addressed to satisfaction? do you recommend that we proceed or that we make changes to Hamideh's branch before proceeding?

added aerosol fv_tracer.res data to gfs_c12/bkg

89fba4d

HamidehGMAO self-assigned this Feb 7, 2025

HamidehGMAO added the SOCA Sea-ice, Ocean, and Coupled Assimilation label Feb 7, 2025

HamidehGMAO requested review from cmgas and travissluka February 7, 2025 20:22

cmgas approved these changes Feb 20, 2025

View reviewed changes

HamidehGMAO requested review from jeromebarre and mer-a-o February 21, 2025 15:37

modified the files so gfs overwrite gfs_aero for sphum..

c6a9b7b

HamidehGMAO added 2 commits March 5, 2025 10:43

consolidated the tracer files int the gfs_aero_c12 bkg data

4ba573f

..

c8da51d

HamidehGMAO changed the title ~~added aerosol fv_tracer.res data to gfs_c12/bkg~~ consolidate gfs data to gfs_aero_c12/bkg for 20201214.21z Mar 10, 2025

jeromebarre approved these changes Mar 14, 2025

View reviewed changes

travissluka approved these changes Mar 14, 2025

View reviewed changes

Merge branch 'develop' into feature/oasim_aod

904a596

consolidate gfs data to gfs_aero_c12/bkg for 20201214.21z #107

Are you sure you want to change the base?

consolidate gfs data to gfs_aero_c12/bkg for 20201214.21z #107

Uh oh!

Conversation

HamidehGMAO commented Feb 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issue(s) addressed

Impact

Checklist

Uh oh!

fmahebert commented Feb 19, 2025

Uh oh!

cmgas left a comment

Choose a reason for hiding this comment

Uh oh!

jeromebarre commented Feb 21, 2025

Uh oh!

danholdaway commented Feb 21, 2025

Uh oh!

jeromebarre commented Feb 21, 2025

Uh oh!

travissluka commented Feb 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HamidehGMAO commented Feb 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HamidehGMAO commented Feb 21, 2025

Uh oh!

mer-a-o commented Feb 21, 2025

Uh oh!

HamidehGMAO commented Feb 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mer-a-o commented Feb 21, 2025

Uh oh!

HamidehGMAO commented Feb 21, 2025

Uh oh!

HamidehGMAO commented Mar 14, 2025

Uh oh!

jeromebarre left a comment

Choose a reason for hiding this comment

Uh oh!

fmahebert commented Apr 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

HamidehGMAO commented Feb 7, 2025 •

edited

Loading

travissluka commented Feb 21, 2025 •

edited

Loading

HamidehGMAO commented Feb 21, 2025 •

edited

Loading

HamidehGMAO commented Feb 21, 2025 •

edited

Loading