Add low resolution dataset for DYAMOND summer

[sriharshakandala]

Add a low-resolution initial conditions dataset for DYAMOND summer simulation.

Checklist:

• I created a new folder $artifact_name
  • I added a README.md in that that folder that
    • describes the data and processing done to it
    • lists the sources of the raw data
    • lists the required citation, licenses
  • If applicable (e.g., for Creative Commons), I added a LICENSE file
  • I added the scripts that retrieve, process, and produce the artifact
  • I added the environment used for such scripts (typically, Project.toml
    and Manifest.toml)
  • I added the OutputArtifacts.toml file containing the information
    needed for package developers to add $artifact_name to their package
• I uploaded the artifact folder to the Caltech cluster (in
  /groups/esm/ClimaArtifacts/artifacts/$artifact_name)
• I added the relevant code to the Overides.toml on the Caltech Cluster
  (in /groups/esm/ClimaArtifacts/artifacts/Overrides.toml)
• I added a link to the main README.md to point to the new artifact

[szy21]

This is not quite right. To get the level -> pressure -> z conversion, you need to follow what is done here for the aerosols. With this, you will have a different z for each grid point, and you will need to interpolate to a fixed z coordinate, as in the aerosol artifact.

After you fix it, Could you make some latitude-pressure plots for different variables so I can take a look at whether it is reasonable?

[akshaysridhar]

Tested on M1 Mac desktop; using locally generated plot script to viz datasets (some variable name choices in the generated dataset are unusual e.g. t as opposed to ta for air temperature)

julia> @time create_initial_conditions(FILE_PATH, "DYAMOND_SUMMER_ICS_p98deg.nc", skip=7)
351.898330 seconds (2.84 G allocations: 129.391 GiB, 2.12% gc time, 1.12% compilation time)

Averaged along `lon` dim:

dyamond_ciwc.pdf
dyamond_clwc.pdf
dyamond_crwc.pdf
dyamond_cswc.pdf
dyamond_p.pdf
dyamond_q.pdf
dyamond_rho.pdf
dyamond_T.pdf
dyamond_u.pdf
dyamond_w.pdf

Example slices at 828m altitude : Spec hum and air temp

[szy21]

Thanks @akshaysridhar. Could you make some lat-lon plots at e.g. surface, 2km, and 5km (the z level doesn't need to be accurate)? I mostly want to make sure things are correct over topography.

[akshaysridhar]

~10m: hus, ta
dyamond_q_10m.pdf
dyamond_T_10m.pdf

~2km : hus, ta
dyamond_q_2023m.pdf
dyamond_T_2023m.pdf

~5km: hus, ta
dyamond_q_4965m.pdf
dyamond_T_4965m.pdf

[szy21]

Please also update the README to describe the processing is done (See the README in the aerosol artifact for example).

[Sbozzolo]

Also, this artifact is missing the Manifest.toml

[sriharshakandala]

@szy21 : Would it be ok to compute rho and rho_e_tot directly in ClimaAtmos after interpolating the required variables? It would be much more efficient!

[szy21]

@szy21 : Would it be ok to compute rho and rho_e_tot directly in ClimaAtmos after interpolating the required variables? It would be much more efficient!

Yes, that’s fine. How is it more efficient though?

[sriharshakandala]

@szy21 : Would it be ok to compute rho and rho_e_tot directly in ClimaAtmos after interpolating the required variables? It would be much more efficient!

Yes, that’s fine. How is it more efficient though?

For this low-resolution case, it reduced the processing time to about 25 sec from 400 sec. I would assume the total_energy and rho calculations are much more optimized and efficient in a distributed setting in ClimaAtmos

[Sbozzolo]

It's not immediately clear to me that computing in atmos would be faster (what's faster: a linear interpolation, or computing some thermodynamic variables? The answer will be different for different resolutions and devices; for GPUs, it is probably preferable to reduce the file size anyway).

More importantly, this is only done at initialization, so performance is not important (unless it's really slow). What's preferable is trying to avoid introducing additional errors, if that can be avoided.

Interpolating and them computing variables is different from computing variables and then interpolating. My preference would go for the first (meaning, compute everything in ClimaAtmos) for two reasons:

1. it ensure consistency in the Thermodynamics and the parameters being used
2. it ensures that the thermodynamic variables are internally consistent (because they are computed one with the other directly)

[akshaysridhar]

https://buildkite.com/clima/climaatmos-target-gpu-simulations/builds/396#019461bf-f7e7-488d-9e35-83a91667450c Following this build I think we can clean up this PR and formalize the p14deg artifact @sriharshakandala.

[szy21]

What's the status of this PR?

[sriharshakandala]

What's the status of this PR?

I have updated it to use the main branch of ClimaInterpolations.jl. I am fine with merging this unless there are any other comments!

[akshaysridhar]

@ph-kev @imreddyTeja Any further comments on this PR? Do we merge this and close #78 ?

[akshaysridhar]

@ph-kev Is it worth generalizing these regridding tools : https://github.com/CliMA/ClimaAnalysis.jl/blob/6b6b6b9abf987b7c375f4ee766a37a73705bd0d6/src/Atmos.jl#L63 in ClimaUtilities? I'm not sure what the best location for these tools would be.

[ph-kev]

For this and altitude, it would be good to also include the short name as well.

[ph-kev]

@ph-kev Is it worth generalizing these regridding tools : https://github.com/CliMA/ClimaAnalysis.jl/blob/6b6b6b9abf987b7c375f4ee766a37a73705bd0d6/src/Atmos.jl#L63 in ClimaUtilities? I'm not sure what the best location for these tools would be.

I think it depends on the use case of the regridding tools. Is it for making NetCDF files or something beyond that? If it is just for NetCDF files, I think it is okay to add them to ClimaUtilities.

@imreddyTeja

[tapios]

Assuming $P = P0*exp(-z / H)$ is not sufficiently accurate to convert pressure to altitude. Need to integrate hydrostatic balance (+ideal gas law) vertically, or, if available, use geopotential/grav for altitude.

[tapios]

Integrating hydrostatic balance is in fact what's being done below.