Add SCHISM compataibility

[mpiannucci]

SCHISM datasets should be compatible with the UGRID accessor, but to ensure this is so we need to test, and fix any issues that arise. This issue also covers adding example notebooks and test cases for SCHISM datasets

[AtiehAlipour-NOAA]

Description:

This issue serves as a general ticket for the development of a subsetting tool for STOFS 3D model outputs. The aim is to add functionality to xarray-subset-grid code for efficiently subsetting data from STOFS 3D outputs while preserving key features of the model.

This issue serves as a starting point for the development of the subsetting tool. Subsequent issues may be created for smaller tasks related to its development.

For STOFS-3D, we use SCHISM as the base model.

Preservation of Mesh Structure: SCHISM use both triangular and quadratic meshes. It's essential that xarray-subset-grid maintains the structural of the mesh after subsetting.

Subset Data for Different Variables at Different Vertical Layers: STOFS 3D outputs contain data across multiple vertical layers for different variables. The xarray-subset-grid should incorporate functionalities to subset data based on different layers and variable names.

[mpiannucci]

Here is the target file: s3://noaa-nos-stofs3d-pds/STOFS-3D-Atl/stofs_3d_atl.20240619/stofs_3d_atl.t12z.fields.out2d_f037_048.nc

[omkar-334]

From Discussions

'stofs_3d_atl.t12z.fields.horizontalVelX_f001_012.nc'
'stofs_3d_atl.t12z.fields.horizontalVelY_f001_012.nc'
'stofs_3d_atl.t12z.fields.salinity_f001_012.nc'
'stofs_3d_atl.t12z.fields.temperature_f001_012.nc'
'stofs_3d_atl.t12z.fields.zCoordinates_f001_012.nc'
'stofs_3d_atl.t12z.fields.out2d_f001_012.nc'
'schout_adcirc_20240619.nc'
As expected, only the last 2 files, out2d and schout are detected as SCHISM and ADCIRC respectively.

According to this solution that Atieh had sent, I tried on out2d and schout.
The problem is that when we rename the variables, the CF metadata remains the same.

In our code, ugrid.py, We obtain the x and y variable names using the ds.cf['mesh_topology']['node_coordinates'] Here the code breaks since it checks for the original variable names after we've renamed.
One solution for this is to rewrite the CF metadata. I've tried this using schoutds.cf['mesh_topology'].assign_attrs(node_coordinates ='x y') but this seems to return a new object and the CF accessor doesn't support item assignment.
out.cf["mesh_topology"]['node_coordinates'] = 'x y' doesn't resolve it either.

Another solution would be not rename the variables.
After dropping and adding the nele dimension,

ds2 = ds2.drop_dims('nele')  
ds2['nele'] = ds['nele']  

This variable disappears - SCHISM_hgrid_face_nodes - (nSCHISM_hgrid_face, nMaxSCHISM_hgrid_face_nodes)
Here, nSCHISM_hgrid_face is renamed to nele. It could be that the new nele does not retain the connectivity and this happens.

[omkar-334]

From Discussion

One solution is to first read files with complete information, and then read files with missing information like temperature for subsetting. We can write a wrapper for this.

'stofs_3d_atl.t12z.fields.horizontalVelX_f001_012.nc'
'stofs_3d_atl.t12z.fields.horizontalVelY_f001_012.nc'
'stofs_3d_atl.t12z.fields.salinity_f001_012.nc'
'stofs_3d_atl.t12z.fields.temperature_f001_012.nc'
'stofs_3d_atl.t12z.fields.zCoordinates_f001_012.nc'

If a file from above is chosen for subsetting, we can grab the respective schout and out2d, given that they exist for every file.
However, files like temperature and salinity are about 7GB. SHould we add this info only if it is needed, or do we chunk it and add to all subsetting jobs?

[ChrisBarker-NOAA]

hmm.

I think that xarray should "lazy load" the data, so the 7GB would only be loading up when saving out the results.

And we want a way for the user to be able to specify what variables they want.

So it should be fine to include everything up front.

[ChrisBarker-NOAA]

One solution for this is to rewrite the CF metadata. I've tried this using > schoutds.cf['mesh_topology'].assign_attrs(node_coordinates ='x y') but this seems to return a new object > and the CF accessor doesn't support item assignment.
out.cf["mesh_topology"]['node_coordinates'] = 'x y' doesn't resolve it either.

I think what you need to do is use .cf['mesh_topology'] to get the xarray variable, and then you can change the attrs directly on that variable.

Another option would be to use:

ds.subset_grid.assign_ugrid_topology() -- it will overwrite the mesh variable attribute for anything you pass in.
(only partially tested, but it should work)

Another solution would be not rename the variables.

is there a reason to rename them? if not, then yes, that's the easier way to go.

[ChrisBarker-NOAA]

One more thought: I think it's a goal for the resulting subset file to be as similar to the original as possible -- e.g. all variables have the same names. So I don't think any renaming should be done, unless absolutely required.

[while the CF standard applies almost no meaning to variable names, a lot of users do -- code tends to simply look for known variable names]

[mpiannucci]

Yes we very much do not want to rename any variables as a part of this process

[saeed-moghimi-noaa]

@josephzhang8 @gseroka @feiye-vims @yangz888 @omkar-334 @AtiehAlipour-NOAA @mpiannucci

Hi Joseph and Greg

Omkar is working on performing subsetting capability for STOFS-3D-Atl. He was trying to do it for adcirc like out puts. He wants to start looking into perform sub setting on native 2D and 3D schism outputs.

If he wants to start with water surface elevation. Would you please point him to the right SCHSIM native file name on STOFS-3D-ATl S3 bucket and file name ( I mean the ones that has quads and triangle elements).

Are those ugrid and cf compliant?

https://noaa-nos-stofs3d-pds.s3.amazonaws.com/README.html

Thanks,
-Saeed

[mpiannucci]

What we are seeing is that the face_node_connectivity is loaded as a float64 array and not integer. This breaks indexing when trying to collect the polygons. Also polygons that are triangles have NaNs and we need to check on the CF/ugrid compliance for that

[ChrisBarker-NOAA]

A bit more detail:

we'd like it to comply with:

https://ugrid-conventions.github.io/ugrid-conventions/#2d-flexible-mesh-mixed-triangles-quadrilaterals-etc-topology

In particular:
The face_node_connectivity should be:

• integers -- indexes into the nodes array.
• The fourth vertex, if it's not there (i.e. a triangle), should use the _FillValue, which should be specified.

I recommend -1 for _FillValue but the spec allows anything that's not a valid index -- so -999, or ....

But it's can't be NaN, as integers don't have NaN

[josephzhang8]

Hi all-

We only recently added UGRID outputs to shadow forecast (Zizang is still working on adding it on NOAA side). Anyway, Dan (@danishyo) and I just checked outputs from the shadow forecast and the format is as @ChrisBarker-NOAA described: int for the table, -1 as fill in value. Dan will share with you all the AWS path, but can someone please add him into the repo? His github ID is:

danishyo

Thanks.

[danishyo]

AWS link is at https://noaa-nos-stofs3d-pds.s3.amazonaws.com/index.html#STOFS-3D-Atl-shadow-VIMS/ Note: only recent output (after 2024/04/01) are ugrid format. Dan Joseph Zhang ***@***.***> 於 2024年7月18日 週四 下午3:56寫道：

…

Hi all- We only recently added UGRID outputs to shadow forecast (Zizang is still working on adding it on NOAA side). Anyway, Dan ***@***.*** <https://github.com/danishyo>) and I just checked outputs from the shadow forecast and the format is as @ChrisBarker-NOAA <https://github.com/ChrisBarker-NOAA> described: int for the table, -1 as fill in value. Dan will share with you all the AWS path, but can someone please add him into the repo? His github ID is: danishyo Thanks. — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM3S4J3UEXW46MILEAOWTP3ZNAMW5AVCNFSM6AAAAABH2WZVTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZXGQ2TOMBQHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>