You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We currently concatenate zarrs when running runfv3 append into the <rundir>/name.zarr. This is convenient for analysis but has several downsides:
The concatenation code can and does corrupt the zarr files when the post-processing is interrupted.
complexity: gsutil-based implementation of appending to a zarr is complex. This constrains the chunk sizes we can use, which then leads to chunk size validation etc.
There are a couple ways to pay down the debt:
reimplement appending with ffspec/normal zarr operations now that gcsfs is more reliable
simply save zarrs into each artifacts// directly and add some functions for opening data
The text was updated successfully, but these errors were encountered:
We currently concatenate zarrs when running
runfv3 append
into the<rundir>/name.zarr
. This is convenient for analysis but has several downsides:gsutil
-based implementation of appending to a zarr is complex. This constrains the chunk sizes we can use, which then leads to chunk size validation etc.There are a couple ways to pay down the debt:
The text was updated successfully, but these errors were encountered: