Skip to content

Commit

Permalink
Add CHELSA-W5E5 tile compression. (#102)
Browse files Browse the repository at this point in the history
* Compress tile files with nccopy.

* Document changes in whatsnew.
  • Loading branch information
juseg authored Jul 15, 2024
1 parent 8e6da89 commit 774b10e
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 3 deletions.
4 changes: 2 additions & 2 deletions doc/whatsnew.rst
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ New features
(aliased ``'cw5e5'``) as a new temperature and precipitation data source in
:func:`hyoga.open.atmosphere` (:issue:`86`, :pull:`87`), aggregated on 30x30
degree tiles (:issue:`88`, :pull:`92`), concatenated yearly (:issue:`99`,
:pull:`100`).
:pull:`100`), and compressed with netCDF-4 (:issue:`101`, :pull:`102`).

.. _CHELSA-W5E5: https://chelsa-climate.org/chelsa-w5e5-v1-0-daily-climate-data-at-1km-resolution/

Expand All @@ -46,7 +46,7 @@ Internal changes
~~~~~~~~~~~~~~~~

- Add aggregators in :mod:`hyoga.open.aggregator` (:issue:`86`, :issue:`88`,
:issue:`99`, :pull:`87`, :pull:`92`, :pull:`100`).
:issue:`99`, :issue:`101`, :pull:`87`, :pull:`92`, :pull:`100`, :pull:`102`).

.. _v0.3.1:

Expand Down
12 changes: 11 additions & 1 deletion hyoga/open/aggregator.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@
"""

import os.path
import subprocess
import tempfile
import xarray as xr
import hyoga.open.downloader

Expand Down Expand Up @@ -171,9 +173,17 @@ def aggregate(self, inputs, output, recipe='avg'):
tile = tile.groupby('time.month')
tile = getattr(tile, recipe)(keep_attrs=True)

# store output as netcdf and return path
# store as uncompressed netcdf
print(f"aggregating {tilepath} ...")
tile.to_netcdf(tilepath)

# nccopy compression beats xarray by far
print(f"compressing {tilepath} ...")
dirname, basename = os.path.split(tilepath)
with tempfile.NamedTemporaryFile(
dir=dirname, prefix=basename+'.') as tmp:
subprocess.run(['nccopy', '-sd6', tilepath, tmp.name])
os.replace(tmp.name, tilepath)

# return multi-tile output pattern
return output

0 comments on commit 774b10e

Please sign in to comment.