Merge pull request #137 from UW-Hydro/develop

release 1.1

Author: arbennett

.gitattributes

@@ -0,0 +1 @@
+metsim/_version.py export-subst

MANIFEST.in

@@ -0,0 +1,2 @@
+include versioneer.py
+include metsim/_version.py

ci/requirements-3.5.yml

@@ -7,6 +7,7 @@ dependencies:
   - pandas
   - netcdf4
   - numba
+  - bottleneck
   - numpy
   - scipy
   - pytest

ci/requirements-3.6.yml

@@ -6,6 +6,7 @@ dependencies:
   - xarray
   - pandas
   - netcdf4
+  - bottleneck
   - numba
   - numpy
   - scipy

docs/conf.py

@@ -19,6 +19,7 @@
 #
 import os
 import sys
+
 import sphinx_rtd_theme
 
 sys.path.insert(0, os.path.abspath('.'))
@@ -34,11 +35,16 @@
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = ['sphinx.ext.autodoc',
+              'sphinx.ext.extlinks',
               'sphinx.ext.mathjax',
               'sphinx.ext.viewcode',
               'sphinx.ext.napoleon',
               'sphinx.ext.autosummary']
 
+extlinks = {'issue': ('https://github.com/UW-Hydro/MetSim/issues/%s', 'GH'),
+            'pull': ('https://github.com/UW-Hydro/MetSim/pull/%s', 'PR'),
+            }
+
 napoleon_google_docstring = False
 napoleon_use_param = False
 napoleon_use_ivar = True

docs/configuration.rst

@@ -57,10 +57,11 @@ current implementation only supports ``mtclim``.
 ``out_precision :: str``: Precision to use when writing output.  Defaults to
 ``f8``.  Can be either ``f4`` or ``f8``.
 
-``annual :: bool``: Whether to chunk up the timeseries into years for
+``time_grouper :: str``: Whether to chunk up the timeseries into pieces for
 processing. This option is useful to set for when you are limited on
-memory.  Each year of output is written as ``{out_prefix}_{year}`` when
-active.
+memory.  Each chunk of output is written as ``{out_prefix}_{date_range}`` when
+active. Any valid ``pandas.TimeGrouper`` string may be used (e.g. use '10AS'
+for 10 year chunks).
 
 ``iter_dims :: list``: The dimensions of input data to iterate over to
 accumulate sites.  Defaults to ``['lat', 'lon']``.
@@ -75,6 +76,11 @@ account when simulating incoming shortwave radiation.  Defaults to ``0``.
 ``mtclim_swe_corr :: bool``: Whether to activate MtClim's SWE correction
 algorithm. Default to ``False``.
 
+``utc_offset :: bool``: Whether to use UTC timecode offsets for shifting
+timeseries. Without this option all times should be considered local to
+the gridcell being processed. Large domain runs probably want to set this
+option to ``True``.
+
 ``lw_cloud :: str``: Type of cloud correction to longwave radiation to apply.
 Can be either ``DEFAULT`` or ``CLOUD_DEARDORFF``.  Defaults to
 ``CLOUD_DEARDORFF``.  Capitalization does not matter.
@@ -109,7 +115,7 @@ Defaults to ``0.45``, range should be between ``0`` and ``1``.
 ``out_vars :: list`` : List of variables to write to output.  Should be a list
 containing valid variables.  The list of valid variables is dependent on which
 simulation method is used, as well as whether disaggregation is used. Defaults
-to ``['temp', 'prec', 'shortwave', 'longwave', 'vapor_pressure', 'red_humid']``.
+to ``['temp', 'prec', 'shortwave', 'longwave', 'vapor_pressure', 'rel_humid']``.
 For more information about input and output variables see the :ref:`data` page.
 
 forcing_vars and state_vars section
@@ -126,7 +132,7 @@ netcdf and data
 The ``in_vars`` section for NetCDF and xarray input acts as a mapping between the variable
 names in the input dataset to the variable names expected by MetSim.  The format
 is given as ``netcdf_varname = metsim_varname``.  The minimum required variables
-given have ``metsim_varname``s corresponding to ``t_min``, ``t_max``, and
+given have ``metsim_varname``\s corresponding to ``t_min``, ``t_max``, and
 ``prec``; these variable names correspond to minimum daily temperature (Celcius),
 maximum daily temperature (Celcius), and precipitation (mm/day).
 
@@ -148,7 +154,7 @@ input style.  Each line is specified as ``varname = scale cdatatype``, where
 floating point scaling factor that should be applied after conversion from
 binary to floating point; the conversion applied by the ``scale`` is applied
 after the value in the input is converted from binary to the ``cdatatype``
-specified for each variable.  Valid ``cdatatype``s are ``signed`` and
+specified for each variable.  Valid ``cdatatype``\s are ``signed`` and
 ``unsigned``.  ``signed`` values are interpreted as values which can be positive
 or negative, whereas ``unsigned`` values are interpreted as values that can only
 be greater than or equal to zero.
@@ -159,6 +165,6 @@ The ``domain_vars`` section is where information about the domain file is given.
 Since the domain file is given as a NetCDF file this section has a similar
 format to that of the NetCDF input file format described above.  That is,
 entries should be of the form ``netcdf_varname = metsim_varname``. The minimum
-required variables have ``metsim_varname``s corresponding to ``lat``, ``lon``,
+required variables have ``metsim_varname``\s corresponding to ``lat``, ``lon``,
 ``mask``, and ``elev``; these variable names correspond to latitude, longitude,
 a mask of valid cells in the domain, and the elevation given in meters.

docs/data.rst

@@ -35,29 +35,26 @@ Second, is the ``elev`` variable. This provides elevation data used for
 calculation of solar geometry. It should be specified in meters, and only needs
 to be given at sites which are marked to be processed via the ``mask`` variable.
 
-It is important to ensure that all valid locations in ``mask`` have data in 
+It is important to ensure that all valid locations in ``mask`` have data in
 ``elev``.  Failure to ensure this will result in errors during runtime.
 
 State file
 ----------
 The state file provides information about the history of each of the grid cells
 to be processed. There are four required variables.
 
-The first two are daily minimum and daily maximum temperatures for the 90 days 
-preceeding the start date specified in the configuration file.  They should be 
-specified as ``t_min`` and ``t_max`` respectively. Similarly precipitation 
-should be given as ``prec``.  These variables are used to generate seasonal 
+The first two are daily minimum and daily maximum temperatures for the 90 days
+preceeding the start date specified in the configuration file.  They should be
+specified as ``t_min`` and ``t_max`` respectively. Similarly precipitation
+should be given as ``prec``.  These variables are used to generate seasonal
 averages which are used in the calculation of shortwave and longwave radiation.
 
-The final required variable is the initial snow water equivalent (SWE) for each
-grid cell. It should be named ``swe`` in the file.
-
 Output Specifications
 =====================
 .. ATTENTION::
-    The ``time`` coordinate in MetSim's output is local to the location of each 
-    cell! This means that for a single time slice in the NetCDF file all locations
-    along a parallel (same latitude) will have the same solar geometry at that time.
+    The ``time`` coordinate in MetSim's output is local to the location of each cell unless the ``utc_offset`` is set to
+    ``True``! This means that for a single time slice in the NetCDF file all locations along a parallel (same latitude)
+    will have the same solar geometry at that time.
 
 The output variables that are available are dependent on the time step being used.  There are two cases:
 
@@ -71,11 +68,11 @@ step:
 * ``t_min`` : Minimum temperature (also a required input value) (C)
 * ``t_max`` : Maximum temperature (also a required input value) (C)
 * ``prec`` : Precipitation (also a required input value) (mm/day)
-* ``swe`` : Snow water equivalent (mm)
 * ``vapor_pressure`` : Vapor pressure (kPa)
 * ``shortwave`` : Shortwave radiation (W/m^2)
 * ``tskc`` : Cloud cover fraction
 * ``pet`` : Potential evapotranpiration (mm/day)
+* ``wind`` : Wind speed (only if given as an input) (m/s)
 
 Sub-daily Output
 ----------------

docs/index.rst

@@ -16,23 +16,23 @@ VIC_ hydrologic model.
 .. _MtClim: http://www.ntsg.umt.edu/project/mtclim
 .. _VIC: https://github.com/UW-Hydro/VIC
 
-MetSim consists of 3 main modules that govern the operation of 3 
+MetSim consists of 3 main modules that govern the operation of 3
 major aspects of its operation:
 
 **1. Management of dataset preprocessing and IO**
 
 The MetSim object provides high level support for setting up jobs
 and infrastructure for running simulation/disaggregation
 steps. It is the main interface through which the other modules
-are accessed. 
+are accessed.
 
 **2. Simulation of meteorological forcings**
 
 The base implementation of the meteorological simulator is
 based off of the algorithms described in [1]_. This component
-has been designed to be flexible in allowing for alternative 
+has been designed to be flexible in allowing for alternative
 implementations which may be specified during the setup of the
-MetSim object.  The default implementation allows for the 
+MetSim object.  The default implementation allows for the
 daily simulation of:
 
  * Mean daily temperature
@@ -46,11 +46,11 @@ daily simulation of:
 
 Daily data from given input or simulated via the forcings generation
 component of MetSim can be disaggregated down to sub-daily values at
-intervals specified in minutes (provided they divide evenly into 24 
+intervals specified in minutes (provided they divide evenly into 24
 hours).  The operation of these algorithms is also described in [1]_.
 
 This documentation is a work in progress.
-If you don't find what you're looking for here, check out MetSim's Github page.  
+If you don't find what you're looking for here, check out MetSim's Github page.
 
 Getting Started
 ===============
@@ -64,7 +64,7 @@ ensure that you have all of the required dependencies:
 - `xarray <http://xarray.pydata.org/>`__ (0.9.1 or later)
 - `pandas <http://pandas.pydata.org/>`__ (0.19.0 or later)
 - `numba <http://numba.pydata.org/>`__ (0.31.0 or later)
-- `netCDF4 <https://github.com/Unidata/netcdf4-python>`__ 
+- `netCDF4 <https://github.com/Unidata/netcdf4-python>`__
 - `scipy <http://scipy.org/>`__
 
 
@@ -86,18 +86,18 @@ Finally, you can install MetSim directly from the source if you desire to::
 
 Basic Usage
 -----------
-MetSim provides a simple command line interface which is primarily operated via 
+MetSim provides a simple command line interface which is primarily operated via
 configuration files.  For more information about the options available to be set
 in the configuration files see the :ref:`configuration` page.
 
 Once installed, MetSim can be used from the command line via:
 
 ``ms /path/to/configuration [-v] [-n #]``
 
-Bracketed flags are optional; ``-v`` activates verbose mode to print messages 
-about the status of a run, and ``-n`` activates parallelism.  The number given 
+Bracketed flags are optional; ``-v`` activates verbose mode to print messages
+about the status of a run, and ``-n`` activates parallelism.  The number given
 after the ``-n`` flag is the number of processes to run. A good rule of thumb is
-to use one less process than the number of processsors (or threads) that the 
+to use one less process than the number of processsors (or threads) that the
 machine you are running on has.
 
 References
@@ -139,3 +139,4 @@ Sitemap
     data
     configuration
     api
+    whats-new

docs/release-proceedure.rst

@@ -0,0 +1,32 @@
+Making a new release of Metsim is easy. Just follow the following steps:
+
+Release per project:
+
+*   Update release notes in docs/whats-new.rst
+
+*   Tag commit
+
+        git tag -a x.x.x -m 'Version x.x.x'
+
+*   and push to github
+
+        git push upstream master --tags
+
+*  Upload to PyPI
+
+        git clean -xfd
+        python setup.py sdist bdist_wheel --universal
+        twine upload dist/*
+
+*   Update conda recipe feedstocks on `conda-forge <https://conda-forge.github.io/>`_.
+
+    *  Update conda-smithy and run conda-smithy rerender
+
+            git clone [email protected]:conda-forge/metsim-feedstock
+            cd metsim-feedstock
+            conda install conda-smithy
+            conda-smithy rerender
+
+    *  Get sha256 hash from pypi.org
+    *  Update version number and hash in recipe
+    *  Check dependencies

docs/whats-new.rst

@@ -0,0 +1,39 @@
+.. currentmodule:: MetSim
+
+What's New
+==========
+
+.. _whats-new.1.1.0:
+
+v1.1.0 (unreleased)
+-------------------
+
+Enhancements
+~~~~~~~~~~~~
+
+- Added option to use forcing start/stop dates to define run length (:issue:`93`).
+  By `Joe Hamman <https://github.com/jhamman>`_.
+- Added option a flexible time grouper when chunking MetSim runs (:issue:`93`).
+  By `Joe Hamman <https://github.com/jhamman>`_.
+- Improved configuration validation by checking for correctness of output variables (:issue:`96`)
+  By `Andrew Bennett <https://github.com/arbennett>`
+- Added option to skip reading ``swe`` variable from state file if it is not
+  going to be used by MtClim. (:issue:`103`). By `Joe Hamman <https://github.com/jhamman>`_.
+- Added support for supplying a glob-like file path or multiple input forcing
+  files (netCDF) (:issue:`126`). By `Joe Hamman <https://github.com/jhamman>`_.
+- Refactored ``mtclim`` and ``disaggregate`` functions to reduce interdependency and
+  increase modularity. By `Andrew Bennett <https://github.com/arbennett>`_.
+- Removed ``swe`` calculations. By `Andrew Bennett <https://github.com/arbennett>`_.
+
+Bug fixes
+~~~~~~~~~
+- Fixed bug where output files were not written with the appropriate calendar
+  encoding attribute (:issue:`97`).
+  By `Joe Hamman <https://github.com/jhamman>`_.
+- Fixed a bug where invalid timesteps were used in subdaily disaggregation.
+  Added a clear error message explaining that subdaily timesteps must be evenly
+  divisible into 24 hours and less than 6 hours in length. (:issue:`110`).
+  By `Joe Hamman <https://github.com/jhamman>`_.
+- Fixed a bug during disaggregation when ``t_min > t_max``.  This now raises
+  an exception.
+  By `Andrew Bennett <https://github.com/arbennett>`_.