Merge pull request #1121 from Unidata/v1.5.7rel

prepare for v1.5.7 release

Author: Jeff Whitaker

Changelog

@@ -1,5 +1,5 @@
- version 1.5.7 (not yet released)
-=================================
+ version 1.5.7 (tag v1.5.7rel)
+==============================
  * don't try to mask vlens with default  _FillValue, since vlens don't have a default _FillValue.
    This gets rid of numpy DeprecationWarning (issue #1099).
  * update docs to reflect the fact that a variable must be in collective mode before writing

README.md

@@ -10,6 +10,10 @@
 ## News
 For details on the latest updates, see the [Changelog](https://github.com/Unidata/netcdf4-python/blob/master/Changelog).
 
+6/22/2021:  Version [1.5.7](https://pypi.python.org/pypi/netCDF4/1.5.7) released.
+Fixed OverflowError on Windows when reading data with dimension sizes greater than 2**32-1.
+Masked arrays no longer returned for vlens.
+
 2/15/2021:  Version [1.5.6](https://pypi.python.org/pypi/netCDF4/1.5.6) released. Added `Dataset.fromcdl` and `Dataset.tocdl`, which require `ncdump` and `ncgen` utilities to be in `$PATH`. Removed python 2.7 support.
 
 12/20/2020: Version [1.5.5.1](https://pypi.python.org/pypi/netCDF4/1.5.5.1) released.

docs/index.html

@@ -641,11 +641,9 @@ <h2 id="groups-in-a-netcdf-file">Groups in a netCDF file</h2>
 object yields summary information about it's contents.</p>
 
 <div class="codehilite"><pre><span></span><code><span class="o">&gt;&gt;&gt;</span> <span class="k">def</span> <span class="nf">walktree</span><span class="p">(</span><span class="n">top</span><span class="p">):</span>
-<span class="o">...</span>     <span class="n">values</span> <span class="o">=</span> <span class="n">top</span><span class="o">.</span><span class="n">groups</span><span class="o">.</span><span class="n">values</span><span class="p">()</span>
-<span class="o">...</span>     <span class="k">yield</span> <span class="n">values</span>
+<span class="o">...</span>     <span class="k">yield</span> <span class="n">top</span><span class="o">.</span><span class="n">groups</span><span class="o">.</span><span class="n">values</span><span class="p">()</span>
 <span class="o">...</span>     <span class="k">for</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">top</span><span class="o">.</span><span class="n">groups</span><span class="o">.</span><span class="n">values</span><span class="p">():</span>
-<span class="o">...</span>         <span class="k">for</span> <span class="n">children</span> <span class="ow">in</span> <span class="n">walktree</span><span class="p">(</span><span class="n">value</span><span class="p">):</span>
-<span class="o">...</span>             <span class="k">yield</span> <span class="n">children</span>
+<span class="o">...</span>         <span class="k">yield from</span> <span class="n">walktree</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
 <span class="o">&gt;&gt;&gt;</span> <span class="nb">print</span><span class="p">(</span><span class="n">rootgrp</span><span class="p">)</span>
 <span class="o">&lt;</span><span class="k">class</span> <span class="err">&#39;</span><span class="nc">netCDF4</span><span class="o">.</span><span class="n">_netCDF4</span><span class="o">.</span><span class="n">Dataset</span><span class="s1">&#39;&gt;</span>
 <span class="n">root</span> <span class="n">group</span> <span class="p">(</span><span class="n">NETCDF4</span> <span class="n">data</span> <span class="n">model</span><span class="p">,</span> <span class="n">file</span> <span class="nb">format</span> <span class="n">HDF5</span><span class="p">):</span>
@@ -987,8 +985,9 @@ <h2 id="writing-data-to-and-retrieving-data-from-a-netcdf-variable">Writing data
 The result will be a numpy scalar array.</p>
 
 <p>By default, netcdf4-python returns numpy masked arrays with values equal to the
-<code>missing_value</code> or <code>_FillValue</code> variable attributes masked.  The
-<code><a href="#Dataset.set_auto_mask">Dataset.set_auto_mask</a></code> <code><a href="#Dataset">Dataset</a></code> and <code><a href="#Variable">Variable</a></code> methods
+<code>missing_value</code> or <code>_FillValue</code> variable attributes masked for primitive and
+enum data types.
+The <code><a href="#Dataset.set_auto_mask">Dataset.set_auto_mask</a></code> <code><a href="#Dataset">Dataset</a></code> and <code><a href="#Variable">Variable</a></code> methods
 can be used to disable this feature so that
 numpy arrays are always returned, with the missing values included. Prior to
 version 1.4.0 the default behavior was to only return masked arrays when the
@@ -1596,7 +1595,7 @@ <h2 id="in-memory-diskless-datasets">In-memory (diskless) Datasets</h2>
 the parallel IO example, which is in <code>examples/mpi_example.py</code>.
 Unit tests are in the <code>test</code> directory.</p>
 
-<p><strong>contact</strong>: Jeffrey Whitaker <a href="ma&#x69;&#108;&#x74;o&#58;&#x6a;&#101;&#102;f&#114;&#x65;&#121;&#46;&#x73;&#x2e;&#x77;&#x68;&#105;&#116;&#97;&#107;&#x65;&#x72;&#64;&#110;&#x6f;aa&#x2e;&#103;&#x6f;&#x76;">&#x6a;&#101;&#102;f&#114;&#x65;&#121;&#46;&#x73;&#x2e;&#x77;&#x68;&#105;&#116;&#97;&#107;&#x65;&#x72;&#64;&#110;&#x6f;aa&#x2e;&#103;&#x6f;&#x76;</a></p>
+<p><strong>contact</strong>: Jeffrey Whitaker <a href="&#x6d;&#x61;&#105;&#x6c;&#116;&#111;&#58;&#106;&#101;&#x66;&#x66;&#114;&#x65;&#x79;&#x2e;&#115;&#46;&#x77;&#x68;&#x69;t&#x61;&#107;&#x65;&#x72;&#64;&#110;oa&#97;&#46;&#103;&#x6f;&#x76;">&#106;&#101;&#x66;&#x66;&#114;&#x65;&#x79;&#x2e;&#115;&#46;&#x77;&#x68;&#x69;t&#x61;&#107;&#x65;&#x72;&#64;&#110;oa&#97;&#46;&#103;&#x6f;&#x76;</a></p>
 
 <p><strong>copyright</strong>: 2008 by Jeffrey Whitaker.</p>
 
@@ -2403,7 +2402,8 @@ <h2 id="in-memory-diskless-datasets">In-memory (diskless) Datasets</h2>
             <div class="docstring"><p><strong><code>set_auto_mask(self, True_or_False)</code></strong></p>
 
 <p>Call <code><a href="#Variable.set_auto_mask">Variable.set_auto_mask</a></code> for all variables contained in this <code><a href="#Dataset">Dataset</a></code> or
-<code><a href="#Group">Group</a></code>, as well as for all variables in all its subgroups.</p>
+<code><a href="#Group">Group</a></code>, as well as for all variables in all its subgroups. Only affects
+Variables with primitive or enum types (not compound or vlen Variables).</p>
 
 <p><strong><code>True_or_False</code></strong>: Boolean determining if automatic conversion to masked arrays
 shall be applied for all variables.</p>
@@ -2744,7 +2744,8 @@ <h2 id="in-memory-diskless-datasets">In-memory (diskless) Datasets</h2>
 <p><strong><code><a href="#Variable.mask">mask</a></code></strong>: If True, data is automatically converted to/from masked
 arrays when missing values or fill values are present. Default is <code>True</code>, can be
 reset using <code><a href="#Variable.set_auto_mask">Variable.set_auto_mask</a></code> and <code><a href="#Variable.set_auto_maskandscale">Variable.set_auto_maskandscale</a></code>
-methods.</p>
+methods. Only relevant for Variables with primitive or enum types (ignored
+for compound and vlen Variables).</p>
 
 <p><strong><code><a href="#Variable.chartostring">chartostring</a></code></strong>: If True, data is automatically converted to/from character
 arrays to string arrays when the <code>_Encoding</code> variable attribute is set.

test/tst_slicing.py

@@ -2,7 +2,7 @@
 from numpy.random import seed, randint
 from numpy.testing import assert_array_equal, assert_equal,\
 assert_array_almost_equal
-import tempfile, unittest, os, random
+import tempfile, unittest, os, random, sys
 import numpy as np
 
 file_name = tempfile.NamedTemporaryFile(suffix='.nc', delete=False).name
@@ -35,8 +35,8 @@ def setUp(self):
         vu[:,::-1,:] = data
 
         v1[:] = data[:, 0, 0]
-        #v2[0:2**31] = 1 # issue 1112 (overflow on windows)
-        v2[2**31] = 1 # issue 1112 (overflow on windows)
+        if sys.maxsize > 2**32:
+            v2[2**31] = 1 # issue 1112 (overflow on windows)
         f.close()
 
     def tearDown(self):
@@ -82,8 +82,8 @@ def test_1d(self):
         v2 = f.variables['data1dx']
         d = data[:,0,0]
         assert_equal(v1[:], d)
-        #assert_equal(v2[:], np.ones(2**31,dtype=np.uint8))
-        assert_equal(v2[2**31], 1)
+        if sys.maxsize > 2**32:
+            assert_equal(v2[2**31], 1)
         assert_equal(v1[4:], d[4:])
         # test return of array scalar.
         assert_equal(v1[0].shape, ())