Use latest HDMF, raise min pandas, dissociate req-min and setup reqs (#1363)

Author: rly

.circleci/config.yml

@@ -354,6 +354,7 @@ jobs:
           command: |
             python -m venv ../venv
             . ../venv/bin/activate
+            # use click<8 until https://github.com/j0057/github-release/issues/62 is resolved
             pip install "click<8" githubrelease
             githubrelease release $CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME \
               create $CIRCLE_TAG --name $CIRCLE_TAG \

.github/PULL_REQUEST_TEMPLATE/release.md

@@ -1,20 +1,26 @@
 Prepare for release of PyNWB [version]
 
 ### Before merging:
-- [ ] Minor releases: Update package versions in `requirements.txt`, `requirements-dev.txt`, `requirements-doc.txt`, `requirements-min.txt` as needed. See https://requires.io/github/NeurodataWithoutBorders/pynwb/requirements/?branch=dev
-- [ ] Check legal file dates and information in `Legal.txt`, `license.txt`, `README.rst`, `docs/source/conf.py`, and any other locations as needed
+- [ ] Major and minor releases: Update package versions in `requirements.txt`, `requirements-dev.txt`,
+  `requirements-doc.txt`, `requirements-min.txt`, `setup.py` as needed
+  See https://requires.io/github/NeurodataWithoutBorders/pynwb/requirements/?branch=dev
+- [ ] Check legal file dates and information in `Legal.txt`, `license.txt`, `README.rst`, `docs/source/conf.py`,
+  and any other locations as needed
 - [ ] Update `setup.py` as needed
 - [ ] Update `README.rst` as needed
-- [ ] Update `src/pynwb/nwb-schema` submodule as needed. Check the version number manually.
+- [ ] Update `src/pynwb/nwb-schema` submodule as needed. Check the version number and commit SHA manually
 - [ ] Update changelog (set release date) in `CHANGELOG.md` and any other docs as needed
-- [ ] Run tests locally including gallery tests and validation tests, and inspect all warnings and outputs (`python test.py -v > out.txt`)
+- [ ] Run tests locally including gallery tests and validation tests, and inspect all warnings and outputs
+  (`python test.py -v > out.txt`)
 - [ ] Test docs locally (`make apidoc`, `make html`)
 - [ ] Push changes to this PR and make sure all PRs to be included in this release have been merged
 - [ ] Check that the readthedocs build for this PR succeeds (build latest to pull the new branch, then activate and
   build docs for new branch): https://readthedocs.org/projects/pynwb/builds/
 
 ### After merging:
 1. Create release by following steps in `docs/source/make_a_release.rst` or use alias `git pypi-release [tag]` if set up
-2. After the CI bot creates the new release (wait ~10 min), update the release notes on the [GitHub releases page](https://github.com/NeurodataWithoutBorders/pynwb/releases) with the changelog
+2. After the CI bot creates the new release (wait ~10 min), update the release notes on the
+   [GitHub releases page](https://github.com/NeurodataWithoutBorders/pynwb/releases) with the changelog
 3. Check that the readthedocs "latest" and "stable" builds run and succeed
-4. Update [conda-forge/pynwb-feedstock](https://github.com/conda-forge/pynwb-feedstock)
+4. Update [conda-forge/pynwb-feedstock](https://github.com/conda-forge/pynwb-feedstock) with the latest version number
+   and SHA256 retrieved from PyPI > PyNWB > Download Files > View hashes for the `.tar.gz` file. Re-render as needed

CHANGELOG.md

@@ -1,5 +1,12 @@
 # PyNWB Changelog
 
+## PyNWB 1.5.1 (May 24, 2021)
+
+## Bug fix:
+- Raise minimum version of pandas from 0.23 to 1.0.5 to be compatible with numpy 1.20, and raise minimum version of
+  HDMF to use the corresponding change in HDMF. @rly (#1363)
+- Update documentation and update structure of requirements files. @rly (#1363)
+
 ## PyNWB 1.5.0 (May 17, 2021)
 
 ### New features:

docs/source/software_process.rst

@@ -43,27 +43,24 @@ lines are covered by the tests.
 Requirement Specifications
 --------------------------
 
-There are 4 kinds of requirements specification in PyNWB.
-
-Setup.py Dependencies
----------------------
-
-There are 4 kinds of requirements specification in PyNWB.
+There are 5 kinds of requirements specification in PyNWB.
 
 The first one is the requirements-min.txt_ file, which lists the package dependencies and their minimum versions for
-installing PyNWB. These dependencies are read by setup.py_ into the `install_requires` key, with the adjustment that
-the `'=='` listed in `requirements-min.txt` are replaced with `'>='` to reflect that they are minimum versions.
+installing PyNWB.
 
-The second one is requirements.txt_ which contain a list of pinned (concrete) dependencies to reproduce
+The second one is requirements.txt_, which lists the pinned (concrete) dependencies to reproduce
 an entire development environment to use PyNWB.
 
-The third one is requirements-dev.txt_ which contain a list of pinned (concrete) dependencies to reproduce
+The third one is requirements-dev.txt_, which lists the pinned (concrete) dependencies to reproduce
 an entire development environment to use PyNWB, run PyNWB tests, check code style, compute coverage, and create test
 environments.
 
-The final one is requirements-doc.txt_ which contain a list of dependencies to generate the documentation for PyNWB.
+The fourth one is requirements-doc.txt_, which lists the dependencies to generate the documentation for PyNWB.
 Both this file and `requirements.txt` are used by ReadTheDocs_ to initialize the local environment for Sphinx to run.
 
+The final one is within setup.py_, which contains a list of package dependencies and their version ranges allowed for
+running PyNWB.
+
 In order to check the status of the required packages, requires.io_ is used to create a badge on the project
 README_. If all the required packages are up to date, a green badge appears.
 

docs/source/update_requirements.rst

@@ -46,7 +46,7 @@ the following scripts:
 
    cd pynwb
 
-   # Set the requirements file to update
+   # Set the requirements file to update: requirements-dev.txt or requirements-doc.txt
    target_requirements=requirements-dev.txt
 
    mkvirtualenv pynwb-requirements

requirements-min.txt

@@ -1,8 +1,7 @@
-# package dependencies and their minimum versions for installing PyNWB
-# the requirements here specify '==' for testing; setup.py replaces '==' with '>='
-h5py==2.9,<3  # support for setting attrs to lists of utf-8 added in 2.9
-hdmf==2.5.5,<3
-numpy==1.16,<1.21
-pandas==0.23,<2
-python-dateutil==2.7,<3
+# minimum versions of package dependencies for installing PyNWB
+h5py==2.9  # support for setting attrs to lists of utf-8 added in 2.9
+hdmf==2.5.6
+numpy==1.16
+pandas==1.0.5
+python-dateutil==2.7
 setuptools

requirements.txt

@@ -1,6 +1,6 @@
 # pinned dependencies to reproduce an entire development environment to use PyNWB
 h5py==2.10.0
-hdmf==2.5.5
+hdmf==2.5.6
 numpy==1.19.3
 pandas==1.1.5
 python-dateutil==2.8.1

setup.py

@@ -12,10 +12,14 @@
 
 schema_dir = 'nwb-schema/core'
 
-with open('requirements-min.txt', 'r') as fp:
-    # replace == with >= and remove trailing comments and spaces
-    reqs = [x.replace('==', '>=').split('#')[0].strip() for x in fp]
-    reqs = [x for x in reqs if x]  # remove empty strings
+reqs = [
+    'h5py>=2.9,<3',
+    'hdmf>=2.5.6,<3',
+    'numpy>=1.16,<1.21',
+    'pandas>=1.0.5,<2',
+    'python-dateutil>=2.7,<3',
+    'setuptools'
+]
 
 print(reqs)