Skip to content

Commit 83125d6

Browse files
rlyrly
authored
Add and test minimum requirements for PyNWB (#1148)
* Update reqs, add min reqs * Add minimum reqs testing on Circle and Azure * Update docs related to requirements files
1 parent 223d9f7 commit 83125d6

File tree

9 files changed

+134
-33
lines changed

9 files changed

+134
-33
lines changed

‎.circleci/config.yml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,6 +163,16 @@ jobs:
163163
- TEST_WHEELINSTALL_ENV: "wheelinstall-py37"
164164
<<: *ci-steps
165165

166+
python35-min-req:
167+
docker:
168+
- image: circleci/python:3.5.9-stretch
169+
environment:
170+
- TEST_TOX_ENV: "py35-min-req"
171+
- COVERAGE_TOX_ENV: ""
172+
- BUILD_TOX_ENV: "build-py35-min-req"
173+
- TEST_WHEELINSTALL_ENV: "wheelinstall-py35-min-req"
174+
<<: *ci-steps
175+
166176
miniconda35:
167177
docker:
168178
- image: continuumio/miniconda3:4.3.27p0
@@ -207,6 +217,13 @@ jobs:
207217
- TEST_TOX_ENV: "gallery-py37"
208218
<<: *gallery-steps
209219

220+
gallery36-min-req:
221+
docker:
222+
- image: circleci/python:3.6.3-jessie
223+
environment:
224+
- TEST_TOX_ENV: "gallery-py36-min-req"
225+
<<: *gallery-steps
226+
210227
deploy-dev:
211228
docker:
212229
- image: circleci/python:3.7.6-stretch
@@ -266,6 +283,8 @@ workflows:
266283
<<: *no_filters
267284
- python37:
268285
<<: *no_filters
286+
- python35-min-req:
287+
<<: *no_filters
269288
- miniconda35:
270289
<<: *no_filters
271290
- miniconda36:
@@ -276,17 +295,21 @@ workflows:
276295
<<: *no_filters
277296
- gallery37:
278297
<<: *no_filters
298+
- gallery36-min-req:
299+
<<: *no_filters
279300
- deploy-dev:
280301
requires:
281302
- flake8
282303
- python35
283304
- python36
284305
- python37
306+
- python35-min-req
285307
- miniconda35
286308
- miniconda36
287309
- miniconda37
288310
- gallery36
289311
- gallery37
312+
- gallery36-min-req
290313
filters:
291314
tags:
292315
ignore:
@@ -301,11 +324,13 @@ workflows:
301324
- python35
302325
- python36
303326
- python37
327+
- python35-min-req
304328
- miniconda35
305329
- miniconda36
306330
- miniconda37
307331
- gallery36
308332
- gallery37
333+
- gallery36-min-req
309334
filters:
310335
tags:
311336
only: /^[0-9]+(\.[0-9]+)*(\.post[0-9]+)?$/

‎azure-pipelines.yml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,14 @@ jobs:
2929
buildToxEnv: 'build-py35'
3030
testWheelInstallEnv: 'wheelinstall-py35'
3131

32+
macOS-py3.5-min-req:
33+
imageName: 'macos-10.13'
34+
pythonVersion: '3.5'
35+
testToxEnv: 'py35-min-req'
36+
coverageToxEnv: ''
37+
buildToxEnv: 'build-py35-min-req'
38+
testWheelInstallEnv: 'wheelinstall-py35-min-req'
39+
3240
Windows-py3.7:
3341
imageName: 'vs2017-win2016'
3442
pythonVersion: '3.7'
@@ -53,6 +61,14 @@ jobs:
5361
buildToxEnv: 'build-py35'
5462
testWheelInstallEnv: 'wheelinstall-py35'
5563

64+
Windows-py3.5-min-req:
65+
imageName: 'vs2017-win2016'
66+
pythonVersion: '3.5'
67+
testToxEnv: 'py35-min-req'
68+
coverageToxEnv: ''
69+
buildToxEnv: 'build-py35-min-req'
70+
testWheelInstallEnv: 'wheelinstall-py35-min-req'
71+
5672
pool:
5773
vmImage: $(imageName)
5874

‎docs/source/software_process.rst‎

Lines changed: 40 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -8,17 +8,20 @@ Software Process
88
Continuous Integration
99
----------------------
1010

11-
PyNWB is tested against Ubuntu, macOS and Windows operating systems.
11+
PyNWB is tested against Ubuntu, macOS, and Windows operating systems.
1212
The project has both unit and integration tests.
1313

14-
* CircleCI runs all PyNWB tests on Ubuntu
15-
* Appveyor runs all PyNWB tests on Windows
16-
* Travis runs all PyNWB tests on macOS
14+
* CircleCI_ runs all PyNWB tests on Ubuntu
15+
* `Azure Pipelines`_ runs all PyNWB tests on Windows and macOS
1716

18-
Each time a PR is published or updated, the project is built, packaged and tested on all support operating systems and python distributions. That way, as a contributor you know if you introduced regressions or coding style inconsistencies.
17+
Each time a PR is published or updated, the project is built, packaged, and tested on all supported operating systems
18+
and python distributions. That way, as a contributor, you know if you introduced regressions or coding style
19+
inconsistencies.
1920

2021
There are badges in the README_ file which shows the current condition of the dev branch.
2122

23+
.. _CircleCI: https://circleci.com/gh/NeurodataWithoutBorders/workflows/pynwb
24+
.. _Azure Pipelines: https://dev.azure.com/NeurodataWithoutBorders/pynwb/_build
2225
.. _README: https://github.com/NeurodataWithoutBorders/pynwb#readme
2326

2427

@@ -40,27 +43,38 @@ lines are covered by the tests.
4043
Requirement Specifications
4144
--------------------------
4245

43-
There are 2 kinds of requirements specification in PyNWB.
46+
There are 4 kinds of requirements specification in PyNWB.
4447

4548
Setup.py Dependencies
4649
---------------------
4750

48-
The first one is the dependencies_ in the `setup.py` file which lists the abstract dependencies for
49-
the PyNWB project. Note that there should not be specific versions of packages in the `setup.py` file.
51+
There are 4 kinds of requirements specification in PyNWB.
5052

51-
Requirements.txt Dependencies
52-
-----------------------------
53+
The first one is the requirements-min.txt_ file, which lists the package dependencies and their minimum versions for
54+
installing PyNWB. These dependencies are read by setup.py_ into the `install_requires` key, with the adjustment that
55+
the `'=='` listed in `requirements-min.txt` are replaced with `'>='` to reflect that they are minimum versions.
5356

54-
The second one is `requirements.txt` which contain a list of pinned (concrete) dependencies to reproduce
55-
an entire development environment to work with PyNWB.
57+
The second one is requirements.txt_ which contain a list of pinned (concrete) dependencies to reproduce
58+
an entire development environment to use PyNWB.
5659

57-
In order to check the status of the required packages requires.io_ is used to create a badge on the project
58-
README_. If all the required packages are up to date,
59-
a green badge appears.
60+
The third one is requirements-dev.txt_ which contain a list of pinned (concrete) dependencies to reproduce
61+
an entire development environment to use PyNWB, run PyNWB tests, check code style, compute coverage, and create test
62+
environments.
63+
64+
The final one is requirements-doc.txt_ which contain a list of dependencies to generate the documentation for PyNWB.
65+
Both this file and `requirements.txt` are used by ReadTheDocs_ to initialize the local environment for Sphinx to run.
66+
67+
In order to check the status of the required packages, requires.io_ is used to create a badge on the project
68+
README_. If all the required packages are up to date, a green badge appears.
6069

6170
If some of the packages are outdated, see :ref:`update_requirements_files`.
6271

63-
.. _dependencies: https://github.com/NeurodataWithoutBorders/pynwb/blob/dev/setup.py
72+
.. _requirements-min.txt: https://github.com/NeurodataWithoutBorders/pynwb/blob/dev/requirements-min.txt
73+
.. _setup.py: https://github.com/NeurodataWithoutBorders/pynwb/blob/dev/setup.py
74+
.. _requirements.txt: https://github.com/NeurodataWithoutBorders/pynwb/blob/dev/requirements.txt
75+
.. _requirements-dev.txt: https://github.com/NeurodataWithoutBorders/pynwb/blob/dev/requirements-dev.txt
76+
.. _requirements-doc.txt: https://github.com/NeurodataWithoutBorders/pynwb/blob/dev/requirements-doc.txt
77+
.. _ReadTheDocs: https://readthedocs.org/projects/pynwb/
6478
.. _requires.io: https://requires.io/github/NeurodataWithoutBorders/pynwb/requirements/?branch=dev
6579

6680

@@ -69,10 +83,16 @@ Versioning and Releasing
6983
-------------------------
7084

7185
PyNWB uses versioneer_ for versioning source and wheel distributions. Versioneer creates a semi-unique release
72-
name for the wheels that are created. It requires a version control system (git in PyNWB's case) to generate a release name.
73-
After all the tests pass, CircleCI creates both wheels(*.whl) and source distribution(*.tgz) for Python 3
74-
and uploads them back to GitHub as a release_. Versioneer makes it possible to get the source distribution from GitHub and create
75-
wheels directly without having to use a version control system because it hardcodes versions in the source distribution.
86+
name for the wheels that are created. It requires a version control system (git in PyNWB's case) to generate a release
87+
name. After all the tests pass, CircleCI creates both a wheel (\*.whl) and source distribution (\*.tar.gz) for Python 3
88+
and uploads them back to GitHub as a release_. Versioneer makes it possible to get the source distribution from GitHub
89+
and create wheels directly without having to use a version control system because it hardcodes versions in the source
90+
distribution.
91+
92+
It is important to note that GitHub automatically generates source code archives in .zip and .tar.gz formats and
93+
attaches those files to all releases as an asset. These files currently do not contain the submodules within PyNWB and
94+
thus do not serve as a complete installation. For a complete source code archive, use the source distribution generated
95+
by CircleCI, typically named `pynwb-{version}.tar.gz`.
7696

7797
.. _versioneer: https://github.com/warner/python-versioneer
7898
.. _release: https://github.com/NeurodataWithoutBorders/pynwb/releases

‎docs/source/update_requirements.rst‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,15 +10,17 @@ The different requirements files introduced in :ref:`software_process` section a
1010
* requirements.txt_
1111
* requirements-dev.txt_
1212
* requirements-doc.txt_
13+
* requirements-min.txt_
1314

1415
.. _requirements.txt: https://github.com/NeurodataWithoutBorders/pynwb/blob/dev/requirements.txt
1516
.. _requirements-dev.txt: https://github.com/NeurodataWithoutBorders/pynwb/blob/dev/requirements-dev.txt
1617
.. _requirements-doc.txt: https://github.com/NeurodataWithoutBorders/pynwb/blob/dev/requirements-doc.txt
18+
.. _requirements-min.txt: https://github.com/NeurodataWithoutBorders/pynwb/blob/dev/requirements-min.txt
1719

1820
requirements.txt
1921
================
2022

21-
`Requirements.txt` of the project can be created or updated and then captured using
23+
`requirements.txt` of the project can be created or updated and then captured using
2224
the following script:
2325

2426
.. code::
@@ -63,3 +65,12 @@ the following scripts:
6365
6466
deactivate
6567
rmvirtualenv pynwb-requirements
68+
69+
70+
requirements-min.txt
71+
====================
72+
73+
Minimum requirements should be updated manually if a new feature or bug fix is added in a dependency that is required
74+
for proper running of PyNWB. Minimum requirements should also be updated if a user requests that PyNWB be installable
75+
with an older version of a dependency, all tests pass using the older version, and there is no valid reason for the
76+
minimum version to be as high as it is.

‎requirements-dev.txt‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
codecov==2.0.15
2-
coverage==4.5.4
3-
flake8==3.7.8
4-
tox==3.14.0
5-
-r requirements.txt
2+
coverage==5.0.3
3+
flake8==3.7.9
4+
tox==3.14.3

‎requirements-doc.txt‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,4 @@ sphinx
22
matplotlib
33
sphinx_rtd_theme
44
sphinx-gallery
5-
Glymur==0.8.19 # required for allensdk 1.3.0 to work
6-
allensdk==1.3.0
7-
-r requirements.txt
5+
allensdk==1.3.2

‎requirements-min.txt‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# these minimum requirements specify '==' for testing; setup.py replaces '==' with '>='
2+
h5py==2.9 # support for setting attrs to lists of utf-8 added in 2.9
3+
hdmf==1.5.3,<2
4+
numpy==1.16
5+
pandas==0.23
6+
python-dateutil==2.7

‎setup.py‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
# -*- coding: utf-8 -*-
22

33
from setuptools import setup, find_packages
4-
import re
54

65
import versioneer
76

@@ -13,10 +12,10 @@
1312

1413
schema_dir = 'nwb-schema/core'
1514

16-
# copy requirements from requirements.txt, ignore all pinned version info, but keep pinned version for hdmf
17-
reqs_re = re.compile("[<=>]+")
18-
with open('requirements.txt', 'r') as fp:
19-
reqs = [reqs_re.split(x.strip())[0] if not x.startswith('hdmf') else x.strip() for x in fp.readlines()]
15+
with open('requirements-min.txt', 'r') as fp:
16+
# replace == with >= and remove trailing comments and spaces
17+
reqs = [x.replace('==', '>=').split('#')[0].strip() for x in fp]
18+
reqs = [x for x in reqs if x] # remove empty strings
2019

2120
print(reqs)
2221

‎tox.ini‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,14 @@ commands =
2727
coverage run test.py --pynwb
2828
coverage html -d tests/coverage/htmlcov
2929

30+
# Test with python 3.5, pinned dev reqs, and minimum run requirements
31+
[testenv:py35-min-req]
32+
basepython = python3.5
33+
deps =
34+
-rrequirements-dev.txt
35+
-rrequirements-min.txt
36+
commands = {[testenv]commands}
37+
3038
# Envs that builds wheels and source distribution
3139
[testenv:build]
3240
commands =
@@ -45,6 +53,13 @@ commands = {[testenv:build]commands}
4553
basepython = python3.7
4654
commands = {[testenv:build]commands}
4755

56+
[testenv:build-py35-min-req]
57+
basepython = python3.5
58+
deps =
59+
-rrequirements-dev.txt
60+
-rrequirements-min.txt
61+
commands = {[testenv:build]commands}
62+
4863
# Envs that will only be executed on CI that does coverage reporting
4964
[testenv:coverage]
5065
passenv = CODECOV_TOKEN
@@ -67,6 +82,10 @@ commands = python -c "import pynwb"
6782
deps = null
6883
commands = python -c "import pynwb"
6984

85+
[testenv:wheelinstall-py35-min-req]
86+
deps = null
87+
commands = python -c "import pynwb"
88+
7089
# Envs that will execute gallery tests
7190
[testenv:gallery]
7291
install_command =
@@ -89,3 +108,11 @@ commands = {[testenv:gallery]commands}
89108
basepython = python3.7
90109
deps = {[testenv:gallery]deps}
91110
commands = {[testenv:gallery]commands}
111+
112+
[testenv:gallery-py36-min-req]
113+
basepython = python3.6
114+
deps =
115+
-rrequirements-dev.txt
116+
-rrequirements-min.txt
117+
-rrequirements-doc.txt
118+
commands = {[testenv:gallery]commands}

0 commit comments

Comments
 (0)