Configure for conda install

[monocongo]

We already have a pip install working, do the same for Anaconda environments, so we can issue a command such as

$ conda install indices_python

Guidance on how this is done: http://conda-test.pydata.org/docs/build_tutorials/pkgs.html

[monocongo]

In order to install as a conda module we'll first need to make it available via PyPI, as mentioned here. This will require updating the setup.py and creating a .pypirc file, as well as both test and production PyPI registrations.

[monocongo]

Package now available (registered/uploaded) on PyPI.

Steps taken to make this happen:

Prerequisites

• Register accounts on pypi.org and test.pypi.org
• Install setuptools, twine, conda-build into the current conda environment

1. Move into the main/top-level directory for the project, hereafter referred to as $PROJECT_HOME

$ cd $PROJECT_HOME

2. Run setuptools to create a source tarball and binary wheel. These files will be created in the $PROJECT_HOME/dist directory.

$ python setup.py sdist bdist_wheel

3. Upload and register the package to the test PyPI repository using twine. In order to complete this on Windows you need to use the DOS console (cmd.exe), since there's an issue with the authentication step hanging when running within Cygwin or the Windows git client bash console (see this issue for details).

$ twine upload --repository-url https://test.pypi.org/legacy/ $PROJECT_HOME/dist/*
Uploading distributions to https://test.pypi.org/legacy/
Enter your username: monocongo
Enter your password:

4. Verify that the test upload/resistration was succesful well by checking the test PyPI repository using this URL: https://test.pypi.org/project/climate-indices/

5. Repeat the above twine command with test.pypi.org replaced with upload.pypi.org:

$ twine upload --repository-url https://upload.pypi.org/legacy/ $PROJECT_HOME/dist/*
Uploading distributions to https://upload.pypi.org/legacy/
Enter your username: monocongo
Enter your password:

6. The package should now show up here: https://pypi.org/project/climate-indices/

[monocongo]

The next part is to build the conda package so that climate_indices can be installed into a conda environment using the command conda install climate_indices. The below follows the instructions provided here.

1. Run conda skeleton to create a build recipe (YAML file, meta.yaml) for the package.

We'll change into our home directory and run a conda skeleton comand which fetches the package from PyPI, creates a directory with the name of the package (climate_indices), and then creates the YAML file in this directory.

$ cd
$ conda skeleton pypi climate_indices

2. Copy the standard conda build recipe scripts into the package's directory (~/climate_indices).

These two scripts are build1.sh (for Linux) and bld.bat (for Windows).

Now the directory ~/climate_indices contains a complete "recipe" for creating the conda package.

$ ll climate_indices/
total 6
-rw-r--r-- 1 DELL 197121  107 Sep 10 10:02 bld.bat
-rw-r--r-- 1 DELL 197121   81 Sep 10 10:02 build1.sh
-rw-r--r-- 1 DELL 197121 1088 Sep  9 15:29 meta.yaml

3. Run conda-build to create the conda package.

$ conda-build climate_indices

Now a binary distribution for the package as a *.bz2 file is available under the Miniconda/Anaconda distribution's build directory for the current environment (we're using an environment named packaging in this example) and current platform (win-64 in this example, since it was built on a Windows 10 system):

$ ls /c/home/miniconda3/envs/packaging/conda-bld/win-64/*.bz2
climate_indices-1.0.0-py37h39e3cac_0.tar.bz2

4. Test to confirm that the package can be installed from the local build repository

$ conda install --use-local climate_indices
$ conda list

*this step is not yet working on the Windows 10 system where this is being developed, see this issue for details.

5. Build the package for all platforms.

We create a directory to contain the builds for all platforms and convert the original build into builds for all conda-supported platforms.

$ mkdir ~/climate_indices_builds
$ export BUILDS_DIR=~/climate_indices_builds
$ conda convert -f --platform all /c/home/miniconda3/envs/packaging/conda-bld/win-64/climate_indices-1.0.0-py37h39e3cac_0.tar.bz2 -o $BUILDS_DIR

*this step is not yet working on the Windows 10 system where this is being developed, see this issue for details.

6. Upload to Anaconda.org

Install the Anaconda client, login using Anaconda Cloud credentials (create account here), and upload the binary file (the climate-indices-*.bz2 file built above).

$ conda install anaconda-client

$ anaconda login
Using Anaconda API: https://api.anaconda.org
Username: monocongo
Password:
monocongo's login successful

$ anaconda upload ..\conda-bld\win-64\climate_indices-1.0.0-py37h39e3cac_0.tar.bz2
Using Anaconda API: https://api.anaconda.org
Using "monocongo" as upload username
Processing '..\conda-bld\win-64\climate_indices-1.0.0-py37h39e3cac_0.tar.bz2'
Detecting file type...
File type is "conda"
Extracting conda package attributes for upload
Creating package "climate_indices"
Creating release "1.0.0"
Uploading file "monocongo/climate_indices/1.0.0/win-64/climate_indices-1.0.0-py37h39e3cac_0.tar.bz2"
 uploaded 61 of 61Kb: 100.00% ETA: 0.0 minutes
Upload complete

conda package located at:
https://anaconda.org/monocongo/climate_indices

Package is now available: https://anaconda.org/monocongo/climate_indices

[monocongo]

We can now install the package into conda using this command:

conda install -c monocongo climate_indices

Next step is to get this into conda-forge which is a more standard/public repository for Python/conda packages.

We'll start by following the instructions here.

[monocongo]

The first thing I've noticed is that we don't get our scripts included in the package when it installs, instead I only see the modules under the main climate_indices directory. For example:

$ winpty python
Python 3.7.0 (default, Jun 28 2018, 08:04:48) [MSC v.1912 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import climate_indices
>>> from climate_indices import indices
>>> import climate_indices.scripts
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'climate_indices.scripts'

There's probably a better way to layout the project which will enable having the scripts available as executables, so we can allow users to run the processing scripts at the command line, either as straight executables, i.e.

$ process_indices_grid <options>

or as a Python module:

$ python -m process_indices_grid <options>

The point here is to allow a straight install via pip or conda without requiring a source distribution download/clone from git.

This may be the simplest way to handle things.

[monocongo]

We should probably go through this guide before closing this issue.

One item that looks interesting is to use README.rst/MANIFEST.in.

[monocongo]

Some info related to errors experienced when performing the above is included in this StackOverflow question.