Skip to content
This repository was archived by the owner on Aug 17, 2022. It is now read-only.

austin.md

Latest commit

 

History

History
87 lines (54 loc) · 4.09 KB

austin.md

File metadata and controls

87 lines (54 loc) · 4.09 KB
Raw

Instructor: Philip Austin (Earth, Ocean and Atmospheric Sciences, UBC)

Title: Parallelization in Python 3 with large datasets

Duration: 3 hours

Target audience: Researchers interested in Python programming on multiple core machines.

Level: Intermediate

Prerequisites:

Course plan:

  • The objective is to learn how to write shared-memory Python programs that make use of multiple cores on a single node. The tutorial will introduce several python modules that schedule operations and manage data to simplify multiprocessing with Python.
  1. Benchmarking parallel code
  2. Understanding the global interpreter lock (GIL)
  3. Multiprocessing and multithreading with joblib
  4. Checkpointing/restarting multiprocessor jobs
  5. Multithreaded file i/o with zarr and parquet
  6. Writing extensions that release the GIL:
    1. Using numba
    2. Using cython
    3. Using C++ and pybind11 with xtensor-python
  7. Using dask/xarray to analyze out-of-core datasets
  8. Visualizing parallelization with dask
  9. Setting up a conda-forge environment for parallel computing

Setup requirements:

For remote access to Westgrid

  • A laptop with the x2go client installed
  • An account on grex
  • .bashrc which adds python 3.6.1 and g++ 5.2 to the PATH

For your own laptop

I'd encourage you to do a local python install on your laptop, to make sure we don't get hit by bandwidth limitations or competition for cpus on grex nodes.

We will need python 3.6 from conda-forge. To get it:

  1. Download miniconda 3.6 from https://conda.io/miniconda.html

  2. When prompted, set the install directory as something like /Users/phil/mini36, with the install for just yourself, but ignore the installer warning and make this your default python. To reverse that decision it is easy to edit either your .bashrc/.bash_profile (for OSX) or your PATH environmental varialbe (for Windows 10) and give higher priority to some other python executable. It will make the following step easier, however if this conda-forge python is your default for the class and the install below

  3. Set conda-forge as your default repository and upgrade your python by doing the following. Open a bash terminal (OSX) or a cmd shell (Windows) and type:

      conda config --prepend channels conda-forge

    followed by:

      conda update --all

  4. Once the update completes, copy the contents of https://github.com/phaustin/parallel_python_course/blob/master/conda_packages.txt into a local file named conda_packages.txt and install those packages by typing:

     conda install --file conda_packages.txt

If the install fails

The odds of this going smoothly for everyone are obviously close to zero. If all else fails you can follow along using a browser to point to my html files, and get the jupyter notebooks working at your leisure after class. All the notebooks we work with during the tutorial will be available in the github repository, and from my bison account. Feel free to email me at paustin@eoas.ubc.ca with any questions.

For Thursday morning:

  1. bookmark the lecture notes at https://clouds.eos.ubc.ca/~phil/courses/parallel_python/

  2. clone the course git repository with the notebooks by typing:

    git clone https://github.com/phaustin/parallel_python_course.git

  3. My anaconda channel is phaustin