|
| 1 | + |
| 2 | +<!--ts--> |
| 3 | + * [Basic Usage in Nix Expressions:](#basic-usage-in-nix-expressions) |
| 4 | + * [mkPython / mkPythonShell](#mkpython--mkpythonshell) |
| 5 | + * [buildPythonPackage / buildPythonApplication](#buildpythonpackage--buildpythonapplication) |
| 6 | + * [buildPythonPackage from GitHub](#buildpythonpackage-from-github) |
| 7 | + * [Examples for Tensorflow / PyTorch](#examples-for-tensorflow--pytorch) |
| 8 | + * [Tensorflow with SSE/AVX/FMA support](#tensorflow-with-sseavxfma-support) |
| 9 | + * [Tensorflow via wheel (newer versions, quicker builds)](#tensorflow-via-wheel-newer-versions-quicker-builds) |
| 10 | + * [Recent PyTorch with nixpkgs dependencies, overlays, and custom python](#recent-pytorch-with-nixpkgs-dependencies-overlays-and-custom-python) |
| 11 | + * [Using overrides](#using-overrides) |
| 12 | + * [Fixing packages via overrides](#fixing-packages-via-overrides) |
| 13 | + * [Include poetry2nix overrides](#include-poetry2nix-overrides) |
| 14 | + |
| 15 | +<!-- Added by: grmpf, at: Sat 04 Jul 2020 12:18:42 PM UTC --> |
| 16 | + |
| 17 | +<!--te--> |
| 18 | + |
| 19 | + |
| 20 | +## Basic Usage in Nix Expressions: |
| 21 | +### mkPython / mkPythonShell |
| 22 | +build a python environment from a list of requirements |
| 23 | +```nix |
| 24 | +let |
| 25 | + mach-nix = import (builtins.fetchGit { |
| 26 | + url = "https://github.com/DavHau/mach-nix/"; |
| 27 | + ref = "2.1.0"; |
| 28 | + }); |
| 29 | +in mach-nix.mkPython { |
| 30 | + requirements = builtins.readFile ./requirements.txt; |
| 31 | +} |
| 32 | +``` |
| 33 | + |
| 34 | +### buildPythonPackage / buildPythonApplication |
| 35 | +Build a python package from its source code and a list of requirements |
| 36 | +```nix |
| 37 | +let |
| 38 | + mach-nix = import (builtins.fetchGit { |
| 39 | + url = "https://github.com/DavHau/mach-nix/"; |
| 40 | + ref = "2.1.0"; |
| 41 | + }); |
| 42 | +in mach-nix.buildPythonPackage { |
| 43 | + pname = "my-package"; |
| 44 | + version = "1.0.0"; |
| 45 | + src = /project-path; |
| 46 | + requirements = builtins.readFile /project-path/requirements.txt; |
| 47 | +} |
| 48 | +``` |
| 49 | + |
| 50 | +### buildPythonPackage from GitHub |
| 51 | +Build a python package from its source code and a list of requirements |
| 52 | +```nix |
| 53 | +let |
| 54 | + mach-nix = import (builtins.fetchGit { |
| 55 | + url = "https://github.com/DavHau/mach-nix/"; |
| 56 | + ref = "2.1.0"; |
| 57 | + }); |
| 58 | +in mach-nix.buildPythonPackage rec { |
| 59 | + pname = "projectname"; |
| 60 | + version = "1.0.0"; |
| 61 | + src = builtins.fetchGit{ |
| 62 | + url = "https://github.com/user/projectname"; |
| 63 | + ref = "master"; |
| 64 | + # rev = "put_commit_hash_here"; |
| 65 | + }; |
| 66 | + doCheck = false; |
| 67 | + doInstallCheck = false; |
| 68 | + requirements = builtins.readFile "${src}/requirements.txt"; |
| 69 | +} |
| 70 | +``` |
| 71 | + |
| 72 | +## Examples for Tensorflow / PyTorch |
| 73 | + |
| 74 | +### Tensorflow with SSE/AVX/FMA support |
| 75 | +I have a complex set of requirements including tensorflow. I'd like to have tensorflow with the usual nix features enabled like SSE/AVX/FMA which I cannot get from pypi. Therefore I must take tensorflow from nixpkgs. For everything else I keep the default, which means wheels are preferred. This allows for quicker installation of dependencies. |
| 76 | +```nix |
| 77 | +let |
| 78 | + mach-nix = import (builtins.fetchGit { |
| 79 | + url = "https://github.com/DavHau/mach-nix/"; |
| 80 | + ref = "2.1.0"; |
| 81 | + }); |
| 82 | +in mach-nix.mkPython { |
| 83 | +
|
| 84 | + requirements = '' |
| 85 | + # bunch of other requirements |
| 86 | + tensorflow |
| 87 | + ''; |
| 88 | +
|
| 89 | + providers = { |
| 90 | + # force tensorflow to be taken from nixpkgs |
| 91 | + tensorflow = "nixpkgs"; |
| 92 | + }; |
| 93 | +} |
| 94 | +``` |
| 95 | +This only works if the restrictions in `requirements.txt` allow for the tensorflow version from nixpkgs. |
| 96 | + |
| 97 | +### Tensorflow via wheel (newer versions, quicker builds) |
| 98 | +I'd like to install a more recent version of tensorflow which is not available from nixpkgs. Also I don't like long build times and therefore I want to install tensorflow via wheel. Usually most wheels work pretty well out of the box, but the tensorflow wheel has an issue which I need to fix with an override. |
| 99 | +```nix |
| 100 | +let |
| 101 | + mach-nix = import (builtins.fetchGit { |
| 102 | + url = "https://github.com/DavHau/mach-nix/"; |
| 103 | + ref = "2.1.0"; |
| 104 | + }); |
| 105 | +in mach-nix.mkPython { |
| 106 | +
|
| 107 | + requirements = '' |
| 108 | + # bunch of other requirements |
| 109 | + tensorflow == 2.2.0rc4 |
| 110 | + ''; |
| 111 | +
|
| 112 | + # no need to specify provider settings since wheel is the default anyways |
| 113 | +
|
| 114 | + # Fix the tensorflow wheel |
| 115 | + overrides_post = [( pythonSelf: pythonSuper: { |
| 116 | + tensorflow = pythonSuper.tensorflow.overridePythonAttrs ( oldAttrs: { |
| 117 | + postInstall = '' |
| 118 | + rm $out/bin/tensorboard |
| 119 | + ''; |
| 120 | + }); |
| 121 | + })]; |
| 122 | +} |
| 123 | +
|
| 124 | +``` |
| 125 | + |
| 126 | +### Recent PyTorch with nixpkgs dependencies, overlays, and custom python |
| 127 | +I'd like to use a recent version of Pytorch from wheel, but I'd like to build the rest of the requirements from sdist or nixpkgs, since I've already written overlays for those packages which I'd like to continue using. Also I require python 3.6 |
| 128 | +```nix |
| 129 | +let |
| 130 | + mach-nix = import (builtins.fetchGit { |
| 131 | + url = "https://github.com/DavHau/mach-nix/"; |
| 132 | + ref = "2.1.0"; |
| 133 | + }); |
| 134 | +overlays = []; # some very useful overlays |
| 135 | +in mach-nix.mkPython rec { |
| 136 | +
|
| 137 | + requirements = '' |
| 138 | + # bunch of other requirements |
| 139 | + torch == 1.5.0 |
| 140 | + ''; |
| 141 | +
|
| 142 | + providers = { |
| 143 | + # disallow wheels by default |
| 144 | + _default = "nixpkgs,sdist"; |
| 145 | + # allow wheels only for torch |
| 146 | + torch = "wheel"; |
| 147 | + }; |
| 148 | +
|
| 149 | + # Include my own overlay. |
| 150 | + pkgs = import <nixpkgs> { config = { allowUnfree = true; }; inherit overlays; }; |
| 151 | +
|
| 152 | + # Select custom python version (Must be taken from pkgs with the overlay applied) |
| 153 | + python = pkgs.python36; |
| 154 | +} |
| 155 | +``` |
| 156 | + |
| 157 | +## Using overrides |
| 158 | +### Fixing packages via overrides |
| 159 | +See previous example for tensorflow wheel |
| 160 | + |
| 161 | +### Include poetry2nix overrides |
| 162 | +I have a complex requirements.txt which includes `imagecodecs`. It is available via wheel, but I prefer to build everything from source. This package has complex build dependencies and is not available from nixpkgs. Luckily poetry2nix` overrides make it work. The peotry2nix overrides depend on nixpkgs-unstable. |
| 163 | +```nix |
| 164 | +let |
| 165 | + mach-nix = import (builtins.fetchGit { |
| 166 | + url = "https://github.com/DavHau/mach-nix/"; |
| 167 | + ref = "2.1.0"; |
| 168 | + }); |
| 169 | +in mach-nix.mkPython rec { |
| 170 | +
|
| 171 | + requirements = '' |
| 172 | + # bunch of other requirements |
| 173 | + imagecodecs |
| 174 | + ''; |
| 175 | +
|
| 176 | + providers = { |
| 177 | + _default = "sdist"; |
| 178 | + }; |
| 179 | +
|
| 180 | + # Use unstable channel |
| 181 | + pkgs = import <unstable> {}; |
| 182 | +
|
| 183 | + # Import overrides from poetry2nix |
| 184 | + # Caution! Use poetry2nix overrides only in `overrides_post`, not `overrides_pre`. |
| 185 | + overrides_post = [ |
| 186 | + ( |
| 187 | + import (builtins.fetchurl { |
| 188 | + url = "https://raw.githubusercontent.com/nix-community/poetry2nix/1cfaa4084d651d73af137866622e3d0699851008/overrides.nix"; |
| 189 | + }) { inherit pkgs; } |
| 190 | + ) |
| 191 | + ]; |
| 192 | +} |
| 193 | +``` |
0 commit comments