You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* fix non-python packages passed via packagesExtra
* refactor examples and readme for 3.0.0
* add changelog for 3.0.0
* bump version to 3.0.0 and update nixpkgs and pypi-deps-db dependencies
Copy file name to clipboardExpand all lines: Changelog.md
+76Lines changed: 76 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,79 @@
1
+
# 3.0.0 (14 Oct 2020)
2
+
flakes pypi gateway, R support, new output formats, more packages for python 3.5/3.6, improved providers nixpkgs/wheel
3
+
4
+
### IMPORTANT NOTICE
5
+
The UI has been reworked. It is backward compatible with a few exceptions. Most importantly, when importing mach-nix, an attribute set must be passed. It can be empty. Example:
6
+
```nix
7
+
let
8
+
mach-nix = import (builtins.fetchGit {
9
+
url = "https://github.com/DavHau/mach-nix/";
10
+
ref = "refs/tags/3.0.0";
11
+
}) {
12
+
# optionally bring your own nixpkgs
13
+
# pkgs = import <nixpkgs> {};
14
+
15
+
# or specify the python version
16
+
# python = "python38";
17
+
};
18
+
in
19
+
...
20
+
```
21
+
22
+
### Features
23
+
- Flakes gateway to pypi. Get a shell with arbitrary python packages. Example:
- R support (experimental): R packages can be passed via `packagesExtra`. Mach-nix will setup rpy2 accordingly. See [usage example](https://github.com/DavHau/mach-nix/blob/master/examples.md#r-and-python).
46
+
47
+
- Non-python packages can be passed via `packagesExtra` to include them into the environment.
48
+
49
+
50
+
### Improvements
51
+
- rework the logic for inheriting dependencies from nixpkgs
52
+
- fixes.nix: allow alternative mod function signature with more arguments:
This will generate the python environment into `./env`. To activate it, execute:
78
77
```bash
@@ -99,7 +98,7 @@ You can call mach-nix directly from a nix expression
99
98
let
100
99
mach-nix = import (builtins.fetchGit {
101
100
url = "https://github.com/DavHau/mach-nix/";
102
-
ref = "refs/tags/2.4.1";
101
+
ref = "refs/tags/3.0.0";
103
102
});
104
103
in
105
104
mach-nix.mkPython {
@@ -113,31 +112,35 @@ mach-nix.mkPython {
113
112
find more examples under [./examples.md](/examples.md)
114
113
115
114
### Advanced
116
-
Mach-nix can be fine tuned with additional arguments by importing it via `builtins.fetchGit`. Examples can be found in [./examples.md](/examples.md). There are 4 different methods which can be invoked:
115
+
Mach-nix can be fine tuned with additional arguments. Examples can be found in [./examples.md](/examples.md).
116
+
117
+
Functions for building python environments:
117
118
1.**mkPython** - builds a python environment for a given `requirements.txt`.
118
-
1.**mkPythonShell** - returns the python environment suitable for nix-shell.
119
+
1.**mkPythonShell** - builds a python environment suitable for nix-shell.
120
+
1.**mkDockerImage** - builds a layered docker image containing a python environment.
121
+
1.**mkNixpkgs** - returns nixpkgs which is conform to the given requirements.
122
+
1.**mkOverlay** - returns an overlay function to make nixpkgs conform to the given requirements .
123
+
1.**mkPythonOverrides** - produces pythonOverrides to make python conform to the given requirements.
124
+
125
+
Functions for building python packages or applications:
119
126
1.**buildPythonPackage** - build a single python package from a source code while automatically detecting requirements.
120
127
1.**buildPythonApplication** - same as **buildPythonPackage**, but package will not be importable by other python packages.
121
128
122
129
**buildPythonPackage** and **buildPythonApplication** accept the same arguments like their equally named partners in nixpkgs, plus the arguments of **mkPython**. If name/version/requirements arguments are omitted, mach-nix attempts to detect them automatically. See [./examples.md](/examples.md).
123
130
124
-
**mkPython** and **mkPythonShell** take exactly the following arguments:
131
+
**mkPython** and all other **mk...** functions take exactly the following arguments:
125
132
126
133
#### Required Arguments:
127
134
-**requirements** (string): Text content of a typical `requirements.txt`.
128
135
129
136
#### Optional Arguments:
130
-
-**disable_checks** (bool): Disable tests wherever possible to decrease build time and failures due to nix incompatible tests
131
-
-**extra_pkgs** (list) Include packages which are not available from pypi. Can contain tarball-URLs, paths, or `mach-nix.buildPythonPackage` calls.
132
-
-**overrides_pre** (list): list of pythonOverrides to apply before the machnix overrides. Use this to include additional packages which can then be selected inside the `requirements`
133
-
-**overrides_post** (list): list of pythonOverrides to apply after the machnix overrides. Use this to fixup packages.
134
-
-**pkgs** (set): pass custom nixpkgs (20.03 or higher is required for wheel support). Base it on `mach-nix.nixpkgs.path` to avoid incompatibilities.
135
137
-**providers** (set): define provider preferences (see examples below)
136
-
-**pypi_deps_db_commit** (string): commit hash of a specific version of the dependency graph ([pypi-deps-db](https://github.com/DavHau/pypi-deps-db)).
137
-
-**pypi_deps_db_sha256** (string): sha256 hash obtained via `nix-prefetch-url --unpack https://github.com/DavHau/pypi-deps-db/tarball/<pypi_deps_db_commit>`
138
-
-**python** (set): select custom python version. To prevent compatibility issues, only take python packages from the nixpkgs version used by mach-nix. For example: `mach-nix.nixpkgs.python36`
139
-
-**_provider_defaults** (set): builtin provider defaults. Disable them by passing {}
138
+
-**packagesExtra** (list) Add extra packages. Can contain tarball-URLs or paths of python source code, packages built via `mach-nix.buildPythonPackage`, or R Packages.
140
139
-**_** (set): use underscore argument to easily modify arbitrary attributes of packages. For example to add built inputs use `_.{package}.builtInputs.add = [...]`. Or to remove patches use `_.{package}.patches = [...]`.
140
+
-**overridesPre** (list): (advanced) list of pythonOverrides to apply before the machnix overrides. Use this to include additional packages which can then be selected inside the `requirements`
141
+
-**overridesPost** (list): (advanced) list of pythonOverrides to apply after the machnix overrides. Use this to fixup packages.
142
+
-**tests** (bool): Whether to enable tests (default: false)
143
+
-**_providerDefaults** (set): builtin provider defaults. Disable them by passing {}
141
144
142
145
#### Configure Providers
143
146
**Providers** allow you to configure the origin for your packages on a granular basis.
Copy file name to clipboardExpand all lines: examples.md
+42-19Lines changed: 42 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -35,15 +35,22 @@ This page contains basic and advanced examples for using mach-nix inside a nix e
35
35
<!--te-->
36
36
37
37
### Import mach-nix
38
-
(every mach-nix expression should begin like this)
38
+
every mach-nix expression should begin like this:
39
39
```nix
40
40
let
41
41
mach-nix = import (builtins.fetchGit {
42
42
url = "https://github.com/DavHau/mach-nix/";
43
-
ref = "refs/tags/2.4.1";
43
+
ref = "refs/tags/3.0.0";
44
44
}) {
45
45
# optionally bring your own nixpkgs
46
46
# pkgs = import <nixpkgs> {};
47
+
48
+
# optionally specify the python version
49
+
# python = "python38";
50
+
51
+
# optionally update pypi data revision from https://github.com/DavHau/pypi-deps-db
52
+
# pypiDataRev = "some_revision";
53
+
# pypiDataSha256 = "some_sha256";
47
54
};
48
55
in
49
56
...
@@ -57,20 +64,26 @@ mach-nix.mkPython { # replace with mkPythonShell if shell is wanted
57
64
}
58
65
```
59
66
60
-
#### Include packages from arbitrary sources.
61
-
`extra_pkgs` accepts python packages built via `mach-nix.buildPythonPackage`. Alternatively, paths or URLs can be passed which are then automatically wrapped in a `mach-nix.buildPythonPackage` call.
67
+
#### Include extra packages.
68
+
`packagesExtra` accepts:
69
+
- python packages built via `mach-nix.buildPythonPackage`.
70
+
- R Packages from `nixpkgs.rPackages` (see R example further down)
71
+
- python package source trees as paths or derivations
72
+
- URLs pointing to a tarball archives containing a python source tree
"https://github.com/psf/requests/tarball/2a7832b5b06d" # from tarball url
67
79
./some/local/project # from local path
68
80
mach-nix.buildPythonPackage { ... }; # from package
69
81
];
70
82
}
71
83
```
72
-
Alternatively, if requirements are not needed, extra_pkgs can be passed directly to mkPython
84
+
Alternatively, if requirements are not needed, packagesExtra can be passed directly to mkPython
73
85
```nix
86
+
...
74
87
mach-nix.mkPython [
75
88
"https://github.com/psf/requests/tarball/2a7832b5b06d" # from tarball url
76
89
./some/local/project # from local path
@@ -79,37 +92,42 @@ mach-nix.mkPython [
79
92
```
80
93
81
94
### buildPythonPackage / buildPythonApplication
82
-
These functions can be used to manually build individual python modules or applications. Those can either be used directly, or fed as `extra_pkgs` of `mkPython`.
95
+
These functions can be used to manually build individual python modules or applications. Those can either be used directly, or fed as `packagesExtra` of `mkPython`.
83
96
Whenever `requirements` are not explicitly specified, they will be extracted automatically from the packages setup.py/setup.cfg. The same goes for the `name` and `version`.
For example the package web2ldap depends on another python package `ldap0` which fails to build because of missing dependencies.
154
173
```nix
174
+
...
155
175
with mach-nix.nixpkgs;
156
176
mach-nix.mkPython {
157
177
@@ -168,6 +188,7 @@ mach-nix.mkPython {
168
188
`imagecodecs` is available via wheel, but if one wants to build it from source, dependencies will be missing since there is no nixpkgs candidate available.
169
189
poetry2nix luckily maintains overrides for this package. They can be included into the mach-nix build like this.
170
190
```nix
191
+
...
171
192
mach-nix.mkPython rec {
172
193
173
194
requirements = ''
@@ -181,7 +202,7 @@ mach-nix.mkPython rec {
181
202
182
203
# Import overrides from poetry2nix
183
204
# Caution! Use poetry2nix overrides only in `overrides_post`, not `overrides_pre`.
Tensorflow from pypi does not provide any hardware optimization support. To get a SSE/AVX/FMA enabled version, set the provider for tensorflow to `nixpkgs`.
198
219
199
220
```nix
221
+
...
200
222
mach-nix.mkPython {
201
223
202
224
requirements = ''
@@ -213,6 +235,7 @@ This only works if the restrictions in `requirements.txt` allow for the tensorfl
213
235
### Tensorflow via wheel (newer versions, quicker builds)
214
236
Install recent tensorflow via wheel
215
237
```nix
238
+
...
216
239
mach-nix.mkPython {
217
240
218
241
requirements = ''
@@ -228,6 +251,7 @@ mach-nix.mkPython {
228
251
### Recent PyTorch with nixpkgs dependencies, and custom python
229
252
Recent pytorch version, Build dependencies from source
230
253
```nix
254
+
...
231
255
mach-nix.mkPython rec {
232
256
233
257
requirements = ''
@@ -241,21 +265,17 @@ mach-nix.mkPython rec {
241
265
# allow wheels only for torch
242
266
torch = "wheel";
243
267
};
244
-
245
-
# Select custom python version (Must be taken from pkgs with the overlay applied)
246
-
python = mach-nix.nixpkgs.python36;
247
268
}
248
269
```
249
270
250
271
## JupyterLab
251
272
252
273
### Starting point for a geospatial environment
253
274
```nix
275
+
...
254
276
let
255
277
pyEnv = mach-nix.mkPython rec {
256
278
257
-
python = "python37";
258
-
259
279
requirements = ''
260
280
jupyterlab
261
281
geopandas
@@ -280,10 +300,11 @@ mkShell rec {
280
300
```
281
301
282
302
## Docker
283
-
For every python environment a docker image is available via the `dockerImage` attribute of the `mkPython` result
303
+
Docker images can be built by using `mkDockerImage` instead of `mkPython`. It accepts the same arguments.
284
304
### JupyterLab Docker Image
285
305
Assuming the following expression under `./jupyter-docker.nix`:
0 commit comments