Version 2.0.0 (#20)

See Changelog.md for changes.

* adapt data interface for new sdist dependency graph format

* change wording in readme

* wheel support + concept providers

* overlay support WIP

* overlay support

* extras support for nixpkgs

* machnix as overrides

* granular provider settings + custom autoPatchelfHook

* improve overlay generator, remove prefer_new

* unify nixpkgs src

* update interface, bump version and examples

* fixes and updated docs

Author: DavHau

Changelog.md

@@ -0,0 +1,32 @@
+# 2.0.0
+## Features:
+ - **Python Wheels**: Wheels are supported for linux including manylinux1, manylinux2010, manylinux2014
+ - **Nixpkgs python packages**: Nixpkgs is now treated as a separate provider. That means packages can be taken directly from there, independent on their availability from pypi.
+ - **Configurable Providers**: The user can now freely decide where packages should be taken from either via a default or on a per package basis. Currently available provider are `nixpkgs`, `sdist`, `wheel`.
+ - **Core architecture - Python Overrides**: The core of mach nix now produces python overrides instead of nixpkgs overlays which makes it much more composible and allows for a less strange API.
+ - **Overlay support**: `mkPython` and lower level interfaces now accept new parameter `pkgs` and `python` which indirectly enables you to use any nixpkgs overlays (But don't use this for modifying python packages. Better do this with **Python Overrides**)
+ - **Python Overrides support**: `mkPython` now accepts new parameter `overrides_pre` and `overrides_post` each allowing to inject a list of python overrides to apply before and after the mach-nix internal overrides. This allows you to fixup packages or add your own modifications to them.
+ - **Extras support**: All providers now fully support python [extras](https://www.python.org/dev/peps/pep-0508/#extras). That means requirements like for example '`requests[security]`' will be correctly resolved.
+
+## Nix Interface changes:
+ - Removed:  
+    - **mkPythonExpr**: removed in favor of `machNixFile` and `machNix`
+ - Added:  
+    - **machNixFile** which generates a nex expresison
+    - **machNix** which evaluates the generated nix expression to get `overrides` and `select_pkgs`
+ - Changed:  
+    - **mkPython**:
+        - Removed arguments:
+            - `python_attr` in favor if `python`
+            - `prefer_nixpkgs` in favor of `providers`
+            - `nixpkgs_commit` in favor of `pkgs`
+            - `nixpkgs_tarball_sha256` in favor of `pkgs`
+        - Added arguments:
+            - `overrides_pre`: list of python overrides to apply before the machnix overrides
+            - `overrides_post`: list of python overrides to apply after the machnix overrides
+            - `pkgs`: pass custom nixpkgs. Only used for manylinux wheel dependencies.
+            - `providers`: define provider preferences
+            - `_provider_defaults`: builtin provider defaults. Disable them by passing {}
+
+# 1.0.0
+Initial Release

Readme.md

@@ -9,11 +9,11 @@ Mach-nix makes it easy to create and share reproducible python environments. Whi
 You can either install mach-nix via pip or by using nix in case you already have the nix package manager installed.
 ### Installing via pip
 ```shell
-pip install git+git://github.com/DavHau/mach-nix@1.0.0
+pip install git+git://github.com/DavHau/mach-nix@2.0.0
 ```
 ### Installing via nix
 ```shell
-nix-env -if https://github.com/DavHau/mach-nix/tarball/1.0.0 -A mach-nix
+nix-env -if https://github.com/DavHau/mach-nix/tarball/2.0.0 -A mach-nix
 ```
 
 ## Basic usage
@@ -44,8 +44,7 @@ If you are comfortable with writing nix expressions, you don't need to install t
 let
   mach-nix = import (builtins.fetchGit {
     url = "https://github.com/DavHau/mach-nix/";
-    ref = "master";
-    rev = "fa2bb2d33fb9b9dc3113046e4fcc16088f56981a";
+    ref = "2.0.0";
   });
 in
 mach-nix.mkPython {
@@ -70,7 +69,7 @@ mach-nix.mkPython {
 The general mechanism can be broken down into the following:
 
 ###  Dependency resolution
-Mach-nix contains a  dependency graph of nearly all python packages available on pypi.org. With this, mach-nix is able to do dependency resolution offline within seconds.
+Mach-nix contains a  dependency graph of nearly all python packages available on pypi.org. This allows mach-nix to resolve dependencies offline within seconds.
 
 The dependency graph data can be found here: https://github.com/DavHau/pypi-deps-db  
 The dependency graph is updated on a daily basis by this set of tools: https://github.com/DavHau/pypi-crawlers  
@@ -100,31 +99,34 @@ The overlay is generated by applying the following strategy to each required pyt
    - If no version of the required package is found in nixpkgs, the package is built from scratch by using `buildPythonPackage`.
 
 ## Advanced Usage (Nix Only)
-Mach-nix can be fine tuned with additional arguments by importing it via `builtins.fetchGit`. Examples can be found in [./example](./example/). There are 3 different methods which can be invoked:
-1. **mkPythonExpr** - returns a derivation containing a nix expression in $out/share/expr.nix which describes the python environment defined by your inputs
-1. **mkPython** - returns the python environment derivation. It basically just evaluates the expression coming from **mkPythonExpr**. 
+Mach-nix can be fine tuned with additional arguments by importing it via `builtins.fetchGit`. Examples can be found in [./example](./example/). There are 4 different methods which can be invoked:
+1. **machNixFile** - returns a single file derivation containing a nix expression in $out/share/mach_nix_file.nix which contains python overrides and a package selector.
+1. **MachNix** - evaluates **machNixFile** and returns `overrides` and `select_pkgs` which you can use to satisfy your requirements. 
+1. **mkPython** - builds a python environment for a given `requirements.txt`.
 1. **mkPythonShell** - returns the python environment suitable for nix-shell.
 
-All 3 methods take the following set of inputs which are processed in [./mach_nix/nix/expression.nix](./mach_nix/nix/expression.nix):
+Both **mkPython** and **mkPythonShell** take the following arguments:
 
 ### Required Arguments:
  - **requirements** (string): Text content of a typical `requirements.txt`.
 
 ### Optional Arguments:
- - **python_attr** (string): Select one of (python27, python35, python36, python37, python38). (default: python37)
- - **prefer_nixpkgs** (bool): Prefer python package versions from nixpkgs instead of newer ones. Decreases build time. (default: true)
- - **disable_checks** (bool): Disable tests wherever possible to decrease build time.
- - **nixpkgs_commit** (string): commit hash of nixpkgs version to use python packages from
- - **nixpkgs_tarball_sha256** (string): sha256 hash of the unpacked tarball for the selected nixpkgs version. (obtained via `nix-prefetch-url --unpack https://github.com/nixos/nixpkgs/tarball/<nixpkgs_commit>`)
+ - **disable_checks** (bool): Disable tests wherever possible to decrease build time and failures due to nix incompatible tests
+ - **overrides_pre** (list): list of pythonOverrides to apply before the machnix overrides
+ - **overrides_post** (list): list of pythonOverrides to apply after the machnix overrides
+ - **pkgs** (set): pass custom nixpkgs version (20.03 or higher is required for wheel support)
+ - **providers** (set): define provider preferences
  - **pypi_deps_db_commit** (string): commit hash of a specific version of the dependency graph ([pypi-deps-db](https://github.com/DavHau/pypi-deps-db)).
  - **pypi_deps_db_sha256** (string): sha256 hash obtained via `nix-prefetch-url --unpack https://github.com/DavHau/pypi-deps-db/tarball/<pypi_deps_db_commit>`
+ - **python** (set): select custom python to base overrides on. Should be from nixpkgs >= 20.03
+ - **_provider_defaults** (set): builtin provider defaults. Disable them by passing {}
+
 
 ## Contributing
 Contributions to this project are welcome in the form of GitHub PRs. If you are planning to make any considerable changes, you should first present your plans in a GitHub issue so it can be discussed.
 
 ## Limitations
- - Currently mach-nix does not provide any functionality which supports you in publishing python projects, like [Poetry](https://python-poetry.org/) does for example.  
- - Mach-nix does not yet support python packages in form of bdist_wheels, therefore some packages from pypi are not available.
+ - Currently mach-nix does not provide any functionality which supports you in publishing python projects, like [Poetry](https://python-poetry.org/) does for example.
 
 ## Alternative / Similar Software:
  - [Poetry](https://python-poetry.org/)

debug/build_overrides.nix

@@ -0,0 +1,15 @@
+let
+  nixpkgs_commit = builtins.readFile ../mach_nix/nix/NIXPKGS_COMMIT;  # nixpkgs version to use python packages from
+  nixpkgs_tarball_sha256 = builtins.readFile ../mach_nix/nix/NIXPKGS_SHA256;
+  pkgs = import (builtins.fetchTarball {
+    name = "nixpkgs";
+    url = "https://github.com/nixos/nixpkgs/tarball/${builtins.readFile ../mach_nix/nix/NIXPKGS_COMMIT}";
+    sha256 = "${builtins.readFile ../mach_nix/nix/NIXPKGS_SHA256}";
+  }) { config = {}; overlays = []; };
+  python = pkgs.python37;
+  manylinux1 = [ pkgs.pythonManylinuxPackages.manylinux1 ];
+  result = import ./overrides.nix;
+  overrides = result.overrides manylinux1 pkgs.autoPatchelfHook;
+  py = pkgs.python37.override { packageOverrides = overrides; };
+in
+py.withPackages (ps: result.select_pkgs ps)

debug/debug.py

@@ -1,39 +1,46 @@
+import json
 import os
 import subprocess as sp
 import tempfile
 from os.path import realpath, dirname
 
+import toml
+
 from mach_nix.generate import main
 
 pwd = dirname(realpath(__file__))
 
 os.environ['py_ver_str'] = '3.7.5'
-os.environ['out_file'] = f'{pwd}/expr.nix'
+os.environ['out_file'] = f'{pwd}/overrides.nix'
 os.environ['disable_checks'] = 'true'
-os.environ['prefer_nixpkgs'] = 'true'
+
+with open(pwd + "/../mach_nix/provider_defaults.toml") as f:
+    provider_settings = toml.load(f)
+provider_settings.update(dict(
+    _default="wheel,nixpkgs,sdist",
+))
+os.environ['providers'] = json.dumps(provider_settings)
 
 nixpkgs_json = tempfile.mktemp()
 cmd = f'nix-build {pwd}/nixpkgs-json.nix -o {nixpkgs_json}'
-print(cmd)
 sp.check_call(cmd, shell=True)
 os.environ['nixpkgs_json'] = nixpkgs_json
 
 pypi_deps_db = tempfile.mktemp()
-
 cmd = f'nix-build {pwd}/pypi-deps-db.nix -o {pypi_deps_db}'
 sp.check_call(cmd, shell=True)
-os.environ['pypi_deps_db_data_dir'] = pypi_deps_db + "/data"
+os.environ['pypi_deps_db_src'] = pypi_deps_db
 
-for key in ('NIXPKGS_COMMIT', 'NIXPKGS_TARBALL_SHA256'):
+for key in ('NIXPKGS_COMMIT', 'NIXPKGS_SHA256'):
     with open(f"{pwd}/../mach_nix/nix/{key}") as f:
         os.environ[key.lower()] = f.read()
 
-for key in ('PYPI_FETCHER_COMMIT', 'PYPI_FETCHER_TARBALL_SHA256'):
+for key in ('PYPI_FETCHER_COMMIT', 'PYPI_FETCHER_SHA256'):
     with open(f"{pypi_deps_db}/{key}") as f:
         os.environ[key.lower()] = f.read()
 
-os.environ['requirements'] = 'requests'
-
+with open(pwd + "/reqs.txt") as f:
+    os.environ['requirements'] = f.read()
 
 # generates and writes nix expression into ./debug/expr.nix
 main()

debug/nixpkgs-json.nix

@@ -1,6 +1,13 @@
 let
-  nixpkgs_src = (import ../mach_nix/nix/nixpkgs-src.nix).stable;
-  python = (import nixpkgs_src { config = {}; }).python37;
+  nixpkgs_commit = builtins.readFile ../mach_nix/nix/NIXPKGS_COMMIT;  # nixpkgs version to use python packages from
+  nixpkgs_tarball_sha256 = builtins.readFile ../mach_nix/nix/NIXPKGS_SHA256;
+  target_nixpkgs_src = builtins.fetchTarball {
+    name = "nixpkgs";
+    url = "https://github.com/nixos/nixpkgs/tarball/${nixpkgs_commit}";
+    sha256 = "${nixpkgs_tarball_sha256}";
+  };
+  #nixpkgs_src = (import ../mach_nix/nix/nixpkgs-src.nix).stable;
+  python = (import target_nixpkgs_src { config = {}; }).python37;
 in
-with import nixpkgs_src { config = {}; };
+with import target_nixpkgs_src { config = {}; overlays = []; };
 import ../mach_nix/nix/nixpkgs-json.nix { inherit pkgs python; }

debug/pypi-deps-db.nix

@@ -1,7 +1,7 @@
 let
   pkgs =  import (import ../mach_nix/nix/nixpkgs-src.nix).stable {};
   commit = builtins.readFile ../mach_nix/nix/PYPI_DEPS_DB_COMMIT;
-  sha256 = builtins.readFile ../mach_nix/nix/PYPI_DEPS_DB_TARBALL_SHA256;
+  sha256 = builtins.readFile ../mach_nix/nix/PYPI_DEPS_DB_SHA256;
   src = builtins.fetchTarball {
     name = "pypi-deps-db-src";
     url = "https://github.com/DavHau/pypi-deps-db/tarball/${commit}";

default.nix

@@ -1,9 +1,23 @@
 let
-  pkgs = import (import ./mach_nix/nix/nixpkgs-src.nix).stable { config = {}; };
+  pkgs = import (import ./mach_nix/nix/nixpkgs-src.nix) { config = {}; overlays = []; };
   python = import ./mach_nix/nix/python.nix { inherit pkgs; };
-  python_deps = (pkgs.lib.attrValues (import ./mach_nix/nix/python-deps.nix { inherit python; fetchurl = pkgs.fetchurl; }));
+  python_deps = (builtins.attrValues (import ./mach_nix/nix/python-deps.nix { inherit python; fetchurl = pkgs.fetchurl; }));
+  mergeOverrides = with pkgs.lib; overrides:
+    if length overrides == 0
+    then a: b: {}  # return dummy overrides
+    else
+      if length overrides == 1
+      then elemAt overrides 0
+      else
+        let
+          last = head ( reverseList overrides );
+          rest = reverseList (tail ( reverseList overrides ));
+        in
+          composeExtensions (mergeOverrides rest) last;
+  autoPatchelfHook = import ./mach_nix/nix/auto_patchelf_hook.nix {inherit (pkgs) fetchurl makeSetupHook writeText;};
 in
 rec {
+  # the mach-nix cmdline tool derivation
   mach-nix = python.pkgs.buildPythonPackage rec {
     pname = "mach-nix";
     version = builtins.readFile ./mach_nix/VERSION;
@@ -13,12 +27,47 @@ rec {
     doCheck = false;
   };
 
-  # call this to generate a nix expression defnining a python environment
-  mkPythonExpr = args: import ./mach_nix/nix/expression.nix args;
+  inherit mergeOverrides;
 
-  # call this to generate a python environment
-  mkPython = args: import "${mkPythonExpr args}/share/expr.nix";
+  # User might want to access it to choose python version
+  nixpkgs = pkgs;
+
+  # call this to generate a nix expression which contains the python overrides
+  machNixFile = args: import ./mach_nix/nix/mach.nix args;
+
+  # Returns `overrides` and `select_pkgs` which satisfy your requirements
+  machNix = args: import "${machNixFile args}/share/mach_nix_file.nix";
 
   # call this to use the python environment with nix-shell
   mkPythonShell = args: (mkPython args).env;
+
+  # (High level API) generates a python environment with minimal user effort
+  mkPython =
+    {
+      requirements,  # content from a requirements.txt file
+      disable_checks ? true,  # Disable tests wherever possible to decrease build time.
+      overrides_pre ? [],  # list of pythonOverrides to apply before the machnix overrides
+      overrides_post ? [],  # list of pythonOverrides to apply after the machnix overrides
+      pkgs ? nixpkgs,  # pass custom nixpkgs. Only used for manylinux wheel dependencies
+      providers ? {},  # define provider preferences
+      pypi_deps_db_commit ? builtins.readFile ./mach_nix/nix/PYPI_DEPS_DB_COMMIT,  # python dependency DB version
+      pypi_deps_db_sha256 ? builtins.readFile ./mach_nix/nix/PYPI_DEPS_DB_SHA256,
+      python ? pkgs.python3,  # select custom python to base overrides on. Should be from nixpkgs >= 20.03
+      _provider_defaults ? with builtins; fromTOML (readFile ./mach_nix/provider_defaults.toml)
+    }:
+    let
+      py = python.override { packageOverrides = mergeOverrides overrides_pre; };
+      result = machNix {
+        inherit requirements disable_checks providers pypi_deps_db_commit pypi_deps_db_sha256 _provider_defaults;
+        python = py;
+      };
+      overrides_machnix = result.overrides pkgs.pythonManylinuxPackages.manylinux1 autoPatchelfHook;
+      py_final = python.override { packageOverrides = mergeOverrides (
+        overrides_pre ++ [
+          overrides_machnix
+        ] ++ overrides_post
+      );};
+    in
+      py_final.withPackages (ps: result.select_pkgs ps)
+    ;
 }

example/advanced-usage.nix

@@ -1,7 +1,7 @@
 let
   mach-nix = import (builtins.fetchGit {
     url = "https://github.com/DavHau/mach-nix/";
-    ref = "1.0.0";
+    ref = "2.0.0";
   });
 in mach-nix.mkPythonExpr {
   requirements = builtins.readFile ./requirements.txt;

example/simple-usage-expression.nix

@@ -1,7 +1,7 @@
 let
   mach-nix = import (builtins.fetchGit {
     url = "https://github.com/DavHau/mach-nix/";
-    ref = "1.0.0";
+    ref = "2.0.0";
   });
 in mach-nix.mkPythonShell {
   requirements = builtins.readFile ./requirements.txt;