Overriding packages for use with magix

[mgttlinger]

I'm prototyping a small Haskell application using magix. I modified an existing Haskell library and want magix to use it for my application.

Is there a way to get magix to include an overlay or otherwise override the package I want to use?

[dschrempf]

That's a good point.

It is possible by using a modified Nixpkgs repository, and provide it on the command line (see magix --help):

  -n,--nixpkgs-path NIXPKGS_PATH
                           Path of Nixpkgs repository to use (default: extracted
                           from '$NIX_PATH')

However, I did not implement this option to be used as a directive (i.e., #nixpkgs NIXPKGS_PATH).

Is this enough for you? Do you think we should implement a directive so we can change this directly in the script source files?

I think other options (like using overlays etc.) go against the keep it simple philosophy of this project. It is not meant to be a replacement of Nix the package manager :-P.

[mgttlinger]

I see the value in keeping things simple and you might be right that this is a level of complexity where a full blown nix build for my application might be called for.

As far as implementing a directive, my vision would rather have been an extension of the existing dependency specification, where instead of the package name, you have a kind of package expression that, e.g. allows you to get the dependency from a flake to allow github:mgttlinger/package#dependency instead of dependency. This would also enable an easy way to use dependencies that are not currently packaged in nixpkgs.

Whether this is viable and desired in magix is your judgement. I thank you for magix and the effort invested either way!

[dschrempf]

I like the idea using Flake-like URLs to specify dependencies. It will be a bit more involved to implement though. Feel free to contribute; I am leaving this ticket open for now. Maybe I find the time at some point to do this myself!

[mgttlinger]

Okay, I think I have a proof of concept implemented. It seems to work and didn't turn out to be that complicated although I'm sure there are some corner cases I'm not thinking of. If you have time to give feedback and suggestions, I will write documentation and create a PR: https://github.com/mgttlinger/magix/tree/flake-ref

[mgttlinger]

It's worth noting that one such corner case are forked nixpkgs flakes as there the packages do not live in the packages output but the legacyPackages

[mgttlinger]

Okay never mind it doesn't really work 😅
P.S.: It seems to work fine for bash but GHC doesn't seem to pick up the libraries included this way so it seems more work is needed but the idea works in principle. I guess the issue might be due to GHC versions needing to match between the flake and the GHC version used in the derivation magix constructs.

[mgttlinger]

Ok, GHC versions needing to match is indeed the issue here. It works nicely otherwise. I assume Python might have similar issues.

An added benefit from this feature is the possibility to specify local paths as flake references, which in my use case is great as I can iterate the changes to the forked library and my script using the changed library in parallel.

[dschrempf]

Thank you! I will have a look soon.

[dschrempf]

I've had a short look. I think the way you solve this is great. I do wonder a bit if we should use a proper Nix AST to build the derivations, and not a template. These expressions quickly become unwieldable.

Also, the error messages might be bad. What if the flake does not provide the package etc., we should probably have a look.

[mgttlinger]

W.r.t. the error messages if the package is missing, I'd argue that they are what a nix user is used to and if supplemented, e.g. by a logging line like "using flake {flake expression}", will probably be alright as far as debugging help needed to figure things out.

Template vs Nix AST: Tough call. I would lean towards the status quo as it is, but if things get more complicated due to ensuring GHC version match, I agree that the text-based simplicity has to be sacrificed in favour of an AST solution

[dschrempf]

I agree.

With respect to GHC version. I think you need the Haskell packages to be in the Haskell package set (otherwise you will get conficting dependencies). One can do this using overridePackages, or a proper overlay, and that is where things get really complicated.

[mgttlinger]

Yeah that makes sense.
I think there are multiple use cases of the feature demanding different solutions (in the Python and Haskell case, as Bash should just work as is):

1. Using libraries not in nixpkgs: They need to compile in the same package set with the same GHC, but don't need to be in the package set. This could be achieved by "simply" using the same nixpkgs version
2. Using modified libraries in nixpkgs: They need proper overriding, as they might also affect other dependencies, but could also work like 1 in case they do not affect other dependencies.

I will play around with generating overrides in this way next week and see how ugly this gets.

[mgttlinger]

Okay, I have an unpushed, uncleaned WIP version where it works, but it makes a few changes where I would like your opinion before polishing.

I changed the underlying nix expression to a flake build where I add all the flake-style dependencies as inputs and set their nixpkgs to follow the one magix uses. Then I create an overlay, adding/overriding all the packages of the flake inputs. Of course, this requires the underlying build to use flakes, so the feature has to be enabled on the user's system. I'd say the added complexity in the code base is reasonable, but I assume this adds a lot of complexity for the usage and might still be prone to version clashes. In addition it adds impurity as the referenced flakes might change but magix won't rebuild which might or might not be the intended function.

Would you say this is a viable solution?

[mgttlinger]

I also had a look at using hnix as nix AST but they don't currently implement the relevant flake related feature as far as I saw.

[dschrempf]

Thanks! I apologize for the delay, but time is a bit tight here at the moment. I didn't forget this issue!