Performance bottleneck

[shajra]

Hi Nicolas,

For a few years, I've been maintaining my own Direnv script based on ideas from the Nix integration page on the Direnv wiki. In the meantime, the field of projects filling this space expanded a lot (Lorri, Sorri, Nixify, for example). So I took a moment to dive into each to decide if I should continue maintaining my own, or just use something maintained by someone else.

In the process of doing that, I found differences in performance with both Lorri and Sorri. The evaluation time for these were both at times around 2x the speed of just going into a Nix shell directly. That overhead isn't as perceptible with Lorri, because it runs as a background process. But it's a more noticeable hit for Sorri, when first building out an environment.

The slowdown seems to be almost entirely due to slipping in tracing on reading files with overrides. I have a toy Haskell project I was playing around with, and just entering a Nix Shell was taking me ~30 seconds (because of some Haskell.nix-based dependencies). That's pretty bad, but at least I have caching. The first-time hit is kind of annoying, though. And if changing files frequently, this could get pretty annoying. When I switched from lightweight auto-detection (no tracing) to full auto-detection, the only file I picked up for the extra 30 seconds of evaluation was the sources.json file from my usage of Niv (that tool is great, thanks for it). Over a minute of evaluation time is pretty extreme for a toy project (granted, Nix has terrible evaluation times, and Haskell.nix only stresses that).

I don't have a PR, but hopefully I've described the problem clearly enough. But that gets to something else I was hoping to run by you.

In all this exploration, I did end up iterating on my own personal project, and though I originally had no desire to rewrite something someone else has worked on, I think this is what I have ended up doing, just to prove out all the ideas I was thinking should be implemented.

Could you have a look at this project? https://github.com/shajra/direnv-nix-lorelei. I'd really appreciate your feedback. I think it has all the functionality Sorri does. The readme file has a small writeup comparing it with prior art. The idea I wanted to play around with was calling Lorri directly, rather than copying/pasting code from that project. I think I pulled that off okay. Also I rolled in a lot of features from all the other scripts that are out there. I did, though, deviate from your approach of scaffolding projects with a script called from .envrc. My approach to installation/configuration is more along the conventional lines of installing into ~/.config/direnv/lib. I do, though, use Nix to directly tie in all the dependencies needed by the script (so users can have confidence it will work upon installation).

There's so many of these projects out there. I empathize with new users confused by yet-another-option. So if you like what you see with Lorelei, I'm curious if you'd be open to joining forces behind one project.

[nmattia]

hey @shajra ,

Thanks for the message!

The evaluation time for these were both at times around 2x the speed of just going into a Nix shell directly.

@zimbatm told me over and over again that using the scopedImport hurts performance, quite a lot. Could that be it? What's the performance if you replace scopedImport with import in the sorri wrapper?

I did end up iterating on my own personal project,

So if I understand correctly, you do not track every single file that's used in the nix eval? How do you decide which files are tracked?

I empathize with new users confused by yet-another-option.

Same here... sorri is not even really out there yet, it was a side project at work that needed a repo 😅

---

I've skimmed through the lorelei readme, I have a few questions:

Additionally files to watch for cache invalidation can be automatically detected, though this detection is not comprehensive.

I'm guessing this is what you are talking about above, i.e. you approximate the set of files to track, compromising accuracy for performance ?

Rather than copying code from Lorri, we actually call Lorri code directly as a library.

I haven't had a look at the lorelei source yet, how do you actually call the lorri code as a library?

[zimbatm]

import memoizes the result, but not scopedImport. If you have multiple import <nixpkgs> it can become costly quite quickly.

IMO the best solution would be to instrument the Nix C++ code to emit all the files that it touched. I'm actually looking for a developer who could implement that feature at the moment.

[shajra]

hey @shajra ,

Thanks for the message!

The evaluation time for these were both at times around 2x the speed of just going into a Nix shell directly.

@zimbatm told me over and over again that using the scopedImport hurts performance, quite a lot. Could that be it? What's the performance if you replace scopedImport with import in the sorri wrapper?

I think I did that in the patch I use within Lorelei of Lorri (https://github.com/shajra/direnv-nix-lorelei/blob/user/shajra/next/nix/remove_trace.patch). But the way that I did it doesn't include the overrides, which means I don't get the tracing of extra files read when I do this patching.

In Lorelei, I give the users the option of using the patched version or not with a --auto-watch-deep switch.

So if I understand correctly, you do not track every single file that's used in the nix eval? How do you decide which files are tracked?

I have these options:

    -a --auto-watch-content  watch autodetected files for contect
			     changes
    -A --auto-watch-mtime    watch autodetected files for
			     modification times
    -d --auto-watch-deep     deeper searching for -a and -A options
    -w --watch-content PATH  watch a file's content for changes
    -W --watch-mtime PATH    watch a file's modification time

So you can auto-watch the build log if you like. And when you auto-watch, you can do it the slow way, or the fast way. And if you don't auto-watch at all, you can explicitly set files to watch as a user.

And just to give users more options, when you set files to watch, you can watch them for either true content changes (based on a content hash), or just to trigger a rebuild based on changes to modification time.

I think that covers every possibility I've seen in any of these scripts.

I've skimmed through the lorelei readme, I have a few questions:

Additionally files to watch for cache invalidation can be automatically detected, though this detection is not comprehensive.

I'm guessing this is what you are talking about above, i.e. you approximate the set of files to track, compromising accuracy for performance ?

Even if you do full-on Lorri-style auto-watching, it doesn't quite catch that Cabal files have changed, at least not for me it didn't. I even tried Lorri itself, if I recall.

Rather than copying code from Lorri, we actually call Lorri code directly as a library.

I haven't had a look at the lorelei source yet, how do you actually call the lorri code as a library?

Nothing too fancy as far as Nix goes. Lorri's daemon as a Rust daemon just does a process call to Nix CLI commands. And to support calling these commands, they have some Nix files in their code base. You might recognize these files, because it looks like you copied code directly from these files. So I just use your Niv project to get Lorri. And then I just import these files from that source. And I just figured out the calling convention by reading the Lorri source code (pretty sure I figured that out correctly, though I was hoping that a Lorri committer could have a look to verify that).

Here's the part of Lorelei where I do all this: https://github.com/shajra/direnv-nix-lorelei/blob/user/shajra/next/nix/default.nix#L26-L41

I was able to do this trick for all the parts of the Lorri code that was in Nix expressions. But as is probably obvious, I didn't have any good way to call logic within their Rust code without writing Rust myself (which was just a step too far for me). The code I had to replicate on my side was the code to comb through the Nix build log to find files to watch.

Otherwise I was able to get the real Lorri code to

• turn a user's Nix expression to one that I use to get a better environment to make a Nix shell from with richer GC root protection
• turn environment I get from this better expression into a nicely manicured environment for Direnv.