Pure nix yarn.lock parsing

[bouk]

Hi there,

I've implemented a pure nix yarn.lock parser: https://gist.github.com/bouk/95f565b744168a3a0fcf396c1ebb805e

I was thinking this can be used in js2nix so it can work without having to generate and commit a yarn.lock.nix file. This would make the user experience a lot nicer since you'd only have to run yarn update to upgrade a version for example, without having to then run js2nix as well. Let me know if this is useful and if you need any help integrating it!

[olebedev]

Hi @bouk, wow, it looks awesome! 🙂

so it can work without having to generate and commit a yarn.lock.nix file. This would make the user experience a lot nicer since you'd only have to run yarn update to upgrade a version for example, without having to then run js2nix as well

Yeah, this I a really good developer experience I think and the js2nix supports it already. You might notice that there is yarn.lock.nix file in this repository and it must be committed into the repository because it is being used to bootstrap the tool itself - dependencies for the js2nix CLI are provided via the yarn.lock.nix file + js2nix Nix library. Other than that, there is no need to commit generated yarn.lock.nix files for the consumers of this project and the Nix expression can be purely (purity here is a requirement, see project's goals section here) generated on-fly. However, it is possible to manually pre-generate such files, if you want to avoid the Import From Derivation situation for some reason. For more details see the first section here.

The tool was implemented in JS initially for a single reason, to make the PoC asap and iterate over with a reasonable pace. I would love to see it implemented in Nix where possible/reasonable. I am not sure, however, if this is something that can be easily done in Nix language: besides a pure representation of a yarn.lock file the resulting Nix expression addressed the dependencies cycles problem, described here. I wouldn't have time re-implementing cycles hosting it in Nix somewhen in nearest future, but I would be happy to review and merge this changes.

A potentially interesting aspect of having on-fly yarn.lock file representation in Nix could be avoid IFD, but as long as any builtins.readFile operation is considered as IFD it doesn't really make a big difference.

Thanks for sharing your implementation, it's very interesting digging into reading through it!

[bouk]

Yes exactly, it'd like to avoid IFD whenever possible, performance being the main reason! The readFile would cause an IFD if the yarn.lock file was generated as well, so if you're packaging up your own code that should be fine.

I'll look at the cyclic dependency thing, it should be solvable. But I'm glad you're interested, I'll have to see if I can implement a function that generates the equivalent of the whole yarn.lock.nix file.

[olebedev]

The readFile would cause an IFD if the yarn.lock file was generated as well, so if you're packaging up your own code that should be fine.

Oh, you are right. I read it again and it seems as long as you readFile not from derivation (f drv) it should be fine.

Wow, that actually looks exciting if we can remove IFD from this tool. The IFD is not a blocker for us at the moment but it brings some complexity into pipelines and definitely slows down parallel Nix build execution. Thanks for your suggestion, I look forward to getting the change in 🙂

[bouk]

Thinking about it some more, I think the IFD would only appear if you are packaging an external source, that is you're using js2nix on a project fetched from GitHub or something—in that case you'd need to vendor the yarn.lock.