Sync with `call-flake.nix` and other tech debt, ideas

[roberth]

Nix itself also has an implementation of significant parts of the logic in flake-compat.

While this is somewhat necessary due to a few differing technical requirements, the overlap is currently swept under the rug instead of taken advantage of.

I would suggest to

• carefully factor parts of the flake-compat logic into library functions with few dependencies (perhaps including logic from allNodes)

• do the same for call-flake.nix

• compare, figure out the identical ones, and "mark" them as such,
  either:

  • a clearly marked section in the file
  • a separate file

  I have a slight preference for the latter. Nix can have a two-file call-flake, I'm sure.

Another thing to pay attention to is that we have an internal, hidden and rich allNodes structure that's not the typical _type = "flake"; view of the flakes, but allows sources to be accessed separately. I feel like that structure will have a larger role to play.
Possible the inputs should also be part of that, I don't know.

(Disclaimer, this is of course somewhat vibes based after having worked with these files before. Some of the latter ideas might not work out, idk)

[adrian-gierakowski]

can call-flake.nix which ships with nix be accessed from outside?

[roberth]

It can not, and that's a good thing for reducing the interface size of the in practice unpinnable and "uncomposable" dependency - which is Nix itself.

[adrian-gierakowski]

ok, so if we extract common functions in both nix and flake-compat, we could maybe run them through the same test suite?
then we'd have to re-implement some logic which is currently implanted in C++ in nix, like the locking logic needed to properly implement #76

As I mentioned here this would be quite a bit of work. Could you share your long term vision which could motivate this effort?

[adrian-gierakowski]

btw. I'm all for separating functions\polyfills into lib within flake-compat as it would make it easier to see the core algo

[roberth]

ok, so if we extract common functions in both nix and flake-compat, we could maybe run them through the same test suite?

That should absolutely be a goal.

we'd have to re-implement some logic which is currently implanted in C++ in nix

Not sure, but we could work with already-locked test files.
Also for the core logic, local path flake inputs will often be good enough, so that can be somewhat light-weight. OTOH those are kind of special, so we'll probably want some git+file:// setup too. (Highly similar to git+https)

quite a bit of work. Could you share your long term vision which could motivate this effort?

The refactoring is worthwhile for making the task of implementing changes like #76 and NixOS/nix#7730 easier.
Sharing the testing burden also helps with that.

[roberth]

we'd have to re-implement

Now I see this was in reference to #84, specifically the mapping between the inputs graph and nodes, whereas I was thinking of the action of writing out a lock file and what comes before that.

I think more concretely what this refactor means is that it's more focused on computations around an individual flake rather than this single mapAttrs over the nodes. So instead of a mapping from nodes to evaluated flakes, you have a mapping on the inputs graph.
The idea is that most feature can then be implemented in those terms, and nodes is only accessed for acquiring data about a flake, not mapped over.
This is then the responsibility of any flake that has a url or is the root flake. The more I think of it, the more it becomes like what I proposed here in NixOS/nix#7730 (comment).

[adrian-gierakowski]

Now I see this was in reference to #84, specifically the mapping between the inputs graph and nodes, whereas I was thinking of the action of writing out a lock file and what comes before that.

I think more concretely what this refactor means is that it's more focused on computations around an individual flake rather than this single mapAttrs over the nodes. So instead of a mapping from nodes to evaluated flakes, you have a mapping on the inputs graph.

so if I understand correctly then you are thinking about re-implementing a large chunk of what's currently handles in cpp nix to pure nix, since call-flake.nix is not concerned with writing out a lock file and does not have access to input graph but gets the lockfile handed to it on a platter

I'm going to read through NixOS/nix#7730 (comment) now and then we can align on some actions points. Would be good to flesh this out sometime before now and upcoming ocean sprint :)