Flake input `git+file:./${submodule}` no longer works

[bryango]

Describe the bug

Flake input of a git submodule in a relative subdirectory git+file:./${submodule}

• works on 2.18.1 (nixpkgs.nixVersions.nix_2_18)
• but no longer works on 2.19.2 (nixpkgs.nixVersions.nix_2_19)

Update: it seems that the regression happens since #9061. An example of this flake ref:

https://github.com/bryango/cheznix/blob/c445c8c82697de2bde53c65e9acacb9790fcd54e/flake.nix#L20-L23

The breakage would be acceptable to me if there is any alternative syntax that works, but currently I could not find anything that can replace git+file:./${submodule}.

Steps To Reproduce

$ nix --version
nix (Nix) 2.19.2

## suppose we have `inputs.nixpkgs.url = git+file:./nixpkgs;`
$ nix flake update nixpkgs
fatal: '/nixpkgs' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
warning: could not read HEAD ref from repo at 'file://file:./nixpkgs', using 'master'
fatal: '/nixpkgs' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
error:
       … while updating the lock file of flake 'git+file:///home/$USER/.config/home-manager?ref=refs/heads/master&rev=c445c8c82697de2bde53c65e9acacb9790fcd54e'

       … while updating the flake input 'nixpkgs'

       … while fetching the input 'git+file://file:./nixpkgs'

       error: program 'git' failed with exit code 128

Expected behavior

I believe submodule sub-flake is quite common and git+file:./${submodule} should still work.

nix-env --version output

evaluating file '/derivation-internal.nix'
nix-env (Nix) 2.19.2
System type: x86_64-linux
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux
Features: gc, signed-caches
System configuration file: /etc/nix/nix.conf
User configuration files: /home/$USER/.config/nix/nix.conf:/etc/xdg/nix/nix.conf
Store directory: /nix/store
State directory: /nix/var/nix
Data directory: /nix/store/0k52aibl2flync375nfw9a0kdjzvrzrc-nix-2.19.2/share

Additional context

N/A

Priorities

Add 👍 to issues you find important.

[roberth]

In the upcoming release 2.20, the git fetcher is overhauled, with improved submodule support, but the submodule flag for the fetcher is not enabled by default.
@edolstra I vaguely recall that we'd enable submodules by default in fetchTree. Did we decide that?
Otherwise, we could solve this flake problem by having a flake meta-attribute and "re-fetch" themselves with changed flags such as submodules = true;.

[bryango]

I tried the latest static binary from hydra which is built from 8e865f3, and it has a similar regression:

$ latestStatic=/nix/store/x1vkrxsxakk9wj7x0sydwifqa22jvqpx-nix-static-x86_64-unknown-linux-musl-2.20.0pre20240106_8e865f3
$ nix copy --from https://cache.nixos.org "$latestStatic"
$ nix shell "$latestStatic" ## below is from within the nix shell

$ nix --version
nix (Nix) 2.20.0pre20240106_8e865f3

$ nix flake update nixpkgs
fatal: '/nixpkgs' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
warning: could not read HEAD ref from repo at 'file://file:./nixpkgs', using 'master'
error:
       … while updating the lock file of flake 'git+file:///home/$USER/.config/home-manager?ref=refs/heads/master&rev=c445c8c82697de2bde53c65e9acacb9790fcd54e'

       … while updating the flake input 'nixpkgs'

       … while fetching the input 'git+file://file:./nixpkgs'

       error: fetching 'refs/heads/master:refs/heads/master' from 'file://file:./nixpkgs': failed to resolve path 'file://file:./nixpkgs': No such file or directory

[bryango]

I tried a bisect using hydra static builds, and it seems that the breaking change is made:

• strictly after: https://hydra.nixos.org/build/237666981
• and before: https://hydra.nixos.org/build/238054861

... which narrows down the commits to this range: 301623f...d070d8b. Looks like it is indeed related to fetchTree.

Update: I think it is precisely:

• fetchTree cleanup #9061

Update: I found that reverting these lines fixes the issue:

• 856fe13#diff-4de86ddf9ae3dd43a49cc14cc56e2f6b29f51fde7a0603e3ab7c5cd34599b5d7L296-L308

But I need to understand why. Then I can make a PR to try to fix it.

[simonzkl]

Before I open a new issue, I also noticed that this broke:

nix build '.?ref=my-branch#my-package'

fatal: ambiguous argument 'my-branch#m': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
error:
       … while fetching the input 'git+file:///home/foo/project?ref=my-branch%23m'

       error: program 'git' failed with exit code 128

Is this the same issue by any chance?

It's possible to work around this issue with:

nix build '.?ref=my-branch&#my-package'

[bryango]

nix build '.?ref=my-branch#my-package'

fatal: ambiguous argument 'my-branch#m': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
error:
       … while fetching the input 'git+file:///home/foo/project?ref=my-branch%23m'

       error: program 'git' failed with exit code 128

Is this the same issue by any chance?

I think this is a different issue, though it is also related to flake URI parsing.

[bryango]

@roberth @fricklerhandwerk would you be interested in taking a look at #9897? 🥺 It is a super simple PR which fixes the issue for the 2.19 series (but not 2.20 yet, due to new complications from submodules & switching to libgit2).