nix store --realize returns invalid folder

[yawnt]

Describe the bug

When I run nix build or nix-store --realize $DRV the returned output path is not valid.
Maybe I'm doing something very wrong, but this is quite puzzling 🤔 . I would really appreciate any guidance! Thanks

Steps To Reproduce

1. Download https://github.com/yawnt/declarative-nix-flatpak/tree/bug
2. Run nix build
3. DRV=$(fgrep -o -H "keyboard.json" /nix/store/*PlexHTPC*drv | uniq | cut -d":" -f 1)
4. nix-store --realize $DRV
5. the above command will return a folder in /nix/store, if you try to ls that folder is is missing

Expected behavior

The path returned by nix-store --realize should be the correct install output.

nix-env --version output
nix-env (Nix) 2.11.0

Additional context

Interesting bit is that the path returned by nix-store --realize $DRV exists during the build (I can ls it), but it disappears afterwards.

# during build
tv.plex.PlexHTPC> ++ ls /nix/store/znw930rx83djssg6r2ks0rxjw979czgb-tv.plex.PlexHTPC
tv.plex.PlexHTPC> tv.plex.PlexHTPC

# after
$ ls /nix/store/znw930rx83djssg6r2ks0rxjw979czgb-tv.plex.PlexHTPC                                                                        
ls: cannot access '/nix/store/znw930rx83djssg6r2ks0rxjw979czgb-tv.plex.PlexHTPC': No such file or directory

# but i can find it in another directory in /nix/store
$ ls /nix/store/ykmdh1mkrv7w8pb1w4gg5xzciv737jmg-tv.plex.PlexHTPC:
tv.plex.PlexHTPC

The issue doesn't manifest if I remove patchPhase from the derivation in flake.nix.

[thufschmitt]

https://github.com/yawnt/declarative-nix-flatpak/blob/2b3ef443f1632a2883cc54f39a8734618dddc71e/flake.nix#L43 makes me err towards this being an instance of #4859 (although it should probably have failed 🤔 )

[yawnt]

Ah, interesting indeed! What's weird is that I had actually simplified the repro further by only building the app without the wrapper script. In that instance the result symlink was valid, and it contained the expected output. It just pointed to the "other" folder and not the one specified in the .drv file. Is it possible that this is what was happening there as well, since you mentioned seeing a valid result folder?

Did you happen to make any further progress in investigating #4859? I've been trying to figure out how to debug this one, but I am currently at a loss as I'm not intimately familiar with the codebase. Trying to fix that though 😄

Edit: just caught up with all the issues and related linked ones. it does look like it's the same type of bug, I suppose the only fix would be implementing what you described in the comment in #4859!

[thufschmitt]

Did you happen to make any further progress in investigating #4859?

Unfortunately not. There's a plan for a fix it (#4859 (comment)), but I never found the time to implement it.

[yawnt]

I can't really promise anything because I'm not very familiar with the codebase, but if you could update the pointers to the code (the ones here are outdated), I can look into giving it a shot?

[thufschmitt]

(the ones here are outdated)

That'll teach me to send non permalinks ;)

I've directly updated the original comment to fix the links.

[yawnt]

Hi,

i was taking a quick look at this. If I understand it correctly, the best way is to make sure that state.store->makeFixedOutputPath invoked while creating the derivation takes into account outputReferences. I noticed that the signature for makeFixedOutputPath is actually:

StorePath Store::makeFixedOutputPath(
    FileIngestionMethod method,
    const Hash & hash,
    std::string_view name,
    const StorePathSet & references,
    bool hasSelfReference) const

so wouldn't the change be to extract outputReferences from the JSON, convert those into a StorePathSet and then to pass those as references with hasSelfReference = true?

PS: I also feel like it would be good to potential print a warning or fail if the output path is different from the one in the drv, what do you think?

Thanks

Edit: on 2nd thought probably hasSelfReference is still false

[yawnt]

A simpler repro (I'm still investigating 😄 ):

flake.nix

{
  description = "A very basic flake";

  outputs = { self, nixpkgs }: {

    defaultPackage.x86_64-linux = 
      let
        pkgs = import nixpkgs { system = "x86_64-linux"; };
        src = pkgs.runCommand "src" {} "mkdir $out && touch $out/file;";
        dep = pkgs.stdenvNoCC.mkDerivation {
          pname = "potato";
          version = "0.0.1";
          builder = ./builder.sh;
          link = src;
          outputHashMode = "recursive";
          outputHash = "sha256-NNJr32jofEiAzb5V01+5zleWRul/5iznEKhIfWXu5Sk=";
        };
      in 
        pkgs.runCommand "list" {} "echo ${dep} > $out ";
  };
}

builder.sh

source $stdenv/setup

set -x

t=$(mktemp -d)

echo hello > $t/hello

ln -s $link $t/link

mkdir -p $out

mv $t/* $out/