Nix segfaults if experimental feature `ca-derivations` is enabled

[doronbehar]

Describe the bug

I have this ~/.config/nix/nix.conf:

experimental-features = nix-command flakes ca-derivations ca-references
substituters = https://cache.nixos.org https://cache.ngi0.nixos.org/
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= cache.ngi0.nixos.org-1:KqH5CBLNSyX184S9BKZJo1LxrxJ9ltnY2uAs5c/f1MA=

And I get this error when I run any nix build or nix develop command:

[22 built, 7 copied (2.9 MiB), 1.5 MiB DL]terminate called after throwing an instance of 'nix::EndOfFile'
  what():  error: unexpected end-of-file
zsh: abort (core dumped)  nix build

Steps To Reproduce

1. Run nix develop inside a git checkout of https://github.com/doronbehar/pistol .
2. Experience the error mentioned above.

Backtrace:

#0  0x00007fdc93f04baa in raise () from /nix/store/s9qbqh7gzacs7h68b2jfmn9l6q4jwfjz-glibc-2.33-59/lib/libc.so.6
#1  0x00007fdc93eef523 in abort () from /nix/store/s9qbqh7gzacs7h68b2jfmn9l6q4jwfjz-glibc-2.33-59/lib/libc.so.6
#2  0x00007fdc9428d7ec in ?? () from /nix/store/lwkpcfphv5nwymi0bvnvfga6q4p72v8i-gcc-10.3.0-lib/lib/libstdc++.so.6
#3  0x00007fdc942987c6 in ?? () from /nix/store/lwkpcfphv5nwymi0bvnvfga6q4p72v8i-gcc-10.3.0-lib/lib/libstdc++.so.6
#4  0x00007fdc942978a9 in ?? () from /nix/store/lwkpcfphv5nwymi0bvnvfga6q4p72v8i-gcc-10.3.0-lib/lib/libstdc++.so.6
#5  0x00007fdc942981e1 in __gxx_personality_v0 () from /nix/store/lwkpcfphv5nwymi0bvnvfga6q4p72v8i-gcc-10.3.0-lib/lib/libstdc++.so.6
#6  0x00007fdc940a0da3 in _Unwind_RaiseException_Phase2 () from /nix/store/s9qbqh7gzacs7h68b2jfmn9l6q4jwfjz-glibc-2.33-59/lib/libgcc_s.so.1
#7  0x00007fdc940a1301 in _Unwind_RaiseException () from /nix/store/s9qbqh7gzacs7h68b2jfmn9l6q4jwfjz-glibc-2.33-59/lib/libgcc_s.so.1
#8  0x00007fdc94298ab7 in __cxa_throw () from /nix/store/lwkpcfphv5nwymi0bvnvfga6q4p72v8i-gcc-10.3.0-lib/lib/libstdc++.so.6
#9  0x00007fdc9448ec9d in nix::FdSource::readUnbuffered(char*, unsigned long) [clone .cold] () from /nix/store/p4l4jzknn5mxbvn0p10gwy8zz0qgby5n-nix-2.5.1/lib/libnixutil.so
#10 0x00007fdc944f6472 in nix::BufferedSource::read(char*, unsigned long) () from /nix/store/p4l4jzknn5mxbvn0p10gwy8zz0qgby5n-nix-2.5.1/lib/libnixutil.so
#11 0x00007fdc944f7668 in nix::Source::operator()(char*, unsigned long) () from /nix/store/p4l4jzknn5mxbvn0p10gwy8zz0qgby5n-nix-2.5.1/lib/libnixutil.so
#12 0x00007fdc9474bad1 in unsigned long nix::readNum<unsigned long>(nix::Source&) () from /nix/store/p4l4jzknn5mxbvn0p10gwy8zz0qgby5n-nix-2.5.1/lib/libnixstore.so
#13 0x00007fdc9483f1bc in nix::RemoteStore::Connection::processStderr(nix::Sink*, nix::Source*, bool) () from /nix/store/p4l4jzknn5mxbvn0p10gwy8zz0qgby5n-nix-2.5.1/lib/libnixstore.so
#14 0x00007fdc9483fe17 in nix::ConnectionHandle::processStderr(nix::Sink*, nix::Source*, bool) [clone .constprop.0] () from /nix/store/p4l4jzknn5mxbvn0p10gwy8zz0qgby5n-nix-2.5.1/lib/libnixstore.so
#15 0x00007fdc9484595e in nix::RemoteStore::queryRealisationUncached(nix::DrvOutput const&, nix::Callback<std::shared_ptr<nix::Realisation const> >) ()
   from /nix/store/p4l4jzknn5mxbvn0p10gwy8zz0qgby5n-nix-2.5.1/lib/libnixstore.so
#16 0x00007fdc9487d07d in nix::Store::queryRealisation(nix::DrvOutput const&, nix::Callback<std::shared_ptr<nix::Realisation const> >) ()
   from /nix/store/p4l4jzknn5mxbvn0p10gwy8zz0qgby5n-nix-2.5.1/lib/libnixstore.so
#17 0x00007fdc9487d67a in nix::Store::queryRealisation(nix::DrvOutput const&) () from /nix/store/p4l4jzknn5mxbvn0p10gwy8zz0qgby5n-nix-2.5.1/lib/libnixstore.so
#18 0x00007fdc9440e612 in nix::getBuiltPaths(nix::ref<nix::Store>, nix::ref<nix::Store>, std::vector<nix::DerivedPath, std::allocator<nix::DerivedPath> > const&)::{lambda(nix::DerivedPathBuilt const&)#2}::operator()(nix::DerivedPathBuilt const&) const () from /nix/store/p4l4jzknn5mxbvn0p10gwy8zz0qgby5n-nix-2.5.1/lib/libnixcmd.so
#19 0x00007fdc94400fa6 in nix::getBuiltPaths(nix::ref<nix::Store>, nix::ref<nix::Store>, std::vector<nix::DerivedPath, std::allocator<nix::DerivedPath> > const&) ()
   from /nix/store/p4l4jzknn5mxbvn0p10gwy8zz0qgby5n-nix-2.5.1/lib/libnixcmd.so
#20 0x00007fdc944016b8 in nix::build(nix::ref<nix::Store>, nix::ref<nix::Store>, nix::Realise, std::vector<std::shared_ptr<nix::Installable>, std::allocator<std::shared_ptr<nix::Installable> > > const&, nix::BuildMode) () from /nix/store/p4l4jzknn5mxbvn0p10gwy8zz0qgby5n-nix-2.5.1/lib/libnixcmd.so
#21 0x000055e5efac3f96 in CmdBuild::run(nix::ref<nix::Store>) ()
#22 0x00007fdc943f2060 in nix::StoreCommand::run() () from /nix/store/p4l4jzknn5mxbvn0p10gwy8zz0qgby5n-nix-2.5.1/lib/libnixcmd.so
#23 0x000055e5efb4ffcf in nix::mainWrapped(int, char**) ()
#24 0x00007fdc94a54228 in nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) ()
   from /nix/store/p4l4jzknn5mxbvn0p10gwy8zz0qgby5n-nix-2.5.1/lib/libnixmain.so
#25 0x000055e5efa22bb6 in main ()

Expected behavior

No segfault or crash.

nix-env --version output

nix-env (Nix) 2.5.1

Additional context

Have I miss-followed the instructions at https://discourse.nixos.org/t/content-addressed-nix-call-for-testers/12881 ?

cc @Ericson2314 .

[thufschmitt]

I suspect that might due to a bad interaction between the client who has these experimental features enabled and the daemon who doesn’t (because they are only specified in your local Nix config). Maybe try with the experimental-features set in /etc/nix/nix.conf?

(Not that it’s supposed to happen ofc. It should nicely degrade to a state without ca-derivations support. But testing all the possible combinations is somewhere between tricky and impossible so it might very possibly have gone through)

[doronbehar]

It seems to stop segfaulting, but I noticed now I get the warning: unknown experimental feature 'ca-references' when running nix commands.

[armeenm]

It seems to stop segfaulting, but I noticed now I get the warning: unknown experimental feature 'ca-references' when running nix commands.

761ac9d removed the option entirely so that's normal. You can safely remove it from your config as well.

[doronbehar]

OK so it'd be nice to update that old discourse post, now that ca-references was removed. I changed the title of this issue to reflect that the issue happens only if the user config enables ca-derivations and the nix-daemon doesn't have it enabled.

[armeenm]

OK so it'd be nice to update that old discourse post

Agreed. @thufschmitt

[thufschmitt]

it'd be nice to update that old discourse post, now that ca-references was removed

Indeed, thanks for reminding me of this. Should be good now

[doronbehar]

Apparently I still experience this issue with ca-derivations enabled only in /etc/nix/nix.conf :(. Checked and the backtrace is the same.

[Et7f3]

minimal repro here:
flake.lock:

{
  "nodes": {
    "flake-compat": {
      "flake": false,
      "locked": {
        "lastModified": 1650374568,
        "narHash": "sha256-Z+s0J8/r907g149rllvwhb4pKi8Wam5ij0st8PwAh+E=",
        "owner": "edolstra",
        "repo": "flake-compat",
        "rev": "b4a34015c698c7793d592d66adbab377907a2be8",
        "type": "github"
      },
      "original": {
        "owner": "edolstra",
        "repo": "flake-compat",
        "type": "github"
      }
    },
    "nixpkgs": {
      "locked": {
        "lastModified": 1660908602,
        "narHash": "sha256-SwZ85IPWvC4NxxFhWhRMTJpApSHbY1u4YK2UFWEBWvY=",
        "owner": "NixOS",
        "repo": "nixpkgs",
        "rev": "495b19d5b3e62b4ec7e846bdfb6ef3d9c3b83492",
        "type": "github"
      },
      "original": {
        "id": "nixpkgs",
        "ref": "nixos-unstable",
        "type": "indirect"
      }
    },
    "root": {
      "inputs": {
        "flake-compat": "flake-compat",
        "nixpkgs": "nixpkgs"
      }
    }
  },
  "root": "root",
  "version": 7
}

flake.nix:

{
  description = "Minimal repro of eof bug";

  inputs.nixpkgs.url = "nixpkgs/nixos-unstable";

  inputs.flake-compat = {
    url = "github:edolstra/flake-compat";
    flake = false;
  };

  outputs = { self, nixpkgs, ... }:
    let

      # Generate a version based on date
      version = builtins.substring 0 12 self.lastModifiedDate;
      vendorSha256 = "sha256-MMOEZ/DcRKulNAgGsZi5pm0cXQnNgGkeqIVSC98pm9c=";

      # System types to support.
      supportedSystems =
        [ "x86_64-linux" "x86_64-darwin" "aarch64-linux" "aarch64-darwin" "arm-linux" ];

      # Helper function to generate an attrset '{ x86_64-linux = f "x86_64-linux"; ... }'.
      forAllSystems = nixpkgs.lib.genAttrs supportedSystems;

      # Nixpkgs instantiated for supported system types.
      nixpkgsFor = forAllSystems (system: import nixpkgs { inherit system; });

    in {

      # Provide some binary packages for selected system types.
      packages = forAllSystems (system:
        let pkgs = nixpkgsFor.${system};
        in {
           eof = pkgs.hello;
        });
    };
}

$ nix --extra-experimental-features ca-derivations realisation info .#eof
this path will be fetched (0.05 MiB download, 0.22 MiB unpacked):
  /nix/store/mg35qkhk7wqbhhykpakds4fsm1riy8ga-hello-2.12.1
terminate called after throwing an instance of 'nix::EndOfFile'
  what():  error: unexpected end-of-file
Aborted (core dumped)

in /etc/nixos/configuration.nix:

  nix.extraOptions = ''                                                                                                                                                                                                
   experimental-features = nix-command flakes                                                                                                                                                                       
  '';

[thufschmitt]

Thanks for the repro @Et7f3 , I'll see whether I can reproduce it with it