error `/no-such-path` on NixOS

[avdv]

Trying to run nativelink on a NixOS system (or inside a Docker image created with nix' dockerTools) often fails to find executables.

For instance using

cpu_count: {
  query_cmd: "nproc",
},

fails with:

2025-11-14T15:22:21.945166Z  INFO nativelink_worker::worker_utils: Spawning process, cmd: "nproc", property_name: "cpu_count"
    at nativelink-worker/src/worker_utils.rs:66
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"

  2025-11-14T15:22:21.945372Z ERROR nativelink_worker::local_worker: Error, err: Error { code: NotFound, messages: ["No such file or directory (os error 2)", "Error executing property_name cpu_count command"] }
    at nativelink-worker/src/local_worker.rs:649
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"

Also trying to execute actions remotely fail with basically the same problem.

The reason for this is that commands are run using .env_clear(). On other systems that resets PATH to some default value (e.g. /usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:.) but on NixOS these directories mostly just don't exist and hence the default PATH is just /no-such-path.

For remote execution, I can work around by using

additional_environment: {
     PATH: {value: '${PATH}'},
},

But that only works since the value is expanded explicitly from the environment and the query_cmd is not. Using an absolute path to nproc is not easily possible, as it depends on the nix store path of coreutils (inside a Docker image).

Clearing the environment when running an action is the right thing to do, but probably not during configuration. Thoughts?

IMO, setting up PATH explicitly is better than depending on some default values and the tools being installed in there. So instead of a hacky way of doing things, maybe there could be an explicit configuration that also is taken into account when caching? (not sure that's possible)

[MarcusSorealheis]

@avdv Thanks for reporting this issue. We will certainly look into it. I'm not sure if commencing with nix develop with flakes and commands is obviated by NixOS because I do not use it, but did you start your session with this command?

Otherwise, I will try to dig in over the weekend to reproduce.

[avdv]

Hi @MarcusSorealheis thanks for the quick response!

I'm not sure if commencing with nix develop with flakes and commands is obviated by NixOS because I do not use it, but did you start your session with this command?

Although I use nix develop all the time on NixOS, not in this case. What I did was create a docker image using nix, based on the nativelink v0.7.6 docker image. I added bash and coreutils nix packages and set up the PATH environment variable of the image to include them. Here's a flake.nix which illustrates this and could be used to reproduce:

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs?ref=nixos-25.05";

  outputs = { self, nixpkgs }:
    let
      lib = nixpkgs.lib;
      pkgs = nixpkgs.legacyPackages.x86_64-linux;

      inherit (pkgs) coreutils dockerTools jq mkShell stdenvNoCC runCommand runtimeShell bash;

      nativelinkImage = dockerTools.pullImage {
        imageName = "ghcr.io/tracemachina/nativelink";
        imageDigest = "sha256:0bc4eb8c30646c30354af64aa0d5baaec6a4f230059d728caadf80221678b0d8";
        hash = "sha256-24Ok1f2TZ4rBiwQYJVnHg+mvoK4JCGitGfJRvBefqgc=";
        finalImageName = "ghcr.io/tracemachina/nativelink";
        finalImageTag = "v0.7.6";
      };

      # extract the path to the nativelink binary from the docker image above and
      # create a script to call it. This is used as the entrypoint for the
      # nativelinkDockerImage below.
      nativelink =
        runCommand "nativelink"
          {
            nativeBuildInputs = [ jq ];
            meta.mainProgram = "nativelink";
          }
          ''
            # Get config file name from manifest
            CONFIG=$( tar -xOf ${nativelinkImage} manifest.json | jq -r '.[0].Config' )

            # Extract entrypoint from config
            nl=$( tar -xOf ${nativelinkImage} "$CONFIG" | jq -r '.config.Entrypoint // empty | join(" ")' )

            mkdir -p "$out/bin"

            cat > "$out/bin/nativelink" <<EOF
            #!${runtimeShell}
            exec $nl "\$@"
            EOF

            chmod +x "$out/bin/nativelink"
          '';

      dockerImage = dockerTools.streamNixShellImage {
        name = "base";
        drv = mkShell.override { stdenv = stdenvNoCC; } {
          env = {
            PATH = lib.makeBinPath [
              bash
              coreutils
            ];
          };
          nativeBuildInputs = [
            bash
          ];
        };
        tag = "latest";
      };

      nativelinkDockerImage = dockerImage.override (o: {
        name = "base-nativelink";
        fromImage = nativelinkImage;
        config = o.config // {
          Entrypoint = [ (lib.getExe nativelink) ];
        };
      });
    in
    {
      apps.x86_64-linux.default = {
        program = "${nativelinkDockerImage}";
        type = "app";
        meta.description = "nix run | docker load";
      };
    };
}

One can use nix run | docker load (or podman load) in order to build and load the image to the local container storage.

That can be used to run nativelink in a container passing in a config file:

$ docker run -v $(pwd)/config.json5:/config.json5:ro localhost/base-nativelink /config.json5

It's only important that all paths in the config should be inside /build.

[palfrey]

IMO, setting up PATH explicitly is better than depending on some default values and the tools being installed in there. So instead of a hacky way of doing things, maybe there could be an explicit configuration that also is taken into account when caching? (not sure that's possible)

I've just been digging through all the cases of env_clear(). There are some security-related reasons to want to do that in general, although in most scenarios for Nativelink there's usually trust of people able to send commands to workers. Just removing it entirely would probably break some things that rely on that though.

Possible solution: have a config option for a allowlist of environment variables we don't wipe (default empty) that you could then add PATH to and we'd keep the Nix PATH whatever that's set to. This would avoid disrupting anyone relying on the current behaviour, while also allowing cases like yours. Thoughts @avdv?

[avdv]

Possible solution: have a config option for a allowlist of environment variables we don't wipe (default empty) that you could then add PATH to and we'd keep the Nix PATH whatever that's set to. This would avoid disrupting anyone relying on the current behaviour, while also allowing cases like yours. Thoughts @avdv?

Yes, that would work for me. (and feels less hacky 😄 )