Fixed-output derivations with runtime dependencies

[ncfavier]

There are at least two (bad) reasons a fixed-output derivation (FOD) can, in principle, end up having a runtime dependency on Nix store paths, which cause different issues:

1. The output of the FOD just so happens to contain the store hash of a derivation that was used to build it. See mistral-vibe: init at 1.3.3 #474854 (comment) for an example of this happening in nixpkgs, which caused fetchFromGitHub to fail when fetching the nixpkgs source.
2. The FOD has a post-processing phase that causes store hashes to be embedded in the output. This is bad because the output of a FOD should be... fixed, and not dependent on whatever version of e.g. the stdenv was used to build it. For an example of this, see the motivation for fetchgit: inherit allowedRequisites in mkDerivation #177326, which added allowedRequisites = [] to some fetchers precisely to ban such cases. (I believe this is now obsolete because Nix forbids FODs to have any runtime dependencies.)

While both reasons are bad, the question is what to do when this happens anyway. For reason 2, the current behaviour of Nix (banning FODs with runtime dependencies) makes the most sense. However, for reason 1, it is too strict, as shown by the linked issue: in that case, the runtime dependency is accidental, so it would make more sense to assume that FODs can't have runtime dependencies and discard any dependencies that are found (e.g. using unsafeDiscardReferences).

I don't know how to solve both problems at once. How do we prevent accidental hash introductions in nixpkgs from causing fetchFromGitHub to fail, while also disallowing actual runtime dependencies in FODs? Maybe some heuristic like "if the FOD has a nontrivial postPatch or whatever phase, disallow references, otherwise discard them" could work, but that sounds hacky. Another idea would be to add a CI check in nixpkgs to detect hardcoded hashes, but then the question is which hashes should be banned. I don't see a way to detect just those hashes that might become runtime dependencies, and banning all valid Nix hashes is unfeasible. Maybe it would suffice to ban valid Nix hashes that are prefixed with /nix/store/?

What do people think? cc @roberth as you were involved in #177326.

[roberth]

A third case would be hypothetical: some sort of lock format or something that's designed around the Nix store. Not even flakes does this though.

The scenarios 1 and 2 seem complete to me. Disallowing or discarding references is the right thing to do.

Work is ongoing to integrate FOD checking into Hydra, so finding any breakage should become easier soon-ish.
See Relevant STF update on discourse

@Ericson2314 does it have a tracking issue?