Content-addressed derivation fails to build on aarch64-darwin

[reckenrode]

Content-addressed derivations fail to build on aarch64-darwin. If I build for --system x86_64-darwin, it gets past this point and fails in another place (which I suspect is not a Nix issue but possibly a nixpkgs one). The error message indicates that mkdir was killed. The system logs indicate that it was due to invalid codesigning.

Steps To Reproduce

1. Given the following flake:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-21.11";
    utils.url = "github:gytis-ivaskevicius/flake-utils-plus/v1.3.1";
  };

  outputs = inputs@{ self, nixpkgs, utils, ... }:
    utils.lib.mkFlake {
      inherit self inputs;
      channels.nixpkgs.config.contentAddressedByDefault = true;
      outputsBuilder = channels: {
        devShell =
          let
            inherit (channels.nixpkgs) mkShell;
          in
          mkShell {
            buildInputs = [ channels.nixpkgs.hello ];
          };
      };
  };
}

2. Run nix develop in a terminal; and
3. Wait for the build failure.

The following error should occur eventually.

$ nix develop
error: builder for '/nix/store/2j9xml2n5iyq0xi898855nryx79x0694-bootstrap-stage0-stdenv-darwin.drv' failed with exit code 137;
       last 1 log lines:
       > /nix/store/ckzrg0f0bdyx8rf703nc61r3hz5yys9q-builder.sh: line 7:  4752 Killed: 9               mkdir $out
       For full logs, run 'nix log /nix/store/2j9xml2n5iyq0xi898855nryx79x0694-bootstrap-stage0-stdenv-darwin.drv'.
error: 1 dependencies of derivation '/nix/store/xiwy4mqmxalqqsd95li2q8zwp9g482yz-apple-framework-CoreFoundation-11.0.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/4x2xdvvslix1swqfpmbai5wln0d5jc2v-bootstrap-stage0-binutils-wrapper-.drv' failed to build
error: 1 dependencies of derivation '/nix/store/8n3lk013mawprnkqnljrdjkq4hbmnc72-bootstrap-stage0-binutils-wrapper-.drv' failed to build
error: 1 dependencies of derivation '/nix/store/b5lxsx61qx0b5pfdh7bbblx6h5xsjay9-bootstrap-stage0-clang-wrapper-11.1.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/wnrvv1x16b82v089kc1r8qqgb472nm7h-signing-utils.drv' failed to build
error: 1 dependencies of derivation '/nix/store/743bn5n7vr3i9fqnzdawk8ymzszgkgn8-adv_cmds-119.drv' failed to build
error: 1 dependencies of derivation '/nix/store/iq3jzdm77p446373v85am48k3crb9yny-bash-5.1-p8.drv' failed to build
error: 1 dependencies of derivation '/nix/store/pqvldlwsnx9ajyqa5b44bjsdv056lg2c-bash-5.1-p8.drv' failed to build
error: 1 dependencies of derivation '/nix/store/rkbqck3zjpra48lxsc2n9i202m55qrmy-bootstrap-stage0-clang-wrapper-11.1.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/wm0pnx0p94lqmmxdwgi0470apvi8y4wv-bootstrap-stage0-clang-wrapper-11.1.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/07xc1bj3sd052vbpay96drszqrr2y4ki-bootstrap-stage1-stdenv-darwin.drv' failed to build
error: 1 dependencies of derivation '/nix/store/sjwm5yz4yhmc40wnl718a58pb8796jv8-bootstrap-stage1-stdenv-darwin.drv' failed to build
error: 1 dependencies of derivation '/nix/store/0j5rpmkspck04qhgknc4dprkszccz2hq-bootstrap-stage2-stdenv-darwin.drv' failed to build
error: 1 dependencies of derivation '/nix/store/dgb7a1c033qkwxvf39f06dhzgn2gkyhm-bootstrap-stage2-stdenv-darwin.drv' failed to build
error: 1 dependencies of derivation '/nix/store/gbkipahps4zgri7z6fw9xxbl9hl2b6jp-bootstrap-stage3-stdenv-darwin.drv' failed to build
error: 1 dependencies of derivation '/nix/store/l9ihjhisj5jkkzgfj55r3hsz7qnbz076-bootstrap-stage3-stdenv-darwin.drv' failed to build
error: 1 dependencies of derivation '/nix/store/xbhils85q8qmkp6wgbixihvzp26wnl2b-cmake-boot-3.21.2.drv' failed to build
error: 1 dependencies of derivation '/nix/store/k20ryspk1p8lds02d0x7sdspsy7k20wq-flex-2.6.4.drv' failed to build
error: 1 dependencies of derivation '/nix/store/xdgm84k4c88lahcpyh9qlxia5bplb2gg-gmp-6.2.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/vgrcvmanclyzp5h7579rsaray6gn09v2-nuke-references.drv' failed to build
error: 1 dependencies of derivation '/nix/store/hgwvb36i6m6v38il6hvc2174ypzp6v72-perl-5.34.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/dg9w64rys4nn35j7ykm76c858zyhs3pq-nix-shell-env.drv' failed to build

This is what shows up in the system logs when mkdir is killed.

default	22:29:55.505930-0500	analyticsd	Received event: com.apple.stability.crash {"exceptionCodes":"0x0000000000000032, 0x00000001006f8000(\n    50,\n    4302274560\n)EXC_BAD_ACCESSSIGKILL (Code Signature Invalid)UNKNOWN_0x32 at 0x00000001006f8000","incidentID":"834082B0-F989-40F3-A5A8-DD1FE06ECFDA","logwritten":1,"process":"mkdir","responsibleApp":"nix","terminationReasonExceptionCode":"0x2","terminationReasonNamespace":"CODESIGNING"}
default	22:29:55.506124-0500	analyticsd	Aggregated. Transform: StabilityCrashNumerator3WithBundleVersion Dirty: 1 Event: com.apple.stability.crash {"exceptionCodes":"0x0000000000000032, 0x00000001006f8000(\n    50,\n    4302274560\n)EXC_BAD_ACCESSSIGKILL (Code Signature Invalid)UNKNOWN_0x32 at 0x00000001006f8000","incidentID":"834082B0-F989-40F3-A5A8-DD1FE06ECFDA","logwritten":1,"process":"mkdir","responsibleApp":"nix","terminationReasonExceptionCode":"0x2","terminationReasonNamespace":"CODESIGNING","timestamp":1644377395504899}
default	22:29:55.506320-0500	analyticsd	Aggregated. Transform: StabilityCrashNumerator3 Dirty: 1 Event: com.apple.stability.crash {"exceptionCodes":"0x0000000000000032, 0x00000001006f8000(\n    50,\n    4302274560\n)EXC_BAD_ACCESSSIGKILL (Code Signature Invalid)UNKNOWN_0x32 at 0x00000001006f8000","incidentID":"834082B0-F989-40F3-A5A8-DD1FE06ECFDA","logwritten":1,"process":"mkdir","responsibleApp":"nix","terminationReasonExceptionCode":"0x2","terminationReasonNamespace":"CODESIGNING","timestamp":1644377395504899}
default	22:29:55.506488-0500	analyticsd	Aggregated. Transform: StabilityCrashNumerator3WithIncidentID Dirty: 1 Event: com.apple.stability.crash {"exceptionCodes":"0x0000000000000032, 0x00000001006f8000(\n    50,\n    4302274560\n)EXC_BAD_ACCESSSIGKILL (Code Signature Invalid)UNKNOWN_0x32 at 0x00000001006f8000","incidentID":"834082B0-F989-40F3-A5A8-DD1FE06ECFDA","logwritten":1,"process":"mkdir","responsibleApp":"nix","terminationReasonExceptionCode":"0x2","terminationReasonNamespace":"CODESIGNING","timestamp":1644377395504899}
default	22:29:55.507496-0500	analyticsd	Persisted Transform: StabilityCrashNumerator3 Dirty: 1
default	22:29:55.508203-0500	analyticsd	Persisted Transform: StabilityCrashNumerator3WithIncidentID Dirty: 1
default	22:29:55.512196-0500	analyticsd	Persisted Transform: StabilityCrashNumerator3WithBundleVersion Dirty: 1
default	22:29:55.512928-0500	analyticsd	budget cache persisted: <private>
default	22:29:55.513041-0500	analyticsd	[TransformManager::checkpoint] 60 transforms persisted. 0 failures.
default	22:29:55.985265-0500	analyticsd	Received event: com.apple.stability.crash {"exceptionCodes":"0x0000000000000001, 0x0000000000000000(\n    1,\n    0\n)EXC_BAD_ACCESSSIGSEGVKERN_INVALID_ADDRESS at 0x0000000000000000","incidentID":"2B2B1066-49DB-4031-B71D-167264D31A9A","logwritten":1,"process":"nix","responsibleApp":"nix","terminationReasonExceptionCode":"0xb","terminationReasonNamespace":"SIGNAL"}
default	22:29:55.985603-0500	analyticsd	Aggregated. Transform: StabilityCrashNumerator3WithBundleVersion Dirty: 1 Event: com.apple.stability.crash {"exceptionCodes":"0x0000000000000001, 0x0000000000000000(\n    1,\n    0\n)EXC_BAD_ACCESSSIGSEGVKERN_INVALID_ADDRESS at 0x0000000000000000","incidentID":"2B2B1066-49DB-4031-B71D-167264D31A9A","logwritten":1,"process":"nix","responsibleApp":"nix","terminationReasonExceptionCode":"0xb","terminationReasonNamespace":"SIGNAL","timestamp":1644377395984555}
default	22:29:55.985816-0500	analyticsd	Aggregated. Transform: StabilityCrashNumerator3 Dirty: 1 Event: com.apple.stability.crash {"exceptionCodes":"0x0000000000000001, 0x0000000000000000(\n    1,\n    0\n)EXC_BAD_ACCESSSIGSEGVKERN_INVALID_ADDRESS at 0x0000000000000000","incidentID":"2B2B1066-49DB-4031-B71D-167264D31A9A","logwritten":1,"process":"nix","responsibleApp":"nix","terminationReasonExceptionCode":"0xb","terminationReasonNamespace":"SIGNAL","timestamp":1644377395984555}
default	22:29:55.986102-0500	analyticsd	Aggregated. Transform: StabilityCrashNumerator3WithIncidentID Dirty: 1 Event: com.apple.stability.crash {"exceptionCodes":"0x0000000000000001, 0x0000000000000000(\n    1,\n    0\n)EXC_BAD_ACCESSSIGSEGVKERN_INVALID_ADDRESS at 0x0000000000000000","incidentID":"2B2B1066-49DB-4031-B71D-167264D31A9A","logwritten":1,"process":"nix","responsibleApp":"nix","terminationReasonExceptionCode":"0xb","terminationReasonNamespace":"SIGNAL","timestamp":1644377395984555}
default	22:29:55.988064-0500	analyticsd	Persisted Transform: StabilityCrashNumerator3 Dirty: 1
default	22:29:55.989104-0500	analyticsd	Persisted Transform: StabilityCrashNumerator3WithIncidentID Dirty: 1
default	22:29:55.991449-0500	analyticsd	Persisted Transform: StabilityCrashNumerator3WithBundleVersion Dirty: 1
default	22:29:55.992733-0500	analyticsd	budget cache persisted: <private>
default	22:29:55.992886-0500	analyticsd	[TransformManager::checkpoint] 60 transforms persisted. 0 failures.

Expected behavior

The expectation is that the build should succeed (or at least fail in the same place that it does on x86_64-darwin).

nix-env --version output

nix-env (Nix) 2.6.0

Additional context

n/a

[ConnorBaker]

I would like to add that I ran into the same issue with nix 2.7.0 on aarch64-darwin (macOS 12.3.1).

-------------------------------------
Translated Report (Full Report Below)
-------------------------------------

Process:               mkdir [24675]
Path:                  /Volumes/VOLUME/*/mkdir
Identifier:            mkdir
Version:               ???
Code Type:             ARM-64 (Native)
Parent Process:        bash [24674]
Responsible:           nix [23752]
User ID:               302

Date/Time:             2022-04-17 18:19:51.4829 -0400
OS Version:            macOS 12.3.1 (21E258)
Report Version:        12
Anonymous UUID:        FA07D672-5149-3553-BA9E-FF08385552A9

Sleep/Wake UUID:       665667E5-8375-4C66-AB44-9A53BC99895A

Time Awake Since Boot: 44000 seconds
Time Since Wake:       19536 seconds

System Integrity Protection: enabled

Crashed Thread:        0

Exception Type:        EXC_BAD_ACCESS (SIGKILL (Code Signature Invalid))
Exception Codes:       UNKNOWN_0x32 at 0x00000001009a4000
Exception Codes:       0x0000000000000032, 0x00000001009a4000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Reason:    Namespace CODESIGNING, Code 2 

VM Region Info: 0x1009a4000 is in 0x1009a4000-0x1009ac000;  bytes after start: 0  bytes before end: 32767
      REGION TYPE                    START - END         [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      mapped file                 100998000-1009a4000    [   48K] r--/rwx SM=COW  ...t_id=bbe5a9fb
--->  mapped file                 1009a4000-1009ac000    [   32K] r-x/rwx SM=COW  ...t_id=bbe5a9fb
      VM_ALLOCATE (reserved)      1009ac000-1009b0000    [   16K] rw-/rwx SM=NUL  ...(unallocated)

Thread 0 Crashed:
0   dyld                          	       0x100b44014 dyld3::MachOFile::isMachO(Diagnostics&, unsigned long long) const + 20
1   dyld                          	       0x100b252e0 dyld4::Loader::mapSegments(Diagnostics&, dyld4::RuntimeState&, char const*, unsigned long long, dyld4::Loader::CodeSignatureInFile const&, bool, dyld3::Array<dyld4::Loader::Region> const&, bool, bool, dyld4::Loader::FileValidationInfo const&) + 1096
2   dyld                          	       0x100b252e0 dyld4::Loader::mapSegments(Diagnostics&, dyld4::RuntimeState&, char const*, unsigned long long, dyld4::Loader::CodeSignatureInFile const&, bool, dyld3::Array<dyld4::Loader::Region> const&, bool, bool, dyld4::Loader::FileValidationInfo const&) + 1096
3   dyld                          	       0x100b2ab6c invocation function for block in dyld4::JustInTimeLoader::makeJustInTimeLoaderDisk(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, bool, unsigned int) + 68
4   dyld                          	       0x100b2a50c dyld4::JustInTimeLoader::withRegions(dyld3::MachOAnalyzer const*, void (dyld3::Array<dyld4::Loader::Region> const&) block_pointer) + 292
5   dyld                          	       0x100b2aac0 invocation function for block in dyld4::JustInTimeLoader::makeJustInTimeLoaderDisk(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, bool, unsigned int) + 480
6   dyld                          	       0x100b2fd3c dyld4::SyscallDelegate::withReadOnlyMappedFile(Diagnostics&, char const*, bool, void (void const*, unsigned long, bool, dyld4::FileID const&, char const*) block_pointer) const + 132
7   dyld                          	       0x100b2a8ac dyld4::JustInTimeLoader::makeJustInTimeLoaderDisk(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, bool, unsigned int) + 204
8   dyld                          	       0x100b246e4 invocation function for block in dyld4::Loader::getLoader(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&) + 1384
9   dyld                          	       0x100b24b54 invocation function for block in dyld4::Loader::forEachResolvedAtPathVar(dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, dyld4::ProcessConfig::PathOverrides::Type, bool&, void (char const*, dyld4::ProcessConfig::PathOverrides::Type, bool&) block_pointer) + 320
10  dyld                          	       0x100b11f98 dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 168
11  dyld                          	       0x100b4f6e0 dyld3::MachOAnalyzer::forEachRPath(void (char const*, bool&) block_pointer) const + 128
12  dyld                          	       0x100b23b8c dyld4::Loader::forEachResolvedAtPathVar(dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, dyld4::ProcessConfig::PathOverrides::Type, bool&, void (char const*, dyld4::ProcessConfig::PathOverrides::Type, bool&) block_pointer) + 708
13  dyld                          	       0x100b23800 invocation function for block in dyld4::Loader::forEachPath(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, void (char const*, dyld4::ProcessConfig::PathOverrides::Type, bool&) block_pointer) + 148
14  dyld                          	       0x100b19ec4 dyld4::ProcessConfig::PathOverrides::forEachPathVariant(char const*, dyld3::Platform, bool, bool&, void (char const*, dyld4::ProcessConfig::PathOverrides::Type, bool&) block_pointer) const + 672
15  dyld                          	       0x100b23754 dyld4::Loader::forEachPath(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, void (char const*, dyld4::ProcessConfig::PathOverrides::Type, bool&) block_pointer) + 172
16  dyld                          	       0x100b23f74 dyld4::Loader::getLoader(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&) + 864
17  dyld                          	       0x100b28b44 invocation function for block in dyld4::JustInTimeLoader::loadDependents(Diagnostics&, dyld4::RuntimeState&, dyld4::Loader::LoadOptions const&) + 380
18  dyld                          	       0x100b45270 invocation function for block in dyld3::MachOFile::forEachDependentDylib(void (char const*, bool, bool, bool, unsigned int, unsigned int, bool&) block_pointer) const + 148
19  dyld                          	       0x100b11f98 dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 168
20  dyld                          	       0x100b450b8 dyld3::MachOFile::forEachDependentDylib(void (char const*, bool, bool, bool, unsigned int, unsigned int, bool&) block_pointer) const + 172
21  dyld                          	       0x100b288ac dyld4::JustInTimeLoader::loadDependents(Diagnostics&, dyld4::RuntimeState&, dyld4::Loader::LoadOptions const&) + 164
22  dyld                          	       0x100b155bc dyld4::prepare(dyld4::APIs&, dyld3::MachOAnalyzer const*) + 1092
23  dyld                          	       0x100b15068 start + 484


Thread 0 crashed with ARM Thread State (64-bit):
    x0: 0x00000001009a4000   x1: 0x000000016f589698   x2: 0x0000000000008000   x3: 0x0000000000040012
    x4: 0x0000000000000003   x5: 0x0000000000000000   x6: 0x0000000000000000   x7: 0x0000000000000000
    x8: 0x0000000100b8862c   x9: 0x0000000100b89ea8  x10: 0x000000001e000000  x11: 0x0800000000008000
   x12: 0x0000000000000001  x13: 0x0000000000000001  x14: 0x00000000000006c0  x15: 0x0000000000000000
   x16: 0x00000000000000c5  x17: 0x6ae100016f588eb8  x18: 0x0000000000000000  x19: 0x000000016f589698
   x20: 0x0000000100894060  x21: 0x000000000000c000  x22: 0x0000000000000003  x23: 0x000000016f589188
   x24: 0x0000000000000020  x25: 0x0000000000000000  x26: 0x000000016f588f4c  x27: 0x0000000000000000
   x28: 0x00000001009a4000   fp: 0x000000016f588b10   lr: 0xf053800100b252e0
    sp: 0x000000016f588b00   pc: 0x0000000100b44014 cpsr: 0x00001000
   far: 0x00000001009a4000  esr: 0x92000007 (Data Abort) byte read Translation fault

Binary Images:
       0x100b10000 -        0x100b6ffff dyld (*) <fbb89662-e6f2-3434-b542-f75185ac5e74> /usr/lib/dyld
               0x0 - 0xffffffffffffffff ??? (*) <00000000-0000-0000-0000-000000000000> ???

External Modification Summary:
  Calls made by other processes targeting this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by all processes on this machine:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=704K resident=0K(0%) swapped_out_or_unallocated=704K(100%)
Writable regions: Total=9232K written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=9232K(100%)

                                VIRTUAL   REGION 
REGION TYPE                        SIZE    COUNT (non-coalesced) 
===========                     =======  ======= 
STACK GUARD                       56.0M        1 
Stack                             8176K        1 
VM_ALLOCATE                         16K        1 
VM_ALLOCATE (reserved)              16K        1         reserved VM address space (unallocated)
__DATA                              32K        2 
__DATA_CONST                       112K        2 
__LINKEDIT                         240K        2 
__TEXT                             464K        2 
dyld private memory               1024K        1 
mapped file                         80K        2 
===========                     =======  ======= 
TOTAL                             65.9M       15 
TOTAL, minus reserved VM space    65.9M       15 

[ConnorBaker]

I was able to reproduce with this minimal example by running both nix build and nix develop:

{
  description = "A flake for building Hello World";
  inputs.nixpkgs.url = github:NixOS/nixpkgs/nixpkgs-unstable;
  outputs = { self, nixpkgs }:
    let 
      system = "aarch64-darwin";
    in
      with import nixpkgs { inherit system; config.contentAddressedByDefault = true; };
      { packages.${system}.default = stdenv.mkDerivation { name = "hello"; buildInputs = [hello]; }; };
}

Console log:

-------------------------------------
Translated Report (Full Report Below)
-------------------------------------

Process:               mkdir [72565]
Path:                  /Volumes/VOLUME/*/mkdir
Identifier:            mkdir
Version:               ???
Code Type:             ARM-64 (Native)
Parent Process:        Exited process [72561]
Responsible:           nix [85739]
User ID:               301

Date/Time:             2022-04-19 10:34:32.1334 -0400
OS Version:            macOS 12.3.1 (21E258)
Report Version:        12
Anonymous UUID:        FA07D672-5149-3553-BA9E-FF08385552A9

Sleep/Wake UUID:       F1A6072E-B80B-44BF-989B-2A0E49A8D7E2

Time Awake Since Boot: 88000 seconds
Time Since Wake:       11172 seconds

System Integrity Protection: enabled

Crashed Thread:        0

Exception Type:        EXC_BAD_ACCESS (SIGKILL (Code Signature Invalid))
Exception Codes:       UNKNOWN_0x32 at 0x00000001021dc000
Exception Codes:       0x0000000000000032, 0x00000001021dc000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Reason:    Namespace CODESIGNING, Code 2 

VM Region Info: 0x1021dc000 is in 0x1021dc000-0x1021e8000;  bytes after start: 0  bytes before end: 49151
      REGION TYPE                    START - END         [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      VM_ALLOCATE                 1021d8000-1021dc000    [   16K] rw-/rwx SM=PRV  
--->  mapped file                 1021dc000-1021e8000    [   48K] r--/rwx SM=COW  ...t_id=fa7f7a09
      GAP OF 0x1b4000 BYTES
      __TEXT                      10239c000-1023fc000    [  384K] r-x/r-x SM=COW  /usr/lib/dyld

Thread 0 Crashed:
0   dyld                          	       0x1023d2510 dyld3::MachOFile::compatibleSlice(Diagnostics&, void const*, unsigned long, char const*, dyld3::Platform, bool, dyld3::GradedArchs const&) + 76
1   dyld                          	       0x1023b6940 invocation function for block in dyld4::JustInTimeLoader::makeJustInTimeLoaderDisk(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, bool, unsigned int) + 96
2   dyld                          	       0x1023b6940 invocation function for block in dyld4::JustInTimeLoader::makeJustInTimeLoaderDisk(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, bool, unsigned int) + 96
3   dyld                          	       0x1023bbd3c dyld4::SyscallDelegate::withReadOnlyMappedFile(Diagnostics&, char const*, bool, void (void const*, unsigned long, bool, dyld4::FileID const&, char const*) block_pointer) const + 132
4   dyld                          	       0x1023b68ac dyld4::JustInTimeLoader::makeJustInTimeLoaderDisk(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, bool, unsigned int) + 204
5   dyld                          	       0x1023b06e4 invocation function for block in dyld4::Loader::getLoader(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&) + 1384
6   dyld                          	       0x1023b0b54 invocation function for block in dyld4::Loader::forEachResolvedAtPathVar(dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, dyld4::ProcessConfig::PathOverrides::Type, bool&, void (char const*, dyld4::ProcessConfig::PathOverrides::Type, bool&) block_pointer) + 320
7   dyld                          	       0x10239df98 dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 168
8   dyld                          	       0x1023db6e0 dyld3::MachOAnalyzer::forEachRPath(void (char const*, bool&) block_pointer) const + 128
9   dyld                          	       0x1023afb8c dyld4::Loader::forEachResolvedAtPathVar(dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, dyld4::ProcessConfig::PathOverrides::Type, bool&, void (char const*, dyld4::ProcessConfig::PathOverrides::Type, bool&) block_pointer) + 708
10  dyld                          	       0x1023af800 invocation function for block in dyld4::Loader::forEachPath(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, void (char const*, dyld4::ProcessConfig::PathOverrides::Type, bool&) block_pointer) + 148
11  dyld                          	       0x1023a5ec4 dyld4::ProcessConfig::PathOverrides::forEachPathVariant(char const*, dyld3::Platform, bool, bool&, void (char const*, dyld4::ProcessConfig::PathOverrides::Type, bool&) block_pointer) const + 672
12  dyld                          	       0x1023af754 dyld4::Loader::forEachPath(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, void (char const*, dyld4::ProcessConfig::PathOverrides::Type, bool&) block_pointer) + 172
13  dyld                          	       0x1023aff74 dyld4::Loader::getLoader(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&) + 864
14  dyld                          	       0x1023b4b44 invocation function for block in dyld4::JustInTimeLoader::loadDependents(Diagnostics&, dyld4::RuntimeState&, dyld4::Loader::LoadOptions const&) + 380
15  dyld                          	       0x1023d1270 invocation function for block in dyld3::MachOFile::forEachDependentDylib(void (char const*, bool, bool, bool, unsigned int, unsigned int, bool&) block_pointer) const + 148
16  dyld                          	       0x10239df98 dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 168
17  dyld                          	       0x1023d10b8 dyld3::MachOFile::forEachDependentDylib(void (char const*, bool, bool, bool, unsigned int, unsigned int, bool&) block_pointer) const + 172
18  dyld                          	       0x1023b48ac dyld4::JustInTimeLoader::loadDependents(Diagnostics&, dyld4::RuntimeState&, dyld4::Loader::LoadOptions const&) + 164
19  dyld                          	       0x1023a15bc dyld4::prepare(dyld4::APIs&, dyld3::MachOAnalyzer const*) + 1092
20  dyld                          	       0x1023a1068 start + 484


Thread 0 crashed with ARM Thread State (64-bit):
    x0: 0x000000016dd45698   x1: 0x00000001021dc000   x2: 0x00000000000086c0   x3: 0x000000016dd458b0
    x4: 0x0000000000000001   x5: 0x0000000000000000   x6: 0x00000001023ef160   x7: 0x0000000000000980
    x8: 0xc2a91a28dc9300bd   x9: 0x00000001023b68e0  x10: 0x0000000000000000  x11: 0x0000000000000001
   x12: 0x000000016dd458b2  x13: 0x0000000000000000  x14: 0x000000023a2b001a  x15: 0x0000000000000000
   x16: 0x0000000000000006  x17: 0x6ae100016dd455c0  x18: 0x0000000000000000  x19: 0x000000016dd45698
   x20: 0x000000016dd458b0  x21: 0x0000000000000001  x22: 0x00000001021dc000  x23: 0x00000001023ef160
   x24: 0x0000000000000000  x25: 0x00000000000086c0  x26: 0x00000001020d8ca8  x27: 0x00000001020b867e
   x28: 0x000000016dd45860   fp: 0x000000016dd45030   lr: 0x2f6f0001023b6940
    sp: 0x000000016dd44e90   pc: 0x00000001023d2510 cpsr: 0x00001000
   far: 0x00000001021dc000  esr: 0x92000007 (Data Abort) byte read Translation fault

Binary Images:
       0x10239c000 -        0x1023fbfff dyld (*) <fbb89662-e6f2-3434-b542-f75185ac5e74> /usr/lib/dyld
               0x0 - 0xffffffffffffffff ??? (*) <00000000-0000-0000-0000-000000000000> ???

External Modification Summary:
  Calls made by other processes targeting this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by all processes on this machine:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=704K resident=0K(0%) swapped_out_or_unallocated=704K(100%)
Writable regions: Total=9216K written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=9216K(100%)

                                VIRTUAL   REGION 
REGION TYPE                        SIZE    COUNT (non-coalesced) 
===========                     =======  ======= 
STACK GUARD                       56.0M        1 
Stack                             8176K        1 
VM_ALLOCATE                         16K        1 
__DATA                              32K        2 
__DATA_CONST                       112K        2 
__LINKEDIT                         240K        2 
__TEXT                             464K        2 
dyld private memory               1024K        1 
mapped file                         48K        1 
===========                     =======  ======= 
TOTAL                             65.9M       13 

[thufschmitt]

After looking a bit at this (as much as possible given that I don’t have a relevant machine handy, I’d say that the issue is that

• At the end of the build, the postLinkSigHook signs all the libraries/binaries
• Then Nix does his hash rewriting dance (the one described here) while relocating the path to its CA location
• But this changes the binaries, so it invalidates the signature
• 💥

I’m not sure what the correct solution would be. A solution could be to change mkDerivation to actually build two derivations for CA:

• First build the actual derivation, without signing anything
• Then build an input-addressed derivation that just copies the content of the previous one, and signs whatever needs to be signed

That’s not really pretty nor efficient, but that would keep most of the benefits of CA derivations (at least early-cutoff would still work as expected), and work

[thufschmitt]

Alternatively, this could be handled internally by Nix, with a hook to sign the binaries after the hash rewriting. This has a couple of potential issues − a mac user could confirm whether they are actual issues or not:

1. That would cause Nix to depend on the OSX code signing tools. If these are part of the base Mac system it’s probably okay, otherwise not so much
2. To keep things really content-addressed, we’d need to use detached signatures. Afaik this
   1. Requires root (maybe not an issue in multi-user mode)
   2. Use a global database that we probably won’t know how to GC

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-29/18903/1

[eliasnaur]

Gentle ping. I'd love to start using CA for my project but this issue is blocking me. Can I do anything to help?

[thufschmitt]

@eliasnaur if you have the time and skills for that, trying out the solution from #6065 (comment) could be very helpful. Outside of that, my macOS knowledge is unfortunately too thin to really see a path

[reckenrode]

I really wanted content-addressed derivations while working on NixOS/nixpkgs#240433. 😭

Fortunately, I have a solution. It’s similar to @thufschmitt’s first option. However, I want to address the second before I discuss how to handle signatures. The detached signatures DB is just a SQLite, so Nix could update it every time a path is added or removed. However, it’s undocumented, so I would be wary of relying on the schema not to change. It’s owned and only writeable by root. While the daemon does run as root today, that could change if a solution to #789 is implemented).

According to the documentation, there are four different ways a file can be signed on macOS.

• LC_CODE_SIGNATURE load commands in Mach-O binaries;
• _CodeSignature directories in bundles;
• Being a file in a bundle with a signature (one of the two above); or
• Extended attributes.

I’m only going to focus on the first one because I don’t think we generate _CodeSignature directories, and the Nix store does not support extended attributes as far as I understand.

The approach I’m suggesting is similar to how Nix handles self-references. Before it calculates a content hash, it would check if the file is a Mach-O binary with a LC_CODE_SIGNATURE load command, zero it out, calculate the hash, move the binary to its final location, then resign the binary.

The following is a series of shell commands manually demonstrating these steps. I don’t know that this is the way it would be implemented in Nix, but this is done using open source tools. The codesign and sigtool binaries comes from sigtool, which is already used in nixpkgs on Darwin.

$ cat test.c
#include <stdio.h>

int main() {
	printf("%s\n", "Hello, content-hashed me!");
}
$ clang test.c -o test
$ codesign -s - -f test
$ otool -l test | rg LC_CODE_SIGNATURE -B 1 -A 3
Load command 16
      cmd LC_CODE_SIGNATURE
  cmdsize 16
  dataoff 33024
 datasize 1536
$ dd if=/dev/zero of=test bs=1 seek=33024 count=1536
$ ./test
fish: Job 1, './test' terminated by signal SIGKILL (Forced quit)
$ sigtool --file test inject
$ ./test
Hello, content-hashed me!
$ otool -l test | rg LC_CODE_SIGNATURE -B 1 -A 3
Load command 16
      cmd LC_CODE_SIGNATURE
  cmdsize 16
  dataoff 33024
 datasize 1536

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/ca-derivations-is-disabled-regardless-of-config-on-mac-os/33550/2