Skip to content

hercules-ci-agent: Fix build on aarch64-darwin - #463879

Merged
roberth merged 1 commit into
NixOS:masterfrom
hercules-ci:fix-hci
Dec 9, 2025
Merged

hercules-ci-agent: Fix build on aarch64-darwin#463879
roberth merged 1 commit into
NixOS:masterfrom
hercules-ci:fix-hci

Conversation

@roberth

@roberth roberth commented Nov 21, 2025

Copy link
Copy Markdown
Member

hercules-ci-agent still needs the fix by @sandydoo in order to build.
I've made it conditional to nullify the possible impact elsewhere.

Long term I'd like to get rid of the TemplateHaskell that's triggering this problem, but that's blocked on a number of issues.

So a workaround is needed.

Things done

  • Built on platform:
    • x86_64-linux
      • noop except empty attribute is now set. Ok for leaf package.
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • Tested, as applicable:
  • Ran nixpkgs-review on this PR. See nixpkgs-review usage.
  • Tested basic functionality of all binary files, usually in ./result/bin/.
  • Nixpkgs Release Notes
    • Package update: when the change is major or breaking.
  • NixOS Release Notes
    • Module addition: when adding a new NixOS module.
    • Module update: when the change is significant.
  • Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

Add a 👍 reaction to pull requests you find important.

@ofborg ofborg Bot added the 6.topic: darwin Running or building packages on Darwin label Nov 21, 2025
@nixpkgs-ci nixpkgs-ci Bot added 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-darwin: 5001+ This PR causes many rebuilds on Darwin and must target the staging branches. 10.rebuild-linux: 1 This PR causes 1 package to rebuild on Linux. 6.topic: haskell General-purpose, statically typed, purely functional programming language labels Nov 21, 2025
@roberth
roberth force-pushed the fix-hci branch 2 times, most recently from d8a57d8 to 544c4e8 Compare November 21, 2025 23:53
@roberth
roberth changed the base branch from master to staging November 21, 2025 23:53
@nixpkgs-ci nixpkgs-ci Bot closed this Nov 21, 2025
@nixpkgs-ci nixpkgs-ci Bot reopened this Nov 21, 2025
Comment thread pkgs/development/compilers/ghc/common-hadrian.nix Outdated
@roberth

roberth commented Nov 25, 2025

Copy link
Copy Markdown
Member Author

Is there anything else to be done?
I'd like to merge this fix soon.

@wolfgangwalther

Copy link
Copy Markdown
Contributor

It's quite unsatisfactory to have such a targeted and unprincipled hack without any path forward to remove it anytime soon.

If older compilers are not affected by this, can't we just build hercules-ci-agent with GHC 9.8.4 for now?

@roberth

roberth commented Nov 25, 2025

Copy link
Copy Markdown
Member Author

I agree that the solution isn't pretty necessarily, but loading a Haskell program + RTS + nix libraries into a single process global symbol namespace works fine, as demonstrated by hercules-ci-agent itself.
If you find that still uncomfortable, I hope that it helps that at least it is targeted and won't impact anything that doesn't request this behavior.

can't we just build hercules-ci-agent with GHC 9.8.4 for now?

This would require hydra to build a significant portion of this package set, whereas currently it only has to build ghc itself at this version.

@sandydoo

sandydoo commented Nov 26, 2025

Copy link
Copy Markdown
Contributor

If older compilers are not affected by this, can't we just build hercules-ci-agent with GHC 9.8.4 for now?

9.8.4 is also affected. I believe 9.8.2 is the latest in the 9.8 series without the linker changes that cause the build failure.

@sternenseemann

Copy link
Copy Markdown
Member

Why was this working fine with 9.8.4 then?

@sandydoo

Copy link
Copy Markdown
Contributor

Why was this working fine with 9.8.4 then?

@sternenseemann it actually worked fine with both 9.8.4 and 9.10.3.
These versions use the new linker machinery, but the build issues only appeared after the darwin team updated the default apple SDK. This included an update to dyld, which would be the likely culprit imo.
I've bisected the commits to roughly that timeline, but I don't have the compute to give an exact commit.

@wolfgangwalther

Copy link
Copy Markdown
Contributor

These versions use the new linker machinery, but the build issues only appeared after the darwin team updated the default apple SDK. This included an update to dyld, which would be the likely culprit imo.

cc @NixOS/darwin-core, maybe you have some input to this as well.

@reckenrode

Copy link
Copy Markdown
Contributor

The only thing we build from dyld are a couple of tools (dsc_extractor and dyld_info). I doubt it’s that. The issue is a missing symbol (ensureSlot) from libnixexpr.dylib.

@wolfgangwalther

Copy link
Copy Markdown
Contributor

I'm not convinced, yet, that this the right thing to do. Imho, this should be fully bisected. I started that, but didn't finish. I have it down to:

# good: [144f1a826e74ba3f00506f110e03579723f82baf] Merge staging-next into staging
git bisect good 144f1a826e74ba3f00506f110e03579723f82baf
# bad: [4fe7117887b41d2937db7f1dfc1e00e844585307] Merge staging-next into staging
git bisect bad 4fe7117887b41d2937db7f1dfc1e00e844585307
git bisect view --oneline
4fe7117887b4 Merge staging-next into staging
de3f35346121 tinysparql: 3.9.2 -> 3.10.0 (#443953)
010b5fabc647 luit: 20240910 -> 20250912 (#442818)
510eca52dd91 go_1_25: 1.25.2 -> 1.25.3 (#451802)
b80587454ae2 go_1_25: 1.25.2 -> 1.25.3
c771cbb68d1c haskell.compiler.ghc924Binary: drop (#440976)
a356d4e02d8c {spandsp,spandsp3}: Enable tests, adopt (#451464)
f9cc041eded0 {spandsp,spandsp3}: Adopt by NGI team
8ab72653f2cb {spandsp,spandsp3}: Enable tests
d67a386a3115 flatbuffers: 25.2.10 -> 25.9.23 (#445727)
fce8451e33a0 python3Packages.anyio: disable racy test (#451746)
d88e47c845b6 python3Packages.anyio: disable racy test
c69dd11cb98b Merge staging-next into staging
5275582fbea1 haskell.compiler.ghc963Binary: drop (#444249)
c9c734465795 haskell.compiler.ghc924Binary: drop
da915f81efab haskell.compiler.ghc967: bootstrap with GHC 9.4.8
5df6c9a28f1d haskell.compiler.ghc963Binary: drop
ac5228c6c025 haskell.compiler.ghc9101: drop
14488d405124 compilers/ghc/common-hadrian: remove unused condition
c0998dca6b2a openexr: Disable known-broken tests on big-endian (#448580)
f4c44275faf0 Merge staging-next into staging
88aa4d379221 Merge remote-tracking branch 'origin/staging-next' into staging
8dc688a8acc5 Merge staging-next into staging
bb7964b6828c Revert "ruby: avoid rebuilding default ruby on linux for now"
a5843ee23c79 Merge staging-next into staging
8751f8a68999 Merge staging-next into staging
ce95c22e5723 rsync: Remove `fakeroot` from `checkInputs` (#446259)
b358f449c6eb Merge staging-next into staging
5cbc433bd4f0 Merge staging-next into staging
2aa5a48364d8 python3Packages.cryptography{,.vectors}: 46.0.1 -> 46.0.2 (#451290)
c4d77c3a0654 Merge staging-next into staging
9d4602f5feb7 Merge staging-next into staging
13c9e5fb30b2 publicsuffix-list: 0-unstable-2025-08-28 -> 0-unstable-2025-10-08 (#451192)
793760ff9576 python3Packages.cryptography.vectors: fix build with uv_build 0.9.2
cf8b55260ff7 python3Packages.cryptography{,.vectors}: 46.0.1 -> 46.0.2
cf1c986380ea Merge staging-next into staging
ebf10ab4c5dd python3Packages.django-bootstrap3: update uv-build.patch (#451275)
6bfef18a25d1 Merge staging-next into staging
6ff2233f5e7f Merge staging-next into staging
e50f11fc6b2b Merge staging-next into staging
578e2c18d3cc python3Packages.django-bootstrap3: update uv-build.patch
fa4affd32ef6 Merge staging-next into staging
c28ce32e9829 publicsuffix-list: 0-unstable-2025-08-28 -> 0-unstable-2025-10-08
6144e8c63362 Merge staging-next into staging
a99a2d70cd64 Merge staging-next into staging
ecf4aea56312 Merge staging-next into staging
c885df7bc72a Merge staging-next into staging
75548e2ff0cf python312Packages.regex: 2025.7.34 -> 2025.9.18 (#446701)
50d68ebe51a8 python3Packages.numpy: Disable borked test on ppc64 (#450059)
ed4dbafb102a fontforge: 20230101 -> 20251009 (#450770)
5ba080519b02 libinput: make wacom support optional (#450827)
202e16ba6595 Merge staging-next into staging
fe1146646d18 uv: 0.9.0 -> 0.9.2 (#450911)
68ddbb7ebb61 openssh: 10.1p1 -> 10.2p1 (#450658)
ebd7ce5ea406 uv: 0.9.0 -> 0.9.2
6513f647db3e Merge staging-next into staging
88f245ca0ce0 python3Packages.ffmpy: remove uv patch
46e42dc7ee38 jj-pre-push: remove uv patch
56806cd8a163 Merge staging-next into staging
e92426ff58ba libinput: make wacom support optional
eda556d94b12 treewide: remove usages of obsolete pie hardening flag (#449771)
d235d45dcb75 libunistring: 1.4 -> 1.4.1 (#450431)
4a6e2ebd8a50 mkDerivation: warn for obsolete pie hardening flag usage
7c3bff4f9bd5 python3Packages.bcrypt: 4.3.0 -> 5.0.0, adopt (#447841)
f87f0618cbce fontforge: 20230101 -> 20251009
cee57b93b696 Merge staging-next into staging
013328f016a3 python3Packages.xmltodict: 0.14.2 -> 1.0.2 (#445729)
eb681f8139a9 netbox_4_4: init at 4.4.2 (#440403)
94830c05ed6f openssh: 10.1p1 -> 10.2p1
ceae63a76a47 Merge staging-next into staging
31b743d9652d python3Packages.numpy: Disable borked test on ppc64
537add7b3de5 python312: 3.12.11 -> 3.12.12 (#450493)
b7cdd086a8b7 Merge staging-next into staging
0579a4756924 catch2_3: 3.10.0 -> 3.11.0 (#450170)
ec3cafaa31c0 Merge staging-next into staging
236ccf8c358e python312: 3.12.11 -> 3.12.12
38cbba4f627c uv: 0.8.23 -> 0.9.0 (#449568)
15d73fb56b11 apple-sdk: update darwinMinVersion to 14.0 and darwinSdkVersion to 14.4 (#447364)
d37154c920e3 apple-sdk_{11,12,13}: add to aliases.nix
0f6b88a896e6 apple-sdk_{11,12,13}: drop
9b456509dc6e xemu: use default SDK on Darwin (14.4)
9d540d1b8c77 wine: use default SDK on Darwin (14.4)
3c201e0f0e9e whisper-cpp: use default SDK on Darwin (14.4)
e61f6d162f21 swift: mark broken until the 5.10.1 update can be done
7cc243d3d52e spotifyd: use default SDK on Darwin (14.4)
ec30ce8c6530 spidermonkey: drop SDK version workaround patch
109017913fe2 spacebar: use default SDK on Darwin (14.4)
6ab924b0c2e1 shipwright: use default SDK on Darwin (14.4)
da7a8d12f023 servo: use default SDK on Darwin (14.4)
e963c19b5f47 sdrangel: use default SDK on Darwin (14.4)
619a93bc87cc sdl3: use default SDK on Darwin (14.4)
f86b1e9a0fad screen-pipe: use default SDK on Darwin (14.4)
1a0918ae7784 rstudio: use default SDK on Darwin (14.4)
46efd7786aea rpm: use default SDK on Darwin (14.4)
f6b75da2d534 qt5.*: use default SDK on Darwin (14.4)
968bc7dfb4ef qemu: use default SDK on Darwin (14.4)
28f33de7dd9e python3Packages.torchvision: use default SDK on Darwin (14.4)
ca24684803ea python3Packages.torch: use default SDK on Darwin (14.4)
e2f070607421 python3Packages.mlx: use default SDK on Darwin (14.4)
ccd7288e2604 python3Packages.materialx: use default SDK on Darwin (14.4)
f02b8cf7b9c1 ponyc: use default SDK on Darwin (14.4)
a0686081fe12 nvtopPackages.apple: use default SDK on Darwin (14.4)
d068cfff50fc nchat: use default SDK on Darwin (14.4)
51370732ee2d llama-cpp: use default SDK on Darwin (14.4)
5f5b7dc7d2ec livkit-webrtc: use default SDK on Darwin (14.4)
58a2ab47bea9 ladybird: use default SDK on Darwin (14.4)
b9eab368aff3 koboldcpp: use default SDK on Darwin (14.4)
8d2b795e205e kanata: use default SDK on Darwin (14.4)
4c07c99f2b30 gst_all_1: use default SDK on Darwin (14.4)
a148418ba6a6 gopass-jsonapi: use default SDK on Darwin (14.4)
1557189e93c8 gopass: use default SDK on Darwin (14.4)
7da765f439b2 go: use default SDK on Darwin (14.4)
72289dfaa66c ghostunnel: use default SDK on Darwin (14.4)
20e89e336ebc drawterm: use default SDK on Darwin (14.4)
2fbd3f4aab2e dotnetCorePackages.dotnet_{8,9,10}.vmr: use default SDK on Darwin (14.4)
ea758132a153 darwin.text_cmds: use default SDK on Darwin (14.4)
13be5191fb93 darwin.system_cmds: use default SDK on Darwin (14.4)
8010c610db66 darwin.removefile: drop workaround for older Clang versions
0ba2bde18f3a darwin.network_cmds: use default SDK on Darwin (14.4)
4d961ec1504f darwin.libutil: use default SDK on Darwin (14.4)
006f7e423442 darwin.libresolv: update build for the 14.4 SDK
c8a5838007a9 darwin.file_cmds: update build for the 14.4 SDK
55f32f567c1e darwin.dyld: use default SDK on Darwin (14.4)
674bfaab7038 darwin.dyld: work around lack of LLVM 21 support in Meson
29b634dd2c7c darwin.copyfile: drop workaround for older Clang versions
b68b26b3ba6e darwin.bootstrap_cmds: use default SDK on Darwin (14.4)
7b4d6379222b darwin.PowerManagement: fix build with 14.4 SDK
e5fd8d4c6164 choose-gui: use default SDK on Darwin (14.4)
f2895523473d build-support/build-mozzila-mach: drop SDK version workaround patch
e001ae3c6346 bitwarden-desktop: use default SDK on Darwin (14.4)
1fd0904a4146 bespokesynth: use default SDK on Darwin (14.4)
7dffcb50385a balena-cli: use default SDK on Darwin (14.4)
8e6ef95a4bdc azahar: use default SDK on Darwin (14.4)
b572ad29440e ares: use default SDK on Darwin (14.4)
e99a7321592f _1fps: use default SDK on Darwin (14.4)
c422ce6874c3 doc/release-notes-25.11: add note about Darwin platform updates
56ac9bffa3b5 doc/stdenv/platform-notes: update Xcode version to SDK version table
2ab175889abb lib.systems.darwinMinVersion: 11.3 -> 14.0
cf45eef8ecf8 lib.systems.darwinSdkVersion: 11.3 -> 14.4
31cf83ad2c0b (HEAD) cc-wrapper: always include libc++ in the search path (#445095)
adebddbdbee8 python3Packages.libpass: 1.9.2 -> 1.9.3 (#450425)
497793b53596 netbox_4_3: mark as EOL
5518f1e35f24 netbox_4_4: init at 4.4.2
724b8f5bf7cb python3Packages.mkdocstrings-python: 1.16.12 -> 1.18.2
d74462059b9c python3Packages.inline-snapshot: 0.24.0 -> 0.28.0
38202296686b libunistring: 1.4 -> 1.4.1
a8fc9df4d3ee python3Packages.libpass: skip tests broken by bcrypt 5.0.0
1bbe4fe35d6b python3Packages.bcrypt: add dotlambda to maintainers
375da609d8cf python3Packages.bcrypt: 4.3.0 -> 5.0.0
6eac86d4992a python3Packages.libpass: 1.9.2 -> 1.9.3
0e141008b41a Merge staging-next into staging
5c2b79edfa75 cc-wrapper: make availability warnings into errors (#445119)
e7fbed853bed treewide: remove usages of obsolete pie hardening flag
fdc93627c624 go: restrict default pie patch to known good GOARCH (#449995)
7b3a342260ec go: restrict default pie patch to known good GOARCH
bdbc0951532d python3Packages.sphinx-autodoc-typehints: 3.2.0 -> 3.4.0 (#450248)
6cf50d7e1e0a python3Packages.sphinx-autodoc-typehints: 3.2.0 -> 3.4.0
fcc958fe89e4 Merge staging-next into staging
76a8ffa6d8d2 cc-wrapper: use `-cxx-isystem` instead of `-isystem` on Clang
bd0dd33de11c cc-wrapper: always include libc++ in the search path
2e4679703a4d Merge staging-next into staging
65dc0d0ef849 Merge staging-next into staging
0e947838b56c catch2_3: 3.10.0 -> 3.11.0
a068c0eb03dc Merge staging-next into staging
608b4fed3f02 gn: 0-unstable-2025-07-29 -> 0-unstable-2025-08-29 (#449075)
66b379da343b python3Packages.sigstore-models: fix build with uv_build 0.9.0
39f481527f37 python3Packages.marimo: fix build with uv_build 0.9.0
98b7ba06e353 python3Packages.githubkit: fix build with uv_build 0.9.0
e4ed1873a68a python3Packages.ffmpy: fix build with uv_build 0.9.0
555d58793034 python3Packages.django-bootstrap5: fix build with uv_build 0.9.0
976b80484b73 python3Packages.django-bootstrap4: fix build with uv_build 0.9.0
f8a5a66b69b7 python3Packages.django-bootstrap3: fix build with uv_build 0.9.0
1cb13614d99a python3Packages.construct-classes: fix build with uv_build 0.9.0
ca4492361ff9 jj-pre-push: fix build with uv_build 0.9.0
7f562f1d546b uv: 0.8.24 -> 0.9.0
1319af86d909 uv: 0.8.23 -> 0.8.24
5f0e71686d48 python312Packages.regex: add maintainer
3117d6599873 gn: 0-unstable-2025-07-29 -> 0-unstable-2025-08-29
ec87d448408c openexr: Disable known-broken tests on big-endian
aef75df9ee70 moltenvk: allow unchecked API usage
ed1bf3bb645a moltenvk: use the nixpkgs deployment target instead of the hardcoded one
204543831464 doc/stdenv/platform-notes: add section about availability checks
369cc5c66b1e cc-wrapper: make availability warnings into errors
275f6c82702a python312Packages.regex: 2025.7.34 -> 2025.9.18
19c9c011200e rsync: Remove `fakeroot` from `checkInputs`
869c1a5fe717 python3Packages.xmltodict: 0.14.2 -> 1.0.2
0c174928ec2c flatbuffers: 25.2.10 -> 25.9.23
41a2afff7c4d tinysparql: 3.9.2 -> 3.10.0
00ec9c2a4047 luit: 20240910 -> 20250912

@nixpkgs-ci nixpkgs-ci Bot added 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. and removed 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-darwin: 5001+ This PR causes many rebuilds on Darwin and must target the staging branches. 10.rebuild-linux: 1 This PR causes 1 package to rebuild on Linux. labels Dec 8, 2025
@roberth

roberth commented Dec 8, 2025

Copy link
Copy Markdown
Member Author

Thank you @wolfgangwalther. I haven't quite figured that one out either. The bad commit produced a different problem, and some more recent commits actually built successfully somehow.

The good news is I've found a new workaround that doesn't require any changes to GHC or anything like that.
It's isolated to hercules-ci-agent, and it gets the job done.

@roberth
roberth changed the base branch from staging to master December 9, 2025 00:09
@nixpkgs-ci nixpkgs-ci Bot closed this Dec 9, 2025
@nixpkgs-ci nixpkgs-ci Bot reopened this Dec 9, 2025
Comment thread pkgs/development/haskell-modules/configuration-nix.nix Outdated
@wolfgangwalther

Copy link
Copy Markdown
Contributor

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 463879
Commit: a4984c95280507c632f70ae6b82f86b14b20485d


x86_64-darwin

✅ 6 packages built:
  • haskellPackages.hercules-ci-agent
  • haskellPackages.hercules-ci-agent.doc
  • haskellPackages.hercules-ci-cli
  • haskellPackages.hercules-ci-cli.doc
  • hci
  • hercules-ci-agent

aarch64-darwin

✅ 7 packages built:
  • haskellPackages.hercules-ci-agent
  • haskellPackages.hercules-ci-agent.doc
  • haskellPackages.hercules-ci-cli
  • haskellPackages.hercules-ci-cli.doc
  • hci
  • hci.doc
  • hercules-ci-agent

@wolfgangwalther wolfgangwalther left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Diff LGTM

@nixpkgs-ci nixpkgs-ci Bot added the 12.approvals: 1 This PR was reviewed and approved by one person. label Dec 9, 2025
@roberth
roberth added this pull request to the merge queue Dec 9, 2025
Merged via the queue into NixOS:master with commit 5751e2b Dec 9, 2025
28 of 31 checks passed
@roberth
roberth deleted the fix-hci branch December 9, 2025 13:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: darwin Running or building packages on Darwin 6.topic: haskell General-purpose, statically typed, purely functional programming language 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. 12.approvals: 1 This PR was reviewed and approved by one person.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Build failure: cachix on darwin

5 participants