Skip to content

Copy instead of link haskell-backend binaries in nix derivation #4106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 24, 2025

Conversation

juliankuners
Copy link
Contributor

@juliankuners juliankuners commented Apr 24, 2025

Following up on #4096, evm-semantics#2739, evm-semantics#2745, and kontrol#1013, I measured the install time of kontrol with kup on a fresh Ubuntu virtual machine. Compared to a previous measurement, the install time has been reduced to 10 minutes from previously 20 minutes.

During the latest measurement I noticed that Haskell dependencies are still being downloaded, even though the import-from-derivation anti-pattern has already been removed in #4096. I investigated and noticed that the Haskell dependencies are also considered runtime dependencies. This is caused by the withZ3 wrapper that links to binaries in Haskell derivations. These Haskell derivations also contain references to Haskell dependencies, namely kore and hs-backend-booster.

This pull request changes the withZ3 wrapper to copy the respective binary prior to wrapping.

E.g., for kore-rpc-booster, this reduces the total size to download by more than 5GB, when installing from nix cache. In addition, these downloads were spread among many smaller derivations to download.

Output of `nix-tree` before the change:

┌─────────────────────────────────────────────────────┬────────────────────────────────────────────────────┬────────────────────────────────────────────────────┐
│kore-rpc-booster                  5.15 GiB (5.15 GiB)│hs-backend-booster-0.1.0         5.12 GiB (5.08 GiB)│kore-0.1.0                       4.89 GiB (1.31 GiB)│
│                                                     │z3-4.13.4                      72.03 MiB (32.24 MiB)│kore-rpc-types-0.1.0            3.34 GiB (80.65 MiB)│
│                                                     │bash-5.2p26                     32.51 MiB (1.54 MiB)│json-rpc-1.0.4                  3.21 GiB (15.82 MiB)│
│                                                     │                                                    │aeson-pretty-0.8.10              3.04 GiB (21.2 MiB)│
│                                                     │                                                    │deriving-aeson-0.2.9           3.02 GiB (605.28 KiB)│
│                                                     │                                                    │aeson-2.1.2.1                   3.02 GiB (52.64 MiB)│
│                                                     │                                                    │monad-logger-0.3.40               2.98 GiB (4.6 MiB)│
│                                                     │                                                    │stm-conduit-4.0.1                2.97 GiB (2.49 MiB)│
│                                                     │                                                    │conduit-extra-1.3.6               2.95 GiB (5.0 MiB)│
│                                                     │                                                    │cereal-conduit-0.8.0            2.88 GiB (441.2 KiB)│
│                                                     │                                                    │conduit-1.3.5                    2.88 GiB (13.4 MiB)│
│                                                     │                                                    │semialign-1.3                    2.86 GiB (3.73 MiB)│
│                                                     │                                                    │mono-traversable-1.0.17.0       2.85 GiB (22.06 MiB)│
│                                                     │                                                    │recursion-schemes-5.2.2.5        2.82 GiB (6.79 MiB)│
│                                                     │                                                    │cryptonite-0.30                 2.81 GiB (54.26 MiB)│
│                                                     │                                                    │free-5.2                        2.81 GiB (22.79 MiB)│
│                                                     │                                                    │vector-algorithms-0.9.0.1       2.81 GiB (25.24 MiB)│
│                                                     │                                                    │witherable-0.4.2                   2.8 GiB (4.6 MiB)│
│                                                     │                                                    │indexed-traversable-instances- 2.79 GiB (358.65 KiB)│
│                                                     │                                                    │bitvec-1.1.5.0                  2.78 GiB (10.49 MiB)│
│                                                     │                                                    │vector-0.13.1.0                 2.77 GiB (49.75 MiB)│
│                                                     │                                                    │semigroupoids-6.0.1             2.77 GiB (12.16 MiB)│
│                                                     │                                                    │language-c-0.9.3                2.76 GiB (63.64 MiB)│
│                                                     │                                                    │profunctors-5.6.2               2.75 GiB (10.68 MiB)│
│                                                     │                                                    │memory-0.18.0                    2.74 GiB (5.69 MiB)│
│                                                     │                                                    │basement-0.0.16                 2.74 GiB (28.09 MiB)│
│                                                     │                                                    │bifunctors-5.6.2                2.73 GiB (16.04 MiB)│
│                                                     │                                                    │optparse-applicative-0.18.1.0    2.72 GiB (8.72 MiB)│
│                                                     │                                                    │streaming-commons-0.2.2.6        2.72 GiB (4.91 MiB)│
│                                                     │                                                    │attoparsec-0.14.4               2.72 GiB (17.16 MiB)│
│                                                     │                                                    │QuickCheck-2.14.3               2.72 GiB (19.19 MiB)│
│                                                     │                                                    │network-run-0.2.8                2.72 GiB (1.29 MiB)│
│                                                     │                                                    │hpp-0.6.5                       2.71 GiB (13.35 MiB)│
└─────────────────────────────────────────────────────┴────────────────────────────────────────────────────┴────────────────────────────────────────────────────┘
/nix/store/y51pfqn8fcmnimhkswfc1fda2pan3kmj-hs-backend-booster-0.1.0                                                                                             
NAR Size: 108.81 MiB | Closure Size: 5.12 GiB | Added Size: 5.08 GiB                                                                                             
Immediate Parents (1): kore-rpc-booster                                                                                                                          

Output of `nix-tree` after the change:

┌─────────────────────────────────────────────────────┬────────────────────────────────────────────────────┬────────────────────────────────────────────────────┐
│kore-rpc-booster              113.87 MiB (113.87 MiB)│z3-4.13.4                      72.03 MiB (32.24 MiB)│gcc-13.2.0-lib                  39.78 MiB (8.81 MiB)│
│                                                     │elfutils-0.191                 58.27 MiB (16.82 MiB)│glibc-2.39-52                  30.96 MiB (28.88 MiB)│
│                                                     │gmp-with-cxx-6.3.0            40.51 MiB (741.04 KiB)│                                                    │
│                                                     │ncurses-6.4                     34.51 MiB (3.54 MiB)│                                                    │
│                                                     │bash-5.2p26                     32.51 MiB (1.54 MiB)│                                                    │
│                                                     │secp256k1-0.4.1                 32.22 MiB (1.25 MiB)│                                                    │
│                                                     │zlib-1.3.1                    31.09 MiB (125.16 KiB)│                                                    │
│                                                     │libffi-3.4.6                   31.03 MiB (71.87 KiB)│                                                    │
│                                                     │glibc-2.39-52                  30.96 MiB (28.88 MiB)│                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
│                                                     │                                                    │                                                    │
└─────────────────────────────────────────────────────┴────────────────────────────────────────────────────┴────────────────────────────────────────────────────┘
/nix/store/ih51sgk8g57fnkbd5r82ddi8k5vln8cl-z3-4.13.4                                                                                                            
NAR Size: 32.24 MiB | Closure Size: 72.03 MiB | Added Size: 32.24 MiB                                                                                            
Immediate Parents (1): kore-rpc-booster                                                                                                                          

@juliankuners juliankuners requested a review from jberthold April 24, 2025 12:14
Copy link
Member

@jberthold jberthold left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great discovery!

@automergerpr-permission-manager automergerpr-permission-manager bot merged commit 362dab3 into master Apr 24, 2025
6 checks passed
@automergerpr-permission-manager automergerpr-permission-manager bot deleted the nix-runtime-dependencies branch April 24, 2025 23:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants