Policy for third-party hardware donation

[zimbatm]

Sometimes, it's easier for organizations or individuals to lend out hardware (rather than open collective). There is an opportunity to gain access to build capacity. And different kinds of hardware (e.g., GPU, Riscv5, MIPS, ...)

Before pursuing this, let's discuss what that would look like.

What are the requirements on our side?

Some threads:

• Some level of trust for the binary cache key
• What channel of communication / API do we use to remote control the server? (Restart, KVM, ...)

[zowoq]

Anything exotic will be a problem for the hercules agent as it is haskell. Just as a remote builder for buildbot/hydra sould mean the cache key isn't an issue?

[Mic92]

We would still need to trust the build results.

[zowoq]

We would still need to trust the build results.

I don't understand your point? Isn't trusting the build results a given?

[Mic92]

We would still need to trust the build results.

I don't understand your point? Isn't trusting the build results a given?

I think we should communicate how builders for different architectures are secured. i.e. Hetzner will have a safer access policies than the machines in someones basement. Then users can decide if they are OK with this.

[zimbatm]

Remote builders sounds good.

One requirement could be that we are the only admins on the machine. It doesn't prevent physical tampering but reduces the attack surface if the host provider gets hacked.

[ConnorBaker]

I’m not familiar with how the infrastructure you’ve set up for builds vs caching works, but one of the concerns I’ve had when trying to stand up infrastructure for consistently building CUDA packages, and for serving a binary cache, but there can be a lot of traffic between nodes. It was enough of a bottleneck between the three desktops in my basement that I moved over to 10GBe networking for everything, and I’m still saturating it.

I don’t know if the remote build protocol takes into account closure size or data locality when deciding which machines should build different things, but there can be a lot of movement on the network, which can be a bottleneck or, in the case of cloud providers, a hefty egress fee.

So a couple of questions from me:

1. Do the machines you’ve set up for Hydra also serve as a cache? If they are, have you run into issues with the amount of egress you’re doing?
2. Is your Cachix the main binary cache for projects under this umbrella? If so, how large is it (if you’re willing or able to disclose)?
3. Are the machines working as hydra builders all on the same network (or Hetzner region)?
4. What are your thoughts on ephemeral Azure builders? I’ve found the HBv3 spot instances in US East extremely competitive price-wise, and have been looking into tooling to automate scaling them up and down.
5. File-level deduplication in the cache would be fairly important, as CUDA enablement involves realizing a new copy of a derivation rather than building on an existing one —- most projects don’t separate the code generation for CPUs from GPUs well enough for us to be able to re-use existing portions of builds. Is there anything like that set up currently?

I’d love to learn more about any of the challenges you all have faced setting up and maintaining this infrastructure!

[zowoq]

remote build protocol takes into account closure size or data locality when deciding which machines should build different things

No, it doesn't.

Do the machines you’ve set up for Hydra also serve as a cache?

No.

Is your Cachix the main binary cache for projects under this umbrella? If so, how large is it (if you’re willing or able to disclose)?

Yes. 1tb, sponsored by Cachix.

Are the machines working as hydra builders all on the same network (or Hetzner region)?

Yes, our linux machines are all in HEL1. Though that isn't exactly intentional, we usually choose based on price. We also have two macos builders in FSN1.

What are your thoughts on ephemeral Azure builders? I’ve found the HBv3 spot instances in US East extremely competitive price-wise, and have been looking into tooling to automate scaling them up and down.

Haven't used them. IIRC there way some discussion about Azure stuff in the nixos org, maybe with the infra team or the foundation?

File-level deduplication in the cache would be fairly important, as CUDA enablement involves realizing a new copy of a derivation rather than building on an existing one —- most projects don’t separate the code generation for CPUs from GPUs well enough for us to be able to re-use existing portions of builds. Is there anything like that set up currently?

No.

I’d love to learn more about any of the challenges you all have faced setting up and maintaining this infrastructure!

I don't think we've had any real technical challenges so far, basically just limited by funding. Once we started the opencollective we just expanded as the funding increased. Building cuda, rocm, etc is probably going to be the first time we've really needed to give thought to some of these topics.

[zimbatm]

Yes, our linux machines are all in HEL1.

See also https://docs.hetzner.com/robot/general/traffic/ Each server comes with 10TB of egress.

[Mic92]

Yes, our linux machines are all in HEL1.

See also https://docs.hetzner.com/robot/general/traffic/ Each server comes with 10TB of egress.

No, for 1Gbit-Links and physical machines, it's unmetered. The traffic limit only applies to VMs.

[zimbatm]

To go back on topic, I am recapping this to:

• We happily accept if you want to donate hardware to extend our build configuration.
• The machine needs to be connected to the Internet, with SSH access.
• We ask that we control the system configuration (so it's part of the nix-community/infra repo).

It could be added to the donation page if we agree on this.

As a personal note, I would love to see GPU, and MIPS hardware

[zimbatm]

In case of esotheric hardware, we also need one or more people doing the work to fix nixpkgs so that we can keep the machine up to date.

[Mic92]

Mips seems a bit dead at this point, even https://mips.com/ now sells your riscv64 cores instead. Some simple gpu runner would be nice for integration tests.

[zimbatm]

Related: https://discourse.nixos.org/t/plct-lab-intends-to-collaborate-on-risc-v-support/48875

[ConnorBaker]

Continuing from #1335 (comment):

@zowoq sorry for the late reply, I'm moving soon and organizing everything has been quite an experience.

Hetzner

A more accurate amount on the Hetzner side: 315 euros / month (though it could be less, at the moment).

I have one RX170 instance (hereafter ubuntu-hetzner) which I've set up as an aarch64-linux builder for our Hercules CI instance. I also use it regularly for nixpkgs-review runs. That is about 169 euros / month.

I have one AX102 instance (hereafter nixos-cantcache-me) which I use to host https://cantcache.me, an attic instance with a cache for CUDA-related work (https://cantcache.me/cuda/nix-cache-info). I use it as a cache for my nixpkgs-review runs, as I'm planning on moving from my desktop builders to ephemeral instances on Azure, and a large amount can be cached between CI runs. This costs about 104 euros / month, plus about 43 euros / month for the 10Gb uplink.

I only recently set up monitoring on nixos-cantcache-me (which runs NixOS), but haven't yet for ubuntu-hetzner (which runs Ubuntu). Utilization wise, nixos-cantcache-me is only under load when I'm running nixpkgs-review or when Hercules CI is running on my desktop builders (described below). ubuntu-hetzner is under load when I'm running nixpkgs-review or when Hercules CI is running on it.

Hercules CI runs on a schedule, so load increases due to CI starting and load decreases due to CI finishing are somewhat predictable (meaning we could benefit from ephemeral instances).

Local machines

My desktop builders are three machines I have in my basement, and they are all running NixOS. I use them every day for work on Nixpkgs (lots of nixpkgs-review runs) or hobby projects, and run Hercules CI on them.

Common specs:

• They're networked directly through a 10 Gbe switch (https://www.zyxel.com/global/en/products/switch/10-12-port-10g-multi-gigabit-lite-l3-smart-managed-switch-xs1930-series/specifications).

• Each has either onboard 10 Gbe or a 10 Gbe NIC.

• Each has 96 GB of DDR5-6800 RAM.

• Each has ZRAM enabled with ZSTD compression (avoiding pesky out-of-memory when building PyTorch, among other things, which allocates a huge number of empty pages when compiling flash attention).

• Each has all drives in a ZFS RAID 0 array, optimized for speed.

• nixos-desktop

  • Intel i9-13900k
  • ZOTAC GAMING GeForce RTX 4090 Trinity OC
  • ~4.53 TB disk space
    • 1 x Samsung 990 PRO 1 TB
    • 2 x Samsung 980 PRO 2 TB

• nixos-build01

  • AMD Ryzen 9 7950X3D
  • ~7.25 TB disk space
    • 4 x Samsung 990 PRO 2 TB

• nixos-ext

  • AMD Ryzen 9 7950X3D
  • ~7.25 TB disk space
    • 4 x Samsung 990 PRO 2 TB

Questions

Since the majority of the load on these machines are my running nixpkgs-review (Hercules CI builds a very limited subset of packages), I'm curious: what do you all use to validate PRs? Is there some common infrastructure for nixpkgs-review runs?

As an example, for each of my PRs, I try to run the following nixpkgs-review instances:

• x86_64-linux with { allowUnfree = false; cudaSupport = false; }
• x86_64-linux with { allowUnfree = true; cudaSupport = true; cudaCapabilities = [ "7.5" ]; }
• aarch64-linux with { allowUnfree = false; cudaSupport = false; }
• aarch64-linux with { allowUnfree = true; cudaSupport = true; cudaCapabilities = [ "7.5" ]; }
• aarch64-linux with { allowUnfree = true; cudaSupport = true; cudaCapabilities = [ "7.2" ]; }
  • This is to test support for Jetson devices
• x86_64-darwin with { allowUnfree = false; cudaSupport = false; }
• aarch64-darwin with { allowUnfree = false; cudaSupport = false; }

That's seven instances: luckily I use a MacBook Pro for as my laptop and have a Jetson, so I can build for both those platforms. But I'm curious how others handle this!

I built all my local machines because I need them to speed up nixpkgs-review runs; I'd love to contribute them to the community, so everyone can benefit from them, but I'm moving and not sure what to do with them physically.