NixOS 25.05 – Zero Hydra Failures

[leona-ya]

Hi, we are Leona Maroni & Tristan Ross, the release managers for NixOS 25.05 ("Warbler").

We want to invite everyone to participate in the Zero Hydra Failures Project. The goal is to reduce the amount of Hydra failures as much as possible before the release planned for 2025-05-23.

The complete timeline can be found in the release schedule (#390768)

The mission

Nixpkgs is the official package repository with the most packages. With all ecosystem changes, new build failures occur. Our goal is to explicitly take time to stabilize the master branch and after the branch-off (planned for 2025-05-16) the release-25.05 in time for release. We build all free packages in our CI and build platform Hydra in jobsets. Reducing the build failures in these jobsets as much as possible is called the Zero Hydra Failures campaign.

Workflow

Please note that some failures can come up again after they were fixed once, because of world-rebuilds in staging (see the Staging Workflow section below). This can especially happen before the 2025-05-03 (planned) where the last breaking changes with many rebuilds will be merged into master.

Finding broken Packages

Jobsets

Relevant jobsets to check for failing jobs are:

• nixos:trunk-combined (Linux packages & NixOS tests) with the master branch
• nixpkgs:trunk (Linux & Darwin packages) with the master branch

Eval reports

Evaluation reports provide a structural overview of the most impactful failing builds.The s https://github.com/nix-community/nix-review-tools and were automated over at https://github.com/malob/nix-review-tools-reports.

1. Navigate to https://malob.github.io/nix-review-tools-reports/.
2. Open the relevant jobset: see the previous section on which jobset to select based on what you are looking at (Linux, Darwin or NixOS tests).
3. Browse the latest reports for build failures.
   
4. Follow the links to the build failure on hydra and read the logs

ZERO Hydra Failures

The platform automatically crawls Hydra and lists packages by maintainer and lists the most important dependencies (failing packages with the most dependents). It also graphs the general trend per platform.

1. Navigate to https://zh.fail

For the record, we started ZHF here (this will be updated with the last breaking staging changes):

Failing builds on aarch64-darwin:	1638
Failing builds on aarch64-linux:	2235
Failing builds on x86_64-darwin:    1932
Failing builds on x86_64-linux:	    2281
Total failed builds	8087

Check on packages you maintain

1. Clone nixpkgs and checkout the master branch
2. Run

   nix-build maintainers/scripts/build.nix --argstr maintainer <name>
   

Hydra

Hydra is Nixpkgs CI platform, where all active branches are built and pushed into the cache, after which channels can originate from its build results.

1. Open the jobset you want to investigate. Most likely this is nixpkgs:trunk
2. Select the latest evaluation
   
3. The interesting tabs are "Still Failing Jobs" and "Newly Failing Jobs". Directly failing jobs are marked with a red cross, while transitively failing ones are greyed out.
   
4. Use the search form to scope the package list to things relevant to you and that you can test.
   
5. Click on a job you want to investigate. You can see the logs of the latest build here.

Submit fixes

1. Search through PRs to make sure none provided a fix yet. If there is one, please take the time and help review the change.
2. If there is no open PR, troubleshoot why it's failing and fix it.
3. Pull Request the fix against the master branch and wait potential review & change requests
   • Add the 0.kind: ZHF Fixes label, so people can better browse these fixes.
   • If your PR causes more than ~500 rebuilds, it is generally preferred to target staging to avoid compute churn for users on master.
   • If no reviewer is automatically added to your PR, check the Git history or the maintainers and ping them (in the pull request) or add them (if you have the rights) as reviewers.
   • If, after a while, no one has reviewed the PR, you can post it in the review requests room on Matrix or in the topic on Discourse to get more attention.
   • If, after an (extra) while, nothing really happened, you can mention your PR in the Nixpkgs / NixOS contributions matrix room or mention @NixOS/nixos-release-managers on the PR.

Always link back to this issue by mentioning the issue number in the description of your pull request:

ZHF: #403336

Backporting

After 2025-05-16

1. Apply the relevant backport label to land the fix in the release branch:

   • Changes to master get backported into release-25.05.
   • Changes to staging get backported into staging-25.05.

2. If the backport action fails, follow the manual backporting steps. Make sure to use git cherry-pick -x <rev> on all commits intended for backport.

Staging Workflow

Especially when you already are an experienced contributor, we welcome you to contribute to the staging process. PRs that have a large number of rebuilds (typically 500+ up to a full world rebuild) also have a high potential of breakages. In the weeks before the release this is less of an issue, but you we are happy to have more helping hands here. The relevant Hydra Jobset for the staging-next branch is nixpkgs:staging-next and coordination happens in the Staging Matrix room.

Broken packages

Everything we cannot fix in time will need to be marked broken on the respective platforms, so that Hydra will not retry builds over and over, thereby wasting compute resources.

Set meta.broken and add a reference and/or explanation, like this:

meta = {
  # ref to issue/explanation
  broken = stdenv.isDarwin; # only broken on darwin
  # broken = true; # broken on all platforms.
};

Orphaned Packages

You can read about failing packages without a maintainer here: https://zh.fail/failed/by-maintainer/_.html (orphaned packages).

If you're new to NixOS, adopting an orphaned package is a great way to get involved and contribute to the community. By doing so, you'll not only help improve the overall quality of the NixOS ecosystem, but you'll also gain valuable experience working with Nix, the language and tool that powers the package management system.

By adopting an orphaned package, you'll be taking on a responsibility that can be both challenging and rewarding. You'll need to understand the package's code and dependencies, make sure it builds and works correctly, and respond to any issues or pull requests that come up. This process can be a great learning experience, as you'll be exposed to a wide variety of programming languages and libraries.

Moreover, by adopting an orphaned package, you'll be making a tangible impact on the NixOS community. Your contributions will be greatly appreciated by users who depend on that package, and you'll be helping to ensure that NixOS releases are as stable, secure and up-to-date as possible.

Closing

This is a great way to help Nixpkgs and NixOS overall, and it's a great time for new contributors to start their Nixpkgs adventure.

As with the feature freeze issue, please keep the discussion here to a minimum so we don't ping all maintainers (although relevant comments can of course be added here if they are directly ZHF-related) and ping the release managers (@NixOS/nixos-release-managers) in the respective issues.

As always, if something is unclear to you, or you have suggestions, questions, feedback, don't hesitate to reach out to us. You can contact us in the release management matrix room.

Thank you all for your work ❤

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/zero-hydra-failures-25-05-edition/63731/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/lets-have-a-great-25-05-release-cycle/61813/4