WIP: Fix ostree install during bootc container build

[tristandruyen]

Description

Similar previous PR: #1646
Resolves: #1645

Context

This PR tries to get the underlying use-case behind #1645 resolved, which is to run the installer in a bootc container build.

While systemd isn't running during the install, we still want to enable all the systemd stuff so it works with systemd once booted for "real".

Bootc also seems to break some other assumptions the installer makes, most notably /root & /usr/local are simlinks to /var.

What I did so far

• quick & dirty changes to make it work, not a finished design on how to solve this properly
• needs workarounds applied after running the installer, but at least in runs through without any fatal errors during container-build now
• for the current workarounds needed during/before/after running the installer in the container build see --no-start-daemon flag does not work with ostree-based installation #1645 (comment)

Status

• With the applied patches nix installs successfully in a bootc container with the some further workarounds seen in the above mentioned issue
• While it logs a warning during the self test:

  WARN SelfTest([ShellFailed { shell: Sh, command: "\"sh\" \"-lc\" \"exec nix build --option substitute false --option post-build-hook \\'\\' --no-link --expr \\'derivation { name = \\\"self-test-sh-1757964187755\\\"; system = \\\"x86_64-linux\\\"; builder = \\\"/bin/sh\\\"; args = [\\\"-c\\\" \\\"echo hello > \\\\$out\\\"]; }\\'\"", output: Output { status: ExitStatus(unix_wait_status(256)), stdout: "", stderr: "error: cannot connect to socket at '/nix/var/nix/daemon-socket/socket': No such file or directory\n" } }, ShellFailed { shell: Bash, command: "\"bash\" \"-lc\" \"exec nix build --option substitute false --option post-build-hook \\'\\' --no-link --expr \\'derivation { name = \\\"self-test-bash-1757964187846\\\"; system = \\\"x86_64-linux\\\"; builder = \\\"/bin/sh\\\"; args = [\\\"-c\\\" \\\"echo hello > \\\\$out\\\"]; }\\'\"", output: Output { status: ExitStatus(unix_wait_status(256)), stdout: "", stderr: "error: cannot connect to socket at '/nix/var/nix/daemon-socket/socket': No such file or directory\n" } }])
  

  Nix seems to work fine once booted "for real", though I haven't tested deeply yet beyond a simple nix run

open questions

• the installer fails when reloading systemd, currently I skipped this in the ostree planner when --no-start-daemon is set, but that's obviously the wrong approach, might need an extra arg like --skip-daemon-reload
• how to properly solve the issues with simlinked paths
  • DETERMINATE_NIXD_BINARY_PATH & roots $HOME cause issues as bootc containers have a simlink from /usr/local to /var/usrlocal & /root to /var/roothome
  • currently worked-around dirtily by changing DETERMINATE_NIXD_BINARY_PATH & HOME
  • possible solutions
    • detect simlinks in all places where it matters
    • make the location for everything that matters configurable with a flag e.g. --nixd-binary-path && `--
    • somehow detect that we're running in a bootc container build change the defaults automagically
• What I did with StartOrEnableSystemdUnit feels suboptimal, is there a better name or should this just be split into 2 actions?
• currently there's some manual cleanup needed in the container build after running the installer, specifically tmpfiles.d & sysuers.d need to be created for bootc lint to run without warnings. Some/all of this should likely be handled in the planner (See --no-start-daemon flag does not work with ostree-based installation #1645 (comment))
• the container images built from this process only work without intervention on a brand-new install, not on updates as they expect /var/nix to be populated, and /var is only copied over during install, this could be left as an exercise to the image builder to migrate somehow....

I am open to clean it up once the above design decisions have an answer from maintainers.

Checklist

• Formatted with cargo fmt
• Built with nix build
• Ran flake checks with nix flake check
• Added or updated relevant tests (leave unchecked if not applicable)
  • updated fixtures due to action renaming
  • seems there are no tests for ostree, should likely be added? part of this pr ?
• Added or updated relevant documentation (leave unchecked if not applicable)
  • error logs already suggested --no-start daemon when it didn't work
  • rest of needed external changes unclear
• Linked to related issues (leave unchecked if not applicable)

Validating with install.determinate.systems

If a maintainer has added the upload to s3 label to this PR, it will become available for installation via install.determinate.systems:

curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix/pr/$PR_NUMBER | sh -s -- install

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}