unit ordering cycle on shutdown w/ userborn and secrets for users

[zackw]

I get these errors every time I reboot my server: (word wrapped for readability)

systemd[1]: systemd-tmpfiles-setup-dev.service: Found ordering cycle:
    local-fs-pre.target/stop
        after systemd-fsck@dev-disk-by\x2dpartlabel-tinka\x2dvar.service/stop 
        after var.mount/stop
        after sops-install-secrets-for-users.service/stop
        after userborn.service/stop
        after systemd-tmpfiles-setup-dev.service/stop 
      - after local-fs-pre.target
systemd[1]: systemd-tmpfiles-setup-dev.service: Job local-fs-pre.target/stop deleted
    to break ordering cycle starting with systemd-tmpfiles-setup-dev.service/stop

I think all you have to do to reproduce this is have a system with userborn enabled and sops-nix providing at least one secret marked neededForUsers = true, but let me know if you can't make it happen and I'll try to come up with a minimized NixOS config that does it.

[SuperSandro2000]

I cannot reproduce this on a system with nixos-unstable, userborn and the root password provisioned via sops.nix.

Could it be that you unlock a disk with sops and it is caused by that? The machine I looked at does not do that and does not have a luks file system.

[zackw]

No, the system does not have encrypted disks. The secrets provisioned by sops-nix are a password hash (for root), several SSH authorized keys files, and SSH host keys files. (Yes, I know most people use the autogenerated host keys as the sops-nix decryption keys. I think that's actually a mistake.)

It might also be relevant that the system uses an ephemeral root filesystem (hand-rolled, not impermanence) and the master decryption key is in /var.

fileSystems."/var".neededForBoot = true;
sops.age.keyFile = "/var/lib/sops/tinka.key";

Perhaps the fix is to make sops-install-secrets-for-users.service not have an ordering requirement on partitions which are marked neededForBoot?

[SuperSandro2000]

local-fs.target was only added recently in #905. It could be that that needs some more refinement.

Maybe you can set fileSystems.<name>.noCheck to avoid fsck?

Perhaps the fix is to make sops-install-secrets-for-users.service not have an ordering requirement on partitions which are marked neededForBoot?

I think that is induced by

sops-nix/modules/sops/secrets-for-users/default.nix

Line 52 in 56b2406

(lib.lists.optional (cfg.age.keyFile != null) cfg.age.keyFile)

[zackw]

/var is a real filesystem, setting noCheck on it would not be a good plan.

sops-nix/modules/sops/secrets-for-users/default.nix

Lines 49 to 54 in 56b2406

	unitConfig.RequiresMountsFor = lib.concatLists [
	(lib.lists.optional (cfg.gnupg.home != null) cfg.gnupg.home)
	cfg.gnupg.sshKeyPaths
	(lib.lists.optional (cfg.age.keyFile != null) cfg.age.keyFile)
	cfg.age.sshKeyPaths
	];

... This is the Right Thing in the general case, just not this specific one, and I can't think of any good way to filter it down to filesystems that aren't required for boot. And I get the impression from the systemd.unit manpage that there's no way to remove an entry from the RequiresMountsFor list in an override file. So I am out of ideas :-(