Summary
sops-install-secrets validates the entire manifest up front and aborts on the first error, before mounting the secrets fs or decrypting anything. As a result, a single secret with an owner that does not resolve to an existing user causes zero secrets to be installed — /run/secrets.d is never even created.
A cosmetic misconfiguration in one unrelated secret therefore takes down every secret-consuming service on the machine.
Observed
On a NixOS host, one secret was declared with owner = "grmpf" while that user was not declared on that particular machine (users.mutableUsers = false, so it could never appear). Activation output:
setting up secrets...
sops-install-secrets: manifest is not valid: failed to lookup user 'grmpf': user: unknown user grmpf
Activation script snippet 'setupSecrets' failed (1)
None of the 10 secrets in the manifest were installed. /run/secrets.d did not exist, /run/secrets was never symlinked. The age key (/var/lib/sops-nix/key.txt) was present and valid, and all encrypted source files existed in the store — 9 of the 10 secrets were owner = "root" and had nothing wrong with them.
Downstream fallout on that host, all from the one bad owner:
yggdrasil.service — 243/CREDENTIALS, LoadCredential= source missing → node off the overlay network entirely (the symptom that started the investigation)
systemd-networkd.service — 243/CREDENTIALS (wireguard private key credential), plus its four sockets
hyprspace.service, p2p-ssh-iroh.service — crash-looping, hundreds of restarts
zerotier — survived only because it fell back to a pre-existing on-disk identity; its init script had already logged identity secret has changed, backing up old identity
sshd-keygen generated a throwaway host key into the missing secret's parent path, creating /run/secrets as a real directory, which will then also block sops-install-secrets from placing its symlink on a subsequent run
The machine was left reachable only by luck. On a headless host with owner pointing at a user that is missing for any reason, this is a remote lockout.
Root cause
validateManifest returns on the first failing secret:
|
|
|
for i := range m.Secrets { |
|
secret := &m.Secrets[i] |
|
if err := app.validateSecret(secret); err != nil { |
|
return err |
for i := range m.Secrets {
secret := &m.Secrets[i]
if err := app.validateSecret(secret); err != nil {
return err
}
...
}
validateSecret → validateOwner does a hard user.Lookup:
|
|
|
func validateOwner(owner string) (int, error) { |
|
lookedUp, err := user.Lookup(owner) |
|
if err != nil { |
|
return 0, fmt.Errorf("failed to lookup user '%s': %w", owner, err) |
|
} |
|
ownerNr, err := strconv.ParseUint(lookedUp.Uid, 10, 64) |
|
if err != nil { |
|
return 0, fmt.Errorf("cannot parse uid %s: %w", lookedUp.Uid, err) |
|
} |
|
return int(ownerNr), nil |
And this all happens before any installation work — validateManifest is called at main.go:1305, whereas MountSecretFs and the decryption/install path only start at main.go:1325:
|
return fmt.Errorf("cannot rename temporary file %s to %s: %w", tempFile.Name(), templatePath, err) |
|
} |
|
tempfileRemoved = true |
|
return nil |
|
} |
|
|
|
func writeTemplates(targetDir string, templates []template, keysGID int, userMode bool) error { |
|
for _, template := range templates { |
|
if err := writeTemplate(targetDir, template, keysGID, userMode); err != nil { |
|
return err |
|
} |
|
} |
|
return nil |
|
} |
|
|
|
func installSecrets(args []string) error { |
|
opts, err := parseFlags(args) |
|
if err != nil { |
|
return err |
|
} |
|
|
|
manifest, err := readManifest(opts.manifest) |
|
if err != nil { |
So the failure mode is strictly all-or-nothing, and the "nothing" case is silent apart from one activation log line.
Expected behaviour
Install as many secrets as possible. A per-secret problem should degrade only that secret.
Concretely, my suggestion:
- In the secret loop, collect per-secret errors instead of returning the first one. Skip the secrets that failed validation, install the rest.
- Log each skipped secret loudly (name + reason).
- Still exit non-zero at the end, with all collected errors aggregated (
errors.Join), so activation still reports failure and CI/deploys still notice — but the machine keeps working.
Errors that are genuinely manifest-global (mutually exclusive gnupgHome/sshKeyPaths/ageKeyFile, unreadable/undecryptable sops files affecting everything, template placeholder resolution) can reasonably stay fatal. It's the per-secret metadata validation — owner, group, mode, format — that should be isolated.
An unresolvable owner in particular has an obvious safe degradation even short of skipping: fall back to owner = root (i.e. the same treatment -ignore-passwd already applies at main.go:605), warn, and install the secret anyway. The file lands with tighter-than-requested ownership, which is fail-safe — no secret is exposed to anyone who was not already root. That would have kept every one of the 10 secrets, including the 9 that were pure root, in place.
Additional suggestion: catch it at eval time
This class of mistake is also detectable in the NixOS module, which would be strictly better than any runtime handling. There is already an implicit coupling — group defaults to config.users.users.${owner}.group:
|
group = lib.mkOption { |
|
type = with lib.types; nullOr str; |
|
default = if config.owner != null then users.${config.owner}.group else null; |
|
defaultText = lib.literalMD "{option}`config.users.users.\${owner}.group`"; |
but that only throws when group is left at its default. In my case group was set explicitly (group = "root"), so eval sailed through and the problem surfaced only at activation on the target machine.
An assertion alongside the existing owner/uid ones at modules/sops/default.nix:449-459, roughly:
{
assertion = secret.owner != null -> config.users.users ? ${secret.owner};
message = "sops.secrets.${secret.name}.owner refers to user '${secret.owner}', which is not declared in users.users";
}
would turn this into a build-time error. Note this only helps for statically declared users, so it complements rather than replaces the runtime robustness above (users created outside users.users would need an escape hatch, e.g. skipping the assertion when users.mutableUsers is true).
Environment
- sops-nix rev
f1406619a3884cd5c47992a70b8b35c9c0fcb4c9
- nixpkgs 26.11 (
26.11.20260804.e72e4f2)
- NixOS, systemd 261.1, x86_64-linux
- Secrets declared indirectly via clan-core
clan.core.vars, which generates sops.secrets with explicit owner/group/mode — worth mentioning because generator-driven setups make it easy for an owner to be templated onto a machine that doesn't declare that user.
Summary
sops-install-secretsvalidates the entire manifest up front and aborts on the first error, before mounting the secrets fs or decrypting anything. As a result, a single secret with anownerthat does not resolve to an existing user causes zero secrets to be installed —/run/secrets.dis never even created.A cosmetic misconfiguration in one unrelated secret therefore takes down every secret-consuming service on the machine.
Observed
On a NixOS host, one secret was declared with
owner = "grmpf"while that user was not declared on that particular machine (users.mutableUsers = false, so it could never appear). Activation output:None of the 10 secrets in the manifest were installed.
/run/secrets.ddid not exist,/run/secretswas never symlinked. The age key (/var/lib/sops-nix/key.txt) was present and valid, and all encrypted source files existed in the store — 9 of the 10 secrets wereowner = "root"and had nothing wrong with them.Downstream fallout on that host, all from the one bad owner:
yggdrasil.service—243/CREDENTIALS,LoadCredential=source missing → node off the overlay network entirely (the symptom that started the investigation)systemd-networkd.service—243/CREDENTIALS(wireguard private key credential), plus its four socketshyprspace.service,p2p-ssh-iroh.service— crash-looping, hundreds of restartszerotier— survived only because it fell back to a pre-existing on-disk identity; its init script had already loggedidentity secret has changed, backing up old identitysshd-keygengenerated a throwaway host key into the missing secret's parent path, creating/run/secretsas a real directory, which will then also blocksops-install-secretsfrom placing its symlink on a subsequent runThe machine was left reachable only by luck. On a headless host with
ownerpointing at a user that is missing for any reason, this is a remote lockout.Root cause
validateManifestreturns on the first failing secret:sops-nix/pkgs/sops-install-secrets/main.go
Lines 731 to 735 in f140661
validateSecret→validateOwnerdoes a harduser.Lookup:sops-nix/pkgs/sops-install-secrets/main.go
Lines 574 to 584 in f140661
And this all happens before any installation work —
validateManifestis called atmain.go:1305, whereasMountSecretFsand the decryption/install path only start atmain.go:1325:sops-nix/pkgs/sops-install-secrets/main.go
Lines 1305 to 1327 in f140661
So the failure mode is strictly all-or-nothing, and the "nothing" case is silent apart from one activation log line.
Expected behaviour
Install as many secrets as possible. A per-secret problem should degrade only that secret.
Concretely, my suggestion:
errors.Join), so activation still reports failure and CI/deploys still notice — but the machine keeps working.Errors that are genuinely manifest-global (mutually exclusive
gnupgHome/sshKeyPaths/ageKeyFile, unreadable/undecryptable sops files affecting everything, template placeholder resolution) can reasonably stay fatal. It's the per-secret metadata validation —owner,group,mode, format — that should be isolated.An unresolvable
ownerin particular has an obvious safe degradation even short of skipping: fall back toowner = root(i.e. the same treatment-ignore-passwdalready applies atmain.go:605), warn, and install the secret anyway. The file lands with tighter-than-requested ownership, which is fail-safe — no secret is exposed to anyone who was not already root. That would have kept every one of the 10 secrets, including the 9 that were pureroot, in place.Additional suggestion: catch it at eval time
This class of mistake is also detectable in the NixOS module, which would be strictly better than any runtime handling. There is already an implicit coupling —
groupdefaults toconfig.users.users.${owner}.group:sops-nix/modules/sops/default.nix
Lines 121 to 124 in f140661
but that only throws when
groupis left at its default. In my casegroupwas set explicitly (group = "root"), so eval sailed through and the problem surfaced only at activation on the target machine.An assertion alongside the existing owner/uid ones at
modules/sops/default.nix:449-459, roughly:would turn this into a build-time error. Note this only helps for statically declared users, so it complements rather than replaces the runtime robustness above (users created outside
users.userswould need an escape hatch, e.g. skipping the assertion whenusers.mutableUsersis true).Environment
f1406619a3884cd5c47992a70b8b35c9c0fcb4c926.11.20260804.e72e4f2)clan.core.vars, which generatessops.secretswith explicitowner/group/mode— worth mentioning because generator-driven setups make it easy for anownerto be templated onto a machine that doesn't declare that user.