Sanitizing systemd timer service

[LecrisUT]

I have followed the instructions on the systemd timer certificate renewal and everything works great but, there are some errors/warnings that pop-up.

• Failed to get unit file state for {INSTANCE_NAME}.service: No such file or directory
• cert-renewer@{INSTANCE_NAME}.service: Control process exited, code=exited status=1

The latter issue is because of ExecStartPre, and although the comments and documentation say that ExecCondition requires systemd version of >= 243, my CentOS version 239 (239-45.el8) runs it just fine. Maybe the default can be ExecCondition and have the user change it if needed?

For the former one I've used a variation of the lighttpd instruction, and I'm greeted with that message whenever it executes. Any suggestions on how sanitize the log for that issue?

[tashian]

Hi @LecrisUT,

Hmm, the first one is most likely coming from the ExecStartPost in [email protected], which tries to reload or restart a service with the same name as the variable you called {INSTANCE_NAME} above. The goal is to have it silently continue if that service doesn't exist.

Here's the latest version of it:

; Try to reload or restart the service that parallels this cert-renewer, if it is enabled
; if the service doesn't exist, forge ahead.
ExecStartPost=/usr/bin/env bash -c "if ! systemctl --quiet is-enabled %i.service ; then exit 0; fi; systemctl try-rel
oad-or-restart %i"

(I recently updated this line to avoid the error you got. Could you check your [email protected] and confirm that it matches the above snippet?)

The second log message is normal.

I'm glad you're using ExecCondition, it's the preferred option. But, for now we will keep it as ExecStartPost for the broadest compatibility. I tested a couple of distros manually in the past, and I definitely ran into some widely-adopted distros that do not (yet) have ExecCondition support.

[LecrisUT]

Also I was using that version of the [email protected]. I was not using the same name of {INSTANCE_NAME} to be more understandable and I was concerned of the variable including @ in it as well, but I guess that's a non-issue.

Would using a different User in the override change also the user who performs the commands in the main @.service? And I suppose the overridden ExectStartPost comes before the main one?

[LecrisUT]

Interesting, so the override confs on ExecStartPost and ExecStartPre are added to rather than replaced? The error for that message also appears early in the log, before the main executable's if i remember correctly.

[tashian]

That's correct, and you can have as many ExecStartPre and ExecStartPost as you want.

[LecrisUT]

Ok I've found an example on how to override the initial one, simply add ExecStartPost= to clear the initial one. I think using that would be better than silent failing since it would be clearer if the original ExecStartPost is working as intended when it's not overridden.

The other bit of QOL improvement would be to get rid of the logs when ExecCondition=1. Having this log spam every 5 minutes, makes it hard to troubleshoot.

Starting Certificate renewer for 389ds...
[email protected]: Control process exited, code=exited status=1
[email protected]: Skipped due to 'exec-condition'.
Condition check resulted in Certificate renewer for 389ds being skipped.

I've tried setting LogLevelMax but it doesn't do the trick, and I think the way to do it is to setup journald.conf for the cert-renewer service.

[LecrisUT]

Update: configuring journald.conf with LogLevelMax=notice only spits our the Control process exited, code=exited status=1 while LogLevelMax=warning disables all of that. I wanted to setup a separate namespace, like SpammerNotice or such for the [email protected] and add LogNamespace= to the cert-renewer so that the other logs are not interfered. Problem I've found is this feature was only introduced in systemd 245 (at least from tracking down the documentation of it), and has not been back-ported to CentOS's 239-45.el8.

So at the minimum, when someone finds this post, they now know how to reduce the log spam, i.e:

$ cat /etc/systemd/[email protected]
[Journal]
MaxLevelStore=warning
$ sudo systemctl enable --now [email protected]
$ cat /etc/systemd/system/[email protected]
...
LogNamespace=SpammerWarning
...

Also, you can safely add this option for future setup as it doesn't give out any errors when this feature is not implemented.

[LecrisUT]

I think the name of overrides.conf in e.g. /etc/systemd/system/[email protected]/overrides.conf should be override.conf. That is so that when we systemctl edit it points to this file. This is the behaviour in CentOS's systemd at least.

[tashian]

Good catch! I will make the change.

[tashian]

systemd allows any *.conf file but it's nice to know there's a default name.