GracefulNodeShutdown does not appear to work

[pdf]

k3s version:

$ k3s --version
k3s version v1.36.3+k3s1 (5aed4d7b)
go version go1.26.5

OS version:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 26.04 LTS
Release:        26.04
Codename:       resolute
$ uname -a
Linux k0 7.0.0-29-generic #29-Ubuntu SMP PREEMPT_DYNAMIC Fri Jul 17 20:52:35 UTC 2026 x86_64 GNU/Linux

According to the upstream docs, the GracefulNodeShutdown feature gate is enabled by default since Kubernetes v1.21.

The documentation suggests that this relies on systemd inhibitor locks, and prior discussions (#8594) seem to suggest that this feature should work.

When setting the relevant kubelet configuration via the method suggested in the k3s docs, via the following drop-in file:

$ cat /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/90-graceful-shutdown.conf 
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
shutdownGracePeriod: 90s
shutdownGracePeriodCriticalPods: 30s

I can see that the settings are successfully applied to the kublet:

kubectl get --raw "/api/v1/nodes/k0/proxy/configz" | jq '{kubeletconfig: (.kubeletconfig | with_entries(select(.key | test("shutdownGracePeriod"))))}'
{
  "kubeletconfig": {
    "shutdownGracePeriod": "1m30s",
    "shutdownGracePeriodCriticalPods": "30s"
  }
}

However, there is no systemd inhibitor created:

systemd-inhibit --list
WHO                          UID USER PID  COMM            WHAT     WHY                                                       MODE 
ModemManager                 0   root 1004 ModemManager    sleep    ModemManager needs to reset devices                       delay
Unattended Upgrades Shutdown 0   root 1015 unattended-upgr shutdown Stop ongoing upgrades or perform upgrades before shutdown delay

2 inhibitors listed.

Rebooting or shutting down the node does not terminate/drain workloads, the machine simply immediately powers down leaving pods in running state until the cluster notices the node is missing f(expected I guess, given that the inhibitor is not created, and processes are torn down rapidly).

Is this feature expected to work on current k3s?

[brandond]

At the very top of the docs you linked (https://kubernetes.io/docs/concepts/cluster-administration/node-shutdown/), there is a caution about the unattended-upgrades package:

The unattended-upgrades package from Debian conflicts with node graceful shutdown in its normal configuration. If you use the default configuration of unattended-upgrades, which customizes the server shutdown grace period, then the kubelet fails to obtain the necessary lock to handle shutdown events properly. This happens if the shutdownGracePeriod value is greater than 30 seconds.

You are using Ubuntu (which is based on Debian), have unattended-upgrades enabled (as per your list of inhibitors), and your shutdownGracePeriod value is greater than 30 seconds.

Can you confirm that you have read and understood the upstream docs, and addressed the conflict as suggested therein?

[pdf]

That's about obtaining the lock during shutdown, because the default InhibitDelayMaxSec set by unattended-upgrades is 30 seconds (this can be overridden).

The problem is that k3s doesn't register an inhibitor at all. Other k8s distributions correctly register the inhibitor when a grace period is configured in the kubelet.

[brandond]

So have you done what the docs say, or not?

I'm not aware of any issues with it, and users have reported success with it in the past, so unless you can show that you are not simply running into the issue with unattended-upgrades, I think the issue is on your side.

fwiw, I don't believe you'll see an inhibitor active unless there are pods currently preventing the node from shutting down. But its been a minute since I tried it myself.

[pdf]

IMO the upstream recommendation is sub-optimal - having a maximum inhibit timer is sensible to avoid unbounded inhibition of the shutdown process, I have InhibitDelayMaxSec set to 300s (longer than the maximum configured kubelet grace period):

busctl get-property org.freedesktop.login1 /org/freedesktop/login1 org.freedesktop.login1.Manager InhibitDelayMaxUSec
t 300000000

The inhibitor needs to be registered before the shutdown event, I can't see how this could possibly work reliably if it was registered JIT because by the time kubelet receives a signal, the shutdown would already be under way. As I say - on other distributions, this works as expected, for example see the results on k0s in the same lab:

$ systemd-inhibit --list
WHO                          UID USER PID  COMM            WHAT     WHY                                                       MODE 
ModemManager                 0   root 1044 ModemManager    sleep    ModemManager needs to reset devices                       delay
UPower                       0   root 2677 upowerd         sleep    Pause device polling                                      delay
Unattended Upgrades Shutdown 0   root 1024 unattended-upgr shutdown Stop ongoing upgrades or perform upgrades before shutdown delay
kubelet                      0   root 1299 kubelet         shutdown Kubelet needs time to handle node shutdown                delay

EDIT: For reference, so that you can see precisely why the upstream warning exists, this is the entire contents of the unattended-upgrades file in question:

$ cat /usr/lib/systemd/logind.conf.d/unattended-upgrades-logind-maxdelay.conf
[Login]
# delay
InhibitDelayMaxSec=30

Hence the warning specifically about grace periods longer than 30 seconds. Overriding this file only disables the maximum inhibit seconds, it does not affect the registration of inhibitors at all.

[brandond]

Seems to work fine for me... which would seem to confirm that the issue is due to your grace periods being longer than the configured max delay on your node, as previously suggested.

root@ip-172-31-23-114:~# mkdir -p /var/lib/rancher/k3s/agent/etc/kubelet.conf.d
cat <<EOF >/var/lib/rancher/k3s/agent/etc/kubelet.conf.d/90-graceful-shutdown.conf
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
shutdownGracePeriod: 15s
shutdownGracePeriodCriticalPods: 10s
EOF

root@ip-172-31-23-114:~# systemd-inhibit
WHO                          UID USER PID COMM            WHAT     WHY                                                       MODE
ModemManager                 0   root 999 ModemManager    sleep    ModemManager needs to reset devices                       delay
Unattended Upgrades Shutdown 0   root 924 unattended-upgr shutdown Stop ongoing upgrades or perform upgrades before shutdown delay

2 inhibitors listed.

root@ip-172-31-23-114:~# curl -sL get.k3s.io | sh -s -
[INFO]  Finding release for channel stable
[INFO]  Using v1.36.3+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.36.3%2Bk3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.36.3%2Bk3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink '/etc/systemd/system/multi-user.target.wants/k3s.service' → '/etc/systemd/system/k3s.service'.
[INFO]  systemd: Starting k3s

root@ip-172-31-23-114:~# systemd-inhibit
WHO                          UID USER PID  COMM            WHAT     WHY                                                       MODE
ModemManager                 0   root 999  ModemManager    sleep    ModemManager needs to reset devices                       delay
Unattended Upgrades Shutdown 0   root 924  unattended-upgr shutdown Stop ongoing upgrades or perform upgrades before shutdown delay
kubelet                      0   root 1742 k3s-server      shutdown Kubelet needs time to handle node shutdown                delay

3 inhibitors listed.