kubelet-wrapper fails to start kubelet on k8s 1.17

[cann0nf0dder]

I've submitted the workaround for it in the k8s mainstream repo, sharing here for visibility as well.
kubernetes/kubernetes#90331

What happened:
After upgrade to kubernetes 1.17.5 kubelet does not start
I couldn't find documentation about change to kubelet startup in the release notes.
Found few people running into the same issue on slack. No solution found.

What you expected to happen:
Kubelet starts fine same as on 1.16 branch.

How to reproduce it (as minimally and precisely as possible):
Upgrade to 1.17.5 on CoreOS

Anything else we need to know?:
Workaround provided below to share my finding with the community

Standard CoreOS kubelet.service

[Unit]
Description=kubelet
Wants=rpc-statd.service

[Service]
User=root
EnvironmentFile=/etc/kubernetes/kubelet.env
Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
  --mount volume=resolv,target=/etc/resolv.conf \
  --mount volume=etc-cni-net,target=/etc/cni/net.d \
  --mount volume=var-lib-cni,target=/var/lib/cni \
  --mount volume=opt-cni-bin,target=/opt/cni/bin \
  --mount volume=var-log,target=/var/log \
  --mount volume=root-docker,target=/root/.docker \
  --mount volume=etc-k8s-cfg,target=/etc/kubernetes/config \
  --mount volume=var-lib-calico,target=/var/lib/calico \
  --volume var-lib-calico,kind=host,source=/var/lib/calico \
  --volume resolv,kind=host,source=/etc/resolv.conf \
  --volume etc-cni-net,kind=host,source=/etc/cni/net.d \
  --volume var-lib-cni,kind=host,source=/var/lib/cni \
  --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
  --volume var-log,kind=host,source=/var/log \
  --volume root-docker,kind=host,source=/root/.docker \
  --volume etc-k8s-cfg,kind=host,source=/etc/kubernetes/config \
  --insecure-options=image"

ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
ExecStartPre=/bin/mkdir -p /etc/kubernetes/pki
ExecStartPre=/bin/mkdir -p /opt/cni/bin
ExecStartPre=/bin/mkdir -p /var/lib/cni
ExecStartPre=/bin/mkdir -p /etc/cni/net.d
ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/pki/ca.crt"
ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
ExecStart=/usr/lib/coreos/kubelet-wrapper \
--config=/etc/kubernetes/config/kubelet.yaml \
--cni-conf-dir=/etc/cni/net.d \
--exit-on-lock-contention \
--kubeconfig=/etc/kubernetes/kubeconfig \
--lock-file=/var/run/lock/kubelet.lock \
--network-plugin=cni

ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid

After upgrading to 1.17.5 kubelet doesn't start with the following logs:

Apr 21 08:21:38 nodename kubelet-wrapper[1841]: + exec /usr/bin/rkt run --uuid-file-save=/var/cache/kubelet-pod.uuid --mount volume=resolv,target=/etc/resolv.conf --mount volume=etc-cni-net,target=/etc/cni/net.d --mount>
Apr 21 08:21:40 nodename kubelet-wrapper[1841]: --config=/etc/kubernetes/config/kubelet.yaml: command not supported
Apr 21 09:03:13 nodename kubelet-wrapper[971]: Usage:
Apr 21 09:03:13 nodename kubelet-wrapper[971]:   kubelet [command]
Apr 21 09:03:13 nodename kubelet-wrapper[971]: Available Commands:
Apr 21 09:03:13 nodename kubelet-wrapper[971]:   help                     Help about any command
Apr 21 09:03:13 nodename kubelet-wrapper[971]:   kube-apiserver
Apr 21 09:03:13 nodename kubelet-wrapper[971]:   kube-controller-manager
Apr 21 09:03:13 nodename kubelet-wrapper[971]:   kube-proxy
Apr 21 09:03:13 nodename kubelet-wrapper[971]:   kube-scheduler
Apr 21 09:03:13 nodename kubelet-wrapper[971]:   kubectl                  kubectl controls the Kubernetes cluster manager
Apr 21 09:03:13 nodename kubelet-wrapper[971]:   kubelet

I've noticed that the coreos specific kubelet-wrapper expects one of the following commands before the parameters:
kubelet, kube-apiserver, kube-controller-manager, kube-proxy, kubelet
I've gone ahead and added kubelet to first line under the kubelet-wrapper and I was able to start the kubelet and make first api-server upgrade successful.

Workaround kubelet.service config:

ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
ExecStart=/usr/lib/coreos/kubelet-wrapper \
kubelet --config=/etc/kubernetes/config/kubelet.yaml \
--cni-conf-dir=/etc/cni/net.d \
--exit-on-lock-contention \
--kubeconfig=/etc/kubernetes/kubeconfig \
--lock-file=/var/run/lock/kubelet.lock \
--network-plugin=cni

Don't know if this is a lack of documentation on the recent kubelet change in k8s or coreos kubelet-wrapper specific issue, I thought I'll share it here for comments/thoughts.

Environment:

Kubernetes version (use kubectl version): 1.16.6->1.17.5 upgrade
Cloud provider or hardware configuration: CoreOS
OS (e.g: cat /etc/os-release):
cat /etc/os-release
NAME=Fedora
VERSION="28 (Twenty Eight)"
ID=fedora
VERSION_ID=28
PLATFORM_ID="platform:f28"
PRETTY_NAME="Fedora 28 (Twenty Eight)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:fedoraproject:fedora:28"
HOME_URL="https://fedoraproject.org/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=28
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=28
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
Kernel (e.g. uname -a):
Linux 4.19.86-coreos #1 SMP Mon Dec 2 20:13:38 -00 2019 x86_64 x86_64 x86_64 GNU/Linux