Failing to Rejoin to Existing Cluster #12205

Silvenga · 2025-04-22T23:06:19Z

Silvenga
Apr 22, 2025

Environmental Info:
K3s Version: k3s version v1.29.15+k3s1 (35a4723)

Node(s) CPU architecture, OS, and Version:

Linux gk1 6.1.0-33-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.133-1 (2025-04-10) x86_64 GNU/Linux

Cluster Configuration:

3 servers, 1 agent

Describe the bug:

One of the server nodes went corrupt, but after re-installing it, joining the existing cluster fails.

I see the bootstrap succeeding, but then later I see:

failed to verify certificate: x509: certificate signed by unknown authority

from Etcd. Is the snapshot too large?

Steps To Reproduce:

export K3S_TOKEN=<blah>
curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=v1.29 sh -s - \
  --disable traefik \
  --disable local-storage \
  --disable servicelb \
  --server https://kube-api.service.<blah>:6443 \
  --tls-san kube-api.service.<blah> \
  --flannel-backend wireguard-native \
  --kubelet-arg=allowed-unsafe-sysctls=net.ipv4.conf.all.src_valid_mark,net.ipv4.ip_forward,net.ipv6.conf.all.forwarding

Expected behavior:

Actual behavior:

Additional context / logs:

Apr 22 17:44:12 gk1 sh[18391]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Apr 22 17:44:13 gk1 k3s[18395]: time="2025-04-22T17:44:13-05:00" level=info msg="Starting k3s v1.29.15+k3s1 (35a47239)"
Apr 22 17:44:13 gk1 k3s[18395]: time="2025-04-22T17:44:13-05:00" level=info msg="Managed etcd cluster not yet initialized"
Apr 22 17:44:13 gk1 k3s[18395]: time="2025-04-22T17:44:13-05:00" level=info msg="Reconciling bootstrap data between datastore and disk"
Apr 22 17:44:13 gk1 k3s[18395]: time="2025-04-22T17:44:13-05:00" level=info msg="Migrating bootstrap data to new format"
Apr 22 17:44:13 gk1 k3s[18395]: time="2025-04-22T17:44:13-05:00" level=info msg=start
Apr 22 17:44:13 gk1 k3s[18395]: time="2025-04-22T17:44:13-05:00" level=info msg="schedule, now=2025-04-22T17:44:13-05:00, entry=1, next=2025-04-23T00:00:00-05:00"
Apr 22 17:44:13 gk1 k3s[18395]: time="2025-04-22T17:44:13-05:00" level=info msg="Failed to test data store connection: failed to get etcd status: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""
Apr 22 17:44:13 gk1 k3s[18395]: time="2025-04-22T17:44:13-05:00" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
Apr 22 17:44:13 gk1 k3s[18395]: time="2025-04-22T17:44:13-05:00" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259"
Apr 22 17:44:13 gk1 k3s[18395]: time="2025-04-22T17:44:13-05:00" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true"
Apr 22 17:44:13 gk1 k3s[18395]: time="2025-04-22T17:44:13-05:00" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-config=/var/lib/rancher/k3s/server/etc/cloud-config.yaml --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --controllers=*,-route,-service --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect-resource-name=k3s-cloud-controller-manager --node-status-update-frequency=1m0s --profiling=false"
Apr 22 17:44:13 gk1 k3s[18395]: time="2025-04-22T17:44:13-05:00" level=info msg="Server node token is available at /var/lib/rancher/k3s/server/token"
Apr 22 17:44:13 gk1 k3s[18395]: time="2025-04-22T17:44:13-05:00" level=info msg="To join server node to cluster: k3s server -s https://<blah>.41:6443 -t ${SERVER_NODE_TOKEN}"
Apr 22 17:44:13 gk1 k3s[18395]: time="2025-04-22T17:44:13-05:00" level=info msg="Agent node token is available at /var/lib/rancher/k3s/server/agent-token"
Apr 22 17:44:13 gk1 k3s[18395]: time="2025-04-22T17:44:13-05:00" level=info msg="To join agent node to cluster: k3s agent -s https://<blah>.41:6443 -t ${AGENT_NODE_TOKEN}"
Apr 22 17:44:13 gk1 k3s[18395]: time="2025-04-22T17:44:13-05:00" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"
Apr 22 17:44:13 gk1 k3s[18395]: time="2025-04-22T17:44:13-05:00" level=info msg="Run: k3s kubectl"
Apr 22 17:44:14 gk1 k3s[18395]: time="2025-04-22T17:44:14-05:00" level=info msg="Password verified locally for node gk1"
Apr 22 17:44:14 gk1 k3s[18395]: time="2025-04-22T17:44:14-05:00" level=info msg="certificate CN=gk1 signed by CN=k3s-server-ca@1596308020: notBefore=2020-08-01 18:53:40 +0000 UTC notAfter=2026-04-22 22:44:14 +0000 UTC"
Apr 22 17:44:15 gk1 k3s[18395]: time="2025-04-22T17:44:15-05:00" level=info msg="certificate CN=system:node:gk1,O=system:nodes signed by CN=k3s-client-ca@1596308020: notBefore=2020-08-01 18:53:40 +0000 UTC notAfter=2026-04-22 22:44:15 +0000 UTC"
Apr 22 17:44:15 gk1 k3s[18395]: time="2025-04-22T17:44:15-05:00" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1596308020: notBefore=2020-08-01 18:53:40 +0000 UTC notAfter=2026-04-22 22:44:15 +0000 UTC"
Apr 22 17:44:15 gk1 k3s[18395]: time="2025-04-22T17:44:15-05:00" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1596308020: notBefore=2020-08-01 18:53:40 +0000 UTC notAfter=2026-04-22 22:44:15 +0000 UTC"
Apr 22 17:44:15 gk1 k3s[18395]: time="2025-04-22T17:44:15-05:00" level=info msg="Module overlay was already loaded"
Apr 22 17:44:15 gk1 k3s[18395]: time="2025-04-22T17:44:15-05:00" level=info msg="Module nf_conntrack was already loaded"
Apr 22 17:44:15 gk1 k3s[18395]: time="2025-04-22T17:44:15-05:00" level=info msg="Module br_netfilter was already loaded"
Apr 22 17:44:15 gk1 k3s[18395]: time="2025-04-22T17:44:15-05:00" level=info msg="Module iptable_nat was already loaded"
Apr 22 17:44:15 gk1 k3s[18395]: time="2025-04-22T17:44:15-05:00" level=info msg="Module iptable_filter was already loaded"
Apr 22 17:44:15 gk1 k3s[18395]: time="2025-04-22T17:44:15-05:00" level=warning msg="Failed to load kernel module nft-expr-counter with modprobe"
Apr 22 17:44:15 gk1 k3s[18395]: time="2025-04-22T17:44:15-05:00" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
Apr 22 17:44:15 gk1 k3s[18395]: time="2025-04-22T17:44:15-05:00" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
Apr 22 17:44:16 gk1 k3s[18395]: time="2025-04-22T17:44:16-05:00" level=info msg="containerd is now running"
Apr 22 17:44:16 gk1 k3s[18395]: time="2025-04-22T17:44:16-05:00" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"
Apr 22 17:44:16 gk1 k3s[18395]: time="2025-04-22T17:44:16-05:00" level=info msg="Creating k3s-cert-monitor event broadcaster"
Apr 22 17:44:16 gk1 k3s[18395]: time="2025-04-22T17:44:16-05:00" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.conf.all.src_valid_mark,net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --feature-gates=CloudDualStackNodeIPs=true --healthz-bind-address=127.0.0.1 --hostname-override=gk1 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --node-ip=<blah>.41 --node-labels= --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/etc/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"
Apr 22 17:44:16 gk1 k3s[18395]: time="2025-04-22T17:44:16-05:00" level=info msg="Handling backend connection request [gk1]"
Apr 22 17:44:16 gk1 k3s[18395]: time="2025-04-22T17:44:16-05:00" level=info msg="Remotedialer connected to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"
Apr 22 17:44:17 gk1 k3s[18395]: time="2025-04-22T17:44:17-05:00" level=error msg="Sending HTTP/1.1 503 response to 127.0.0.1:52610: runtime core not ready"
Apr 22 17:44:17 gk1 k3s[18395]: time="2025-04-22T17:44:17-05:00" level=info msg="Adding member gk1-aa5a3a23=https://<blah>.41:2380 to etcd cluster [gk2-6e4547d7=https://<blah>.42:2380 gk3-c7c15c7d=https://<blah>.43:2380]"
Apr 22 17:44:17 gk1 k3s[18395]: time="2025-04-22T17:44:17-05:00" level=fatal msg="etcd cluster join failed: rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\""
Apr 22 17:44:17 gk1 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE

On a healthy node, I see

Apr 22 17:53:51 gk2 k3s[1249015]: time="2025-04-22T17:53:51-05:00" level=info msg="Serving HTTP bootstrap from datastore for <blah>.41:49462"
Apr 22 17:54:10 gk2 k3s[1249015]: time="2025-04-22T17:54:10-05:00" level=info msg="Serving HTTP bootstrap from datastore for <blah>.41:33608"
Apr 22 17:54:20 gk2 k3s[1249015]: time="2025-04-22T17:54:20-05:00" level=info msg="Serving HTTP bootstrap from datastore for <blah>.41:33632"
Apr 22 17:54:29 gk2 k3s[1249015]: time="2025-04-22T17:54:29-05:00" level=info msg="Serving HTTP bootstrap from datastore for <blah>.41:53006

(confirms networking is good)

Answered by brandond

Apr 23, 2025

Yeah... something's definitely wrong here; the bootstrap data in the datastore is somehow incomplete so the nodes are generating their own CA data. What I'd probably try is:

Restore a snapshot to server 1
On server 1, run k3s certificate rotate-ca --force --path /var/lib/rancher/k3s/server to force a write of the content on disk to the datastore
Restart server 1
Join additional servers to the cluster

View full answer

brandond · 2025-04-22T23:17:16Z

brandond
Apr 22, 2025
Collaborator

Apr 22 17:44:13 gk1 k3s[18395]: time="2025-04-22T17:44:13-05:00" level=info msg="Starting k3s v1.29.15+k3s1 (35a47239)"
Apr 22 17:44:13 gk1 k3s[18395]: time="2025-04-22T17:44:13-05:00" level=info msg="Managed etcd cluster not yet initialized"
Apr 22 17:44:13 gk1 k3s[18395]: time="2025-04-22T17:44:13-05:00" level=info msg="Reconciling bootstrap data between datastore and disk"
Apr 22 17:44:13 gk1 k3s[18395]: time="2025-04-22T17:44:13-05:00" level=info msg="Migrating bootstrap data to new format"

Migrating bootstrap data to new format

You should not be seeing this message. What version are your other nodes on? K3s has not written bootstrap data in a format that would need migration since K3s v1.22 back in 2021. (#3398).

The fact that you are seeing this message indicates that either your other nodes are on a VERY old version, or the data in etcd on the other nodes is also corrupt. If the nodes are not on a very old version, I would suggest restoring from an etcd snapshot.

Before doing this - and this is a long shot, I don't know what exactly happened to your node - you might try running k3s-killall.sh then rm -rf /var/lib/rancher/k3s/server/db/etcd /var/lib/rancher/k3s/server/tls, and then joining the cluster again.

8 replies

Silvenga Apr 22, 2025
Author

Thanks @brandond! I'll start poking around and try bootstrapping from another node in the cluster (seems to be flavoring just the one).

I know restoring a Etcd snapshot requires going into single server mode, and then rejoining - do you know if that would that clear out all the corruption from the bootstrap data?

brandond Apr 22, 2025
Collaborator

It should. You could always try restoring the snapshot on the currently-broken node, and if that comes up successfully, then you can joint the other two back to it.

Silvenga Apr 23, 2025
Author

An on-demand snapshot was restored onto the "out" node, and started joining the other nodes to that "out" node. Each are failing due to the same issue:

Migrating bootstrap data to new format
...
Failed to test data store connection: failed to get etcd status: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing
...
Adding member gk2-c1ed337f=https://<blah>.42:2380 to etcd cluster [gk1-b0c63c06=https://<blah>.41:2380]
...
x509: certificate signed by unknown authority

Certificates are completely different:

sha1sum /var/lib/rancher/k3s/server/tls/etcd/

gk1
a405d73f2c6060a5763d28fe0391e76eef9a0ae5  client.crt
e8baffb756c07037081d3ae3f8e57aee2557ee96  client.key
361b4fbada31d2d91f3df646a2dd4048b8ae3c8c  peer-ca.crt
9efa288a6d9f0f33a565bdb72727255368def857  peer-ca.key
2e900324077ae093ac4af5377c1e85fac512d301  peer-server-client.crt
4940117b7b04b8b616ab1457b459c69a81532863  peer-server-client.key
d7630827a3628409c9676d2275da69d3a20799c6  server-ca.crt
3d983510d10a186359611b8b215648d636f00c37  server-ca.key
3f6b7a312a78b586bd2b0782c8c3a9cd5f386868  server-client.crt
4b9c6d38455e38d86b146a8452c574ea64bca43e  server-client.key

gk2
e0f2b2f0f3b6c2befcf2dd4832b584ceb191e64e  client.crt
d3c4f9ac698fa804b5114ef8fc94047b8c19f5c9  client.key
c889069628ec163e6a6321f499240a0708999490  peer-ca.crt
3c11ada73ddc03adf7607313789afe91adbdabb7  peer-ca.key
3e24a9f43ed883f2aed0c631d00cd3495a38847f  peer-server-client.crt
a66d2c9b57fa1e1541306df4f1916e1240927026  peer-server-client.key
977a8161304c8d6077f24da24cb623583a82f852  server-ca.crt
ac6cad32d174baea4c96b0f6af7c294b03ce0615  server-ca.key
8d25875dbb5aaf69d82b13ff1c805f426b6b54a5  server-client.crt
d05c03cad1c603d18388df14860e50b1a4d5032b  server-client.key

brandond Apr 23, 2025
Collaborator

Yeah... something's definitely wrong here; the bootstrap data in the datastore is somehow incomplete so the nodes are generating their own CA data. What I'd probably try is:

Restore a snapshot to server 1
On server 1, run k3s certificate rotate-ca --force --path /var/lib/rancher/k3s/server to force a write of the content on disk to the datastore
Restart server 1
Join additional servers to the cluster

Answer selected by Silvenga

Silvenga Apr 25, 2025
Author

@brandond that worked! Thanks for your help!

kubectl get nodes

NAME   STATUS   ROLES                       AGE   VERSION
gk1    Ready    control-plane,etcd,master   2d    v1.29.15+k3s1
gk2    Ready    control-plane,etcd,master   2d    v1.29.15+k3s1
gk3    Ready    control-plane,etcd,master   2d    v1.29.15+k3s1
gk4    Ready    <none>                      94d   v1.29.15+k3s1

No migration of the bootstrap data, and looks like the certs were imported correctly, unlike before:

Apr 24 19:39:27 gk3 k3s[1936]: time="2025-04-24T19:39:27-05:00" level=info msg="Acquiring lock file /var/lib/rancher/k3s/data/.lock"
Apr 24 19:39:27 gk3 k3s[1936]: time="2025-04-24T19:39:27-05:00" level=info msg="Preparing data dir /var/lib/rancher/k3s/data/a581eef12a8ffe1f21d99fbda5fec39763623c78ba13d44e2fee91ebc557306d"
Apr 24 19:39:31 gk3 k3s[1936]: time="2025-04-24T19:39:31-05:00" level=info msg="Starting k3s v1.29.15+k3s1 (35a47239)"
Apr 24 19:39:31 gk3 k3s[1936]: time="2025-04-24T19:39:31-05:00" level=info msg="Managed etcd cluster not yet initialized"
Apr 24 19:39:31 gk3 k3s[1936]: time="2025-04-24T19:39:31-05:00" level=info msg="Reconciling bootstrap data between datastore and disk"
Apr 24 19:39:31 gk3 k3s[1936]: time="2025-04-24T19:39:31-05:00" level=info msg="certificate CN=system:admin,O=system:masters signed by CN=k3s-client-ca@1596308020: notBefore=2020-08-01 18:53:40 +0000 UTC notAfter=2026-04-25 00:39:31 +0000 UTC"
Apr 24 19:39:31 gk3 k3s[1936]: time="2025-04-24T19:39:31-05:00" level=info msg="certificate CN=system:k3s-supervisor,O=system:masters signed by CN=k3s-client-ca@1596308020: notBefore=2020-08-01 18:53:40 +0000 UTC notAfter=2026-04-25 00:39:31 +0000 UTC"
Apr 24 19:39:31 gk3 k3s[1936]: time="2025-04-24T19:39:31-05:00" level=info msg="certificate CN=system:kube-controller-manager signed by CN=k3s-client-ca@1596308020: notBefore=2020-08-01 18:53:40 +0000 UTC notAfter=2026-04-25 00:39:31 +0000 UTC"
Apr 24 19:39:31 gk3 k3s[1936]: time="2025-04-24T19:39:31-05:00" level=info msg="certificate CN=system:kube-scheduler signed by CN=k3s-client-ca@1596308020: notBefore=2020-08-01 18:53:40 +0000 UTC notAfter=2026-04-25 00:39:31 +0000 UTC"
Apr 24 19:39:31 gk3 k3s[1936]: time="2025-04-24T19:39:31-05:00" level=info msg="certificate CN=system:apiserver,O=system:masters signed by CN=k3s-client-ca@1596308020: notBefore=2020-08-01 18:53:40 +0000 UTC notAfter=2026-04-25 00:39:31 +0000 UTC"
Apr 24 19:39:31 gk3 k3s[1936]: time="2025-04-24T19:39:31-05:00" level=info msg="certificate CN=k3s-cloud-controller-manager signed by CN=k3s-client-ca@1596308020: notBefore=2020-08-01 18:53:40 +0000 UTC notAfter=2026-04-25 00:39:31 +0000 UTC"
Apr 24 19:39:31 gk3 k3s[1936]: time="2025-04-24T19:39:31-05:00" level=info msg="certificate CN=kube-apiserver signed by CN=k3s-server-ca@1596308020: notBefore=2020-08-01 18:53:40 +0000 UTC notAfter=2026-04-25 00:39:31 +0000 UTC"
Apr 24 19:39:31 gk3 k3s[1936]: time="2025-04-24T19:39:31-05:00" level=info msg="certificate CN=system:auth-proxy signed by CN=k3s-request-header-ca@1596308020: notBefore=2020-08-01 18:53:40 +0000 UTC notAfter=2026-04-25 00:39:31 +0000 UTC"
Apr 24 19:39:31 gk3 k3s[1936]: time="2025-04-24T19:39:31-05:00" level=info msg="certificate CN=etcd-client signed by CN=etcd-server-ca@1745365305: notBefore=2025-04-22 23:41:45 +0000 UTC notAfter=2026-04-25 00:39:31 +0000 UTC"
Apr 24 19:39:31 gk3 k3s[1936]: time="2025-04-24T19:39:31-05:00" level=info msg="certificate CN=etcd-peer signed by CN=etcd-peer-ca@1745365305: notBefore=2025-04-22 23:41:45 +0000 UTC notAfter=2026-04-25 00:39:31 +0000 UTC"
Apr 24 19:39:31 gk3 k3s[1936]: time="2025-04-24T19:39:31-05:00" level=info msg="certificate CN=etcd-server signed by CN=etcd-server-ca@1745365305: notBefore=2025-04-22 23:41:45 +0000 UTC notAfter=2026-04-25 00:39:31 +0000 UTC"
Apr 24 19:39:31 gk3 k3s[1936]: time="2025-04-24T19:39:31-05:00" level=info msg=start
Apr 24 19:39:31 gk3 k3s[1936]: time="2025-04-24T19:39:31-05:00" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1596308020: notBefore=2020-08-01 18:53:40 +0000 UTC notAfter=2026-04-25 00:39:31 +0000 UTC"
Apr 24 19:39:31 gk3 k3s[1936]: time="2025-04-24T19:39:31-05:00" level=info msg="schedule, now=2025-04-24T19:39:31-05:00, entry=1, next=2025-04-25T00:00:00-05:00"

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failing to Rejoin to Existing Cluster #12205

{{title}}

Replies: 1 comment 8 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Failing to Rejoin to Existing Cluster #12205

Silvenga Apr 22, 2025

Replies: 1 comment · 8 replies

brandond Apr 22, 2025 Collaborator

Silvenga Apr 22, 2025 Author

brandond Apr 22, 2025 Collaborator

Silvenga Apr 23, 2025 Author

brandond Apr 23, 2025 Collaborator

Silvenga Apr 25, 2025 Author

Silvenga
Apr 22, 2025

Replies: 1 comment 8 replies

brandond
Apr 22, 2025
Collaborator

Silvenga Apr 22, 2025
Author

brandond Apr 22, 2025
Collaborator

Silvenga Apr 23, 2025
Author

brandond Apr 23, 2025
Collaborator

Silvenga Apr 25, 2025
Author