What can lead to server/cred/passwd being "newer than datastore"?

[lel-amri]

Hello everyone,

I'm currently debugging a performance issue in my country-wide K3s cluster, and I'm encountering a seemingly unrelated issue with one of my node.

One of my K3s server spits this error at startup, after an unexpected (but seemingly clean) restart:

/var/lib/rancher/k3s/server/cred/passwd newer than datastore and could cause a cluster outage. Remove the file(s) from disk and restart to be recreated from datastore.

Here are the logs that precede:

Oct 19 20:54:48 greenbean k3s[358877]: {"level":"info","ts":"2026-10-19T20:54:48.122062Z","caller":"rafthttp/stream.go:253","msg":"set message encoder","from":"2becb47161ccf748","to":"f994b8c843a1d0f7","stream-type":"stream Message"}
Oct 19 20:54:48 greenbean k3s[358877]: {"level":"info","ts":"2026-10-19T20:54:48.122149Z","caller":"rafthttp/stream.go:278","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"2becb47161ccf748","remote-peer-id":"f994b8c843a1d0f7"}
Oct 19 20:54:48 greenbean k3s[358877]: {"level":"info","ts":"2026-10-19T20:54:48.129917Z","caller":"rafthttp/stream.go:253","msg":"set message encoder","from":"2becb47161ccf748","to":"f994b8c843a1d0f7","stream-type":"stream MsgApp v2"}
Oct 19 20:54:48 greenbean k3s[358877]: {"level":"info","ts":"2026-10-19T20:54:48.129956Z","caller":"rafthttp/stream.go:278","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"2becb47161ccf748","remote-peer-id":"f994b8c843a1d0f7"}
Oct 19 20:54:48 greenbean k3s[358877]: {"level":"warn","ts":"2026-10-19T20:54:48.403839Z","caller":"read/read.go:206","msg":"waiting for ReadIndex response took too long, retrying","sent-request-id":17818669303529580395,"retry-timeout":"500ms"}
Oct 19 20:54:48 greenbean k3s[358877]: {"level":"info","ts":"2026-10-19T20:54:48.403985Z","logger":"raft","caller":"v3@v3.7.0/raft.go:1766","msg":"2becb47161ccf748 no leader at term 26; dropping index reading msg"}
Oct 19 20:54:48 greenbean k3s[358877]: {"level":"warn","ts":"2026-10-19T20:54:48.904187Z","caller":"read/read.go:206","msg":"waiting for ReadIndex response took too long, retrying","sent-request-id":17818669303529580396,"retry-timeout":"500ms"}
Oct 19 20:54:48 greenbean k3s[358877]: {"level":"info","ts":"2026-10-19T20:54:48.904303Z","logger":"raft","caller":"v3@v3.7.0/raft.go:1766","msg":"2becb47161ccf748 no leader at term 26; dropping index reading msg"}
Oct 19 20:54:49 greenbean k3s[358877]: {"level":"warn","ts":"2026-10-19T20:54:49.235104Z","caller":"txn/util.go:93","msg":"apply request took too long","took":"4.996634545s","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/leases/cilium/cilium-operator-resource-lock\"  limit:1","response":"","error":"context canceled"}
Oct 19 20:54:49 greenbean k3s[358877]: {"level":"info","ts":"2026-10-19T20:54:49.235389Z","caller":"traceutil/trace.go:195","msg":"trace","trace_id":1325693625,"operation":"range","detail":"{range_begin:/registry/leases/cilium/cilium-operator-resource-lock; range_end:; }","duration":"4.997052405s","start":"2026-10-19T20:54:44.238296Z","end":"2026-10-19T20:54:49.235349Z","steps":["'agreement among raft nodes before linearized reading'  (duration: 4.996607063s)"],"step_count":1}
Oct 19 20:54:49 greenbean k3s[358877]: {"level":"warn","ts":"2026-10-19T20:54:49.235567Z","caller":"v3rpc/interceptor.go:203","msg":"request stats","start time":"2026-10-19T20:54:44.238256Z","time spent":"4.997268368s","remote":"[::1]:51322","response type":"/etcdserverpb.KV/Range","request count":0,"request size":57,"response count":0,"response size":0,"request content":"key:\"/registry/leases/cilium/cilium-operator-resource-lock\"  limit:1"}
Oct 19 20:54:49 greenbean k3s[358877]: {"level":"warn","ts":"2026-10-19T20:54:49.235867Z","logger":"etcd-client","caller":"v3@v3.7.1-k3s3/retry_interceptor.go:68","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x14db907e8600/[::1]:2379","peer":"Peer{Addr: '[::1]:2379', LocalAddr: '[::1]:51322', AuthInfo: 'tls'}","method":"/etcdserverpb.KV/Range","attempt":0,"error":"rpc error: code = Canceled desc = context canceled"}
Oct 19 20:54:49 greenbean k3s[358877]: E1019 20:54:49.236048  358877 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"context canceled\"}: context canceled" logger="UnhandledError"
Oct 19 20:54:49 greenbean k3s[358877]: E1019 20:54:49.236276  358877 writers.go:130] "Unhandled Error" err="apiserver was unable to write a application/vnd.kubernetes.protobuf response: http: Handler timeout" logger="UnhandledError"
Oct 19 20:54:49 greenbean k3s[358877]: E1019 20:54:49.237194  358877 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
Oct 19 20:54:49 greenbean k3s[358877]: E1019 20:54:49.237547  358877 writers.go:144] "Unhandled Error" err="apiserver was unable to write a fallback application/vnd.kubernetes.protobuf response: http: Handler timeout" logger="UnhandledError"
Oct 19 20:54:49 greenbean k3s[358877]: E1019 20:54:49.239188  358877 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.650524ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/cilium/leases/cilium-operator-resource-lock" result=null
Oct 19 20:54:49 greenbean k3s[358877]: {"level":"warn","ts":"2026-10-19T20:54:49.405222Z","caller":"read/read.go:206","msg":"waiting for ReadIndex response took too long, retrying","sent-request-id":17818669303529580397,"retry-timeout":"500ms"}
Oct 19 20:54:49 greenbean k3s[358877]: {"level":"info","ts":"2026-10-19T20:54:49.405433Z","logger":"raft","caller":"v3@v3.7.0/raft.go:1766","msg":"2becb47161ccf748 no leader at term 26; dropping index reading msg"}
Oct 19 20:54:49 greenbean k3s[358877]: {"level":"warn","ts":"2026-10-19T20:54:49.512574Z","caller":"v3rpc/interceptor.go:203","msg":"request stats","start time":"2026-10-19T20:54:44.522280Z","time spent":"4.990269273s","remote":"[::1]:45024","response type":"/etcdserverpb.Cluster/MemberList","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
Oct 19 20:54:49 greenbean k3s[358877]: {"level":"warn","ts":"2026-10-19T20:54:49.512549Z","logger":"k3s.remotestore","caller":"v3@v3.7.1-k3s3/retry_interceptor.go:68","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x14db9f4dd000/[::1]:2379","peer":"Peer{Addr: '[::1]:2379', LocalAddr: '[::1]:45024', AuthInfo: 'tls'}","method":"/etcdserverpb.Cluster/MemberList","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Oct 19 20:54:49 greenbean k3s[358877]: time="2026-10-19T20:54:49Z" level=warning msg="Failed to retrieve HTTP bootstrap data from datastore; falling back to disk for [REDACTED_IP6_1]:50786: context deadline exceeded"
Oct 19 20:54:49 greenbean k3s[358877]: {"level":"warn","ts":"2026-10-19T20:54:49.906496Z","caller":"read/read.go:206","msg":"waiting for ReadIndex response took too long, retrying","sent-request-id":17818669303529580398,"retry-timeout":"500ms"}
Oct 19 20:54:49 greenbean k3s[358877]: {"level":"info","ts":"2026-10-19T20:54:49.906613Z","logger":"raft","caller":"v3@v3.7.0/raft.go:1766","msg":"2becb47161ccf748 no leader at term 26; dropping index reading msg"}
Oct 19 20:54:50 greenbean k3s[358877]: E1019 20:54:50.171440  358877 leaderelection.go:473] "Error retrieving lease lock" err="Get \"https://[::1]:6444/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=5s\": context deadline exceeded" lock="kube-system/kube-scheduler"
Oct 19 20:54:50 greenbean k3s[358877]: I1019 20:54:50.171529  358877 leaderelection.go:304] "Failed to renew lease" lock="kube-system/kube-scheduler" err="context deadline exceeded"
Oct 19 20:54:50 greenbean k3s[358877]: {"level":"warn","ts":"2026-10-19T20:54:50.172510Z","caller":"txn/util.go:93","msg":"apply request took too long","took":"4.992545573s","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/leases/kube-system/kube-scheduler\"  limit:1","response":"","error":"context canceled"}
Oct 19 20:54:50 greenbean k3s[358877]: {"level":"info","ts":"2026-10-19T20:54:50.172652Z","caller":"traceutil/trace.go:195","msg":"trace","trace_id":2055259015,"operation":"range","detail":"{range_begin:/registry/leases/kube-system/kube-scheduler; range_end:; }","duration":"4.992701404s","start":"2026-10-19T20:54:45.179932Z","end":"2026-10-19T20:54:50.172634Z","steps":["'agreement among raft nodes before linearized reading'  (duration: 4.99254173s)"],"step_count":1}
Oct 19 20:54:50 greenbean k3s[358877]: {"level":"warn","ts":"2026-10-19T20:54:50.172721Z","caller":"v3rpc/interceptor.go:203","msg":"request stats","start time":"2026-10-19T20:54:45.179909Z","time spent":"4.992793313s","remote":"[::1]:51322","response type":"/etcdserverpb.KV/Range","request count":0,"request size":47,"response count":0,"response size":0,"request content":"key:\"/registry/leases/kube-system/kube-scheduler\"  limit:1"}
Oct 19 20:54:50 greenbean k3s[358877]: {"level":"warn","ts":"2026-10-19T20:54:50.173306Z","logger":"etcd-client","caller":"v3@v3.7.1-k3s3/retry_interceptor.go:68","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x14db907e8600/[::1]:2379","peer":"Peer{Addr: '[::1]:2379', LocalAddr: '[::1]:51322', AuthInfo: 'tls'}","method":"/etcdserverpb.KV/Range","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL"}
Oct 19 20:54:50 greenbean k3s[358877]: E1019 20:54:50.173412  358877 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"context canceled\"}: context canceled" logger="UnhandledError"
Oct 19 20:54:50 greenbean k3s[358877]: E1019 20:54:50.173515  358877 writers.go:130] "Unhandled Error" err="apiserver was unable to write a application/vnd.kubernetes.protobuf response: http: Handler timeout" logger="UnhandledError"
Oct 19 20:54:50 greenbean k3s[358877]: E1019 20:54:50.174668  358877 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
Oct 19 20:54:50 greenbean k3s[358877]: E1019 20:54:50.174827  358877 writers.go:144] "Unhandled Error" err="apiserver was unable to write a fallback application/vnd.kubernetes.protobuf response: http: Handler timeout" logger="UnhandledError"
Oct 19 20:54:50 greenbean k3s[358877]: E1019 20:54:50.176230  358877 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="2.954671ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler" result=null
Oct 19 20:54:50 greenbean k3s[358877]: {"level":"warn","ts":"2026-10-19T20:54:50.407207Z","caller":"read/read.go:206","msg":"waiting for ReadIndex response took too long, retrying","sent-request-id":17818669303529580399,"retry-timeout":"500ms"}
Oct 19 20:54:50 greenbean k3s[358877]: {"level":"info","ts":"2026-10-19T20:54:50.407369Z","logger":"raft","caller":"v3@v3.7.0/raft.go:1766","msg":"2becb47161ccf748 no leader at term 26; dropping index reading msg"}
Oct 19 20:54:50 greenbean k3s[358877]: time="2026-10-19T20:54:50Z" level=info msg="Connected to proxy" url="wss://[REDACTED_IPV6_2]:6443/v1-k3s/connect"
Oct 19 20:54:50 greenbean k3s[358877]: time="2026-10-19T20:54:50Z" level=info msg="Remotedialer connected to proxy" url="wss://[REDACTED_IPV6_2]:6443/v1-k3s/connect"
Oct 19 20:54:50 greenbean k3s[358877]: {"level":"warn","ts":"2026-10-19T20:54:50.650761Z","logger":"etcd-client","caller":"v3@v3.7.1-k3s3/retry_interceptor.go:68","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x14db907e8600/[::1]:2379","peer":"Peer{Addr: '[::1]:2379', LocalAddr: '[::1]:51322', AuthInfo: 'tls'}","method":"/etcdserverpb.KV/Range","attempt":0,"error":"rpc error: code = Canceled desc = context canceled"}
Oct 19 20:54:50 greenbean k3s[358877]: E1019 20:54:50.650910  358877 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"context canceled\"}: context canceled" logger="UnhandledError"
Oct 19 20:54:50 greenbean k3s[358877]: E1019 20:54:50.651010  358877 writers.go:130] "Unhandled Error" err="apiserver was unable to write a application/vnd.kubernetes.protobuf response: http: Handler timeout" logger="UnhandledError"
Oct 19 20:54:50 greenbean k3s[358877]: {"level":"warn","ts":"2026-10-19T20:54:50.651044Z","caller":"txn/util.go:93","msg":"apply request took too long","took":"4.998905693s","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/leases/hetzner/karpenter-leader-election\"  limit:1","response":"","error":"context canceled"}
Oct 19 20:54:50 greenbean k3s[358877]: {"level":"info","ts":"2026-10-19T20:54:50.652284Z","caller":"traceutil/trace.go:195","msg":"trace","trace_id":1802485548,"operation":"range","detail":"{range_begin:/registry/leases/hetzner/karpenter-leader-election; range_end:; }","duration":"5.000140063s","start":"2026-10-19T20:54:45.652103Z","end":"2026-10-19T20:54:50.652243Z","steps":["'agreement among raft nodes before linearized reading'  (duration: 4.998902075s)"],"step_count":1}
Oct 19 20:54:50 greenbean k3s[358877]: {"level":"warn","ts":"2026-10-19T20:54:50.652378Z","caller":"v3rpc/interceptor.go:203","msg":"request stats","start time":"2026-10-19T20:54:45.652084Z","time spent":"5.000269383s","remote":"[::1]:51322","response type":"/etcdserverpb.KV/Range","request count":0,"request size":54,"response count":0,"response size":0,"request content":"key:\"/registry/leases/hetzner/karpenter-leader-election\"  limit:1"}
Oct 19 20:54:50 greenbean k3s[358877]: E1019 20:54:50.651969  358877 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
Oct 19 20:54:50 greenbean k3s[358877]: E1019 20:54:50.652483  358877 writers.go:144] "Unhandled Error" err="apiserver was unable to write a fallback application/vnd.kubernetes.protobuf response: http: Handler timeout" logger="UnhandledError"
Oct 19 20:54:50 greenbean k3s[358877]: E1019 20:54:50.653795  358877 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="3.079841ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/hetzner/leases/karpenter-leader-election" result=null
Oct 19 20:54:50 greenbean k3s[358877]: E1019 20:54:50.721360  358877 leaderelection.go:473] "Error retrieving lease lock" err="Get \"https://[::1]:6444/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=5s\": context deadline exceeded" lock="kube-system/kube-controller-manager"
Oct 19 20:54:50 greenbean k3s[358877]: I1019 20:54:50.721558  358877 leaderelection.go:304] "Failed to renew lease" lock="kube-system/kube-controller-manager" err="context deadline exceeded"
Oct 19 20:54:50 greenbean k3s[358877]: {"level":"warn","ts":"2026-10-19T20:54:50.721699Z","caller":"txn/util.go:93","msg":"apply request took too long","took":"4.991817432s","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/leases/kube-system/kube-controller-manager\"  limit:1","response":"","error":"context canceled"}
Oct 19 20:54:50 greenbean k3s[358877]: {"level":"info","ts":"2026-10-19T20:54:50.721773Z","caller":"traceutil/trace.go:195","msg":"trace","trace_id":386193658,"operation":"range","detail":"{range_begin:/registry/leases/kube-system/kube-controller-manager; range_end:; }","duration":"4.991907589s","start":"2026-10-19T20:54:45.729855Z","end":"2026-10-19T20:54:50.721763Z","steps":["'agreement among raft nodes before linearized reading'  (duration: 4.991815236s)"],"step_count":1}
Oct 19 20:54:50 greenbean k3s[358877]: {"level":"warn","ts":"2026-10-19T20:54:50.721823Z","caller":"v3rpc/interceptor.go:203","msg":"request stats","start time":"2026-10-19T20:54:45.729832Z","time spent":"4.991979159s","remote":"[::1]:51322","response type":"/etcdserverpb.KV/Range","request count":0,"request size":56,"response count":0,"response size":0,"request content":"key:\"/registry/leases/kube-system/kube-controller-manager\"  limit:1"}
Oct 19 20:54:50 greenbean k3s[358877]: {"level":"warn","ts":"2026-10-19T20:54:50.721467Z","logger":"etcd-client","caller":"v3@v3.7.1-k3s3/retry_interceptor.go:68","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x14db907e8600/[::1]:2379","peer":"Peer{Addr: '[::1]:2379', LocalAddr: '[::1]:51322', AuthInfo: 'tls'}","method":"/etcdserverpb.KV/Range","attempt":0,"error":"rpc error: code = Canceled desc = context canceled"}
Oct 19 20:54:50 greenbean k3s[358877]: E1019 20:54:50.721998  358877 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"context canceled\"}: context canceled" logger="UnhandledError"
Oct 19 20:54:50 greenbean k3s[358877]: E1019 20:54:50.722094  358877 writers.go:130] "Unhandled Error" err="apiserver was unable to write a application/vnd.kubernetes.protobuf response: http: Handler timeout" logger="UnhandledError"
Oct 19 20:54:50 greenbean k3s[358877]: I1019 20:54:50.722889  358877 event.go:395] "Event occurred" object="kube-system/kube-controller-manager" fieldPath="" kind="Lease" apiVersion="coordination.k8s.io/v1" type="Normal" reason="LeaderElection" message="greenbean_4e872ae0-d008-4eaf-bc61-ef7683d0e9bd stopped leading"
Oct 19 20:54:50 greenbean k3s[358877]: E1019 20:54:50.723041  358877 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
Oct 19 20:54:50 greenbean k3s[358877]: E1019 20:54:50.723164  358877 writers.go:144] "Unhandled Error" err="apiserver was unable to write a fallback application/vnd.kubernetes.protobuf response: http: Handler timeout" logger="UnhandledError"
Oct 19 20:54:50 greenbean k3s[358877]: E1019 20:54:50.724068  358877 controllermanager.go:413] "leaderelection lost/stopped"
Oct 19 20:54:50 greenbean systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Oct 19 20:54:50 greenbean systemd[1]: k3s.service: Failed with result 'exit-code'.
Oct 19 20:54:50 greenbean systemd[1]: k3s.service: Unit process 38744 (containerd-shim) remains running after unit stopped.
Oct 19 20:54:50 greenbean systemd[1]: k3s.service: Unit process 38802 (containerd-shim) remains running after unit stopped.
Oct 19 20:54:50 greenbean systemd[1]: k3s.service: Unit process 39660 (containerd-shim) remains running after unit stopped.
Oct 19 20:54:50 greenbean systemd[1]: k3s.service: Unit process 39706 (containerd-shim) remains running after unit stopped.
Oct 19 20:54:50 greenbean systemd[1]: k3s.service: Unit process 39756 (containerd-shim) remains running after unit stopped.
Oct 19 20:54:50 greenbean systemd[1]: k3s.service: Unit process 39793 (containerd-shim) remains running after unit stopped.
Oct 19 20:54:50 greenbean systemd[1]: k3s.service: Unit process 44354 (containerd-shim) remains running after unit stopped.
Oct 19 20:54:50 greenbean systemd[1]: k3s.service: Unit process 279605 (containerd-shim) remains running after unit stopped.
Oct 19 20:54:50 greenbean systemd[1]: k3s.service: Unit process 282869 (containerd-shim) remains running after unit stopped.
Oct 19 20:54:50 greenbean systemd[1]: k3s.service: Unit process 283623 (containerd-shim) remains running after unit stopped.
Oct 19 20:54:50 greenbean systemd[1]: k3s.service: Unit process 284536 (containerd-shim) remains running after unit stopped.
Oct 19 20:54:50 greenbean systemd[1]: k3s.service: Consumed 4h 9min 52.277s CPU time.
Oct 19 20:54:56 greenbean systemd[1]: k3s.service: Scheduled restart job, restart counter is at 1.
Oct 19 20:54:56 greenbean systemd[1]: Starting Lightweight Kubernetes...
Oct 19 20:54:56 greenbean k3s[373454]: time="2026-10-19T20:54:56Z" level=info msg="Starting k3s v1.37.1+k3s1 (356c0254)"
Oct 19 20:54:56 greenbean k3s[373454]: time="2026-10-19T20:54:56Z" level=info msg="Managed etcd cluster bootstrap already complete and initialized"
Oct 19 20:54:56 greenbean k3s[373454]: time="2026-10-19T20:54:56Z" level=info msg="Reconciling bootstrap data between datastore and disk"
Oct 19 20:54:56 greenbean k3s[373454]: time="2026-10-19T20:54:56Z" level=info msg="Opening etcd MVCC KV backend database at /var/lib/rancher/k3s/server/db/etcd-tmp/member/snap/db"
Oct 19 20:54:56 greenbean k3s[373454]: {"level":"info","ts":"2026-10-19T20:54:56.327130Z","logger":"k3s.store.bbolt","caller":"backend/backend.go:213","msg":"Opening db file (/var/lib/rancher/k3s/server/db/etcd-tmp/member/snap/db) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: true, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: , ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, MaxSize: 0, NoSync: true, OpenFile: 0x0, Mlock: false, Logger: 0x2361890ee158, NoStatistics: false}"}
Oct 19 20:54:56 greenbean k3s[373454]: {"level":"info","ts":"2026-10-19T20:54:56.375171Z","logger":"k3s.store.bbolt","caller":"bbolt@v1.5.0/db.go:327","msg":"Opening bbolt db (/var/lib/rancher/k3s/server/db/etcd-tmp/member/snap/db) successfully"}
Oct 19 20:54:56 greenbean k3s[373454]: {"level":"info","ts":"2026-10-19T20:54:56.375411Z","logger":"k3s.store","caller":"mvcc/kvstore.go:335","msg":"restored last compact revision","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":1068476}
Oct 19 20:54:56 greenbean k3s[373454]: {"level":"info","ts":"2026-10-19T20:54:56.390878Z","logger":"k3s.store","caller":"mvcc/kvstore.go:409","msg":"kvstore restored","current-rev":1071414}
Oct 19 20:54:56 greenbean k3s[373454]: time="2026-10-19T20:54:56Z" level=info msg="Opened etcd MVCC KV store"
Oct 19 20:54:56 greenbean k3s[373454]: time="2026-10-19T20:54:56Z" level=fatal msg="/var/lib/rancher/k3s/server/cred/passwd newer than datastore and could cause a cluster outage. Remove the file(s) from disk and restart to be recreated from datastore."
Oct 19 20:54:56 greenbean systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE

My control-plane has three nodes, greenbean, redbean and whitebean. At the time, greenbean was the leader, and was started with the following options:

--protect-kernel-defaults
--node-name=greenbean
--node-ip=REDACTED_IPV6_3
--service-cidr=fdd7:d716:5e7d:79e5:7376:6300::/112
--token=hunter2
--agent-token=hunter3
--disable-cloud-controller
--flannel-backend=none
--disable-network-policy
--kube-controller-manager-arg=allocate-node-cidrs=false
--disable=traefik
--disable=local-storage
--disable=servicelb
--disable-helm-controller
--cluster-cidr=fdd7:d716:5e7d:79e5:706f:6473::/96
--cluster-init

Can you think of anything that can lead to this situation? (the passwd file being newer than the etcd database (I assume that's what "datastore" means here))

I'm not making this Q&A an issue, because I don't think there is anything wrong with K3s and strongly believe that there is something wrong with my deployment. Maybe this could lead to an improvement in the K3s documentation, but so far I'm too clueless to figure-out what' s going on here.

[nicholas-greenwood]

yes, datastore is etcd here, though the message is easy to misread. in v1.37.1+k3s1, it means your local passwd has different contents from its copy in the bootstrap record, and an mtime at least 3 whole seconds newer than the timestamp saved with that copy. it isn't comparing against the etcd database file's mtime, and a changed timestamp alone wouldn't trigger it

your etcd-tmp/member/snap/db log shows startup is checking greenbean's local etcd copy. that doesn't tell us whether redbean and whitebean have the same bootstrap data

i'd check the file's modification time before changing anything:

sudo stat /var/lib/rancher/k3s/server/cred/passwd

then line that timestamp up with deployment or restore history and clock corrections. a local rewrite of passwd or an older local datastore copy could explain the mismatch, but the read timeouts don't tell us which side changed or why

[brandond]

Oct 19 20:54:50 greenbean k3s[358877]: {"level":"info","ts":"2026-10-19T20:54:50.407369Z","logger":"raft","caller":"v3@v3.7.0/raft.go:1766","msg":"2becb47161ccf748 no leader at term 26; dropping index reading msg"}

your etcd cluster lost quorum, or at least this node lost connection with its peers, and k3s exited as a result, as the local controllers could not renew their leases with the apiserver.

I can't tell you what modified your passwd file, but K3s will only touch this on startup, when it verifies it against what's in the datastore. If the contents differ, but the one on disk is newer, then K3s refuses to start because it does not know why that might be.

If you didn't modify that file on purpose, then the easiest fix is to just remove it and let k3s replace it with the content from the datastore.