Talos 1.10 -> 1.11 upgrade and etcd issues

[licquia]

I recently upgraded a cluster that had been on Talos 1.7.x, and ran into the "too many learners" issue when moving from 1.10.7 to 1.11.3. I eventually rebuilt the cluster from scratch (fortunately, in sandbox instead of production), and retried the upgrade, stopping at 1.10.7.

There's a follow-up to the blog post above, where the initial fix for the problem was shown to be incomplete. I'm pretty sure that was what caused my problems with Talos 1.11. In that post, they mention being on etcd 3.5.24 or later. I note that Talos 1.10.x never made it past etcd 3.5.21.

I think the mitigation steps in the follow-up should help when the time comes to make the transition to 1.11.x in the future. All the same, should there be more prominent warnings about the pitfalls of the etcd 3.6 transition somewhere in the upgrade docs? Or would the project consider bumping etcd in 1.10.x to etcd 3.5.24 or later?

(I'd be just as happy to be shown the mistake I made, or pointed to docs that I missed if this has been covered.)

[smira]

In fact Talos only has a default version of etcd, and you can take it to the v3.5.x version yourself via machine config update. (You don't need Talos upgrade to update etcd).

In our integration testing we never hit this issue, but of course it might be the case depending on cluster history.

Same with going to v1.11 you can stay on etcd 3.5.x via machine config change to pin etcd version.