LVM Support in talos

[pranav767]

Existing Issue: #12309
We've had some discussion regarding adding LVM capabilities in talos, here
Addressing the first question would be do we even need to introduce LVM in talos.

First of all, CSIs (as far as I know) today happily manage LVM volumes without any Talos involvement, so I'm not sure why we need to introduce any concept of it to Talos.

To be clear Volumes in talos can be defined as a disk/partition/director/overlay mount, talos handles static provisioning really well, for any k8s use case we can use UserVolumes.
Their are few scenarios where the need for LVM arises:

1. When multiple physical disks need to be pooled into a single logical storage pool that workloads can consume as one volume.
2. Gradual storage expansion by adding new disks over time without reconfiguring applications or Pods.
3. Resizing volumes (primarily growing) without recreating disks or redeploying workloads.
4. Strong per-workload storage isolation, preventing one Pod from exhausting disk space and impacting others.
5. Disk replacement and maintenance on a node without stopping workloads or losing data.
6. Use of Volume Layouts like striping or mirroring
7. Fast, point-in-time snapshots for short-term rollback or experimentation.
8. Cloning volumes for development, testing, or CI workloads.

We can have LVM in talos using any csi drivers like csi-driver-lvm or topolvm
But there are multiple limitations which needs to be addressed while using these, topolvm-limitations

I tried using csi-driver-lvm in my initial attempts.

1. First attempt: The cluster broke because csi-driver-lvm was installed without restricting it to worker nodes. The driver attempted pvcreate on the Talos system disk, CSI drivers will destroy the OS disk if allowed.
2. Second attempt: After restricting the driver to worker nodes, the pod started but failed with failed to mkdir "/etc/lvm/backup": read-only file system, as the CSI driver assumed /etc/lvm was writable.
3. Final attempt: Reinstalled the CSI driver with lvm.hostWritePath=/var/lib/lvm, after which the driver started successfully and LVM provisioning worked as expected.

Later setting up a pod and pvc with proper storage class worked fine:

$ kubectl exec -n kube-system -it csi-driver-lvm-kdlx2 \
  -c csi-driver-lvm -- vgs
  VG      #PV #LV #SN Attr   VSize  VFree 
  csi-lvm   2   1   0 wz--n- 19.99g 18.99g

IMHO, this way works fine, we should work on LVM support in Talos if it simplifies the user experience and helps avoid the following issues:

• Reliance on privileged containers, increasing the risk of host compromise
• Regex-based disk selection, which can accidentally wipe the OS disk
• Lack of Talos-level policies and guardrails to prevent operator mistakes
• Assumptions about mutable system paths (e.g. /etc), which break on Talos
• Ad-hoc disk bootstrapping, leading to inconsistent VG layout and capacity reporting
• Absence of declarative intent, making maintenance and recovery more error-prone
• Also, limitations mentioned in topolvm

If these points make a strong case for adding LVM support in Talos, we can further discuss relevant use cases and scenarios, and how such support could improve usability , as done in common config