Upgrade from 1.11.6 to 1.12.0 failing on worker nodes with storage mounts

[vacquah]

I have a cluster of 3 masters / 3 workers, with each of the workers mounted with a 1TB SSD for longorn. Managing everthing in a self hosted omni instance. I haven't had any problems upgrading talos or kuberbetes until now. The upgrade on the masters work fine, but crashes consistently on the worker nodes. The logs indicate the upgrade was successful on the workers but cant mount the storage during the reboot. Looks like an issue with the longhorn storage mounts. I am a bit out of my depth trying to decipher this one. Hoping for some help.

I have the following as patches on each worker node ( this is for just one of them ). Anything changed?

patch # 1

machine:
  network:
    hostname: atlas-03
    interfaces:
      - deviceSelector:
          physical: true
        addresses:
          - 10.1.20.43/24
        routes:
          - network: 0.0.0.0/0
            gateway: 10.1.20.1
        dhcp: false
    nameservers:
      - 100.64.0.1
      - 1.1.1.2
      - 1.0.0.2
    extraHostEntries:
      - ip: 10.1.20.41
        aliases:
          - atlas-01
      - ip: 10.1.20.42
        aliases:
          - atlas-02
      - ip: 10.1.20.43
        aliases:
          - atlas-03
      - ip: 10.1.20.44
        aliases:
          - atlas-04
      - ip: 10.1.20.45
        aliases:
          - atlas-05
      - ip: 10.1.20.46
        aliases:
          - atlas-06
  time:
    disabled: false
    servers:
      - time.cloudflare.com

  kubelet:
    extraMounts:
      - destination: /var/mnt/storage
        type: bind
        source: /var/mnt/storage

patch # 2

machine:
  kubelet:
    extraMounts:
      - destination: /var/mnt/storage
        type: bind
        source: /var/mnt/storage
        options:
          - bind
          - rshared
          - rw
  disks:
      - device: /dev/sda # The name of the disk to use.
        partitions:
          - mountpoint: /var/mnt/storage # Where to mount the partition.

I have this in my longhorn values.yaml

# Storage path
defaultSettings:
  defaultDataPath: "/var/mnt/storage"
  # defaultDataPath: "/var/lib/longhorn/" # Use a writable system path
  backupTarget: "s3://k8s-longhorn@feldspar-place/"
  backupTargetCredentialSecret: longhorn-minio-secret
  

[smira]

We can't really help without logs.

FYI, you're using legacy machine.disks method which was deprecated for quite some time (before 1.11). It still works, but it has its own downsides. See https://docs.siderolabs.com/talos/v1.12/configure-your-talos-cluster/storage-and-disk-management/disk-management/overview

Specifically, you can try to migrate to https://docs.siderolabs.com/talos/v1.12/configure-your-talos-cluster/storage-and-disk-management/disk-management/existing to keep your data intact. In the migration, drop any kubelet's extraMounts.

[smira]

you need to find another stable match for the partition, e.g. by matching a specific disk and then matching by partition index.

use talosctl get dv and talosctl get disks to get an idea

[vacquah]

I get the following:

talosctl get disks -n 10.1.20.43
WARNING: 10.1.20.43: server version 1.11.6 is older than client version 1.12.1
NODE         NAMESPACE   TYPE   ID             VERSION   SIZE     READ ONLY   TRANSPORT   ROTATIONAL   WWID                                   MODEL             SERIAL
10.1.20.43   runtime     Disk   loop0          2         4.1 kB   true
10.1.20.43   runtime     Disk   loop1          2         61 kB    true
10.1.20.43   runtime     Disk   loop2          2         483 kB   true
10.1.20.43   runtime     Disk   loop3          2         66 MB    true
10.1.20.43   runtime     Disk   mmcblk0        2         31 GB    false       mmc                                                                               0xc289becb
10.1.20.43   runtime     Disk   mmcblk0boot0   2         4.2 MB   true        mmc
10.1.20.43   runtime     Disk   mmcblk0boot1   2         4.2 MB   true        mmc
10.1.20.43   runtime     Disk   sda            2         1.0 TB   false       sata                     naa.500a0751e6babfb7                   CT1000MX500SSD1
10.1.20.43   runtime     Disk   sdb            5         22 GB    false       iscsi       true         naa.60000000000000000e00000000010001   VIRTUAL-DISK
10.1.20.43   runtime     Disk   sdc            6         11 GB    false       iscsi       true         naa.60000000000000000e00000000020001   VIRTUAL-DISK
10.1.20.43   runtime     Disk   sdd            3         107 GB   false       iscsi       true         naa.60000000000000000e00000000030001   VIRTUAL-DISK

talosctl get disk sda -o yaml -n 10.1.20.43
WARNING: 10.1.20.43: server version 1.11.6 is older than client version 1.12.1
node: 10.1.20.43
metadata:
    namespace: runtime
    type: Disks.block.talos.dev
    id: sda
    version: 2
    owner: block.DisksController
    phase: running
    created: 1970-01-01T00:00:07Z
    updated: 1970-01-01T00:00:07Z
spec:
    dev_path: /dev/sda
    size: 1000204886016
    pretty_size: 1.0 TB
    io_size: 4096
    sector_size: 512
    readonly: false
    cdrom: false
    model: CT1000MX500SSD1
    modalias: scsi:t-0x00
    wwid: naa.500a0751e6babfb7
    bus_path: /platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/ata2/host1/target1:0:0/1:0:0:0
    sub_system: /sys/class/block
    transport: sata
    symlinks:
        - /dev/disk/by-diskseq/32
        - /dev/disk/by-id/ata-CT1000MX500SSD1_2311E6BABFB7
        - /dev/disk/by-id/wwn-0x500a0751e6babfb7
        - /dev/disk/by-path/platform-fd500000.pcie-pci-0000:01:00.0-ata-2
        - /dev/disk/by-path/platform-fd500000.pcie-pci-0000:01:00.0-ata-2.0
        

talosctl get dv -n 10.1.20.43
WARNING: 10.1.20.43: server version 1.11.6 is older than client version 1.12.1
NODE         NAMESPACE   TYPE               ID             VERSION   TYPE        SIZE     DISCOVERED   LABEL       PARTITIONLABEL
10.1.20.43   runtime     DiscoveredVolume   loop2          1         disk        483 kB   squashfs
10.1.20.43   runtime     DiscoveredVolume   loop3          1         disk        66 MB    squashfs
10.1.20.43   runtime     DiscoveredVolume   mmcblk0        1         disk        31 GB    gpt
10.1.20.43   runtime     DiscoveredVolume   mmcblk0boot0   1         disk        4.2 MB
10.1.20.43   runtime     DiscoveredVolume   mmcblk0boot1   1         disk        4.2 MB
10.1.20.43   runtime     DiscoveredVolume   mmcblk0p1      1         partition   105 MB   vfat         EFI         EFI
10.1.20.43   runtime     DiscoveredVolume   mmcblk0p2      1         partition   1.0 MB                            BIOS
10.1.20.43   runtime     DiscoveredVolume   mmcblk0p3      1         partition   1.0 GB   xfs          BOOT        BOOT
10.1.20.43   runtime     DiscoveredVolume   mmcblk0p4      1         partition   1.0 MB   talosmeta                META
10.1.20.43   runtime     DiscoveredVolume   mmcblk0p5      1         partition   105 MB   xfs          STATE       STATE
10.1.20.43   runtime     DiscoveredVolume   mmcblk0p6      1         partition   30 GB    xfs          EPHEMERAL   EPHEMERAL
10.1.20.43   runtime     DiscoveredVolume   sda            1         disk        1.0 TB   gpt
10.1.20.43   runtime     DiscoveredVolume   sda1           1         partition   1.0 TB   xfs
10.1.20.43   runtime     DiscoveredVolume   sdb            1         disk        22 GB    ext4
10.1.20.43   runtime     DiscoveredVolume   sdc            1         disk        11 GB    ext4
10.1.20.43   runtime     DiscoveredVolume   sdd            1         disk        107 GB   ext4

[vacquah]

I completely removed 2 of my workers - so now have just one worker left at 10.1.20.43

[vacquah]

I came up with this:

apiVersion: v1alpha1
kind: ExistingVolumeConfig
name: storage
discovery:
  volumeSelector:
    match: |
      volume.size > 900u * GB &&
      volume.size < 1100u * GB &&
      volume.partition_index == 1

based on this:

---
node: 10.1.20.43
metadata:
    namespace: runtime
    type: DiscoveredVolumes.block.talos.dev
    id: sda1
    version: 1
    owner: block.DiscoveryController
    phase: running
    created: 1970-01-01T00:00:07Z
    updated: 1970-01-01T00:00:07Z
spec:
    dev_path: /dev/sda1
    type: partition
    device_path: /sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/ata2/host1/target1:0:0/1:0:0:0/block/sda/sda1
    parent: sda
    parent_dev_path: /dev/sda
    size: 1000203091968
    pretty_size: 1.0 TB
    sector_size: 512
    io_size: 4096
    name: xfs
    uuid: e04d057f-e408-40e7-bc0a-27a5798c14e8
    block_size: 4096
    filesystem_block_size: 4096
    probed_size: 999714713600
    partition_uuid: 908d45de-bdc3-49cd-9e57-ece2c839f865
    partition_type: 0fc63daf-8483-4772-8e79-3d69d8477de4
    partition_index: 1

[smira]

I would recommend to patch on parent/disk, but up to you.