Unable to decrypt STATE volume after upgrading from 1.11 to 1.12

[sfackler]

Bug Report

Description

When upgrading Talos nodes running on RPi 4s from v1.11.6 to v1.12.0, Talos seems to lose track of the disk encryption configuration for the STATE partition, leaving the node stuck offline. To recover, I have to reset it back into maintenance mode via the grub entry and reinstall. This seems to be 100% reproducible - I hit it twice on one node and then once on another.

The nodes are still using the deprecated systemDiskEncryption field for this:

    systemDiskEncryption:
        state:
            provider: luks2
            keys:
                - nodeID: {}
                  slot: 0
        ephemeral:
            provider: luks2
            keys:
                - nodeID: {}
                  slot: 0

I also tried adding a hardcoded password in slot 1 before upgrading the second node, but it still hit the same issue.

I can install v1.12.0 directly with the same disk encryption config and it seems to work fine.

Logs

(Sorry for the screenshot - I had to pull these off of a UART probe since the node can't come online)

The VolumeConfig object for the STATE volume looks the same between the nodes I reinstalled 1.12 on to and one I still have on 1.11:

1.12:

node: 192.168.4.11
metadata:
    namespace: runtime
    type: VolumeConfigs.block.talos.dev
    id: STATE
    version: 3
    owner: block.VolumeConfigController
    phase: running
    created: 2025-12-28T20:11:45Z
    updated: 2025-12-28T20:14:58Z
    labels:
        talos.dev/system-volume:
    finalizers:
        - block.VolumeManagerController
spec:
    type: partition
    provisioning:
        wave: -1
        diskSelector:
            match: system_disk
        partitionSpec:
            minSize: 104857600
            maxSize: 104857600
            relativeMaxSize: 0
            grow: false
            label: STATE
            typeUUID: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
        filesystemSpec:
            type: xfs
            label: STATE
    encryption:
        provider: luks2
        keys:
            - slot: 0
              type: nodeID
    locator:
        match: volume.partition_label == "STATE"
    mount:
        targetPath: /system/state
        selinuxLabel: system_u:object_r:system_state_t:s0
        projectQuotaSupport: false
        fileMode: 448

1.11:

node: 192.168.4.12
metadata:
    namespace: runtime
    type: VolumeConfigs.block.talos.dev
    id: STATE
    version: 4
    owner: block.VolumeConfigController
    phase: running
    created: 2025-12-17T15:51:44Z
    updated: 2025-12-17T15:51:49Z
    finalizers:
        - block.VolumeManagerController
spec:
    type: partition
    provisioning:
        wave: -1
        diskSelector:
            match: system_disk
        partitionSpec:
            minSize: 104857600
            maxSize: 104857600
            grow: false
            label: STATE
            typeUUID: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
        filesystemSpec:
            type: xfs
            label: STATE
    encryption:
        provider: luks2
        keys:
            - slot: 0
              type: nodeID
    locator:
        match: volume.partition_label == "STATE"
    mount:
        targetPath: /system/state
        selinuxLabel: system_u:object_r:system_state_t:s0
        projectQuotaSupport: false
        fileMode: 448

Environment

• Talos version: v1.12.0
• Kubernetes version: v1.34.3
• Platform: Raspberry PI 4

[smira]

I don't think it's related to actual encryption. I wonder if with Pi your node UUID changes due to U-boot or other updates

[sfackler]

Would the "volume is encrypted, but no encryption config provided" message still be in the logs if it was due to the UUID changing? The upgrade also in the same way when I added an extra static key in slot 1, but I didn't think to check in those logs if it was also trying slot 1. I could give it a try on the third node that's still on 1.11 if you think that'd be helpful.

For what it's worth, I was able to upgrade these nodes between 1.11.x releases without issue, but I don't know if there was a larger kernel bump in 1.12 or something like that.