Skip to content

bootloader/lkenv,lk: recover devices wedged by duplicated boot partions - #17419

Open
kubiko wants to merge 1 commit into
canonical:masterfrom
kubiko:bootloader-lk-recover-duplicate-boot-part
Open

bootloader/lkenv,lk: recover devices wedged by duplicated boot partions#17419
kubiko wants to merge 1 commit into
canonical:masterfrom
kubiko:bootloader-lk-recover-duplicate-boot-part

Conversation

@kubiko

@kubiko kubiko commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

A device updated by a snapd carrying the duplicate boot image partition bug can be left with the same kernel revision recorded in more than one boot image partition. Fixing the assignment logic stops new duplicates from appearing but does not help such a device: the duplicate is persisted in the snapbootsel partition, no free boot image partition can be found any more and kernel refreshes keep failing. Nothing in the existing code repairs it - RemoveKernelFromBootPartition() looks a partition up by the kernel revision it holds and clears the first match only, so even removing the kernel leaves the second reference behind.

Add DuplicateKernelBootPartitions(), reporting each kernel revision that is referenced by more than one boot image partition together with all the partitions referencing it, and ClearKernelBootPartition(), clearing the value of a boot image partition addressed by label rather than by the value it holds, which is what is needed to clear one specific partition out of several holding the same value.

Use both from lk.ExtractKernelAssets() to repair the boot image matrix before looking for a free boot image partition, so an affected device recovers on its next kernel refresh with no manual intervention.

A redundant reference is cleared only after the two boot image partitions have been confirmed to contain identical content, ensuring we accidentally don't drop into a failed refresh trap. If they differ, then the reference is the only record of where a distinct boot image lives: clearing it could bring the device to the bricked state. The exception here is when a duplicate kernel snap record is not referenced by either the current or try kernel (system for recovery env), in which case it's safe to clear it as it's unreferenced. In other cases, the matrix is left untouched, and the extraction fails as before rather than silently discarding a boot image. The reference in the partition appearing first in the matrix is the one kept, matching the partition that GetKernelBootPartition() resolves the kernel revision to, so the bootloader's view of the device does not change.

Repair is wired into the kernel path only. The recovery system matrix can duplicate as well, but it reserves every assigned recovery system, so it leaks a boot image partition instead of wedging the device, and the boot images of two distinct recovery systems may legitimately be identical, which makes the content check unsafe as a criterion there.

…ions

A device updated by a snapd carrying the duplicate boot image partition
bug can be left with the same kernel revision recorded in more than one
boot image partition. Fixing the assignment logic stops new duplicates
from appearing but does not help such a device: the duplicate is
persisted in the snapbootsel partition, no free boot image partition can
be found any more and kernel refreshes keep failing. Nothing in the
existing code repairs it - RemoveKernelFromBootPartition() looks a
partition up by the kernel revision it holds and clears the first match
only, so even removing the kernel leaves the second reference behind.

Add DuplicateKernelBootPartitions(), reporting each kernel revision that
is referenced by more than one boot image partition together with all the
partitions referencing it, and ClearKernelBootPartition(), clearing the
value of a boot image partition addressed by label rather than by the
value it holds, which is what is needed to clear one specific partition
out of several holding the same value.

Use both from lk.ExtractKernelAssets() to repair the boot image matrix
before looking for a free boot image partition, so an affected device
recovers on its next kernel refresh with no manual intervention.

How much has to be established before a redundant reference can be
cleared depends on whether the kernel revision is referenced for booting
at all. Add IsKernelReferenced() to report whether a kernel revision is
named by snap_kernel or snap_try_kernel. These are the only two variables
that direct the bootloader at the kernel matrix; the recovery system
variables name recovery systems in a separate matrix that this repair
never touches, so they are deliberately not considered.

A kernel revision named by neither is not used to boot the device. The
bootloader never searches the matrix for it, so no boot image partition it
points at can be selected, and the redundant references can be dropped
without comparing the boot image partitions at all - there is no boot
image to lose. This matters in practice, because the revision left
duplicated is frequently one that has already been superseded.

For a kernel revision that is referenced, a redundant reference is only
cleared once the two boot image partitions have been confirmed to hold
identical content. If they differ, then the reference is the only record
of where a distinct boot image lives: clearing it would leave a boot
image the bootloader can no longer find and would let the extraction that
follows overwrite an image that is still needed. In that case the matrix
is left untouched and the extraction fails as before rather than silently
discarding a boot image. The reference in the partition appearing first in
the matrix is the one kept, matching the partition that
GetKernelBootPartition() resolves the kernel revision to, so the
bootloader's view of the device does not change.

Repair is wired into the kernel path only. The recovery system matrix can
duplicate as well, but it reserves every assigned recovery system, so it
leaks a boot image partition instead of wedging the device, and the boot
images of two distinct recovery systems may legitimately be identical,
which makes the content check unsafe as a criterion there.

While at it, factor the boot image partition path resolution that
ExtractKernelAssets() already did out into bootPartitionPath(), as the
repair needs the same lookup.

Signed-off-by: Ondrej Kubik <ondrej.kubik@canonical.com>
@codecov

codecov Bot commented Jul 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 79.81651% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.84%. Comparing base (d7c520b) to head (92eef55).

Files with missing lines Patch % Lines
bootloader/lk.go 69.23% 13 Missing and 7 partials ⚠️
bootloader/lkenv/lkenv.go 95.45% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master   #17419      +/-   ##
==========================================
+ Coverage   78.81%   78.84%   +0.02%     
==========================================
  Files        1406     1403       -3     
  Lines      197084   197108      +24     
  Branches     2498     2498              
==========================================
+ Hits       155330   155404      +74     
+ Misses      32387    32348      -39     
+ Partials     9367     9356      -11     
Flag Coverage Δ
unittests 78.84% <79.81%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

Copy link
Copy Markdown

Thu Jul 30 17:58:02 UTC 2026
The following results are from: https://github.com/canonical/snapd/actions/runs/30556922399

Failures:

Preparing:

  • openstack-ext:ubuntu-18.04-64:tests/nested/classic/hotplug
  • openstack-ext:ubuntu-18.04-64:tests/nested/classic/activate-state
  • openstack-ext:ubuntu-22.04-64:tests/nested/manual/snapd-removes-vulnerable-snap-confine-revs:core
  • openstack-ext:ubuntu-22.04-64:tests/nested/manual/snapd-removes-vulnerable-snap-confine-revs:snapd
  • openstack-arm-ext:ubuntu-24.04-arm-64:tests/nested/manual/optee-fde:hooks
  • openstack-arm-ext:ubuntu-24.04-arm-64:tests/nested/manual/optee-fde:optee
  • openstack-arm-ext:ubuntu-24.04-arm-64:tests/nested/core/core20-basic
  • openstack-ext:ubuntu-24.04-64:tests/nested/manual/snapd-removes-vulnerable-snap-confine-revs:snapd
  • openstack-ext:ubuntu-24.04-64:tests/nested/manual/snapd-removes-vulnerable-snap-confine-revs:core
  • openstack:ubuntu-core-24-64:tests/main/graphical-user-daemons

Executing:

  • openstack:debian-12-64:tests/main/cgroup-devices-v2
  • openstack-ext:ubuntu-18.04-64:tests/nested/manual/minimal-smoke:secboot_enabled
  • openstack-ext:ubuntu-22.04-64:tests/nested/core/core20-fault-inject-on-refresh:kernel_reboot_refresh_gadget_assets
  • openstack-ext:ubuntu-26.04-64:tests/nested/core/core20-degraded
  • openstack-ext:ubuntu-26.04-64:tests/nested/core/core20-factory-reset:fde
  • openstack-ext:ubuntu-26.04-64:tests/nested/core/core20-gadget-reseal
  • openstack-ext:ubuntu-26.04-64:tests/nested/core/core20-kernel-reseal
  • openstack:ubuntu-core-18-64:tests/main/abort
  • openstack:ubuntu-core-20-64:tests/core/services
  • openstack:ubuntu-core-20-64:tests/core/snapd-failover

Restoring:

  • openstack:ubuntu-24.04-64:tests/main/nss-modules:winbind

Skipped tests from snapd-testing-skip

If you wish to have any of the below tests run in your PR, in your PR description, add 'unskip:' followed by a copy-and-pasted list of the below tests you wish to run (unskip plus test list must be valid yaml)

  • openstack:ubuntu-24.04-64:tests/main/i18n
  • openstack:ubuntu-26.04-64:tests/main/apparmor-prompting-flag-restart
  • openstack:ubuntu-26.04-64:tests/main/apparmor-prompting-integration-tests
  • openstack:ubuntu-26.04-64:tests/main/apparmor-prompting-prompt-restoration
  • openstack:ubuntu-26.04-64:tests/main/apparmor-prompting-smoke
  • openstack:ubuntu-26.04-64:tests/main/apparmor-prompting-snapd-startup
  • openstack:ubuntu-26.04-64:tests/main/apparmor-prompting-support
  • openstack:ubuntu-26.04-64:tests/main/i18n
  • openstack:ubuntu-26.04-64:tests/main/interfaces-requests-activates-handlers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant