Skip to content

Releases: longhorn/longhorn

Longhorn v1.11.0

29 Jan 06:00
@derekbit derekbit
v1.11.0
b8a3ced

Choose a tag to compare

View all tags
Longhorn v1.11.0 Latest
Latest

Longhorn v1.11.0 Release Notes

The Longhorn team is excited to announce the release of Longhorn v1.11.0. This release marks a major milestone, with the V2 Data Engine officially entering the Technical Preview stage following significant stability improvements.

Additionally, this version optimizes the stability of the whole system and introduces critical improvements in resource observability, scheduling, and utilization.

For terminology and background on Longhorn releases, see Releases.

Warning

Hotfix

longhorn-instance-manager Image

The longhorn-instance-manager:v1.11.0 image is affected by a regression issue introduced by the new longhorn-instance-manager Proxy service APIs. The bug causes Proxy connection leaks in the longhorn-instance-manager pods, resulting in increased memory usage. To mitigate this issue, replace longhornio/longhorn-instance-manager:v1.11.0 with the hotfixed image longhornio/longhorn-instance-manager:v1.11.0-hotfix-1.

You can apply the update by following these steps:

  1. Update the longhorn-instance-manager image

    • Change the longhorn-instance-manager image tag from v1.11.0 to v1.11.0-hotfix-1 in the appropriate file:
      • For Helm: Update values.yaml
      • For manifests: Update the deployment manifest directly.

  2. Proceed with the installation or upgrade

    • Apply the changes using your standard Helm install/upgrade command or reapply the updated manifest.

longhorn-manager Image

The longhorn-manager:v1.11.0 image is affected by a regression issue introduced by the new Kubernetes Node validator. The bug blocks setting Kubernetes node CNI labels because it waits for the Longhorn webhook server to be running, while the Longhorn webhook server waits for CNI network to be ready. To mitigate this issue, replace longhornio/longhorn-manager:v1.11.0 with the hotfixed image longhornio/longhorn-manager:v1.11.0-hotfix-1.

You can apply the update by following these steps:

  1. Disable the upgrade version check
  • Helm users: Set upgradeVersionCheck to false in the values.yaml file.
  • Manifest users: Remove the --upgrade-version-check flag from the deployment manifest.
  1. Update the longhorn-manager image
  • Change the longhorn-manager image tag from v1.11.0 to v1.11.0-hotfix-1 in the appropriate file:
    • For Helm: Update values.yaml.
    • For manifests: Update the deployment manifest directly.
  1. Proceed with the installation or upgrade
  • Apply the changes using your standard Helm install/upgrade command or reapply the updated manifest.

Deprecation

V2 Backing Image Deprecation

The Backing Image feature for the V2 Data Engine is now deprecated in v1.11.0 and is scheduled for removal in v1.12.0.

Users using V2 volumes for virtual machines are encouraged to adopt the Containerized Data Importer (CDI) for volume population instead.

GitHub Issue #12237

Primary Highlights

V2 Data Engine

Now in Technical Preview Stage

We are pleased to announce that the V2 Data Engine has officially graduated to the Technical Preview stage. This indicates increased stability and feature maturity as we move toward General Availability.

Limitation: While the engine is in Technical Preview, live upgrade is not supported yet. V2 volumes must be detached (offline) before engine upgrade.

Support for ublk Frontend

Longhorn supports configuring UBLK performance parameters globally, per volume, or via StorageClass to improve I/O performance.

GitHub Issue #11039

V1 Data Engine

Faster Replica Rebuilding from Multiple Sources

The V1 Data Engine now supports parallel rebuilding. When a replica needs to be rebuilt, the engine can now stream data from multiple healthy replicas simultaneously rather than a single source. This significantly reduces the time required to restore redundancy for volumes containing tons of scattered data chunks.

GitHub Issue #11331

General

Balance-Aware Algorithm Disk Selection For Replica Scheduling

Longhorn improves the disk selection for the replica scheduling by introducing an intelligent balance-aware scheduling algorithm, reducing uneven storage usage across nodes and disks.

GitHub Issue #10512

Node Disk Health Monitoring

Longhorn now actively monitors the physical health of the underlying disks used for storage by using S.M.A.R.T. data. This allows administrators to identify issues and raise alerts when abnormal SMART metrics are detected, helping prevent failed volumes.

GitHub Issue #12016

Share Manager Networking

Users can now configure an extra network interface for the Share Manager to support complex network segmentation requirements.

GitHub Issue #10269

ReadWriteOncePod (RWOP) Support

Full support for the Kubernetes ReadWriteOncePod access mode has been added.

GitHub Issue #9727

StorageClass allowedTopologies Support

Administrators can now use the allowedTopologies field in Longhorn StorageClasses to restrict volume provisioning to specific zones, regions, or nodes within the cluster.

GitHub Issue #12261

Installation

Important

Ensure that your cluster is running Kubernetes v1.25 or later before installing Longhorn v1.11.0.

You can install Longhorn using a variety of tools, including Rancher, Kubectl, and Helm. For more information about installation methods and requirements, see Quick Installation in the Longhorn documentation.

Upgrade

Important

Ensure that your cluster is running Kubernetes v1.25 or later before upgrading from Longhorn v1.10.x to v1.11.0.

Longhorn only allows upgrades from supported versions. For more information about upgrade paths and procedures, see Upgrade in the Longhorn documentation.

Post-Release Known Issues

For information about issues identified after this release, see Release-Known-Issues.

Resolved Issues in this release

Highlight

Feature

Impro...

Read more

Contributors

mattn, COLDTURNIP, and 70 other contributors
Assets 6
Loading

Longhorn v1.10.2

28 Jan 09:24
@derekbit derekbit
v1.10.2
c8648f5

Choose a tag to compare

View all tags
Longhorn v1.10.2

Longhorn v1.10.2 Release Notes

Longhorn 1.10.2 introduces several improvements and bug fixes that are intended to improve system quality, resilience, stability and security.

We welcome feedback and contributions to help continuously improve Longhorn.

For terminology and context on Longhorn releases, see Releases.

Important Fixes

This release includes several critical stability fixes.

RWX Volume Unavailable After Node Drain

Fixed a race condition where ReadWriteMany (RWX) volumes could remain in the attaching state after node drains, causing workloads to become unavailable.

For more details, see Issue #12231.

Encrypted Volume Cannot Be Expanded Online

Fixed an issue where online expansion of encrypted volumes did not propagate the new size to the dm-crypt device.

For more details, see Issue #12368.

Cloned Volume Cannot Be Attached to Workload

Fixed a bug where cloned volumes could fail to reach a healthy state, preventing attachment to workloads.

For more details, see Issue #12208.

Block Mode Volume Migration Stuck

Fixed a regression in block-mode volume migrations where newly created replicas could incorrectly inherit the lastFailedAt timestamp from source replicas, causing repeated deletion and blocking migration completion.

For more details, see Issue #12312.

Replica Auto Balance Disk Pressure Threshold Stalled

Fixed an issue where replica auto-balance under disk pressure could be blocked if stopped volumes were present on the disk.

For more details, see Issue #12334.

Replicas Accumulate During Engine Upgrade

Fixed a bug where temporary replicas could accumulate during engine upgrade. High etcd latency could cause new replicas to fail verification, leading to accumulation over multiple reconciliation cycles.

For more details, see Issue #12115.

Potential Client Connection and Context Leak

Fixed potential context leaks in the instance manager client and backing image manager client, improving stability and preventing resource exhaustion.

For more details, see Issue #12200 and Issue #12195.

Replica Node Level Soft Anti-Affinity Ignored

Fixed a bug of replica scheduling loop where replicas could be scheduled onto nodes that already host a replica, even when Replica Node-Level Soft Anti-Affinity was disabled.

For more details, see Issue #12251.

Installation

Important

Ensure that your cluster is running Kubernetes v1.25 or later before installing Longhorn v1.10.2.

You can install Longhorn using a variety of tools, including Rancher, Kubectl, and Helm. For more information about installation methods and requirements, see Quick Installation in the Longhorn documentation.

Upgrade

Important

Ensure that your cluster is running Kubernetes v1.25 or later before upgrading from Longhorn v1.9.x to v1.10.2.

Longhorn only allows upgrades from supported versions. For more information about upgrade paths and procedures, see Upgrade in the Longhorn documentation.

Post-Release Known Issues

For information about issues identified after this release, see Release-Known-Issues.

Resolved Issues

Feature

  • [BACKPORT][v1.10.2][FEATURE] Inherit namespace for longhorn-share-manager in FastFailover mode 12245 - @yangchiu
  • [BACKPORT][v1.10.2][FEATURE] [Dependency] aws-sdk-go v1.55.7 is EOL as of 2025-07-31 — plan to migrate to v2? 12181 - @mantissahz @roger-ryao

Improvement

  • [BACKPORT][v1.10.2][IMPROVEMENT] Fix V2 Volume CSI Clone Slowness Caused by VolumeAttachment Webhook Blocking 12329 - @PhanLe1010 @roger-ryao

Bug

Contributors

Contributors

COLDTURNIP, innobead, and 13 other contributors
Loading

Longhorn v1.11.0-rc3

22 Jan 08:51
@derekbit derekbit
v1.11.0-rc3
a43debd

Choose a tag to compare

View all tags
Longhorn v1.11.0-rc3 Pre-release
Pre-release

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Highlight

Feature

Improvement

Read more

Contributors

COLDTURNIP, ehpc, and 31 other contributors
Loading

Longhorn v1.10.2-rc1

22 Jan 02:46
@derekbit derekbit
v1.10.2-rc1
d920984

Choose a tag to compare

View all tags
Longhorn v1.10.2-rc1 Pre-release
Pre-release

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Feature

  • [BACKPORT][v1.10.2][FEATURE] Inherit namespace for longhorn-share-manager in FastFailover mode 12245 - @yangchiu
  • [BACKPORT][v1.10.2][FEATURE] [Dependency] aws-sdk-go v1.55.7 is EOL as of 2025-07-31 — plan to migrate to v2? 12181 - @mantissahz @roger-ryao

Improvement

  • [BACKPORT][v1.10.2][IMPROVEMENT] Fix V2 Volume CSI Clone Slowness Caused by VolumeAttachment Webhook Blocking 12329 - @PhanLe1010 @roger-ryao

Bug

  • [BACKPORT] Replica rebuild, clone and restore fail, traffic being sent to HTTP proxy 12518 - @yangchiu @derekbit
  • [BACKPORT][v1.10.2][BUG] Healthy replica could be deleted unexpectedly after reducing volume's number of replicas 12512 - @yangchiu @shuo-wu
  • [BACKPORT][v1.10.2][BUG] Data locality enabled volume fails to remove an existing running replica after numberOfReplicas reduced 12509 - @derekbit @chriscchien
  • [BACKPORT][v1.10.2][BUG] System backup may fail to be created or deleted 12479 - @yangchiu @mantissahz
  • [BACKPORT][v1.10.2][BUG] Some default settings in questions.yaml are placed incorrectly. 12222 - @derekbit @roger-ryao
  • [BACKPORT][v1.10.2][BUG] Auto balance feature may lead to volumes falling into a replica deletion-recreation loop 12482 - @shuo-wu @roger-ryao
  • [BACKPORT][v1.10.2][BUG] Single replica volume could get stuck in attaching/detaching loop after the replica node rebooted 12494 - @COLDTURNIP @yangchiu
  • [BACKPORT][v1.10.2][BUG] Potential Instance Manager Client Context Leak 12200 - @derekbit @chriscchien
  • [BACKPORT][v1.10.2][BUG] SnapshotBack proxy request might be sent to incorrect instance-manager pod 12476 - @derekbit @chriscchien
  • [BACKPORT][v1.10.2][BUG] unknown OS condition in node CR is not properly removed during upgrade 12451 - @COLDTURNIP @roger-ryao
  • [BACKPORT][v1.10.2][BUG] RWX volume becomes unavailable after drain node 12231 - @yangchiu @mantissahz
  • [BACKPORT][v1.10.2][BUG] Should not rebuild the v2 volume if this volume never be used. 12239 - @mantissahz
  • [BACKPORT][v1.10.2][BUG] Test case test_recurring_jobs_when_volume_detached_unexpectedly failed: backup completed but progress did not reach 100% 12156 - @yangchiu @mantissahz
  • [BACKPORT][v1.10.2][BUG] mounting error is not properly hanedled during CSI node publish volume 12382 - @COLDTURNIP @yangchiu
  • [BACKPORT][v1.10.2][BUG] Encrypted Volume Cannot Be Expanded Online 12368 - @yangchiu @mantissahz
  • [BACKPORT][v1.10.2][BUG] The auo generated backing image pod name is complained by kubelet 12357 - @COLDTURNIP @yangchiu
  • [BACKPORT][v1.10.2][BUG] tests.test_cloning.test_cloning_basic fails at msater-head 12342 - @c3y1huang
  • [BACKPORT][v1.10.2][Bug] A cloned volume cannot be attached to a workload 12208 - @yangchiu @PhanLe1010
  • [BACKPORT][v1.10.2][BUG] Block Mode Volume Migration Stuck 12312 - @COLDTURNIP @yangchiu @shuo-wu
  • [BACKPORT][v1.10.2][BUG] Replica auto balance disk pressure threshold stalled with stopped volumes 12334 - @c3y1huang @chriscchien
  • [BACKPORT][v1.10.2][BUG] short name mode is enforcing, but image name longhornio/longhorn-manager:v1.10. │ │ 0 returns ambiguous list 12270 - @yangchiu
  • [BACKPORT][v1.10.2][BUG] Replicas accumulate during engine upgrade 12115 - @c3y1huang @chriscchien
  • [BACKPORT][v1.10.2][BUG] Potential BackingImageManagerClient Connection and Context Leak 12195 - @derekbit @chriscchien
  • [BACKPORT][v1.10.2][BUG] Longhorn ignores Replica Node Level Soft Anti-Affinity when auto balance is set to best-effort 12251 - @c3y1huang @chriscchien
  • [BACKPORT][v1.10.2][BUG] invalid memory address or nil pointer dereference (again) 12234 - @chriscchien @bachmanity1
  • [BACKPORT][v1.10.2][BUG] Request Header Or Cookie Too Large in Web UI with OIDC auth 12213 - @chriscchien @houhoucoop

Contributors

Contributors

COLDTURNIP, innobead, and 10 other contributors
Loading

Longhorn v1.11.0-rc2

15 Jan 06:56
@derekbit derekbit
v1.11.0-rc2
a3aa927

Choose a tag to compare

View all tags
Longhorn v1.11.0-rc2 Pre-release
Pre-release

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Highlight

Feature

Improvement

Read more

Contributors

COLDTURNIP, ehpc, and 31 other contributors
Loading

Longhorn v1.10.1

12 Nov 08:04
@derekbit derekbit
v1.10.1
452bc55

Choose a tag to compare

View all tags
Longhorn v1.10.1

Longhorn v1.10.1 Release Notes

Longhorn 1.10.1 introduces several improvements and bug fixes that are intended to improve system quality, resilience, stability and security.

We welcome feedback and contributions to help continuously improve Longhorn.

For terminology and context on Longhorn releases, see Releases.

Warning

HotFix

The longhorn-manager:v1.10.1 image is affected by

To mitigate the issues, replace longhorn-manager:v1.10.1 with the hotfixed image longhorn-manager:v1.10.1-hotfix-2.

Follow these steps to apply the update:

  1. Disable the upgrade version check

    • Helm users: Set upgradeVersionCheck to false in the values.yaml file.
    • Manifest users: Remove the --upgrade-version-check flag from the deployment manifest.

  2. Update the longhorn-manager image

    • Change the image tag from v1.10.1 to v1.10.1-hotfix-2 in the appropriate file:
      • For Helm: Update values.yaml
      • For manifests: Update the deployment manifest directly.

  3. Proceed with the upgrade

    • Apply the changes using your standard Helm upgrade command or reapply the updated manifest.

Upgrade

If your Longhorn cluster was initially deployed with a version earlier than v1.3.0, the Custom Resources (CRs) were created using the v1beta1 APIs. While the upgrade from Longhorn v1.8 to v1.9 automatically migrates all CRs to the new v1beta2 version, a manual CR migration is strongly advised before upgrading from Longhorn v1.9 to v1.10.

Certain operations, such as an etcd or CRD restore, may leave behind v1beta1 data. Manually migrating your CRs ensures that all Longhorn data is properly updated to the v1beta2 API, preventing potential compatibility issues and unexpected behavior with the new Longhorn version.

Following the manual migration, verify that v1beta1 has been removed from the CRD stored versions to ensure completion and a successful upgrade.

For more details, see Kubernetes official document for CRD storage version, and Issue #11886.

Migration Requirement Before Longhorn v1.10 Upgrade

Before upgrading from Longhorn v1.9 to v1.10, perform the following manual CRD storage version migration.

Note: If your Longhorn installation uses a namespace other than longhorn-system, replace longhorn-system with your custom namespace throughout the commands.

# Temporarily disable the CR validation webhook to allow updating read-only settings CRs.
kubectl patch validatingwebhookconfiguration longhorn-webhook-validator \
  --type=merge \
  -p "$(kubectl get validatingwebhookconfiguration longhorn-webhook-validator -o json | \
  jq '.webhooks[0].rules |= map(if .apiGroups == ["longhorn.io"] and .resources == ["settings"] then
    .operations |= map(select(. != "UPDATE")) else . end)')"

# Migrate CRDs that ever stored v1beta1 resources
migration_time="$(date +%Y-%m-%dT%H:%M:%S)"
crds=($(kubectl get crd -l app.kubernetes.io/name=longhorn -o json | jq -r '.items[] | select(.status.storedVersions | index("v1beta1")) | .metadata.name'))
for crd in "${crds[@]}"; do
  echo "Migrating ${crd} ..."
  for name in $(kubectl -n longhorn-system get "$crd" -o jsonpath='{.items[*].metadata.name}'); do
    # Attach additional annotations to trigger v1beta1 resource updating in the latest storage version.
    kubectl patch "${crd}" "${name}" -n longhorn-system --type=merge -p='{"metadata":{"annotations":{"migration-time":"'"${migration_time}"'"}}}'
 done
 # Clean up the stored version in CRD status
 kubectl patch crd "${crd}" --type=merge -p '{"status":{"storedVersions":["v1beta2"]}}' --subresource=status
done

# Re-enable the CR validation webhook.
kubectl patch validatingwebhookconfiguration longhorn-webhook-validator \
  --type=merge \
  -p "$(kubectl get validatingwebhookconfiguration longhorn-webhook-validator -o json | \
  jq '.webhooks[0].rules |= map(if .apiGroups == ["longhorn.io"] and .resources == ["settings"] then
    .operations |= (. + ["UPDATE"] | unique) else . end)')"

Migration Verification

After running the script, verify the CRD stored versions using this command:

kubectl get crd -l app.kubernetes.io/name=longhorn -o=jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.storedVersions}{"\n"}{end}'

Crucially, all Longhorn CRDs MUST have only "v1beta2" listed in storedVersions (i.e., "v1beta1" must be completely absent) before proceeding to the v1.10 upgrade.

Example of successful output:

backingimagedatasources.longhorn.io: ["v1beta2"]
backingimagemanagers.longhorn.io: ["v1beta2"]
backingimages.longhorn.io: ["v1beta2"]
backupbackingimages.longhorn.io: ["v1beta2"]
backups.longhorn.io: ["v1beta2"]
backuptargets.longhorn.io: ["v1beta2"]
backupvolumes.longhorn.io: ["v1beta2"]
engineimages.longhorn.io: ["v1beta2"]
engines.longhorn.io: ["v1beta2"]
instancemanagers.longhorn.io: ["v1beta2"]
nodes.longhorn.io: ["v1beta2"]
orphans.longhorn.io: ["v1beta2"]
recurringjobs.longhorn.io: ["v1beta2"]
replicas.longhorn.io: ["v1beta2"]
settings.longhorn.io: ["v1beta2"]
sharemanagers.longhorn.io: ["v1beta2"]
snapshots.longhorn.io: ["v1beta2"]
supportbundles.longhorn.io: ["v1beta2"]
systembackups.longhorn.io: ["v1beta2"]
systemrestores.longhorn.io: ["v1beta2"]
volumeattachments.longhorn.io: ["v1beta2"]
volumes.longhorn.io: ["v1beta2"]

With these steps completed, the Longhorn upgrade to v1.10 should now proceed without issues.

Troubleshooting CRD Upgrade Failures During Upgrade to Longhorn v1.10

If you did not apply the required pre-upgrade migration steps and the CRs are not fully migrated to v1beta2, the longhorn-manager Pods may fail to operate correctly. A common error message for this issue is:

Upgrade failed: cannot patch "backingimagedatasources.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "backingimagedatasources.longhorn.io" is invalid: status.storedVersions[0]: Invalid value: "v1beta1": missing from spec.versions; v1beta1 was previously a storage version, and must remain in spec.versions until a storage migration ensures no data remains persisted in v1beta1 and removes v1beta1 from status.storedVersions

To fix this issue, you must perform a forced downgrade back to the exact Longhorn v1.9.x version that was running before the failed upgrade attempt.

Downgrade Procedure (kubectl Installation)

If Longhorn was installed using kubectl, you must patch the current-longhorn-version setting before downgrading. Replace v1.9.x with the original version before upgrade in the following commands.

# Attaching annotation to allow patching current-longhorn-version.
kubectl patch settings.longhorn.io current-longhorn-version -n longhorn-system --type=merge -p='{"metadata":{"annotations":{"longhorn.io/update-setting-from-longhorn":""}}}'
# Temporarily override current version to allow old version installation
# Replace the value `"v1.9.x" to the original version before upgrade.
kubectl patch settings.longhorn.io current-longhorn-version -n longhorn-system --type=merge -p='{"value":"v1.9.x"}'

After modifying current-longhorn-version, you can proceed to downgrade to the original Longhorn v1.9.x deployment.

Downgrade Procedure (Helm Installation)

If Longhorn was installed using Helm, the downgrade is allowed by disabling the preUpgradeChecker.upgradeVersionCheck flag.

Post-Downgrade

Once the downgrade is complete and the Longhorn system is stable on the v1.9.x version, you must immediately follow the steps outlined in the Migration Requirement Before Longhorn v1.10 Upgrade. This step is crucial to migrate all remaining v1beta1 CRs to v1beta2 before attempting the Longhorn v1.10 upgrade again.

Important Fixes

This release includes several critical stability and performance improvements:

Goroutine Leak in Instance Manager (V2 Data Engine)

Fixed a goroutine leak in the instance manager when using the V2 data engine. This issue could lead to increased memory usage and potential stability problems over time.

For ...

Read more

Contributors

COLDTURNIP, innobead, and 13 other contributors
Loading

Longhorn v1.10.1-rc1

30 Oct 07:27
@derekbit derekbit
v1.10.1-rc1
5e6dee0

Choose a tag to compare

View all tags
Longhorn v1.10.1-rc1 Pre-release
Pre-release

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Feature

  • [BACKPORT][v1.10.1][FEATURE] node storage scheduled metrics 11955 - @yangchiu

Improvement

  • [BACKPORT][v1.10.1][IMPROVEMENT] CSIStorageCapacity objects must show schedulable (allocatable) capacity 12036 - @chriscchien @bachmanity1
  • [BACKPORT][v1.10.1][IMPROVEMENT] improve error logging for failed mounting during node publish volume 12033 - @COLDTURNIP @roger-ryao
  • [BACKPORT][v1.10.1][IMPROVEMENT] Improve Helm Chart defaultSettings handling with automatic quoting and multi-type support 12020 - @derekbit @chriscchien
  • [BACKPORT][v1.10.1][IMPROVEMENT] Avoid repeat engine restart when there are replica unavailable during migration 11945 - @yangchiu @shuo-wu
  • [BACKPORT][v1.10.1][IMPROVEMENT] Adjust maximum of GuaranteedInstanceManagerCPU to a big value 11968 - @mantissahz
  • [BACKPORT][v1.10.1][IMPROVEMENT] Add usage metrics for Longhorn installation variant 11795 - @derekbit

Bug

Misc

Contributors

Contributors

COLDTURNIP, innobead, and 10 other contributors
Loading

Longhorn v1.10.0

25 Sep 08:14
@derekbit derekbit
v1.10.0
92b8338

Choose a tag to compare

View all tags
Longhorn v1.10.0

Longhorn v1.10.0 Release Notes

Longhorn v1.10.0 is a major release focused on improving stability, performance, and the overall user experience. This version introduces significant enhancements to our core features, including the V2 Data Engine, and streamlines configuration for easier management.

The key highlights include improvements to the V2 Data Engine, enhanced resilience, simplified configuration, and better observability.

We welcome feedback and contributions to help continuously improve Longhorn.

For terminology and context on Longhorn releases, see Releases.

Warning

HotFix

The longhorn-manager:v1.10.0 image is affected by a regression issue introduced by the new share-manager pod backoff logic. This bug may cause a nil pointer dereference panic in the longhorn-manager, leading to repeated crashes and failure to deploy new share-manager pods after an upgrade. To mitigate this issue, replace longhorn-manager:v1.10.0 with the hotfixed image longhorn-manager:v1.10.0-hotfix-1.

You can apply the update by following these steps:

  1. Disable the upgrade version check

    • Helm users: Set upgradeVersionCheck to false in the values.yaml file.
    • Manifest users: Remove the --upgrade-version-check flag from the deployment manifest.

  2. Update the longhorn-manager image

    • Change the image tag from v1.10.0 to v1.10.0-hotfix-1 in the appropriate file:
      • For Helm: Update values.yaml
      • For manifests: Update the deployment manifest directly.

  3. Proceed with the upgrade

  • Apply the changes using your standard Helm upgrade command or reapply the updated manifest.

Upgrade

If your Longhorn cluster was initially deployed with a version earlier than v1.3.0, the Custom Resources (CRs) were created using the v1beta1 APIs. While the upgrade from Longhorn v1.8 to v1.9 automatically migrates all CRs to the new v1beta2 version, a manual CR migration is strongly advised before upgrading from Longhorn v1.9 to v1.10.

Certain operations, such as an etcd or CRD restore, may leave behind v1beta1 data. Manually migrating your CRs ensures that all Longhorn data is properly updated to the v1beta2 API, preventing potential compatibility issues and unexpected behavior with the new Longhorn version.

Following the manual migration, verify that v1beta1 has been removed from the CRD stored versions to ensure completion and a successful upgrade.

For more details, see Kubernetes official document for CRD storage version, and Issue #11886.

Migration Requirement Before Longhorn v1.10 Upgrade

Before upgrading from Longhorn v1.9 to v1.10, perform the following manual CRD storage version migration.

Note: If your Longhorn installation uses a namespace other than longhorn-system, replace longhorn-system with your custom namespace throughout the commands.

# Temporarily disable the CR validation webhook to allow updating read-only settings CRs.
kubectl patch validatingwebhookconfiguration longhorn-webhook-validator \
  --type=merge \
  -p "$(kubectl get validatingwebhookconfiguration longhorn-webhook-validator -o json | \
  jq '.webhooks[0].rules |= map(if .apiGroups == ["longhorn.io"] and .resources == ["settings"] then
    .operations |= map(select(. != "UPDATE")) else . end)')"

# Migrate CRDs that ever stored v1beta1 resources
migration_time="$(date +%Y-%m-%dT%H:%M:%S)"
crds=($(kubectl get crd -l app.kubernetes.io/name=longhorn -o json | jq -r '.items[] | select(.status.storedVersions | index("v1beta1")) | .metadata.name'))
for crd in "${crds[@]}"; do
  echo "Migrating ${crd} ..."
  for name in $(kubectl -n longhorn-system get "$crd" -o jsonpath='{.items[*].metadata.name}'); do
    # Attach additional annotations to trigger v1beta1 resource updating in the latest storage version.
    kubectl patch "${crd}" "${name}" -n longhorn-system --type=merge -p='{"metadata":{"annotations":{"migration-time":"'"${migration_time}"'"}}}'
 done
 # Clean up the stored version in CRD status
 kubectl patch crd "${crd}" --type=merge -p '{"status":{"storedVersions":["v1beta2"]}}' --subresource=status
done

# Re-enable the CR validation webhook.
kubectl patch validatingwebhookconfiguration longhorn-webhook-validator \
  --type=merge \
  -p "$(kubectl get validatingwebhookconfiguration longhorn-webhook-validator -o json | \
  jq '.webhooks[0].rules |= map(if .apiGroups == ["longhorn.io"] and .resources == ["settings"] then
    .operations |= (. + ["UPDATE"] | unique) else . end)')"

Migration Verification

After running the script, verify the CRD stored versions using this command:

kubectl get crd -l app.kubernetes.io/name=longhorn -o=jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.storedVersions}{"\n"}{end}'

Crucially, all Longhorn CRDs MUST have only "v1beta2" listed in storedVersions (i.e., "v1beta1" must be completely absent) before proceeding to the v1.10 upgrade.

Example of successful output:

backingimagedatasources.longhorn.io: ["v1beta2"]
backingimagemanagers.longhorn.io: ["v1beta2"]
backingimages.longhorn.io: ["v1beta2"]
backupbackingimages.longhorn.io: ["v1beta2"]
backups.longhorn.io: ["v1beta2"]
backuptargets.longhorn.io: ["v1beta2"]
backupvolumes.longhorn.io: ["v1beta2"]
engineimages.longhorn.io: ["v1beta2"]
engines.longhorn.io: ["v1beta2"]
instancemanagers.longhorn.io: ["v1beta2"]
nodes.longhorn.io: ["v1beta2"]
orphans.longhorn.io: ["v1beta2"]
recurringjobs.longhorn.io: ["v1beta2"]
replicas.longhorn.io: ["v1beta2"]
settings.longhorn.io: ["v1beta2"]
sharemanagers.longhorn.io: ["v1beta2"]
snapshots.longhorn.io: ["v1beta2"]
supportbundles.longhorn.io: ["v1beta2"]
systembackups.longhorn.io: ["v1beta2"]
systemrestores.longhorn.io: ["v1beta2"]
volumeattachments.longhorn.io: ["v1beta2"]
volumes.longhorn.io: ["v1beta2"]

With these steps completed, the Longhorn upgrade to v1.10 should now proceed without issues.

Troubleshooting CRD Upgrade Failures During Upgrade to Longhorn v1.10

If you did not apply the required pre-upgrade migration steps and the CRs are not fully migrated to v1beta2, the longhorn-manager Pods may fail to operate correctly. A common error message for this issue is:

Upgrade failed: cannot patch "backingimagedatasources.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "backingimagedatasources.longhorn.io" is invalid: status.storedVersions[0]: Invalid value: "v1beta1": missing from spec.versions; v1beta1 was previously a storage version, and must remain in spec.versions until a storage migration ensures no data remains persisted in v1beta1 and removes v1beta1 from status.storedVersions

To fix this issue, you must perform a forced downgrade back to the exact Longhorn v1.9.x version that was running before the failed upgrade attempt.

Downgrade Procedure (kubectl Installation)

If Longhorn was installed using kubectl, you must patch the current-longhorn-version setting before downgrading. Replace v1.9.x with the original version before upgrade in the following commands.

# Attaching annotation to allow patching current-longhorn-version.
kubectl patch settings.longhorn.io current-longhorn-version -n longhorn-system --type=merge -p='{"metadata":{"annotations":{"longhorn.io/update-setting-from-longhorn":""}}}'
# Temporarily override current version to allow old version installation
# Replace the value `"v1.9.x" to the original version before upgrade.
kubectl patch settings.longhorn.io current-longhorn-version -n longhorn-system --type=merge -p='{"value":"v1.9.x"}'

After modifying current-longhorn-version, you can proceed to downgrade to the original Longhorn v1.9.x deployment.

Downgrade Procedure (Helm Installation)

If Longhorn was installed using Helm, the downgrade is allowed by disabling the preUpgradeChecker.upgradeVersionCheck flag.

Post-Downgrade

Once the downgrade is complete and the Longhorn system is stable on the v1.9.x version, you must immediately follow the steps outlined in the Migration Requirement Before Longhorn v1.10 Upgrade. This step is crucial to migrate all remaining v1beta1 CRs to v1beta2 before attempting the Longhorn v1.10 upgrade again.

Removal

longhorn.io/v1beta1 API

The v1beta1 Longhorn API version has been removed.

See GitHub Issue #10249 for details.

replica.status.evictionRequested Field

The deprecated replica.status.evictionRequested field has been removed.

See GitHub Issue #7022 for details.

Primary Highlights

New V2 Data Engine Features

Interrupt Mode Support

Interrupt mode has been added to the V2 Data Engine to help reduce CPU usage. This feature is especially beneficial for clusters with idle or low I/O workloads, where conserving CPU resources is more important than minimizing latency.

While interrupt mode lowers CPU consumption, it may introduce slightly higher I/O latency compared to polling mode. In addition, the current implementation uses a hybrid approach, which still ...

Read more

Contributors

COLDTURNIP, innobead, and 28 other contributors
Loading

Longhorn v1.9.2

24 Sep 09:01
@derekbit derekbit
v1.9.2
c1342db

Choose a tag to compare

View all tags
Longhorn v1.9.2

Longhorn v1.9.2 Release Notes

Longhorn 1.9.2 introduces several improvements and bug fixes that are intended to improve system quality, resilience, stability and security.

The Longhorn team appreciates your contributions and expects to receive feedback regarding this release.

Note

For more information about release-related terminology, see Releases.

Installation

Important

Ensure that your cluster is running Kubernetes v1.25 or later before installing Longhorn v1.9.2.

You can install Longhorn using a variety of tools, including Rancher, Kubectl, and Helm. For more information about installation methods and requirements, see Quick Installation in the Longhorn documentation.

Upgrade

Important

Ensure that your cluster is running Kubernetes v1.25 or later before upgrading from Longhorn v1.8.x or v1.9.x (< v1.9.2) to v1.9.2.

Longhorn only allows upgrades from supported versions. For more information about upgrade paths and procedures, see Upgrade in the Longhorn documentation.

Post-Release Known Issues

For information about issues identified after this release, see Release-Known-Issues.

Resolved Issues

Improvement

  • [BACKPORT][v1.9.2][IMPROVEMENT] Add usage metrics for Longhorn installation variant 11805 - @derekbit
  • [BACKPORT][v1.9.2][IMPROVEMENT] SAST Potential dereference of the null pointer in controller/volume_controller.go in longhorn-manager 11782 - @c3y1huang
  • [BACKPORT][v1.9.2][IMPROVEMENT] Collect mount table, process status and process table in support bundle 11726 - @mantissahz @chriscchien
  • [BACKPORT][v1.9.2][IMPROVEMENT] rename the backing image manager to reduce the probability of CR name collision 11567 - @COLDTURNIP @chriscchien
  • [BACKPORT][v1.9.2][IMPROVEMENT] Improve log messages of longhorn-engine, tgt and liblonghorn for troubleshooting 11604 - @yangchiu @derekbit
  • [BACKPORT][v1.9.2][IMPROVEMENT] Misleading log message Deleting orphans on evicted node ... 11501 - @yangchiu @derekbit
  • [BACKPORT][v1.9.2][IMPROVEMENT] Check if the backup target is available before creating a backup, backup backing image, and system backup 11324 - @yangchiu @nzhan126
  • [BACKPORT][v1.9.2][IMPROVEMENT] adjust the hardcoded timeout limitation for backing image downloading 11310 - @COLDTURNIP @chriscchien
  • [BACKPORT][v1.9.2][IMPROVEMENT] Improve longhorn-engine controller log messages 11508 - @derekbit @chriscchien
  • [BACKPORT][v1.9.2][IMPROVEMENT] Make liveness probe parameters of instance-manager pod configurable 11506 - @derekbit @chriscchien
  • [BACKPORT][v1.9.2][IMPROVEMENT] backing image handle node disk deleting events 11488 - @COLDTURNIP @chriscchien
  • [BACKPORT][v1.9.2][IMPROVEMENT] Handle credential secret containing mixed invalid conditions 11327 - @yangchiu @nzhan126
  • [BACKPORT][v1.9.2][IMPROVEMENT] Improve the condition message of engine image check 11193 - @derekbit @chriscchien

Bug

  • [BACKPORT][v1.9.2][BUG] Potential Data Corruption During Volume Resizing When Created from Snapshot 11788 - @yangchiu @PhanLe1010
  • [BUG] [v1.9.x] support bundle stuck at 33% 11744 - @mantissahz @chriscchien
  • [BACKPORT][v1.9.2][BUG] Unable to disable v2-data-engine even though there is no v2 volumes, backing images or orphaned data 11639 - @shuo-wu @chriscchien
  • [BACKPORT][v1.9.2][BUG] Longhorn pvcs are in pending state. 11722 - @yangchiu @derekbit
  • [BUG] Broken link in documentation 11729 - @consideRatio
  • [BACKPORT][v1.9.2][BUG] longhornctl preflight install should load and check iscsi_tcp kernel module. 11710 - @mantissahz @chriscchien
  • [BACKPORT][v1.9.2][BUG] Backing image download gets stuck after network disconnection 11624 - @COLDTURNIP
  • [BACKPORT][v1.9.2][BUG] Volume becomes faulted when its replica node disks run out of space during a write operation 11341 - @mantissahz @chriscchien
  • [BACKPORT][v1.9.2][BUG] Engine process continues running after rapid volume detachment 11606 - @COLDTURNIP @yangchiu @chriscchien
  • [BACKPORT][v1.9.2][BUG] Creating a 2 Gi volume with a 200 Mi backing image is rejected with “volume size should be larger than the backing image size” 11648 - @COLDTURNIP @yangchiu @chriscchien
  • [BACKPORT][v1.9.2][BUG] longhorn-manager repeatedly emits No instance manager for node xxx for update instance state of orphan instance orphan-xxx.. 11599 - @COLDTURNIP @chriscchien
  • [BACKPORT][v1.9.2][BUG] BackupBackingImage may be created from an unready BackingImageManager 11692 - @WebberHuang1118 @roger-ryao
  • [BACKPORT][v1.9.2][BUG] Longhorn fails to create Backing Image Backup on ARM platform 11570 - @COLDTURNIP
  • [BACKPORT][v1.9.2][BUG] remaining unknown OS condition in node CR 11614 - @COLDTURNIP @roger-ryao
  • [BACKPORT][v1.9.2][BUG] Volumes fails to remount when they go read-only 11584 - @derekbit @chriscchien
  • [BACKPORT][v1.9.2][BUG] Dangling Volume State When Live Migration Terminates Unexpectedly 11590 - @PhanLe1010 @chriscchien
  • [BACKPORT][v1.9.2][BUG] Unable to setup backup target in storage network environment: cannot find a running instance manager for node 11482 - @derekbit @chriscchien
  • [BACKPORT][v1.9.2][BUG] Test case test_recurring_jobs_when_volume_detached_unexpectedly failed: backup completed but progress did not reach 100% 11476 - @yangchiu @mantissahz
  • [BACKPORT][v1.9.2][BUG] Recurring Job with 'default' group causes goroutine deadlock on v1.9.1 (Regression of #11020) 11494 - @c3y1huang
  • [BACKPORT][v1.9.2][BUG] Test Case test_replica_auto_balance_node_least_effort Is Sometimes Failed 11391 - @derekbit @chriscchien
  • [BACKPORT][v1.9.2][BUG] Unable to set up S3 backup target if backups already exist 11344 - @mantissahz @chriscchien
  • [BACKPORT][v1.9.2][BUG] longhorn-manager is crashed due to SIGSEGV: segmentation violation 11422 - @derekbit @roger-ryao
  • [BACKPORT][v1.9.2][BUG] Typo in configuration parameter: "offlineRelicaRebuilding" should be "offlineReplicaRebuilding" 11382 - @yangchiu
  • [BUG][UI][v1.9.2-rc2] Unable to Retrieve Volume's Backup List in the Operation 11841 - @houhouhoucoop @roger-ryao

New Contributors

Contributors

Contributors

COLDTURNIP, innobead, and 11 other contributors
Loading

Longhorn v1.10.0-rc4

23 Sep 05:40
@derekbit derekbit
v1.10.0-rc4
bdef70d

Choose a tag to compare

View all tags
Longhorn v1.10.0-rc4 Pre-release
Pre-release

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Highlight

Feature

Improvement

Read more

Contributors

COLDTURNIP, innobead, and 28 other contributors
Loading