Skip to content

CKS (main): an upgrade, scale or start fails if a node still has an ISO attached #14630

Open

Description

@bhouse-nexthop

problem

On main, a CKS upgrade, scale-up or start fails at once if a node VM still has an ISO attached, for example the Kubernetes binaries ISO left by an operation that didn't finish:

ERROR KubernetesClusterUpgradeWorker  Failed to attach binaries ISO for VM : <control node> in the Kubernetes cluster name: <cluster>
com.cloud.exception.InvalidParameterValueException: The specified ISO is already attached to this Instance.
    at com.cloud.template.TemplateManagerImpl.enforceCdromAttachLimits(TemplateManagerImpl.java)
    at com.cloud.template.TemplateManagerImpl.attachIso(TemplateManagerImpl.java)
    at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterActionWorker.attachIsoKubernetesVMs(KubernetesClusterActionWorker.java)

The operation fails before it changes anything, and the cluster goes to Alert. It keeps failing until the ISO is detached from each node by hand.

Cause:

  • KubernetesClusterActionWorker.attachIsoKubernetesVMs calls templateService.attachIso(isoId, vmId, true) on every node, and relies on the attach replacing whatever ISO the node has. That is how attachIso behaves on 4.22.
  • Multiple CD-ROM / ISO Support Per VM #13101 (multiple CD-ROMs per VM) added TemplateManagerImpl.enforceCdromAttachLimits. It refuses an ISO that is already attached, and, with the default of one CD-ROM, any second ISO ("Instance has reached the maximum of 1 attached CD-ROM(s)").

A node can be left holding an ISO when the management server stops during a CKS operation, or when an operator attached one by hand.

versions

main (since #13101). 4.22 is not affected, as its attachIso replaces the attached ISO.

The steps to reproduce the bug

  1. Create a CKS cluster.
  2. Attach the binaries ISO of the cluster's Kubernetes version to one of its node VMs (attachIso), or stop the management server while an upgrade of the cluster is running, after the ISO is attached.
  3. Upgrade or scale up the cluster (after any recovery of the cluster state).
  4. The job fails with "The specified ISO is already attached to this Instance." With a different ISO attached, it fails with "Instance has reached the maximum of 1 attached CD-ROM(s)".

What to do about it?

In attachIsoKubernetesVMs, leave the binaries ISO in place on a node that has it already, and detach any other ISO from the node before attaching, as the attach used to replace it.

Activity

  1. github-actions commented on Nov 10, 2026

    @github-actions

    Warning

    threat detection engine error
    The threat detection engine encountered an error and could not complete analysis. This is a tooling failure, not a security finding.

    What happened

    The threat detection engine failed to produce results.

    Review the workflow run logs for details.

    🎯 Triage report

    On main, a CKS upgrade, scale-up or start fails immediately if a node VM already has an ISO attached, for example the binaries ISO left by an operation that did not finish. Since #13101, attachIso refuses an ISO that is already attached. With the default one-CD-ROM limit it also refuses any second ISO. 4.22 is not affected. A fix is already open as #14637.

    📊 Assessment

    Dimension Value Reasoning
    Type type:bug Regression on main from the CD-ROM limit change (#13101)
    Component component:cks KubernetesClusterActionWorker.attachIsoKubernetesVMs
    Severity Severity:Major Blocks upgrade, scale and start until ISOs are detached by hand
    Labels type:bug, component:cks, Severity:Major
    Coding agent Suitable Fix is specified, and PR #14637 is already open for review

    🔗 Similar issues

    💡 Notes and suggestions
    • Reproduction: attach the cluster's binaries ISO to a node VM, then upgrade or scale the cluster.
    • Check the start path as well, since the report says start also fails.

    Generated by Daily Issue Triage · copilot · haiku55 · ~5 AIC · ⊞ 3.8K · ◷

    Add this agentic workflow to your repo

    To install this agentic workflow, run

    gh aw add githubnext/agentics/workflows/daily-issue-triage.md@139fa09ef17cde4135e696ff8e7af09bc5ddfa6e
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions