Skip to content

Bug: CLI crashes with "'str' object has no attribute 'get'" when updating SharedStorage MountDir #7162

New issue
New issue
Open
Open
Bug: CLI crashes with "'str' object has no attribute 'get'" when updating SharedStorage MountDir#7162
Labels
3.xbug
@dna-oliv

Description

@dna-oliv
dna-oliv
opened on Dec 19, 2025

Bug: CLI crashes with "'str' object has no attribute 'get'" when updating SharedStorage MountDir

Required Info

  • AWS ParallelCluster version: 3.14.0

  • Cluster name: (any cluster with SharedStorage)

  • Arn of the cluster CloudFormation main stack: N/A (reproducible on any cluster)

  • Output of pcluster describe-cluster:

    {
  "version": "3.14.0",
  "clusterStatus": "CREATE_COMPLETE",
  "computeFleetStatus": "RUNNING",
  "scheduler": {
    "type": "slurm"
  }
}

Full cluster configuration (minimal reproduction):

Region: eu-north-1
Image:
  Os: rhel9
HeadNode:
  Imds:
    Secured: true
  InstanceType: c6i.large
  Networking:
    SubnetId: subnet-02022469c3451395b
    AdditionalSecurityGroups:
      - sg-0b6398ec0dbba2ef2
    ElasticIp: true
  Ssh:
    KeyName: my_key

Scheduling:
  Scheduler: slurm
  SlurmQueues:
    - Name: hpc6a
      Networking:
        SubnetIds:
          - subnet-0f51e28e2cd594d96
        AdditionalSecurityGroups:
          - sg-0b6398ec0dbba2ef2
      CapacityType: ONDEMAND
      ComputeSettings:
        LocalStorage:
          RootVolume:
            Size: 200
            VolumeType: gp3
      ComputeResources:
        - Name: hpc6a-48xlarge
          InstanceType: hpc6a.48xlarge
          MinCount: 0
          MaxCount: 8
          DisableSimultaneousMultithreading: true
          Efa:
            Enabled: true
          Networking:
            PlacementGroup:
              Enabled: true

SharedStorage:
  - Name: fsx-lustre
    MountDir: /fsx
    StorageType: FsxLustre
    FsxLustreSettings:
      StorageCapacity: 1200
Imds:
  ImdsSupport: v2.0

Bug description and how to reproduce

Description

When running pcluster update-cluster with a configuration that changes the MountDir of a SharedStorage item, the CLI crashes with an unhandled exception instead of returning a proper validation error.

Error message:

{
  "message": "Cluster update failed.\n'str' object has no attribute 'get'"
}

Steps to reproduce

  1. Create a cluster with SharedStorage that has a MountDir:

    SharedStorage:
  - Name: fsx-lustre
    MountDir: /fsx
    StorageType: FsxLustre
    FsxLustreSettings:
      StorageCapacity: 1200

  2. Update the cluster with a changed MountDir (no other changes required):

    SharedStorage:
  - Name: fsx-lustre
    MountDir: /fsx1    # Changed from /fsx
    StorageType: FsxLustre
    FsxLustreSettings:
      StorageCapacity: 1200

  3. Run:

    pcluster update-cluster --cluster-name <name> --region <region> --cluster-configuration <new-config>

  4. CLI crashes with 'str' object has no attribute 'get'

Conditions

The bug occurs when all three conditions are met:

  1. Cluster compute fleet status is anything other than STOPPED (e.g., RUNNING, UNKNOWN, STARTING, etc.)
  2. SharedStorage MountDir is changed
  3. Slurm scheduler is used

Workaround

Stop the compute fleet before updating MountDir:

pcluster update-compute-fleet --cluster-name <name> --region <region> --status STOP_REQUESTED
# Wait for status to become STOPPED
pcluster describe-compute-fleet --cluster-name <name> --region <region>
# Then update succeeds
pcluster update-cluster --cluster-name <name> --region <region> --cluster-configuration <config>
(.pcluster3140) ➜  pcluster update-cluster --cluster-name repro-cluster --cluster-configuration repro-cluster.yaml --debug
{
  "message": "Cluster update failed.\n'str' object has no attribute 'get'"
}

(.pcluster3140) ➜  pcluster update-compute-fleet --cluster-name repro-cluster --region eu-north-1 --status STOP_REQUESTED
{
  "status": "STOP_REQUESTED",
  "lastStatusUpdatedTime": "2025-12-19T01:24:42.850Z"
}

(.pcluster3140) ➜  pcluster describe-compute-fleet --cluster-name repro-cluster --region eu-north-1
{
  "status": "STOPPED",
  "lastStatusUpdatedTime": "2025-12-19T01:25:02.699Z"
}

(.pcluster3140) ➜  pcluster update-cluster --cluster-name repro-cluster --cluster-configuration repro-cluster.yaml --debug
{
  "cluster": {
    "clusterName": "repro-cluster",
    "cloudformationStackStatus": "UPDATE_IN_PROGRESS",
    "cloudformationStackArn": "arn:aws:cloudformation:eu-north-1:REDACTED:stack/repro-cluster/42742080-dc70-11f0-9148-06a50478435d",
    "region": "eu-north-1",
    "version": "3.14.0",
    "clusterStatus": "UPDATE_IN_PROGRESS",
    "scheduler": {
      "type": "slurm"
    }
  },
  "validationMessages": [
    {
      "level": "INFO",
      "type": "DeletionPolicyValidator",
      "message": "The DeletionPolicy is set to Delete. The storage 'fsx-lustre' will be deleted when you remove it from the configuration when performing a cluster update or deleting the cluster."
    }
  ],
  "changeSet": [
    {
      "parameter": "SharedStorage[fsx-lustre].MountDir",
      "requestedValue": "/fsx1",
      "currentValue": "/fsx"
    }
  ]
}

Root Cause Analysis

Bug Location

File: cli/src/pcluster/config/update_policy_utils.py
Line: 25
Class: SharedStorageChangeInfo.__init__()

Traceback

2025-12-18 17:14:42,145 - INFO - entrypoint.py:260:run() - Handling CLI command update-cluster
2025-12-18 17:14:42,146 - DEBUG - entrypoint.py:261:run() - Parsed CLI arguments: args(REDACTED), extra_args([])
2025-12-18 17:14:42,147 - DEBUG - util.py:170:_assert_node_executable() - Found Node.js executable in /Users/REDACTED/.nvm/versions/node/v20.19.0/bin/node
2025-12-18 17:14:42,208 - DEBUG - util.py:191:_assert_node_version() - Found Node.js version (v20.19.0
)
2025-12-18 17:14:42,210 - INFO - common.py:44:_set_region() - Setting AWS Region to eu-north-1
2025-12-18 17:14:43,057 - INFO - common.py:134:_log_boto3_calls() - Executing boto3 call: region=eu-north-1, service=cloudformation, operation=DescribeStacks, params={'StackName': 'repro-cluster'}
2025-12-18 17:14:43,764 - INFO - import_cdk.py:10:import_cdk() - Start importing
2025-12-18 17:14:43,769 - INFO - common.py:134:_log_boto3_calls() - Executing boto3 call: region=eu-north-1, service=cloudformation, operation=DescribeStacks, params={'StackName': 'repro-cluster'}
2025-12-18 17:14:44,006 - INFO - common.py:134:_log_boto3_calls() - Executing boto3 call: region=eu-north-1, service=ec2, operation=DescribeInstances, params={'Filters': [{'Name': 'tag:parallelcluster:cluster-name', 'Values': ['repro-cluster']}, {'Name': 'instance-state-name', 'Values': ['pending', 'running', 'stopping', 'stopped']}, {'Name': 'tag:parallelcluster:node-type', 'Values': ['HeadNode']}]}
2025-12-18 17:14:44,766 - INFO - cluster.py:488:_validate_and_parse_config() - Validating cluster configuration...
2025-12-18 17:14:44,787 - INFO - common.py:134:_log_boto3_calls() - Executing boto3 call: region=eu-north-1, service=cloudformation, operation=DescribeStackResources, params={'StackName': 'repro-cluster'}
2025-12-18 17:14:45,277 - DEBUG - common.py:190:_validator_execute() - Executing validator OsCustomAmiValidator
2025-12-18 17:14:45,277 - DEBUG - common.py:190:_validator_execute() - Executing validator SecurityGroupsValidator
2025-12-18 17:14:45,277 - DEBUG - common.py:190:_validator_execute() - Executing validator SecurityGroupsValidator
2025-12-18 17:14:45,373 - INFO - common.py:134:_log_boto3_calls() - Executing boto3 call: region=eu-north-1, service=ec2, operation=DescribeSecurityGroups, params={'GroupIds': ['sg-0b6398ec0dbba2ef2']}
2025-12-18 17:14:45,595 - DEBUG - common.py:190:_validator_execute() - Executing validator ElasticIpValidator
2025-12-18 17:14:45,595 - DEBUG - common.py:190:_validator_execute() - Executing validator EbsVolumeThroughputValidator
2025-12-18 17:14:45,595 - DEBUG - common.py:190:_validator_execute() - Executing validator EbsVolumeThroughputIopsValidator
2025-12-18 17:14:45,595 - DEBUG - common.py:190:_validator_execute() - Executing validator InstanceTypeValidator
2025-12-18 17:14:45,600 - INFO - common.py:134:_log_boto3_calls() - Executing boto3 call: region=eu-north-1, service=ec2, operation=DescribeInstanceTypeOfferings, params={}
2025-12-18 17:14:47,093 - DEBUG - common.py:190:_validator_execute() - Executing validator SharedStorageEfsSettingsValidator
2025-12-18 17:14:47,093 - DEBUG - common.py:190:_validator_execute() - Executing validator EbsVolumeThroughputValidator
2025-12-18 17:14:47,093 - DEBUG - common.py:190:_validator_execute() - Executing validator EbsVolumeThroughputIopsValidator
2025-12-18 17:14:47,093 - DEBUG - common.py:190:_validator_execute() - Executing validator PlacementGroupNamingValidator
2025-12-18 17:14:47,098 - INFO - common.py:134:_log_boto3_calls() - Executing boto3 call: region=eu-north-1, service=ec2, operation=DescribeInstanceTypes, params={'InstanceTypes': ['hpc6a.48xlarge']}
2025-12-18 17:14:47,301 - DEBUG - common.py:190:_validator_execute() - Executing validator NameValidator
2025-12-18 17:14:47,301 - DEBUG - common.py:190:_validator_execute() - Executing validator SchedulableMemoryValidator
2025-12-18 17:14:47,302 - DEBUG - common.py:190:_validator_execute() - Executing validator PlacementGroupNamingValidator
2025-12-18 17:14:47,302 - DEBUG - common.py:190:_validator_execute() - Executing validator SecurityGroupsValidator
2025-12-18 17:14:47,302 - DEBUG - common.py:190:_validator_execute() - Executing validator SecurityGroupsValidator
2025-12-18 17:14:47,308 - INFO - common.py:134:_log_boto3_calls() - Executing boto3 call: region=eu-north-1, service=ec2, operation=DescribeSubnets, params={'SubnetIds': ['subnet-0f51e28e2cd594d96']}
2025-12-18 17:14:47,528 - DEBUG - common.py:190:_validator_execute() - Executing validator NameValidator
2025-12-18 17:14:47,528 - DEBUG - common.py:190:_validator_execute() - Executing validator EfaMultiAzValidator
2025-12-18 17:14:47,528 - DEBUG - common.py:190:_validator_execute() - Executing validator EfaValidator
2025-12-18 17:14:47,529 - DEBUG - common.py:190:_validator_execute() - Executing validator MultiAzPlacementGroupValidator
2025-12-18 17:14:47,529 - DEBUG - common.py:190:_validator_execute() - Executing validator DuplicateNameValidator
2025-12-18 17:14:47,529 - DEBUG - common.py:190:_validator_execute() - Executing validator MaxCountValidator
2025-12-18 17:14:47,529 - DEBUG - common.py:190:_validator_execute() - Executing validator QueueSubnetsValidator
2025-12-18 17:14:47,529 - DEBUG - common.py:190:_validator_execute() - Executing validator SlurmNodePrioritiesWarningValidator
2025-12-18 17:14:47,529 - DEBUG - common.py:190:_validator_execute() - Executing validator SingleInstanceTypeSubnetValidator
2025-12-18 17:14:47,529 - DEBUG - common.py:190:_validator_execute() - Executing validator EfaSecurityGroupValidator
2025-12-18 17:14:47,529 - DEBUG - common.py:190:_validator_execute() - Executing validator EfaPlacementGroupValidator
2025-12-18 17:14:47,529 - DEBUG - common.py:190:_validator_execute() - Executing validator ComputeResourceSizeValidator
2025-12-18 17:14:47,529 - DEBUG - common.py:190:_validator_execute() - Executing validator CapacityTypeValidator
2025-12-18 17:14:47,529 - DEBUG - common.py:190:_validator_execute() - Executing validator FeatureRegionValidator
2025-12-18 17:14:47,536 - DEBUG - common.py:190:_validator_execute() - Executing validator ExternalSlurmdbdVsDatabaseIncompatibility
2025-12-18 17:14:47,536 - DEBUG - common.py:190:_validator_execute() - Executing validator ExternalSlurmdbdRequiresCustomMungeKey
2025-12-18 17:14:47,536 - DEBUG - common.py:190:_validator_execute() - Executing validator ExternalSlurmdbdTrafficNotEncrypted
2025-12-18 17:14:47,536 - DEBUG - common.py:190:_validator_execute() - Executing validator DuplicateNameValidator
2025-12-18 17:14:47,536 - DEBUG - common.py:190:_validator_execute() - Executing validator RootVolumeEncryptionConsistencyValidator
2025-12-18 17:14:47,536 - DEBUG - common.py:190:_validator_execute() - Executing validator MaxCountValidator
2025-12-18 17:14:47,536 - DEBUG - common.py:190:_validator_execute() - Executing validator MaxCountValidator
2025-12-18 17:14:47,536 - DEBUG - common.py:190:_validator_execute() - Executing validator SharedStorageNameValidator
2025-12-18 17:14:47,536 - DEBUG - common.py:190:_validator_execute() - Executing validator FsxDraValidator
2025-12-18 17:14:47,536 - DEBUG - common.py:190:_validator_execute() - Executing validator FsxS3Validator
2025-12-18 17:14:47,536 - DEBUG - common.py:190:_validator_execute() - Executing validator FsxPersistentOptionsValidator
2025-12-18 17:14:47,536 - DEBUG - common.py:190:_validator_execute() - Executing validator FsxBackupOptionsValidator
2025-12-18 17:14:47,536 - DEBUG - common.py:190:_validator_execute() - Executing validator FsxStorageTypeOptionsValidator
2025-12-18 17:14:47,537 - DEBUG - common.py:190:_validator_execute() - Executing validator FsxStorageCapacityValidator
2025-12-18 17:14:47,537 - DEBUG - common.py:190:_validator_execute() - Executing validator FsxBackupIdValidator
2025-12-18 17:14:47,537 - DEBUG - common.py:190:_validator_execute() - Executing validator DeletionPolicyValidator
2025-12-18 17:14:47,537 - DEBUG - common.py:190:_validator_execute() - Executing validator LogRotationValidator
2025-12-18 17:14:47,537 - DEBUG - common.py:190:_validator_execute() - Executing validator DetailedMonitoringValidator
2025-12-18 17:14:47,547 - INFO - common.py:134:_log_boto3_calls() - Executing boto3 call: region=eu-north-1, service=ec2, operation=DescribeInstanceTypes, params={'InstanceTypes': ['c6i.large']}
2025-12-18 17:14:47,752 - INFO - common.py:134:_log_boto3_calls() - Executing boto3 call: region=eu-north-1, service=ec2, operation=DescribeSubnets, params={'SubnetIds': ['subnet-02022469c3451395b']}
2025-12-18 17:14:47,926 - INFO - import_cdk.py:16:import_cdk() - Import complete in 4 seconds
2025-12-18 17:14:47,971 - INFO - common.py:134:_log_boto3_calls() - Executing boto3 call: region=eu-north-1, service=ec2, operation=DescribeImages, params={'ImageIds': ['ami-0aeb582419dd31854']}
2025-12-18 17:14:48,198 - DEBUG - common.py:190:_validator_execute() - Executing validator RegionValidator
2025-12-18 17:14:48,198 - DEBUG - common.py:190:_validator_execute() - Executing validator ClusterNameValidator
2025-12-18 17:14:48,198 - DEBUG - common.py:190:_validator_execute() - Executing validator ArchitectureOsValidator
2025-12-18 17:14:48,198 - DEBUG - common.py:190:_validator_execute() - Executing validator InstanceTypeBaseAMICompatibleValidator
2025-12-18 17:14:48,198 - DEBUG - common.py:190:_validator_execute() - Executing validator InstanceTypeOSCompatibleValidator
2025-12-18 17:14:48,199 - DEBUG - common.py:190:_validator_execute() - Executing validator DuplicateNameValidator
2025-12-18 17:14:48,199 - DEBUG - common.py:190:_validator_execute() - Executing validator DuplicateNameValidator
2025-12-18 17:14:48,199 - DEBUG - common.py:190:_validator_execute() - Executing validator SharedStorageNameValidator
2025-12-18 17:14:48,199 - DEBUG - common.py:190:_validator_execute() - Executing validator SharedStorageMountDirValidator
2025-12-18 17:14:48,199 - DEBUG - common.py:190:_validator_execute() - Executing validator FeatureRegionValidator
2025-12-18 17:14:48,199 - DEBUG - common.py:190:_validator_execute() - Executing validator FsxArchitectureOsValidator
2025-12-18 17:14:48,199 - DEBUG - common.py:190:_validator_execute() - Executing validator ExistingFsxNetworkingValidator
2025-12-18 17:14:48,213 - DEBUG - common.py:190:_validator_execute() - Executing validator NumberOfStorageValidator
2025-12-18 17:14:48,213 - DEBUG - common.py:190:_validator_execute() - Executing validator NumberOfStorageValidator
2025-12-18 17:14:48,213 - DEBUG - common.py:190:_validator_execute() - Executing validator NumberOfStorageValidator
2025-12-18 17:14:48,213 - DEBUG - common.py:190:_validator_execute() - Executing validator NumberOfStorageValidator
2025-12-18 17:14:48,213 - DEBUG - common.py:190:_validator_execute() - Executing validator NumberOfStorageValidator
2025-12-18 17:14:48,214 - DEBUG - common.py:190:_validator_execute() - Executing validator NumberOfStorageValidator
2025-12-18 17:14:48,214 - DEBUG - common.py:190:_validator_execute() - Executing validator NumberOfStorageValidator
2025-12-18 17:14:48,214 - DEBUG - common.py:190:_validator_execute() - Executing validator ManagedFsxMultiAzValidator
2025-12-18 17:14:48,214 - DEBUG - common.py:190:_validator_execute() - Executing validator MultiAzEbsVolumeValidator
2025-12-18 17:14:48,214 - DEBUG - common.py:190:_validator_execute() - Executing validator MultiAzRootVolumeValidator
2025-12-18 17:14:48,214 - DEBUG - common.py:190:_validator_execute() - Executing validator DuplicateMountDirValidator
2025-12-18 17:14:48,214 - DEBUG - common.py:190:_validator_execute() - Executing validator OverlappingMountDirValidator
2025-12-18 17:14:48,214 - DEBUG - common.py:190:_validator_execute() - Executing validator HeadNodeLaunchTemplateValidator
2025-12-18 17:14:48,225 - INFO - common.py:134:_log_boto3_calls() - Executing boto3 call: region=eu-north-1, service=ec2, operation=RunInstances, params={'InstanceType': 'c6i.large', 'MinCount': 1, 'MaxCount': 1, 'ImageId': 'ami-0aeb582419dd31854', 'NetworkInterfaces': [{'DeviceIndex': 0, 'NetworkCardIndex': 0, 'InterfaceType': 'interface', 'Groups': ['sg-0b6398ec0dbba2ef2'], 'SubnetId': 'subnet-02022469c3451395b'}], 'DryRun': True, 'TagSpecifications': [], 'KeyName': 'REDACTED-eu-north-1', 'BlockDeviceMappings': [{'DeviceName': '/dev/xvdba', 'VirtualName': 'ephemeral0'}, {'DeviceName': '/dev/xvdbb', 'VirtualName': 'ephemeral1'}, {'DeviceName': '/dev/xvdbc', 'VirtualName': 'ephemeral2'}, {'DeviceName': '/dev/xvdbd', 'VirtualName': 'ephemeral3'}, {'DeviceName': '/dev/xvdbe', 'VirtualName': 'ephemeral4'}, {'DeviceName': '/dev/xvdbf', 'VirtualName': 'ephemeral5'}, {'DeviceName': '/dev/xvdbg', 'VirtualName': 'ephemeral6'}, {'DeviceName': '/dev/xvdbh', 'VirtualName': 'ephemeral7'}, {'DeviceName': '/dev/xvdbi', 'VirtualName': 'ephemeral8'}, {'DeviceName': '/dev/xvdbj', 'VirtualName': 'ephemeral9'}, {'DeviceName': '/dev/xvdbk', 'VirtualName': 'ephemeral10'}, {'DeviceName': '/dev/xvdbl', 'VirtualName': 'ephemeral11'}, {'DeviceName': '/dev/xvdbm', 'VirtualName': 'ephemeral12'}, {'DeviceName': '/dev/xvdbn', 'VirtualName': 'ephemeral13'}, {'DeviceName': '/dev/xvdbo', 'VirtualName': 'ephemeral14'}, {'DeviceName': '/dev/xvdbp', 'VirtualName': 'ephemeral15'}, {'DeviceName': '/dev/xvdbq', 'VirtualName': 'ephemeral16'}, {'DeviceName': '/dev/xvdbr', 'VirtualName': 'ephemeral17'}, {'DeviceName': '/dev/xvdbs', 'VirtualName': 'ephemeral18'}, {'DeviceName': '/dev/xvdbt', 'VirtualName': 'ephemeral19'}, {'DeviceName': '/dev/xvdbu', 'VirtualName': 'ephemeral20'}, {'DeviceName': '/dev/xvdbv', 'VirtualName': 'ephemeral21'}, {'DeviceName': '/dev/xvdbw', 'VirtualName': 'ephemeral22'}, {'DeviceName': '/dev/xvdbx', 'VirtualName': 'ephemeral23'}, {'DeviceName': '/dev/sda1', 'Ebs': {'Encrypted': True, 'VolumeType': 'gp3', 'Iops': 3000, 'Throughput': 125, 'DeleteOnTermination': True}}], 'MetadataOptions': {'HttpTokens': 'required'}}
2025-12-18 17:14:48,852 - DEBUG - common.py:190:_validator_execute() - Executing validator SchedulerValidator
2025-12-18 17:14:48,852 - DEBUG - common.py:190:_validator_execute() - Executing validator SchedulerOsValidator
2025-12-18 17:14:48,852 - DEBUG - common.py:190:_validator_execute() - Executing validator HeadNodeImdsValidator
2025-12-18 17:14:48,852 - DEBUG - common.py:190:_validator_execute() - Executing validator RootVolumeSizeValidator
2025-12-18 17:14:48,853 - DEBUG - common.py:190:_validator_execute() - Executing validator EbsVolumeTypeSizeValidator
2025-12-18 17:14:48,853 - DEBUG - common.py:190:_validator_execute() - Executing validator EbsVolumeIopsValidator
2025-12-18 17:14:48,853 - DEBUG - common.py:190:_validator_execute() - Executing validator KeyPairValidator
2025-12-18 17:14:48,865 - INFO - common.py:134:_log_boto3_calls() - Executing boto3 call: region=eu-north-1, service=ec2, operation=DescribeKeyPairs, params={'KeyNames': ['REDACTED-eu-north-1']}
2025-12-18 17:14:49,066 - DEBUG - common.py:190:_validator_execute() - Executing validator MixedSecurityGroupOverwriteValidator
2025-12-18 17:14:49,066 - DEBUG - common.py:190:_validator_execute() - Executing validator MultiNetworkInterfacesInstancesValidator
2025-12-18 17:14:49,066 - DEBUG - common.py:190:_validator_execute() - Executing validator ComputeResourceLaunchTemplateValidator
2025-12-18 17:14:49,079 - INFO - common.py:134:_log_boto3_calls() - Executing boto3 call: region=eu-north-1, service=ec2, operation=RunInstances, params={'InstanceType': 'hpc6a.48xlarge', 'MinCount': 1, 'MaxCount': 1, 'ImageId': 'ami-0aeb582419dd31854', 'Placement': {}, 'NetworkInterfaces': [{'DeviceIndex': 0, 'NetworkCardIndex': 0, 'InterfaceType': 'efa', 'Groups': ['sg-0b6398ec0dbba2ef2'], 'SubnetId': 'subnet-0f51e28e2cd594d96'}], 'DryRun': True, 'TagSpecifications': [], 'BlockDeviceMappings': [{'DeviceName': '/dev/xvdba', 'VirtualName': 'ephemeral0'}, {'DeviceName': '/dev/xvdbb', 'VirtualName': 'ephemeral1'}, {'DeviceName': '/dev/xvdbc', 'VirtualName': 'ephemeral2'}, {'DeviceName': '/dev/xvdbd', 'VirtualName': 'ephemeral3'}, {'DeviceName': '/dev/xvdbe', 'VirtualName': 'ephemeral4'}, {'DeviceName': '/dev/xvdbf', 'VirtualName': 'ephemeral5'}, {'DeviceName': '/dev/xvdbg', 'VirtualName': 'ephemeral6'}, {'DeviceName': '/dev/xvdbh', 'VirtualName': 'ephemeral7'}, {'DeviceName': '/dev/xvdbi', 'VirtualName': 'ephemeral8'}, {'DeviceName': '/dev/xvdbj', 'VirtualName': 'ephemeral9'}, {'DeviceName': '/dev/xvdbk', 'VirtualName': 'ephemeral10'}, {'DeviceName': '/dev/xvdbl', 'VirtualName': 'ephemeral11'}, {'DeviceName': '/dev/xvdbm', 'VirtualName': 'ephemeral12'}, {'DeviceName': '/dev/xvdbn', 'VirtualName': 'ephemeral13'}, {'DeviceName': '/dev/xvdbo', 'VirtualName': 'ephemeral14'}, {'DeviceName': '/dev/xvdbp', 'VirtualName': 'ephemeral15'}, {'DeviceName': '/dev/xvdbq', 'VirtualName': 'ephemeral16'}, {'DeviceName': '/dev/xvdbr', 'VirtualName': 'ephemeral17'}, {'DeviceName': '/dev/xvdbs', 'VirtualName': 'ephemeral18'}, {'DeviceName': '/dev/xvdbt', 'VirtualName': 'ephemeral19'}, {'DeviceName': '/dev/xvdbu', 'VirtualName': 'ephemeral20'}, {'DeviceName': '/dev/xvdbv', 'VirtualName': 'ephemeral21'}, {'DeviceName': '/dev/xvdbw', 'VirtualName': 'ephemeral22'}, {'DeviceName': '/dev/xvdbx', 'VirtualName': 'ephemeral23'}, {'DeviceName': '/dev/sda1', 'Ebs': {'Encrypted': True, 'VolumeType': 'gp3', 'Iops': 3000, 'Throughput': 125, 'DeleteOnTermination': True, 'VolumeSize': 200}}], 'MetadataOptions': {'HttpTokens': 'required'}}
2025-12-18 17:14:49,664 - DEBUG - common.py:190:_validator_execute() - Executing validator RootVolumeSizeValidator
2025-12-18 17:14:49,664 - DEBUG - common.py:190:_validator_execute() - Executing validator EbsVolumeTypeSizeValidator
2025-12-18 17:14:49,664 - DEBUG - common.py:190:_validator_execute() - Executing validator EbsVolumeIopsValidator
2025-12-18 17:14:49,664 - DEBUG - common.py:190:_validator_execute() - Executing validator InstanceArchitectureCompatibilityValidator
2025-12-18 17:14:49,664 - DEBUG - common.py:190:_validator_execute() - Executing validator EfaOsArchitectureValidator
2025-12-18 17:14:49,664 - DEBUG - common.py:190:_validator_execute() - Executing validator PlacementGroupCapacityTypeValidator
2025-12-18 17:14:49,664 - DEBUG - common.py:190:_validator_execute() - Executing validator InstanceTypeBaseAMICompatibleValidator
2025-12-18 17:14:49,664 - DEBUG - common.py:190:_validator_execute() - Executing validator InstanceTypeOSCompatibleValidator
2025-12-18 17:14:49,665 - DEBUG - common.py:190:_validator_execute() - Executing validator InstanceTypeAcceleratorManufacturerValidator
2025-12-18 17:14:49,665 - DEBUG - common.py:190:_validator_execute() - Executing validator InstanceTypePlacementGroupValidator
2025-12-18 17:14:49,665 - DEBUG - common.py:190:_validator_execute() - Executing validator ComputeResourceTagsValidator
2025-12-18 17:14:49,665 - DEBUG - common.py:190:_validator_execute() - Executing validator HeadNodeMemorySizeValidator
2025-12-18 17:14:49,665 - INFO - cluster.py:499:_validate_and_parse_config() - Validation succeeded.
2025-12-18 17:14:49,685 - INFO - common.py:134:_log_boto3_calls() - Executing boto3 call: region=eu-north-1, service=sts, operation=GetCallerIdentity, params={}
2025-12-18 17:14:50,421 - INFO - common.py:134:_log_boto3_calls() - Executing boto3 call: region=eu-north-1, service=s3, operation=HeadBucket, params={'Bucket': 'parallelcluster-redacted-v1-do-not-delete'}
2025-12-18 17:14:50,987 - INFO - common.py:134:_log_boto3_calls() - Executing boto3 call: region=eu-north-1, service=s3, operation=HeadObject, params={'Bucket': 'parallelcluster-redacted-v1-do-not-delete', 'Key': 'parallelcluster/.bootstrapped'}
2025-12-18 17:14:51,177 - INFO - common.py:134:_log_boto3_calls() - Executing boto3 call: region=eu-north-1, service=s3, operation=GetObject, params={'Bucket': 'parallelcluster-redacted-v1-do-not-delete', 'Key': 'parallelcluster/.bootstrapped'}
2025-12-18 17:14:51,368 - INFO - s3_bucket.py:468:_check_default_bucket() - Bucket parallelcluster-redacted-v1-do-not-delete is already bootstrapped with required features.
2025-12-18 17:14:51,375 - INFO - common.py:134:_log_boto3_calls() - Executing boto3 call: region=eu-north-1, service=s3, operation=GetObject, params={'Bucket': 'parallelcluster-redacted-v1-do-not-delete', 'Key': 'parallelcluster/3.14.0/clusters/repro-cluster-5e6268jxpmpaw3c6/configs/cluster-config.yaml', 'VersionId': '51KNj75egNsWlYFOsrubTZsDx6hcY4_s'}
2025-12-18 17:14:51,673 - INFO - common.py:134:_log_boto3_calls() - Executing boto3 call: region=eu-north-1, service=dynamodb, operation=GetItem, params={'TableName': 'parallelcluster-repro-cluster', 'ConsistentRead': True, 'Key': {'Id': 'COMPUTE_FLEET'}}
2025-12-18 17:14:52,258 - CRITICAL - cluster.py:968:update() - 'str' object has no attribute 'get'
2025-12-18 17:14:52,258 - ERROR - entrypoint.py:284:main() - {"message": "Cluster update failed.\n'str' object has no attribute 'get'"}
Traceback (most recent call last):
  File "/Users/REDACTED/workspace/pcluster/.pcluster3140/lib/python3.13/site-packages/pcluster/models/cluster.py", line 916, in update
    target_config, changes, ignored_validation_failures = self.validate_update_request(
                                                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        target_source_config, validator_suppressors, validation_failure_level, force
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/REDACTED/workspace/pcluster/.pcluster3140/lib/python3.13/site-packages/pcluster/models/cluster.py", line 876, in validate_update_request
    changes = self._validate_patch(force, target_config)
  File "/Users/REDACTED/workspace/pcluster/.pcluster3140/lib/python3.13/site-packages/pcluster/models/cluster.py", line 884, in _validate_patch
    patch_allowed, update_changes = patch.check()
                                    ~~~~~~~~~~~^^
  File "/Users/REDACTED/workspace/pcluster/.pcluster3140/lib/python3.13/site-packages/pcluster/config/config_patch.py", line 259, in check
    check_result, reason, action_needed, print_change = change.update_policy.check(change, self)
                                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/Users/REDACTED/workspace/pcluster/.pcluster3140/lib/python3.13/site-packages/pcluster/config/update_policy.py", line 80, in check
    if self.condition_checker(change, patch):
       ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/Users/REDACTED/workspace/pcluster/.pcluster3140/lib/python3.13/site-packages/pcluster/config/update_policy.py", line 473, in condition_checker_shared_storage_update_policy
    and not is_compute_fleet_update_supported_for_shared_storage(change)
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/REDACTED/workspace/pcluster/.pcluster3140/lib/python3.13/site-packages/pcluster/config/update_policy.py", line 265, in is_compute_fleet_update_supported_for_shared_storage
    change_info = SharedStorageChangeInfo(change)
  File "/Users/REDACTED/workspace/pcluster/.pcluster3140/lib/python3.13/site-packages/pcluster/config/update_policy_utils.py", line 25, in __init__
    self.storage_type = storage_item.get("StorageType")
                        ^^^^^^^^^^^^^^^^
AttributeError: 'str' object has no attribute 'get'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/REDACTED/workspace/pcluster/.pcluster3140/lib/python3.13/site-packages/pcluster/api/controllers/common.py", line 149, in wrapper
    return func(*args, **kwargs)
  File "/Users/REDACTED/workspace/pcluster/.pcluster3140/lib/python3.13/site-packages/pcluster/api/controllers/common.py", line 101, in _wrapper_http_success_status_code
    return func(*args, **kwargs), status_code
           ~~~~^^^^^^^^^^^^^^^^^
  File "/Users/REDACTED/workspace/pcluster/.pcluster3140/lib/python3.13/site-packages/pcluster/api/controllers/cluster_operations_controller.py", line 391, in update_cluster
    changes, ignored_validation_failures = cluster.update(
                                           ~~~~~~~~~~~~~~^
        target_source_config=cluster_config,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        force=force_update,
        ^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/REDACTED/workspace/pcluster/.pcluster3140/lib/python3.13/site-packages/pcluster/models/cluster.py", line 969, in update
    raise _cluster_error_mapper(e, f"Cluster update failed.\n{e}")
pcluster.models.cluster.ClusterActionError: Cluster update failed.
'str' object has no attribute 'get'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/REDACTED/workspace/pcluster/.pcluster3140/lib/python3.13/site-packages/pcluster/cli/entrypoint.py", line 210, in _run_operation
    return args.func(args)
           ~~~~~~~~~^^^^^^
  File "/Users/REDACTED/workspace/pcluster/.pcluster3140/lib/python3.13/site-packages/pcluster/cli/entrypoint.py", line 116, in dispatch
    return middleware[operation](dispatch_func, body, kwargs)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/REDACTED/workspace/pcluster/.pcluster3140/lib/python3.13/site-packages/pcluster/cli/middleware.py", line 65, in wrapper
    ret = func(dest_func, _body, kwargs)
  File "/Users/REDACTED/workspace/pcluster/.pcluster3140/lib/python3.13/site-packages/pcluster/cli/middleware.py", line 77, in update_cluster
    ret = func(**kwargs)
  File "/Users/REDACTED/workspace/pcluster/.pcluster3140/lib/python3.13/site-packages/pcluster/cli/model.py", line 211, in call
    ret = func(*args, **kwargs)
  File "/Users/REDACTED/workspace/pcluster/.pcluster3140/lib/python3.13/site-packages/pcluster/api/controllers/common.py", line 161, in wrapper
    raise InternalServiceException(str(e)) from e
pcluster.api.errors.InternalServiceException

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/REDACTED/workspace/pcluster/.pcluster3140/lib/python3.13/site-packages/pcluster/cli/entrypoint.py", line 268, in main
    ret = run(sys.argv[1:])
  File "/Users/REDACTED/workspace/pcluster/.pcluster3140/lib/python3.13/site-packages/pcluster/cli/entrypoint.py", line 262, in run
    return _run_operation(model, args, extra_args)
  File "/Users/REDACTED/workspace/pcluster/.pcluster3140/lib/python3.13/site-packages/pcluster/cli/entrypoint.py", line 221, in _run_operation
    raise APIOperationException(json.loads(error_encoded))
pcluster.cli.exceptions.APIOperationException: {
  "message": "Cluster update failed.\n'str' object has no attribute 'get'"
}

Explanation

SharedStorageChangeInfo is designed to handle changes at the SharedStorage item level (adding/removing entire storage items), where change.old_value and change.new_value are dictionaries.

However, SHARED_STORAGE_UPDATE_POLICY is also applied to nested property changes like MountDir, where values are strings:

# Expected by SharedStorageChangeInfo: SharedStorage item change
change.key = "SharedStorage"
change.old_value = {"Name": "fsx", "MountDir": "/fsx", "StorageType": "FsxLustre", ...}

# Actual: MountDir property change
change.key = "MountDir"
change.old_value = "/fsx"      # string
change.new_value = "/fsx1"     # string

The code calls .get("StorageType") on a string, causing the AttributeError.

Suggested Fix

Add type checking in SharedStorageChangeInfo.__init__() or in the calling functions:

Option 1: Guard in SharedStorageChangeInfo

def __init__(self, change):
    old_value = change.old_value
    new_value = change.new_value

    storage_item = new_value if new_value is not None else old_value

    # Guard against non-dict values (nested property changes)
    if not isinstance(storage_item, dict):
        self.storage_type = None
        self.storage_settings = {}
        self.is_external = False
        self.is_mount = False
        self.is_unmount = False
        return

    # ... rest of existing code

Option 2: Guard in calling functions

def is_compute_fleet_update_supported_for_shared_storage(change):
    # Only process SharedStorage item-level changes
    if not isinstance(change.old_value, dict) and not isinstance(change.new_value, dict):
        return False
    change_info = SharedStorageChangeInfo(change)
    return change_info.is_external and (...)

Additional context

  • The same issue likely affects is_login_fleet_update_supported_for_shared_storage() at line 251
  • Other fields using SHARED_STORAGE_UPDATE_POLICY may also be affected (check cluster_schema.py)

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.xbug

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions