Multi-AZ Support: please define expected behavior

[snemir2]

For a ParallelCluster (latest version) configured with multiple subnets defined for compute queues, what is the expected node-to-subnet allocation behavior?
1. Does the order of subnets matter?
• For example, if I define subnets in az1, az2, and az3, will new compute nodes always launch in az1 first unless that AZ is out of capacity/IP addresses?
2. If an AZ (e.g., az1) is out of capacity for the requested instance type, will ParallelCluster automatically retry launching in the next subnet (az2 or az3)?
3. Are there limitations or constraints to be aware of?
• How many subnets across different AZs can be defined?
• Will it retry across all available subnets until one has capacity, or stop after one or two attempts?

⸻

Context

This question is in response to frequent instance capacity issues in a particular region. I’m trying to understand whether defining more subnets across multiple AZs can help alleviate these availability problems and if there are limits on it to b consider (adding subnets/AZs across 8 AZs would work so that it will try all 8 till it finds where instance can be created?)

[joehellmersNOAA]

I have been able to utilize multiple subnets for a partition in order to increase the chance of getting the instance type(s) I want.

You do need to use the "Instances" key. You don't have to have more than one "InstanceType" under that key, but if your workload supports multiple instance types then that could help as well.

[snemir2]

To clarify my request: I’m looking for documentation or clarity around how additional subnets are used when multiple subnets are defined in the configuration.
• Specifically, will subnets be used in the order they appear in the config file? (This would be strongly preferred.)
• The goal is predictability: ideally, the head node, login node(s), OneZone EFS, and—when possible—the compute nodes should all be placed in the same Availability Zone(unless that zone is out of capacity, then will take latency/cost penalty for better availability and let compute deploy in a different az)

Perhaps, this is more of a feature request: have compute nodes try AZs in the order that they appear in config file to ensure deterministic behavior.

[joehellmersNOAA]

I think that could be a good feature. However, it should be configurable. Doing some sort of round-robin or load balanced use of subnets could be desirable in some circumstances.

My guess is that's the way it works currently.

[snemir2]

@joehellmersNOAA — could you share an example where round-robin placement across subnets/AZs would actually be desirable? (e.g., scenarios like compute ↔ head node traffic across AZs).

From my perspective, aside from capacity/availability considerations, there doesn’t seem to be much benefit to spreading nodes across multiple AZs — especially since the head node is always tied to a single subnet right now. Of course -- aside from avoiding insufficient ec2 instance capacity errors.

[joehellmersNOAA]

That's the only legitimate reason I can think of but that is an important reason when working in the AWS cloud. For most large scale HPC workflows you simply must have nodes when you need them.

I suppose you might be tempted to think it could help with High Availability (HA), but you still would have a single point of failure with the head node. Better to setup a back up cluster in that case.

Maybe it could distribute your S3 read/write operation across AZs and maybe that could help in scenarios where your jobs are doing a lot of S3 activity, like many of mine do. But for HPC you might think about using Directory Buckets (i.e. single AZ) in which case you wouldn't want to spread your jobs over multiple AZs anyway.

Of course you absolutely wouldn't want to have your MPI jobs running across AZs.

I don't know if this uses round-robin or resources balancing, but it does distribute the instance allocation across multiple subnets/AZs

    Networking:
      SubnetIds:
      - <subnet1>
      - <subnet2>
    ComputeResources:
    - Name: myqueue
      Instances:
        - InstanceType: <instance type>
      MinCount: 0
      MaxCount: 30
      DisableSimultaneousMultithreading: false

[himani2411]

Hi,

Thanks for reaching out!

We have extended the parameter Scheduling/SlurmQueues/AllocationStrategy with 2 new values which assign priorities for the Subnets that you provide in cluster-config.yaml in AWS ParallelCluster 3.14.0

EC2 docs on AllocationStrategy https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-fleet-allocation-strategy.html

[himani2411]

With AWS ParallelCluster <3.14.0

1. Does the order of subnets matter?
   • For example, if I define subnets in az1, az2, and az3, will new compute nodes always launch in az1 first unless that AZ is out of capacity/IP addresses?

The order of subnet did not matter for the allocation of instances.

2. If an AZ (e.g., az1) is out of capacity for the requested instance type, will ParallelCluster automatically retry launching in the next subnet (az2 or az3)?
   This is not a simple answer that I can give as it depends on the whole Scheduling configuration you provide. So we create a request to EC2 Create Fleet based on the cluster-configuration you provide and EC2 is the one which evaluates based on the request. For example the ScalingStrategy will have an impact on this request too.

When you submit a job in a compute resource with a single instance type, in a queue that spans multiple Availability Zones, the all-or-nothing EC2 launch API call only succeeds if all of the capacity can be provided in a single Availability Zone.

Are there limitations or constraints to be aware of?
• How many subnets across different AZs can be defined?

You cant have 2 subnets which are covering the same AZ and all the Subnets should be in same VPC. More details are https://docs.aws.amazon.com/parallelcluster/latest/ug/Scheduling-v3.html#yaml-Scheduling-SlurmQueues-Networking-SubnetIds

Will it retry across all available subnets until one has capacity, or stop after one or two attempts?

We do retry with a jitter and exponential backoff to avoid throttling