Summary
On AWS, the deployed subnet (and therefore the EC2 instance) is always placed in the alphabetically-first availability zone that offers the requested instance type, e.g. us-west-2a. When that specific AZ is out of GPU capacity, the deployment fails with InsufficientInstanceCapacity, even though other AZs in the same region have capacity. There is currently no way to choose the AZ, so re-running the deploy deterministically hits the same exhausted zone.
Environment
- Version: v4.0.0
- Cloud: AWS
- Example:
g6.2xlarge in us-west-2
Steps to reproduce
- Run
./deploy-aws.
- Choose a GPU instance type (e.g.
g6.2xlarge) and a region where the -a AZ is capacity-constrained (e.g. us-west-2).
- Let Terraform apply.
Actual behavior
Error: creating EC2 Instance: InsufficientInstanceCapacity: We currently do not have
sufficient g6.2xlarge capacity in the Availability Zone you requested (us-west-2a).
... You can currently get g6.2xlarge capacity by not specifying an Availability Zone
in your request or choosing us-west-2b, us-west-2c, us-west-2d.
Re-running does not help, because the AZ choice is deterministic.
Root cause
In src/terraform/aws/isaac-workstation/main.tf, the subnet's AZ is hard-selected as the first sorted zone that offers the instance type:
availability_zone = try(sort(data.aws_ec2_instance_type_offerings.zones.locations)[0], "not-available")
"Offerings" means the AZ supports the instance type, not that it currently has capacity. So the deployment always lands in the same zone (-a) regardless of live capacity.
Proposed fix
Add an optional availability_zone input (a Terraform variable plus a --availability-zone / --az option on deploy-aws) so the user can pin a specific AZ to route around per-zone capacity shortages. When left empty, behavior is unchanged: auto-select the first offered zone.
I have a fix ready and will open a PR referencing this issue.
Summary
On AWS, the deployed subnet (and therefore the EC2 instance) is always placed in the alphabetically-first availability zone that offers the requested instance type, e.g.
us-west-2a. When that specific AZ is out of GPU capacity, the deployment fails withInsufficientInstanceCapacity, even though other AZs in the same region have capacity. There is currently no way to choose the AZ, so re-running the deploy deterministically hits the same exhausted zone.Environment
g6.2xlargeinus-west-2Steps to reproduce
./deploy-aws.g6.2xlarge) and a region where the-aAZ is capacity-constrained (e.g.us-west-2).Actual behavior
Re-running does not help, because the AZ choice is deterministic.
Root cause
In
src/terraform/aws/isaac-workstation/main.tf, the subnet's AZ is hard-selected as the first sorted zone that offers the instance type:"Offerings" means the AZ supports the instance type, not that it currently has capacity. So the deployment always lands in the same zone (
-a) regardless of live capacity.Proposed fix
Add an optional
availability_zoneinput (a Terraform variable plus a--availability-zone/--azoption ondeploy-aws) so the user can pin a specific AZ to route around per-zone capacity shortages. When left empty, behavior is unchanged: auto-select the first offered zone.I have a fix ready and will open a PR referencing this issue.