Keep gpu_4_queue at 10 minimum with demand scaling to 40

[khluu]

What changed

• keep gpu_4_queue at MinSize = 10 with MaxSize = 40
• disable BuildkiteTerminateInstanceAfterJob so scarce g6.12xlarge capacity is retained between jobs
• enable the standard Buildkite queue autoscaler by making the stack variable-sized
• use a content-addressed copy of the exact Buildkite v6.21.0 template for this stack only; it removes forced blue/green ASG replacement and the fixed-size creation-signal wait

Behavior

• ten instances are the protected baseline and desired capacity remains 10 without additional demand
• the queue autoscaler polls Buildkite every minute and may increase desired capacity up to 40
• automatic scale-in remains disabled for persistent agents, so reducing capacity above 10 is an explicit drain and scale-down operation

Why

AWS was returning InsufficientInstanceCapacity for g6.12xlarge across all four us-west-2 availability zones. Recycling each instance after one job lost capacity that could not reliably be reacquired. The upstream v6.21.0 template also forces a second ASG for parameter changes, which cannot converge while existing scarce capacity remains allocated.

The patched template keeps the existing physical ASG and prevents the baseline capacity from disappearing, while still allowing demand-based scale-out when more agents are needed.

Validation

• terraform validate
• git diff --check
• CloudFormation change set reports AgentAutoScaleGroup as Modify with Replacement: False and MaxSize requiring no recreation
• the only added resource is the standard Autoscaling nested stack
• retained bk-gpu-4-queue-ci-AgentAutoScaleGroup-OEQAPNeflt32
• CloudFormation reached UPDATE_COMPLETE with MinSize=10, MaxSize=40, and BuildkiteTerminateInstanceAfterJob=false
• live ASG initially remained desired/in-service 10 with all ten instances protected from scale-in
• autoscaler EventBridge rule is enabled at rate(1 minute) and its first polls capped desired capacity at the ten-instance minimum
• targeted Terraform plan reports 0 to add, 1 to change, 0 to destroy; the remaining in-place change is the known template URL state drift