Use asg warm pools for faster buildkite job starts

[nitrocode]

https://aws.amazon.com/about-aws/whats-new/2021/04/amazon-ec2-auto-scaling-introduces-warm-pools-accelerate-scale-out-while-saving-money/

https://aws.amazon.com/blogs/compute/scaling-your-applications-faster-with-ec2-auto-scaling-warm-pools/

If we could keep X instances warmed up, we could start jobs much faster without having to set the min count on the asg to something non-zero.

It should be added soon to cloudformation: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html

[chloeruka]

Woah! You're quick. Yeah this feature looks great; especially for Windows instances that take a while to boot. According to the announcement it's not yet available on Cloudformation, but when it is we'll take a look into it.

[nitrocode]

The only param that would require an additional input to use the warm pool would be the min size of it. We use an asg min size of 0 and max size of 10 so a warm pool min size of 3 seems reasonable.

The cloudformation docs have been released.

We could add the following

Parameters:
  WarmPoolMinSize:
    Description: Minimum number of instances in warm pool
    Type: Number
    Default: 0

Conditions:
    UseWarmPool:
      !Not [ !Equals [ !Ref WarmPoolMinSize, 0 ] ]

Resources:
  WarmPool: 
    Type: AWS::AutoScaling::WarmPool
    Condition: UseWarmPool
    Properties:
      AutoScalingGroupName: !Ref AgentAutoScaleGroup
      MinSize: !Ref WarmPoolMinSize
      PoolState: Stopped

What do you folks think?

[theonlysinjin]

This is fantastic.
I've been using the Cloudwatch metrics to scale ASG when there are no idle agents (ie all are busy now), to help beat the agent start time.
Though it seems our agents now start (with the update) in just under 2 minutes which is pretty good.

[dieend]

Warm Pool is now available in CloudFormation https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-autoscaling-warmpool.html

Right now I can't manually update the created ASG from the current template because the template configuring MixedInstancesPolicy

elastic-ci-stack-for-aws/templates/aws-stack.yml

Lines 1038 to 1060 in 7b3d02c

	MixedInstancesPolicy:
	InstancesDistribution:
	OnDemandPercentageAboveBaseCapacity: !If [ SpotPriceSet, 0, !Ref OnDemandPercentage ]
	SpotAllocationStrategy: capacity-optimized
	SpotMaxPrice: !If [SpotPriceSet, !Ref SpotPrice, !Ref "AWS::NoValue"]
	LaunchTemplate:
	LaunchTemplateSpecification:
	LaunchTemplateId: !Ref AgentLaunchTemplate
	Version: !GetAtt "AgentLaunchTemplate.LatestVersionNumber"
	Overrides:
	- InstanceType: !Select [ "0", !Split [ ",", !Join [ ",", [ !Ref InstanceType, "", "", "" ] ] ] ]
	- !If
	- UseInstanceType2
	- InstanceType: !Select [ "1", !Split [ ",", !Join [ ",", [ !Ref InstanceType, "", "", "" ] ] ] ]
	- !Ref "AWS::NoValue"
	- !If
	- UseInstanceType3
	- InstanceType: !Select [ "2", !Split [ ",", !Join [ ",", [ !Ref InstanceType, "", "", "" ] ] ] ]
	- !Ref "AWS::NoValue"
	- !If
	- UseInstanceType4
	- InstanceType: !Select [ "3", !Split [ ",", !Join [ ",", [ !Ref InstanceType, "", "", "" ] ] ] ]
	- !Ref "AWS::NoValue"

We have to remove them if we'd like to use WarmPool

[aiven-amartin]

Any progress/caveats on this issue? Would be really nice to have this option rather than trying to bake git checkouts/other slow workspace parts into the AMI itself.

[alex-shakouri-ai]

This would be helpful on our side as well! Would help out with ensuring we have agents on a warm start up to be able to start jobs faster!

[wolfeidau]

@alex-shakouri-ai We had a look at this and it would be hard to just plug into the existing model.

Currently we aren't looking at re architecting the auto scaling.