Scaler seems to bypass Lifecycle Hooks

[lox]

In my initial testing, it looks like lifecycle hooks didn't seem to be triggered on scale-ins 🤔

@etaoins have you noticed that?

[etaoins]

It appeared to work for me but due to #6 and #7 the ASG always wanted to scale in. I had an agent terminate between two jobs in the same build but never while a job was running. That could’ve just been luck, though.

[lox]

Just did some testing and not seeing lifecycle events firing. I suspect that lifecycle hooks only fire when there are autoscaling rules on an ASG.

[lox]

I wonder if we could use target tracking to scale-in, but use the lamba for scale-out 🤔

[lucasteligioridis]

Is this issue relevant anymore with the new version? @lox

[lox]

It is in the sense that it's still a thing if you try and use the scale-in support in the lambda, but we disable it in Elastic Stack.

I'd still love to know why it doesn't work.

[keithduncan]

This appears to have been fixed some time between last being looked at and now.

In buildkite/elastic-ci-stack-for-aws#964 I’m looking at re-enabling scale in (set desired count to less than current) in the Lambda and using service-side ASG lifecycle hook handlers to orchestrate the agent exit, and the lifecycle hooks are firing:

🎉