feat: support scaling direction-aware cooldown for task auto-scalers

[jtuglu1]

Description

Adds support for configuring different cooldowns for scaling direction. While both scaling actions do cause temporary disruption to ingestion, scaling down can cause more disruption than scaling up due to having less resources than when you started to recover from lag. Therefore, to allow for aggressive scale up while having a more conservative scale-down approach, this adds configuration for cool down period for both directions. Cool down for a specific scaling direction is evaluated as follows:

1. minScaleUpDelay/minScaleDownDelay
2. minTriggerScaleActionFrequencyMillis (marked as deprecated)
3. The default of 600000.

This also does the following:

1. Cleans up the core scaling logic to be more clear/readable
2. Adds a custom ScaleActionSupplier which makes the contract clear for auto-scalers who want to implement it.
3. Fixes a few notification bugs, namely:

• Emitting/logging metrics with skip reasons irrespective of whether we are actually planning to scale or not (e.g. auto-scaler function returns value != -1, but that value is equal to the current task count). Now, we emit a required task count metric with a specific skip reason.

Release note

Adds support for configuring different cooldowns for scaling direction for streaming task auto-scalers.

---

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[kfaraz]

@jtuglu1 , cost-based auto scaler already uses a minScaleDownDelay config for similar purposes. I feel we should simply expose that at the AutoScalerConfig instead of two new min trigger fields. (I never liked the name minTriggerScaleActionFrequencyMillis as it's verbose and still ambiguous.)

The functionality of the minScaleDownDelay may be modified as required by this PR.

cc: @Fly-Style

[jtuglu1]

@jtuglu1 , cost-based auto scaler already uses a minScaleDownDelay config for similar purposes. I feel we should simply expose that at the AutoScalerConfig instead of two new min trigger fields. (I never liked the name minTriggerScaleActionFrequencyMillis as it's verbose and still ambiguous.)

The functionality of the minScaleDownDelay may be modified as required by this PR.

cc: @Fly-Style

Sure, I always want to reduce config bloat. However, I'd prefer to settle on a conventional naming scheme if possible (e.g. minScaleUpDelay) to keep kicking the ambiguous can that is minTriggerScaleActionFrequencyMillis down the road. This way, we can deprecate that config and instead just treat it as an alias for both if unspecified in future releases.

[Fly-Style]

Thank you for your contribution! The patch has nice direction, some polishing and we will be glad to merge it.

[Fly-Style]

Really like the improvement, clear differentiation between scale-up and scale-down, while preserving old behaviour as a fallback. LGTM!