Skip to content

[cluster autoscaling] initial commit for controller - #42171

Merged
dd-mergequeue[bot] merged 21 commits into
mainfrom
celene/cluster_autoscaler_controller
Nov 28, 2025
Merged

[cluster autoscaling] initial commit for controller#42171
dd-mergequeue[bot] merged 21 commits into
mainfrom
celene/cluster_autoscaler_controller

Conversation

@celenechang

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds cluster autoscaling controller

Motivation

Describe how you validated your changes

Additional Notes

@celenechang
celenechang force-pushed the celene/cluster_autoscaler_controller branch from d12f4e7 to b9a71c6 Compare October 20, 2025 23:33
@cit-pr-commenter

cit-pr-commenter Bot commented Oct 20, 2025

Copy link
Copy Markdown

Go Package Import Differences

Baseline: 13e01c0
Comparison: 6c18314

binaryosarchchange
cluster-agentlinuxamd64
+59, -0
+github.com/DataDog/datadog-agent/pkg/clusteragent/autoscaling/cluster
+github.com/DataDog/datadog-agent/pkg/clusteragent/autoscaling/cluster/model
+github.com/aws/karpenter-provider-aws/pkg/apis
+github.com/aws/karpenter-provider-aws/pkg/apis/v1
+github.com/awslabs/operatorpkg/metrics
+github.com/awslabs/operatorpkg/object
+github.com/awslabs/operatorpkg/option
+github.com/awslabs/operatorpkg/serrors
+github.com/awslabs/operatorpkg/status
+github.com/awslabs/operatorpkg/unstructured
+github.com/evanphx/json-patch/v5
+github.com/mitchellh/hashstructure/v2
+gomodules.xyz/jsonpatch/v2
+k8s.io/cloud-provider/api
+k8s.io/component-helpers/resource
+k8s.io/csi-translation-lib
+k8s.io/csi-translation-lib/plugins
+sigs.k8s.io/controller-runtime
+sigs.k8s.io/controller-runtime/pkg/builder
+sigs.k8s.io/controller-runtime/pkg/cache
+sigs.k8s.io/controller-runtime/pkg/cache/internal
+sigs.k8s.io/controller-runtime/pkg/certwatcher
+sigs.k8s.io/controller-runtime/pkg/certwatcher/metrics
+sigs.k8s.io/controller-runtime/pkg/client
+sigs.k8s.io/controller-runtime/pkg/client/apiutil
+sigs.k8s.io/controller-runtime/pkg/client/config
+sigs.k8s.io/controller-runtime/pkg/cluster
+sigs.k8s.io/controller-runtime/pkg/config
+sigs.k8s.io/controller-runtime/pkg/controller
+sigs.k8s.io/controller-runtime/pkg/controller/controllerutil
+sigs.k8s.io/controller-runtime/pkg/controller/priorityqueue
+sigs.k8s.io/controller-runtime/pkg/conversion
+sigs.k8s.io/controller-runtime/pkg/event
+sigs.k8s.io/controller-runtime/pkg/handler
+sigs.k8s.io/controller-runtime/pkg/healthz
+sigs.k8s.io/controller-runtime/pkg/leaderelection
+sigs.k8s.io/controller-runtime/pkg/log
+sigs.k8s.io/controller-runtime/pkg/manager
+sigs.k8s.io/controller-runtime/pkg/manager/signals
+sigs.k8s.io/controller-runtime/pkg/metrics
+sigs.k8s.io/controller-runtime/pkg/metrics/server
+sigs.k8s.io/controller-runtime/pkg/predicate
+sigs.k8s.io/controller-runtime/pkg/reconcile
+sigs.k8s.io/controller-runtime/pkg/recorder
+sigs.k8s.io/controller-runtime/pkg/source
+sigs.k8s.io/controller-runtime/pkg/webhook
+sigs.k8s.io/controller-runtime/pkg/webhook/admission
+sigs.k8s.io/controller-runtime/pkg/webhook/admission/metrics
+sigs.k8s.io/controller-runtime/pkg/webhook/conversion
+sigs.k8s.io/controller-runtime/pkg/webhook/conversion/metrics
+sigs.k8s.io/karpenter/pkg/apis
+sigs.k8s.io/karpenter/pkg/apis/v1
+sigs.k8s.io/karpenter/pkg/cloudprovider
+sigs.k8s.io/karpenter/pkg/operator/options
+sigs.k8s.io/karpenter/pkg/scheduling
+sigs.k8s.io/karpenter/pkg/utils/env
+sigs.k8s.io/karpenter/pkg/utils/pretty
+sigs.k8s.io/karpenter/pkg/utils/resources
+sigs.k8s.io/karpenter/pkg/utils/volume
cluster-agentlinuxarm64
+59, -0
+github.com/DataDog/datadog-agent/pkg/clusteragent/autoscaling/cluster
+github.com/DataDog/datadog-agent/pkg/clusteragent/autoscaling/cluster/model
+github.com/aws/karpenter-provider-aws/pkg/apis
+github.com/aws/karpenter-provider-aws/pkg/apis/v1
+github.com/awslabs/operatorpkg/metrics
+github.com/awslabs/operatorpkg/object
+github.com/awslabs/operatorpkg/option
+github.com/awslabs/operatorpkg/serrors
+github.com/awslabs/operatorpkg/status
+github.com/awslabs/operatorpkg/unstructured
+github.com/evanphx/json-patch/v5
+github.com/mitchellh/hashstructure/v2
+gomodules.xyz/jsonpatch/v2
+k8s.io/cloud-provider/api
+k8s.io/component-helpers/resource
+k8s.io/csi-translation-lib
+k8s.io/csi-translation-lib/plugins
+sigs.k8s.io/controller-runtime
+sigs.k8s.io/controller-runtime/pkg/builder
+sigs.k8s.io/controller-runtime/pkg/cache
+sigs.k8s.io/controller-runtime/pkg/cache/internal
+sigs.k8s.io/controller-runtime/pkg/certwatcher
+sigs.k8s.io/controller-runtime/pkg/certwatcher/metrics
+sigs.k8s.io/controller-runtime/pkg/client
+sigs.k8s.io/controller-runtime/pkg/client/apiutil
+sigs.k8s.io/controller-runtime/pkg/client/config
+sigs.k8s.io/controller-runtime/pkg/cluster
+sigs.k8s.io/controller-runtime/pkg/config
+sigs.k8s.io/controller-runtime/pkg/controller
+sigs.k8s.io/controller-runtime/pkg/controller/controllerutil
+sigs.k8s.io/controller-runtime/pkg/controller/priorityqueue
+sigs.k8s.io/controller-runtime/pkg/conversion
+sigs.k8s.io/controller-runtime/pkg/event
+sigs.k8s.io/controller-runtime/pkg/handler
+sigs.k8s.io/controller-runtime/pkg/healthz
+sigs.k8s.io/controller-runtime/pkg/leaderelection
+sigs.k8s.io/controller-runtime/pkg/log
+sigs.k8s.io/controller-runtime/pkg/manager
+sigs.k8s.io/controller-runtime/pkg/manager/signals
+sigs.k8s.io/controller-runtime/pkg/metrics
+sigs.k8s.io/controller-runtime/pkg/metrics/server
+sigs.k8s.io/controller-runtime/pkg/predicate
+sigs.k8s.io/controller-runtime/pkg/reconcile
+sigs.k8s.io/controller-runtime/pkg/recorder
+sigs.k8s.io/controller-runtime/pkg/source
+sigs.k8s.io/controller-runtime/pkg/webhook
+sigs.k8s.io/controller-runtime/pkg/webhook/admission
+sigs.k8s.io/controller-runtime/pkg/webhook/admission/metrics
+sigs.k8s.io/controller-runtime/pkg/webhook/conversion
+sigs.k8s.io/controller-runtime/pkg/webhook/conversion/metrics
+sigs.k8s.io/karpenter/pkg/apis
+sigs.k8s.io/karpenter/pkg/apis/v1
+sigs.k8s.io/karpenter/pkg/cloudprovider
+sigs.k8s.io/karpenter/pkg/operator/options
+sigs.k8s.io/karpenter/pkg/scheduling
+sigs.k8s.io/karpenter/pkg/utils/env
+sigs.k8s.io/karpenter/pkg/utils/pretty
+sigs.k8s.io/karpenter/pkg/utils/resources
+sigs.k8s.io/karpenter/pkg/utils/volume

@celenechang
celenechang force-pushed the celene/cluster_autoscaler_controller branch from b9a71c6 to a741e13 Compare October 21, 2025 15:41
@celenechang
celenechang force-pushed the celene/cluster_autoscaler_controller branch from a741e13 to eae5682 Compare October 21, 2025 15:50
Comment thread pkg/clusteragent/autoscaling/cluster/controller_test.go Outdated
Comment thread pkg/clusteragent/autoscaling/cluster/controller.go Outdated
Comment thread pkg/clusteragent/autoscaling/cluster/controller.go Outdated
Comment thread pkg/clusteragent/autoscaling/cluster/controller.go Outdated
Comment thread pkg/clusteragent/autoscaling/cluster/controller.go Outdated
Comment thread pkg/clusteragent/autoscaling/cluster/controller.go Outdated
Comment thread pkg/clusteragent/autoscaling/cluster/controller.go Outdated
Comment thread pkg/clusteragent/autoscaling/cluster/controller.go Outdated
Comment thread cmd/cluster-agent/subcommands/start/command.go
Comment thread pkg/clusteragent/autoscaling/cluster/config_retriever.go Outdated
Comment thread pkg/clusteragent/autoscaling/cluster/config_retriever.go Outdated
Comment thread pkg/clusteragent/autoscaling/cluster/controller.go Outdated
Comment thread pkg/clusteragent/autoscaling/cluster/controller.go Outdated
Comment thread pkg/clusteragent/autoscaling/cluster/controller.go Outdated
@celenechang
celenechang force-pushed the celene/cluster_autoscaler_controller branch from 1357c21 to e9e0970 Compare October 21, 2025 21:49
@github-actions github-actions Bot added the long review PR is complex, plan time to review it label Oct 23, 2025
@celenechang celenechang added the qa/done QA done before merge and regressions are covered by tests label Oct 23, 2025
Comment thread cmd/cluster-agent/subcommands/start/command.go
Comment thread pkg/clusteragent/autoscaling/cluster/config_retriever.go
Comment thread pkg/clusteragent/autoscaling/cluster/config_retriever.go Outdated
Comment thread pkg/clusteragent/autoscaling/cluster/controller.go Outdated
type store = autoscaling.Store[minNodePool]

const datadogCreatedLabelKey = "datadoghq.com/datadog-cluster-autoscaler.created"
const datadogModifiedLabelKey = "datadoghq.com/datadog-cluster-autoscaler.modified"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you like to track the RC version or observed generation? It could help reduce the number of reconciles.

Comment thread pkg/clusteragent/autoscaling/cluster/controller.go Outdated
}

func (c *Controller) patchNodePool(ctx context.Context, np *karpenterv1.NodePool, mnp minNodePool) error {
log.Infof("Patching NodePool: %s", mnp.name)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you choose to use Patch instead of a simple Update? It would be much simpler to just have a function that builds a NodePool from the K8S NodePool + NodePoolRecs

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i went with patch because it seems to go with the general flow of keeping the items in the store minimal, and minimally updating the cluster resource (just a couple fields) compared to the potential size of a nodePoolSpec. i'm not sure it would be simpler, but if it's better to go with update i can change that

Comment thread pkg/clusteragent/autoscaling/cluster/min_node_pool.go Outdated
Comment thread pkg/clusteragent/autoscaling/cluster/provider.go Outdated
Comment thread pkg/clusteragent/autoscaling/cluster/telemetry.go Outdated
@celenechang
celenechang marked this pull request as ready for review November 12, 2025 14:08
@celenechang
celenechang requested review from a team as code owners November 12, 2025 14:08
@L3n41c

L3n41c commented Nov 27, 2025

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +155 to +156
} else {
if isCreatedByDatadog(nodePool.GetLabels()) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Guard against nil NodePool when not in store

When processing a workqueue key for a NodePool that isn’t present in the store and is missing in the cluster (e.g., delete events for unmanaged or already-removed NodePools), syncNodePool still dereferences nodePool.GetLabels(). With nodePool == nil this panics and takes down the controller instead of safely ignoring the event. A nil check is needed before using the NodePool object in this branch.

Useful? React with 👍 / 👎.

Comment on lines +53 to +55
sender, err := senderManager.GetSender("cluster_autoscaling")
sender.DisableDefaultHostname(true)
if err != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Check GetSender error before using sender

StartClusterAutoscaling calls sender.DisableDefaultHostname(true) before verifying the result of senderManager.GetSender. If sender creation fails (for example if the demultiplexer isn’t initialized), GetSender returns an error and nil sender, and this line will panic before returning the error to the caller, preventing the controller from starting cleanly. The error should be handled before dereferencing the sender.

Useful? React with 👍 / 👎.

Comment thread pkg/clusteragent/autoscaling/cluster/controller_test.go Outdated
Comment thread cmd/cluster-agent/subcommands/start/command.go Outdated
Comment thread cmd/cluster-agent/subcommands/start/command.go Outdated
Comment thread pkg/clusteragent/autoscaling/cluster/model/node_pool_internal.go Outdated
vboulineau and others added 2 commits November 27, 2025 23:21
Co-authored-by: Lénaïc Huard <L3n41c@users.noreply.github.com>
@vboulineau
vboulineau force-pushed the celene/cluster_autoscaler_controller branch from 4dac434 to 7844a07 Compare November 27, 2025 22:44
@vboulineau

Copy link
Copy Markdown
Contributor

/merge -@ 8am CET

@dd-devflow-routing-codex

dd-devflow-routing-codex Bot commented Nov 27, 2025

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2025-11-27 22:45:56 UTC ℹ️ Start processing command /merge -@ 8am CET


2025-11-27 22:45:59 UTC ℹ️ MergeQueue: merge request scheduled for Fri, 28 Nov 2025 07:00:00 UTC

Pull Request scheduled to be added to the queue on Fri, 28 Nov 2025 07:00:00 UTC


2025-11-28 07:00:37 UTC ℹ️ MergeQueue: waiting for PR to be ready

This merge request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.


2025-11-28 11:00:04 UTC ⚠️ MergeQueue: This merge request was unqueued

devflow unqueued this merge request: It did not become mergeable within the expected time

@vboulineau
vboulineau force-pushed the celene/cluster_autoscaler_controller branch from 8f8af46 to 6c18314 Compare November 28, 2025 08:55
@vboulineau

Copy link
Copy Markdown
Contributor

/merge

@dd-devflow-routing-codex

dd-devflow-routing-codex Bot commented Nov 28, 2025

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2025-11-28 11:10:56 UTC ℹ️ Start processing command /merge


2025-11-28 11:11:00 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 43m (p90).


2025-11-28 11:11:15 UTCMergeQueue: This merge request is not mergeable, blocked by github

PR can't be merged according to github policy

@vboulineau

Copy link
Copy Markdown
Contributor

/merge

@dd-devflow-routing-codex

dd-devflow-routing-codex Bot commented Nov 28, 2025

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2025-11-28 11:13:51 UTC ℹ️ Start processing command /merge


2025-11-28 11:14:01 UTC ℹ️ MergeQueue: waiting for PR to be ready

This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.


2025-11-28 11:15:08 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in main is approximately 43m (p90).


2025-11-28 11:51:13 UTC ℹ️ MergeQueue: This merge request was merged

@adel121 adel121 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for @DataDog/container-platform owned files. 👍

@dd-mergequeue
dd-mergequeue Bot merged commit c585dbd into main Nov 28, 2025
599 checks passed
@dd-mergequeue
dd-mergequeue Bot deleted the celene/cluster_autoscaler_controller branch November 28, 2025 11:51
@github-actions github-actions Bot modified the milestones: 7.74.0, 7.75.0 Nov 28, 2025
dd-octo-sts Bot pushed a commit that referenced this pull request Nov 28, 2025
### What does this PR do?

Adds cluster autoscaling controller

### Motivation

### Describe how you validated your changes

### Additional Notes

Co-authored-by: vboulineau <vincent.boulineau@datadoghq.com>
(cherry picked from commit c585dbd)
FlorentClarret pushed a commit that referenced this pull request Nov 28, 2025
…#43622)

Backport c585dbd from #42171.

___

### What does this PR do?

Adds cluster autoscaling controller

### Motivation

### Describe how you validated your changes

### Additional Notes

Co-authored-by: Celene <celene@datadoghq.com>
Co-authored-by: vboulineau <vincent.boulineau@datadoghq.com>
rdesgroppes added a commit that referenced this pull request Jul 2, 2026
### What does this PR do?
Remove the `v0.60.0` version pin on `otelgrpc`, thus making `v0.69.0`
actual.

### Motivation
The TODO named kubernetes/apiserver@b887c9eb ("Drop usages of deprecated
otelgrpc methods") as the removal condition.

That commit indeed first landed in `k8s.io/apiserver v0.34.0`.

It turns out that a number of PRs had put the agent way past that
threshold:
| PR     | `k8s.io/apiserver`         |
|--------|----------------------------|
| #37844 | v0.32.2 -> **v0.34.1**     |
| #42171 |         -> v0.35.0-alpha.0 |
| #46808 |         -> v0.35.0         |
| #48796 |         -> v0.35.3         |
| #49973 |         -> v0.35.4         |
| #51049 |         -> v0.35.5         |

Dropping the stale `replace` directive is therefore safe and sound.
gh-worker-dd-mergequeue-cf854d Bot pushed a commit that referenced this pull request Jul 2, 2026
### What does this PR do?
Remove the `v0.60.0` version pin on `otelgrpc`, thus making `v0.69.0` actual.

### Motivation
kubernetes/apiserver@b887c9e ("Drop usages of deprecated otelgrpc methods") was named as the removal condition:
```
// TODO(songy23): remove this once kubernetes/apiserver@b887c9e is brought to agent
```

That commit indeed first landed in `k8s.io/apiserver v0.34.0`.

It turns out that a number of PRs had put the agent way past that threshold:
| PR     | `k8s.io/apiserver`         |
|--------|----------------------------|
| #37844 | v0.32.2 -> **v0.34.1**     |
| #42171 |         -> v0.35.0-alpha.0 |
| #46808 |         -> v0.35.0         |
| #48796 |         -> v0.35.3         |
| #49973 |         -> v0.35.4         |
| #51049 |         -> v0.35.5         |

Dropping the stale `replace` directive is therefore safe and sound.

Co-authored-by: regis.desgroppes <regis.desgroppes@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/autoscaling kind/feature long review PR is complex, plan time to review it qa/done QA done before merge and regressions are covered by tests team/container-autoscaling

Projects

None yet

Development

Successfully merging this pull request may close these issues.