Does keda support scaling of ClusterScoped resources or what am i missing? #6631

vasyly71 · 2025-03-19T23:29:23Z

vasyly71
Mar 19, 2025

Hi,
I tried it with keda 2.16.1 nd KEDA 2.15.1 on EKS cluster 1.30

Attached zip test.zip contains resource

CRD (test-crds.yaml):
- nstest - namespaced CRD
- cltest - cluster scoped crd
Resources
- nstest - example of namespaced resource
- cltest - example of cluster scoped resource
scaled objects
- nstest - scaledobject for namespaced resource
- cltest - scaledobject for cluster scoped resource

when i create all elements above and execute "kubectl get so" i get:

kubectl get so
NAME     SCALETARGETKIND                 SCALETARGETNAME   MIN   MAX   TRIGGERS     AUTHENTICATION   READY   ACTIVE    FALLBACK   PAUSED    AGE
cltest                                   cltest            1     10    prometheus                    False   Unknown   Unknown    Unknown   22h
nstest   mygroup.example.com/v1.nstest   nstest            1     10    prometheus                    True    True      Unknown    Unknown   22h

The only difference between resources is one is namespaced another is cluster scoped and scaledobject for cluster scoped does not become ready.

Here is a log and its fragment, note that first log sais no scale subresource but it is there, and thenScaledObject doesn't have correct scaleTargetRef specification {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"cltest","namespace":"default"}, "namespace": "default", "name": "cltest", "reconcileID": "6a1f18f7-5dbc-4175-93ff-5a9fa7a14a84", "error": "the server could not find the requested resource (get cltests.meta.k8s.io cltest)"}

keda-operator.log

2025-03-19T22:35:31Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"cltest","namespace":"default"}, "namespace": "default", "name": "cltest", "reconcileID": "6a1f18f7-5dbc-4175-93ff-5a9fa7a14a84"}
2025-03-19T22:35:31Z	INFO	Starting workers	{"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication", "worker count": 1}
2025-03-19T22:35:31Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"nstest","namespace":"default"}, "namespace": "default", "name": "nstest", "reconcileID": "f4648b29-0cbd-40d6-9df6-b3c1f511cb9c"}
2025-03-19T22:35:31Z	INFO	Starting workers	{"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "worker count": 1}
2025-03-19T22:35:31Z	INFO	cert-rotation	certs are ready in /certs
2025-03-19T22:35:31Z	INFO	Starting workers	{"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication", "worker count": 1}
2025-03-19T22:35:31Z	INFO	Starting workers	{"controller": "cert-rotator", "worker count": 1}
2025-03-19T22:35:31Z	INFO	cert-rotation	no cert refresh needed
2025-03-19T22:35:31Z	INFO	cert-rotation	Ensuring CA cert	{"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2025-03-19T22:35:31Z	ERROR	Target resource doesn't expose /scale subresource	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"cltest","namespace":"default"}, "namespace": "default", "name": "cltest", "reconcileID": "6a1f18f7-5dbc-4175-93ff-5a9fa7a14a84", "resource": "mygroup.example.com/v1.cltest", "name": "cltest", "error": "the server could not find the requested resource (get cltests.meta.k8s.io cltest)"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).checkTargetResourceIsScalable
	/workspace/controllers/keda/scaledobject_controller.go:401
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:265
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2025-03-19T22:35:31Z	ERROR	ScaledObject doesn't have correct scaleTargetRef specification	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"cltest","namespace":"default"}, "namespace": "default", "name": "cltest", "reconcileID": "6a1f18f7-5dbc-4175-93ff-5a9fa7a14a84", "error": "the server could not find the requested resource (get cltests.meta.k8s.io cltest)"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:194
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2025-03-19T22:35:31Z	INFO	cert-rotation	Ensuring CA cert	{"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2025-03-19T22:35:31Z	INFO	Detected resource targeted for scaling	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"nstest","namespace":"default"}, "namespace": "default", "name": "nstest", "reconcileID": "f4648b29-0cbd-40d6-9df6-b3c1f511cb9c", "resource": "mygroup.example.com/v1.nstest", "name": "nstest"}
2025-03-19T22:35:31Z	ERROR	Reconciler error	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"cltest","namespace":"default"}, "namespace": "default", "name": "cltest", "reconcileID": "6a1f18f7-5dbc-4175-93ff-5a9fa7a14a84", "error": "the server could not find the requested resource (get cltests.meta.k8s.io cltest)"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227

Thank You
Vasyl

zroubalik · 2025-03-26T12:54:50Z

zroubalik
Mar 26, 2025
Maintainer

Are you sure the CRD exposes /scale subresource correctly?

4 replies

vasyly71 Mar 26, 2025
Author

An honest answer would be not at the moment of writing original post.
But provided examples clearly illustrate that behavior of two identical CRDs with difference only in namespaced/cluster scoped result in have different keda behaviour.
Proper implementation of /scale sub resource (i believe you mean label selector) become important when keda creates an HPA object. The part of HPA vcreation does not seem even was reached. And seems problem is HPA does not support cluster scoped targets.

The final question is - does KEDA even expected to work with cluster scoped objects, do you have "test cases" and examples for this. And if answer is no should it be mentioned in the documentation that targets must be namespaced?

zroubalik Mar 26, 2025
Maintainer

The contract we have doesn't care about namespaced/cluster-wide aspect. The only requirement is the /scale subresource defined on that particular resource.

If it works for you on namespaced resource without /scale subresource is interesting, but imho just coincidence.

This subresource is not much about labels or HPA, it is about how you expose information to the CRs status, it should be relatively easy to implement: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#scale-subresource

vasyly71 Mar 27, 2025
Author

Hi,

Thank you for your response.

(Please note that original request hs attached zip with identical CRDS and corresponding CR that differe only in scope - namespaced and cluster, namespaced works with keda clustered not)

Here are crd resource (cluster scoped) and scale objects (almost identical were provided in the original request but without labelSelector which in fact results in improper HPA object creation )

# CRD:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: cltests.mygroup.example.com
spec:
  group: mygroup.example.com
  names:
    kind: cltest
    listKind: cltestList
    plural: cltests
    singular: cltest
  scope: Cluster
  versions:
  - name: v1
    served: true
    storage: true
    subresources:
      scale:
        specReplicasPath: .spec.replicas
        statusReplicasPath: .status.replicas
        labelSelectorPath: .status.labelSelector
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              replicas:
                type: integer
          status:
            type: object
            properties:
              labelSelector:
                type: string            
              replicas:
                type: integer

# CR
apiVersion: mygroup.example.com/v1
kind: cltest
metadata:
  name: cltest
spec:
  replicas: 2
status:
  replicas: 0
  labelSelector: app=cltest

# Keda scale object:
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: cltest
spec:
  scaleTargetRef:
    apiVersion: mygroup.example.com/v1
    kind: cltest
    name: cltest
  minReplicaCount: 1
  maxReplicaCount: 10
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prereq-monitoring-prometheus-server.dev-vasyly-smp-prereq-0.svc.cluster.local
        metricName: ds_replica_forecast
        query: ds_replica_forecast 
        threshold: "1"

Creation of scaled objects above gives this error message below in keda-operator and object is not getting to ready state, i would really appreciate if you could point what what am i missing or doing wrong?

2025-03-27T02:03:58Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"cltest","namespace":"default"}, "namespace": "default", "name": "cltest", "reconcileID": "4b783cd4-a64c-4f7b-be42-d6117ac677f3"}
2025-03-27T02:03:58Z	INFO	Adding Finalizer for the ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"cltest","namespace":"default"}, "namespace": "default", "name": "cltest", "reconcileID": "4b783cd4-a64c-4f7b-be42-d6117ac677f3"}
2025-03-27T02:03:58Z	INFO	KubeAPIWarningLogger	metadata.finalizers: "finalizer.keda.sh": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
2025-03-27T02:03:58Z	ERROR	Target resource doesn't expose /scale subresource	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"cltest","namespace":"default"}, "namespace": "default", "name": "cltest", "reconcileID": "4b783cd4-a64c-4f7b-be42-d6117ac677f3", "resource": "mygroup.example.com/v1.cltest", "name": "cltest", "error": "the server could not find the requested resource (get cltests.meta.k8s.io cltest)"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).checkTargetResourceIsScalable
	/workspace/controllers/keda/scaledobject_controller.go:411
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
	/workspace/controllers/keda/scaledobject_controller.go:270
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
	/workspace/controllers/keda/scaledobject_controller.go:193
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:224
2025-03-27T02:03:58Z	ERROR	ScaledObject doesn't have correct scaleTargetRef specification	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"cltest","namespace":"default"}, "namespace": "default", "name": "cltest", "reconcileID": "4b783cd4-a64c-4f7b-be42-d6117ac677f3", "error": "the server could not find the requested resource (get cltests.meta.k8s.io cltest)"}

Thank You
Vasyl

zroubalik Mar 27, 2025
Maintainer

Thanks for the additional info. So seems like there is an issue with cluster wide CRDs. I will transfer this into an issue

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Does keda support scaling of ClusterScoped resources or what am i missing? #6631

{{title}}

Replies: 1 comment 4 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Does keda support scaling of ClusterScoped resources or what am i missing? #6631

vasyly71 Mar 19, 2025

Replies: 1 comment · 4 replies

zroubalik Mar 26, 2025 Maintainer

vasyly71 Mar 26, 2025 Author

zroubalik Mar 26, 2025 Maintainer

vasyly71 Mar 27, 2025 Author

zroubalik Mar 27, 2025 Maintainer

vasyly71
Mar 19, 2025

Replies: 1 comment 4 replies

zroubalik
Mar 26, 2025
Maintainer

vasyly71 Mar 26, 2025
Author

zroubalik Mar 26, 2025
Maintainer

vasyly71 Mar 27, 2025
Author

zroubalik Mar 27, 2025
Maintainer