Unable to Use dynatrace as a trigger in k8s : context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}

[kekjoshi]

Hi Team,

We are trying to achieve HPA on our EKS cluster with the help of KEDA.

The trigger type we have to use is Dynatrace

Helm chart we have deployed : https://github.com/kedacore/charts/blob/v2.16.1/keda/values.yaml
Scaler Used : https://keda.sh/docs/2.16/scalers/dynatrace/

Below is our scaler manifest

apiVersion: v1
kind: Secret
metadata:
name: dynatrace-keda-secret
namespace: monitoring
type: Opaque
data:
host: "dynatrace-host-base64 encoded"
token: " base64 encoding of api key with metric read and ingest permission"

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: keda-trigger-auth-dynatrace
namespace: monitoring
spec:
secretTargetRef:

• parameter: token
  name: dynatrace-keda-secret
  key: token
• parameter: host
  name: dynatrace-keda-secret
  key: host

---

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: dynatrace-scaledobject
namespace: monitoring
spec:
maxReplicaCount: 2
minReplicaCount: 1
scaleTargetRef:
kind: Deployment
name: "splunk-otel-collector-k8s-cluster-receiver"
triggers:
- type: dynatrace
metadata:
metricSelector: 'builtin:kubernetes.containers:splitBy():fold'
from: 'now-5m'
threshold: '1'
authenticationRef:
name: keda-trigger-auth-dynatrace

Errors we are having on operator pod
25-03-24T17:33:29Z INFO cert-rotation Ensuring CA cert {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2025-03-24T17:33:29Z INFO cert-rotation Ensuring CA cert {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2025-03-24T17:33:29Z INFO cert-rotation no cert refresh needed
2025-03-24T17:33:29Z INFO cert-rotation Ensuring CA cert {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2025-03-24T17:33:29Z INFO cert-rotation Ensuring CA cert {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2025-03-24T17:33:36Z INFO cert-rotation no cert refresh needed
2025-03-24T17:33:36Z INFO cert-rotation Ensuring CA cert {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2025-03-24T17:33:36Z INFO cert-rotation Ensuring CA cert {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2025-03-24T18:24:02Z ERROR dynatrace_scaler error executing Dynatrace query {"type": "ScaledObject", "namespace": "monitoring", "name": "dynatrace-scaledobject", "error": "Get "https://xyz.live.dynatrace.com/api/v2/metrics/query?from=now-5m&metricSelector=builtin%3Akubernetes.containers%3AsplitBy%28%29%3Afold\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scalers.(*dynatraceScaler).GetMetricsAndActivity
/workspace/pkg/scalers/dynatrace_scaler.go:175
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).GetMetricsAndActivityForScaler
/workspace/pkg/scaling/cache/scalers_cache.go:151
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScalerState
/workspace/pkg/scaling/scale_handler.go:758
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScaledObjectState.func1
/workspace/pkg/scaling/scale_handler.go:633

Logs from operstor-metric-server pod

I0324 17:33:31.001715 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0324 17:33:31.001903 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController
I0324 17:33:46.759802 1 client.go:103] "msg"="Waiting for establishing a gRPC connection to KEDA Metrics Server" "logger"="keda_metrics_adapter.provider"
E0324 17:34:46.854791 1 provider.go:90] "msg"="timeout" "error"="timeout while waiting to establish gRPC connection to KEDA Metrics Service server" "logger"="keda_metrics_adapter.provider" "server"="keda-operator.keda.svc.cluster.local:9666"
E0324 17:34:46.854846 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:"timeout while waiting to establish gRPC connection to KEDA Metrics Service server"}: timeout while waiting to establish gRPC connection to KEDA Metrics Service server" logger="UnhandledError"
E0324 17:34:46.856015 1 writers.go:122] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E0324 17:34:46.857111 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout" logger="UnhandledError"
E0324 17:34:46.858211 1 writers.go:135] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E0324 17:34:46.859382 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="103.850165ms" method="GET" path="/apis/external.metrics.k8s.io/v1beta1/namespaces/monitoring/s0-dynatrace" result=null
I0324 17:39:46.839550 1 client.go:103] "msg"="Waiting for establishing a gRPC connection to KEDA Metrics Server" "logger"="keda_metrics_adapter.provider"
I0324 17:40:46.842463 1 client.go:103] "msg"="Waiting for establishing a gRPC connection to KEDA Metrics Server" "logger"="keda_metrics_adapter.provider"
E0324 17:40

Note :
We have keda operator and api server deployed in keda namespace and the scaler manifest in another namespace called as monitoring as we neeed to scale deployment from monitoring namespace.

Can you please help us to resolve the issue.
Thanks

[kekjoshi]

unable to access locally also https://keda.sh/docs/2.16/operate/metrics-server/

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/monitoring/s0-dynatrace??labelSelector=scaledobject.keda.sh%2Fname%3D{dynatrace-scaledobject}"

Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request